DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Dai, Damai; Deng, Chengqi; Zhao, Chenggang; Xu, Ruijian; Gao, Huazuo; Chen, Deli; Li, Jiashi; Zeng, Wangding; Yu, Xingkai; Wu, Yongle; Xie, Zhenda; Li, Y. K.; Huang, Panpan; Luo, Fuli; Ruan, Chong; Sui, Zhifang; Liang, Wenfeng

doi:10.18653/v1/2024.acl-long.70

articleJan 1, 2024GOLD OA

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

DDDamai Dai CDChengqi Deng CZChenggang Zhao RXRuijian Xu HGHuazuo Gao

Indexed incrossref

Abstract

Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Citation impact

178

total citations

FWCI: 56.57
Percentile: 100%
References: 0

Citations per year

Authors

17

Topics & keywords

Topics

Keywords

Computer science
Natural language processing
Artificial intelligence

No related works found for this paper.

Funding

NK
National Key Research and Development Program of China
Award: 2020AAA0106700