Mixture of Experts - Tag - DataSci Ocean

Mixture of Experts

2024

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM 04-24

Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints 04-10