FreedomIntelligence/Apollo-MoE-0.5B
MoE built from dense models vs Qwen/Qwen2-0.5B. FreedomIntelligence/Apollo-MoE-0.5B is a mixture of 7 experts with the same shape as Qwen/Qwen2-0.5B's MLP, but none of them is a copy of it: they come from other models or were trained further. Shared tensors (attention, norms, embeddings): 0 of 284 identical.