huihui-ai/Huihui-MoE-0.8B-2E
MoE built from dense models vs Qwen/Qwen3-0.6B. huihui-ai/Huihui-MoE-0.8B-2E is a mixture of 2 experts assembled from dense models (mergekit-moe style): experts 0, 1 are an exact copy of Qwen/Qwen3-0.6B's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 226 of 226 identical.
1 derivatives checked
- mradermacher/Huihui-MoE-0.8B-2E-GGUF: Same weights, converted