Fu01978/SmolMoE-4x360M-Instruct
MoE built from dense models vs Fu01978/SmolLM2-360M-Instruct-Heretic. Fu01978/SmolMoE-4x360M-Instruct is a mixture of 4 experts assembled from dense models (mergekit-moe style): expert 3 is an exact copy of Fu01978/SmolLM2-360M-Instruct-Heretic's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 184 of 194 identical.
2 derivatives checked
- mradermacher/SmolMoE-4x360M-Instruct-i1-GGUF: Quantization
- mradermacher/SmolMoE-4x360M-Instruct-GGUF: Same weights, converted