Fu01978/Llama-3.2-1B-4B-Quad-MoE
MoE built from dense models vs cutelemonlili/Llama3.2-1B-Instruct_Lean_Code. Fu01978/Llama-3.2-1B-4B-Quad-MoE is a mixture of 4 experts assembled from dense models (mergekit-moe style): expert 2 is an exact copy of cutelemonlili/Llama3.2-1B-Instruct_Lean_Code's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 24 of 98 identical.
2 derivatives checked
- mradermacher/Llama-3.2-1B-4B-Quad-MoE-i1-GGUF: Quantization
- mradermacher/Llama-3.2-1B-4B-Quad-MoE-GGUF: Same weights, converted