Fu01978/Llama-3.2-3B-MoE-4Expert
MoE built from dense models vs DavidAU/Llama-3.2-3B-Instruct-heretic-ablitered-uncensored. Fu01978/Llama-3.2-3B-MoE-4Expert is a mixture of 4 experts assembled from dense models (mergekit-moe style): experts 0, 2 are an exact copy of DavidAU/Llama-3.2-3B-Instruct-heretic-ablitered-uncensored's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 149 of 170 identical.
2 derivatives checked
- mradermacher/Llama-3.2-3B-MoE-4Expert-i1-GGUF: Quantization
- mradermacher/Llama-3.2-3B-MoE-4Expert-GGUF: Same weights, converted