Pinkstack/Superthoughts-lite-v2-MOE-Llama3.2-bf16
MoE built from dense models vs meta-llama/Llama-3.2-1B-Instruct. Pinkstack/Superthoughts-lite-v2-MOE-Llama3.2-bf16 is a mixture of 4 experts with the same shape as meta-llama/Llama-3.2-1B-Instruct's MLP, but none of them is a copy of it: they come from other models or were trained further. Shared tensors (attention, norms, embeddings): 33 of 98 identical.
2 derivatives checked
- mradermacher/Superthoughts-lite-v2-MOE-Llama3.2-bf16-i1-GGUF: Quantization
- mradermacher/Superthoughts-lite-v2-MOE-Llama3.2-bf16-GGUF: Same weights, converted