Sakalti/SakaMoe-3x1.6B-Instruct
MoE built from dense models vs Sakalti/SJT-0.5B. Sakalti/SakaMoe-3x1.6B-Instruct is a mixture of 3 experts assembled from dense models (mergekit-moe style): experts 0, 1 are an exact copy of Sakalti/SJT-0.5B's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 219 of 219 identical.
2 derivatives checked
- mradermacher/SakaMoe-3x1.6B-Instruct-i1-GGUF: Quantization
- mradermacher/SakaMoe-3x1.6B-Instruct-GGUF: Same weights, converted