jsfs11/MixtureofMerges-MoE-2x7b-v6
MoE built from dense models vs mlabonne/AlphaMonarch-7B. jsfs11/MixtureofMerges-MoE-2x7b-v6 is a mixture of 2 experts assembled from dense models (mergekit-moe style): expert 1 is an exact copy of mlabonne/AlphaMonarch-7B's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 0 of 195 identical.
3 derivatives checked
- mradermacher/MixtureofMerges-MoE-2x7b-v6-i1-GGUF: Quantization
- mradermacher/MixtureofMerges-MoE-2x7b-v6-GGUF: Quantization
- jsfs11/MixtureofMerges-MoE-2x7b-SLERPv0.9b: Full fine-tune