jsfs11/MixtureofMerges-MoE-4x7b-v5
MoE built from dense models vs Kukedlc/Neural4gsm8k. jsfs11/MixtureofMerges-MoE-4x7b-v5 is a mixture of 4 experts assembled from dense models (mergekit-moe style): expert 2 is an exact copy of Kukedlc/Neural4gsm8k's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 54 of 195 identical.
3 derivatives checked
- mradermacher/MixtureofMerges-MoE-4x7b-v5-i1-GGUF: Quantization
- mradermacher/MixtureofMerges-MoE-4x7b-v5-GGUF: Quantization
- tensorblock/MixtureofMerges-MoE-4x7b-v5-GGUF: Quantization