CultriX/MoNeuTrix-MoE-4x7B
MoE built from dense models vs CultriX/MoNeuTrix-7B-v1. CultriX/MoNeuTrix-MoE-4x7B is a mixture of 4 experts assembled from dense models (mergekit-moe style): expert 2 is an exact copy of CultriX/MoNeuTrix-7B-v1's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 0 of 195 identical.
2 derivatives checked
- mradermacher/MoNeuTrix-MoE-4x7B-i1-GGUF: Not comparable yet
- mradermacher/MoNeuTrix-MoE-4x7B-GGUF: Not comparable yet