Marsouuu/MistralBase-4x7B-MoE-ECE-PRYMMAL-Martial
MoE built from dense models vs mistralai/Mistral-7B-v0.3. Marsouuu/MistralBase-4x7B-MoE-ECE-PRYMMAL-Martial is a mixture of 4 experts with the same shape as mistralai/Mistral-7B-v0.3's MLP, but none of them is a copy of it: they come from other models or were trained further. Shared tensors (attention, norms, embeddings): 195 of 195 identical.