NeuralNovel/Mini-Mixtral-v0.2
MoE built from dense models vs mistralai/Mistral-7B-Instruct-v0.2. NeuralNovel/Mini-Mixtral-v0.2 is a mixture of 2 experts assembled from dense models (mergekit-moe style): expert 1 is an exact copy of mistralai/Mistral-7B-Instruct-v0.2's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 48 of 195 identical.
2 derivatives checked
- mradermacher/Mini-Mixtral-v0.2-i1-GGUF: Quantization
- mradermacher/Mini-Mixtral-v0.2-GGUF: Quantization