Jacoby746/Proto-Athena-v0.2-4x7B
MoE built from dense models vs BAAI/Infinity-Instruct-7M-Gen-mistral-7B. Jacoby746/Proto-Athena-v0.2-4x7B is a mixture of 4 experts with the same shape as BAAI/Infinity-Instruct-7M-Gen-mistral-7B's MLP, but none of them is a copy of it: they come from other models or were trained further. Shared tensors (attention, norms, embeddings): 195 of 195 identical.
1 derivatives checked
- mradermacher/Proto-Athena-v0.2-4x7B-GGUF: Quantization