Rakuten/RakutenAI-2.0-8x7B
MoE built from dense models vs Rakuten/RakutenAI-7B. Rakuten/RakutenAI-2.0-8x7B is a mixture of 8 experts with the same shape as Rakuten/RakutenAI-7B's MLP, but none of them is a copy of it: they come from other models or were trained further. Shared tensors (attention, norms, embeddings): 0 of 195 identical.
1 derivatives checked
- Rakuten/RakutenAI-2.0-8x7B-instruct: Full fine-tune