Local-Novel-LLM-project/Shadows-MoE
MoE built from dense models vs Local-Novel-LLM-project/Ninja-V3. Local-Novel-LLM-project/Shadows-MoE is a mixture of 2 experts assembled from dense models (mergekit-moe style): experts 0, 1 are an exact copy of Local-Novel-LLM-project/Ninja-V3's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 195 of 195 identical.
2 derivatives checked
- mradermacher/Shadows-MoE-GGUF: Quantization
- mradermacher/Shadows-MoE-i1-GGUF: Quantization