SteelStorage/Umbra-v3-MoE-4x11b
MoE built from dense models vs Himitsui/Kaiju-11B. SteelStorage/Umbra-v3-MoE-4x11b is a mixture of 4 experts assembled from dense models (mergekit-moe style): expert 0 is an exact copy of Himitsui/Kaiju-11B's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 0 of 291 identical.
2 derivatives checked
- mradermacher/Umbra-v3-MoE-4x11b-i1-GGUF: Quantization
- mradermacher/Umbra-v3-MoE-4x11b-GGUF: Quantization