raincandy-u/Llama-3-Aplite-Instruct-4x8B-MoE
MoE built from dense models vs meta-llama/Meta-Llama-3-8B-Instruct. raincandy-u/Llama-3-Aplite-Instruct-4x8B-MoE is a mixture of 4 experts assembled from dense models (mergekit-moe style): expert 0 is an exact copy of meta-llama/Meta-Llama-3-8B-Instruct's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 195 of 195 identical.
2 derivatives checked
- mradermacher/Llama-3-Aplite-Instruct-4x8B-GGUF: Quantization
- mradermacher/Llama-3-Aplite-Instruct-4x8B-i1-GGUF: Quantization