Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-switch-top1
MoE built from dense models vs meta-llama/Llama-3.1-8B-Instruct. Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-switch-top1 is a mixture of 4 experts assembled from dense models (mergekit-moe style): experts 0, 1, 2, 3 are an exact copy of meta-llama/Llama-3.1-8B-Instruct's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 195 of 195 identical.
4 derivatives checked
- Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-sni-switch-top1-router-ffn-lora-aux-anneal-5ep: Adapter
- Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-sni-switch-top1-router-ffn-lora-sft-5ep: Adapter
- Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-lbox-switch-top1-router-ffn-lora-sft-5ep: Adapter
- Jongbin-kr/llama-3.1-8b-instruct-4x1-moe-lbox-switch-top1-router-ffn-lora-aux-anneal-5ep: Adapter
2 other repos hold byte-for-byte identical weight files.