DrRiceIO7/Franken-MoE-Small-Base
MoE built from dense models vs Qwen/Qwen3-0.6B. DrRiceIO7/Franken-MoE-Small-Base is a mixture of 8 experts assembled from dense models (mergekit-moe style): expert 0 is an exact copy of Qwen/Qwen3-0.6B's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 226 of 226 identical.
2 derivatives checked
- DrRiceIO7/Franken-MoE-Small-SFT-Preview: LoRA-style
- DrRiceIO7/Franken-MoE-Small-SFT-Preview-v2: LoRA-style