tardellirs/gemma3moe-16x4-270m-starter
MoE built from dense models vs google/gemma-3-270m. tardellirs/gemma3moe-16x4-270m-starter is a mixture of 16 experts assembled from dense models (mergekit-moe style): experts 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 are an exact copy of google/gemma-3-270m's MLP; the other experts come from other models. Shared tensors (attention, norms, embeddings): 182 of 182 identical.