qing-yao/ppt-pythia-160m-uniform250-previous_mse-seed208-stage1
Full fine-tune vs EleutherAI/pythia-160m. Fine-tune: 100% of the weights changed, across 12 of 12 layers.
Tensors changed: Norms 50/50, Attention 48/48, MLP 48/48
10 derivatives checked
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_delta_shuffle1-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_delta_shuffle2-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_weight_shuffle2_warmup500-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_weight_shuffle1_warmup500-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_delta_shuffle1_warmup2000-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-reuse_previous_mse_warmup2000-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_restore_attention_warmup2000-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_restore_mlp_layernorm_warmup2000-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-previous_mse_weight_shuffle1_warmup2000-seed208-stage2: Full fine-tune
- qing-yao/ppt-pythia-160m-uniform250-initial_weight_shuffle1_warmup2000-seed208-stage2: Full fine-tune