Skip to content

Steps.md

Latest commit

 

History

History
80 lines (67 loc) · 18.5 KB

Steps.md

File metadata and controls

80 lines (67 loc) · 18.5 KB
Raw

Steps for differents parts of the pipeline

Preprocessing data

  1. Scripts can be found in preprocessing directory
  2. The cached data gets stored in - /home/asd/.cache/huggingface/datasets/text/. So make sure there is enough space on the disk

Create vocabulary files

  1. python create_tokenizer.py --file ../../../../BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/wikitext-103-raw/wiki.train.raw --store_files ../config/roberta_test/ --vocab_size 50000

Running code on TPUs

  1. MLM RoBERTa-base - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --output_dir=../../../bucket/model_outputs/wikitext/mono_english_80 --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 80 --warmup_steps 10000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name wikitext_mlm &
  2. MLM with own vocab - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm.py --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --output_dir=../../../bucket/model_outputs/wikitext/mono_english --model_type=roberta --config_name=roberta-base --tokenizer_name=../config/roberta_test --learning_rate 1e-4 --num_train_epochs 80 --warmup_steps 10000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name wikitext_mlm &
  3. Monolingual French model - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --output_dir=../../../bucket/model_outputs/french/mono_french_warmup --model_type=roberta --config_name=camembert-base --tokenizer_name=camembert-base --num_train_epochs 80 --warmup_steps 10000 --learning_rate 1e-4 --do_train --do_eval --train_file=../../../bucket/french_data/processed/fr_gutenberg/train.txt --validation_file=../../../bucket/french_data/processed/fr_gutenberg/valid.txt --cache_dir=../../../bucket/cache --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name monolingual_french &

Word based modifications

  1. Vocabulary permutation TPU - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --cache_dir=../../../bucket/cache --data_cache_dir=../../../bucket/cache --output_dir=../../../bucket/model_outputs/wikitext/english_vocab_permutation --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 40 --warmup_steps 10000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name word_permute_40 --permute_vocabulary --vocab_permutation_file ../synthetic_language_files/word_based/configuration_files/permuted_vocab_seed_42_size_50265.json --vocab_modification random &
  2. Vocabulary permutation TPU without data_cache_dir - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --cache_dir=../../../bucket/cache --output_dir=../../../bucket/model_outputs/wikitext/english_vocab_permutation --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 40 --warmup_steps 20000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name word_permute_60 --permute_vocabulary --vocab_permutation_file ../synthetic_language_files/word_based/configuration_files/permuted_vocab_seed_42_size_50265.json --vocab_modification add &
  3. Resume from checkpoint - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --model_name_or_path=../../../bucket/model_outputs/wikitext/english_vocab_permutation/checkpoint-20000/ --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --cache_dir=../../../bucket/cache --output_dir=../../../bucket/model_outputs/wikitext/english_vocab_permutation --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 40 --warmup_steps 20000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name permute_vocabulary --permute_vocabulary --vocab_permutation_file ../synthetic_language_files/word_based/configuration_files/permuted_vocab_seed_42_size_50265.json --vocab_modification add &
  4. Word modification TPU without data_cache_dir - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --cache_dir=../../../bucket/cache --output_dir=../../../bucket/model_outputs/wikitext/english_word_modification_095 --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 40 --warmup_steps 10000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name word_modification_40 --modify_words --modify_words_probability 0.95 --word_modification add &
  5. Inverted word order - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --cache_dir=../../../bucket/cache --output_dir=../../../bucket/model_outputs/wikitext/english_word_modification_inverted --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 40 --warmup_steps 10000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name inverted_sentence_40 --invert_word_order --word_modification add &
  6. Inverted order French - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --output_dir=../../../bucket/model_outputs/french/french_inverted_warmup --model_type=roberta --config_name=camembert-base --tokenizer_name=camembert-base --num_train_epochs 40 --warmup_steps 10000 --learning_rate 1e-4 --do_train --do_eval --train_file=../../../bucket/french_data/processed/fr_gutenberg/train.txt --validation_file=../../../bucket/french_data/processed/fr_gutenberg/valid.txt --cache_dir=../../../bucket/cache --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name inverted_french --invert_word_order --word_modification add &
  7. One-to-one mapping - nohup python examples/xla_spawn.py --num_cores 8 examples/language-modeling/run_mlm_synthetic.py --train_file=../../../bucket/wikitext-103-raw/wiki.train.txt --validation_file=../../../bucket/wikitext-103-raw/wiki.valid.txt --cache_dir=../../../bucket/cache --output_dir=../../../bucket/model_outputs/wikitext/english_word_modification_one_to_one --model_type=roberta --config_name=roberta-base --tokenizer_name=roberta-base --learning_rate 1e-4 --num_train_epochs 40 --warmup_steps 10000 --do_train --do_eval --save_steps 10000 --logging_steps 50 --per_device_train_batch_size 16 --overwrite_output_dir --run_name one_to_one_mapping_40 --one_to_one_mapping --word_modification add &

Galactic dependencies

  1. Example data for galactic dependencies is in /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/galactic/treebanks-V1.0/treebanks
  2. Converting a CONLLU file to a different word order - GALACTIC_ROOT=$(pwd) bin/gd-translate --input ../../../../data/dependency_parse_data/english/dep_wiki.valid.raw --spec en~hi@N~hi@V --mem 60g. Run from preprocessing/dependency_parsing/gdtreebank. When running slurm, be sure to run the command GALACTIC_ROOT=$(pwd) before starting it.
  3. Uploading to Google Cloud: ./google-cloud-sdk/bin/gsutil cp /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/MNLI/en_hi_N_hi_V/synthetic_dep_flattened_dev_matched-en~fr@N~hi@V.* gs://multilingual-1/syntax_modified_data/english/en_hi_N_hi_V/GLUE/MNLI

Dependency parse of the corpus

  1. Command for converting a corpus to CONLLU file (MLM) - python convert_sentences_to_dependency.py --language en --data /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/wikitext-103-raw/wiki.valid.raw --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english
  2. Command for converting galactic output to corpus (MLM) - python convert_conllu_to_corpus.py --galactic_file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/dep_wiki-en~fr@N~hi@V.conllu --task mlm
  3. Command for converting datasets (like MNLI) to flattened corpus - python convert_dataset_to_sentences.py --data_dir /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/glue_data_new/MNLI --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/MNLI/ --task mnli --truncate 100

MNLI

  1. Convert to flattened corpus - python convert_dataset_to_sentences.py --data_dir /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/glue_data_new/MNLI --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/MNLI/ --task mnli
  2. Convert flattened sentences to dependency (Need to run this for 3 files) - python convert_sentences_to_dependency.py --language en --task mnli --data /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/MNLI/flattened_dev_matched.txt --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/MNLI/
  3. Galactic (Change to gdtreebank) - GALACTIC_ROOT=$(pwd) bin/gd-translate --input ../../../../data/dependency_parse_data/english/MNLI/dep_flattened_dev_matched.txt --spec en~hi@N~hi@V --mem 40g
  4. Convert it back to MNLI format - python convert_conllu_to_corpus.py --galactic_file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/MNLI/dep_flattened_dev_matched-en~hi@N~hi@V.conllu --index_selector /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/MNLI/selected_indices_flattened_dev_matched.txt --supervised_dataset /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/glue_data_new/MNLI/dev_matched.tsv --task mnli

XNLI

  1. Convert to flattened corpus - python convert_dataset_to_sentences.py --data_dir /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/multilingual_nlu/XNLI --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/dependency_parse_data/english/XNLI --task xnli --truncate 50 --language en
  2. Convert monolingual XNLI (train) to JSON - python convert_xnli_tsv_to_json.py --mode train --xnli_dir /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/multilingual_nlu/XNLI/XNLI-MT-1.0/multinli/ --xnli_save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en --language en
  3. Convert monolingual XNLI (dev) to JSON - python convert_xnli_tsv_to_json.py --mode dev --xnli_dir /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/multilingual_nlu/XNLI/XNLI-1.0/ --xnli_save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en --language en
  4. Convert JSON to flat corpus - python convert_json_to_sentences.py --xnli_file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en/train_en.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en
  5. Convert flat corpus to dep - python convert_dataset_to_dependency.py --language en --data /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en/flattened_dev_en.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en/dep/ --task xnli
  6. Convert dep to galactic - bin/gd-translate --input /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en/dep/dep_flattened_dev_en.json --spec en~fr@N~fr@V --mem 40g
  7. Convert galactic to XNLI - python convert_galactic_to_corpus.py --galactic_file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en/dep/dep_flattened_dev_en-en~en@N~en@V.conllu --supervised_dataset /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en/dev_en.json --index_selector /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xnli/en/dep/selected_indices_flattened_dev_en.json --task xnli

NER

  1. Convert monolingual NER dataset to JSON - python convert_token_classification_to_json.py --data_dir /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/multilingual_nlu/xtreme/panx_dataset/ --task_name ner --language en
  2. Convert JSON to flat corpus - python convert_json_to_sentences.py --file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en/dev.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en --task ner
  3. Convert flat corpus to dep - python convert_dataset_to_dependency.py --language en --data /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en/flattened_dev.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en/dep/ --task ner
  4. Convert dep to galactic - bin/gd-translate --input /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en/dep/dep_flattened_dev.json --spec en~en@N~en@V --mem 40g
  5. Convert galactic to NER - python convert_galactic_to_corpus.py --galactic_file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en/dep/dep_flattened_dev-en~en@N~en@V.conllu --supervised_dataset /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en/dev.json --index_selector /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/ner/en/dep/selected_indices_flattened_dev.json --task ner

POS

  1. Convert monolingual POS dataset to JSON - python convert_token_classification_to_json.py --data_dir /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/multilingual_nlu/xtreme/udpos/ --task_name pos --language ar
  2. Convert JSON to flat corpus - python convert_json_to_sentences.py --file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en/dev-en.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en --task pos
  3. Convert flat corpus to dep - python convert_dataset_to_dependency.py --language en --data /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en/flattened_dev-en.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en/dep/ --task pos
  4. Convert dep to galactic - bin/gd-translate --input /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en/dep/dep_flattened_dev-en.json --spec en~fr@N~fr@V --mem 40g
  5. Convert galactic to POS - python convert_galactic_to_corpus.py --galactic_file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en/dep/dep_flattened_dev-en-en~en@N~en@V.conllu --supervised_dataset /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en/dev-en.json --index_selector /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/pos/en/dep/selected_indices_flattened_dev-en.json --task pos

Sentence retrieval (tatoeba)

  1. Convert parallel corpus to JSON - python get_subset_tatoeba.py --filename /n/fs/nlp-asd/asd/asd/BERT_Embeddings_Test/BERT_Embeddings_Test/global_data/multilingual_nlu/xtreme/tatoeba_scraped_4_10/en_hi.tsv --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/tatoeba/en --language1 en --language2 hi --truncate 1000
  2. Convert JSON to flat corpus - python convert_json_to_sentences.py --file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/tatoeba/en/en.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/tatoeba/en/ --task tatoeba
  3. Convert flat corpus to dep - python convert_dataset_to_dependency.py --language en --data /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/tatoeba/en/flattened_en.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/tatoeba/en/dep/ --task tatoeba
  4. Convert dep to galactic - bin/gd-translate --input /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/tatoeba/en/dep/dep_flattened_en.json --spec en~fr@N~fr@V --mem 40g
  5. Convert galactic to tatoeba - python convert_galactic_to_corpus.py --galactic_file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/tatoeba/en/dep/dep_flattened_en-en~en@N~en@V.conllu

XQuAD

  1. Convert dataset to correct JSON format - python convert_xquad_to_json.py --file /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xquad/en/old/train_en.json --save_dir /n/fs/nlp-asd/asd/asd/Projects/Multilingual/data/xquad/en

Running experiments with run_experiment.py (tested locally)

Look at [Debug.md](the debug file) for some example commands, especially for TATOEBA.

  1. For MLM: python run_experiments.py --task mlm --run_type monolingual --language en --other_args output_dir=../data/model_outputs/wikitext/debug/ run_name=en_invert invert_word_order word_modification=replace
  2. For XNLI: python run_experiments.py --task xnli --run_type monolingual --language en --other_args output_dir=../data/model_outputs/wikitext/debug/ run_name=en_invert invert_word_order word_modification=replace model_name_or_path=../data/model_outputs/wikitext/debug/mlm
  3. For NER: python run_experiments.py --task ner --run_type monolingual --language en --other_args output_dir=../data/model_outputs/wikitext/debug/ run_name=en_invert invert_word_order word_modification=replace model_name_or_path=../data/model_outputs/wikitext/debug/mlm
  4. For POS: python run_experiments.py --task pos --run_type monolingual --language en --other_args output_dir=../data/model_outputs/wikitext/debug/ run_name=en_invert invert_word_order word_modification=replace model_name_or_path=../data/model_outputs/wikitext/debug/mlm
  5. For Tatoeba: python run_experiments.py --task tatoeba --run_type monolingual --language en --other_args output_dir=../data/model_outputs/wikitext/debug/ run_name=en_invert invert_word_order word_modification=replace model_name_or_path=../data/model_outputs/wikitext/debug/mlm
  6. For XQuAD: python run_experiments.py --task xquad --run_type monolingual --language en --other_args output_dir=../data/model_outputs/wikitext/debug/ run_name=en_invert invert_word_order word_modification=replace model_name_or_path=../data/model_outputs/wikitext/debug/mlm