Skip to content

Latest commit

 

History

History
74 lines (45 loc) · 2.73 KB

tpu_training_EN.md

File metadata and controls

74 lines (45 loc) · 2.73 KB

한국어 | English

How to Use TPU for Pretraining ELECTRA

1. Apply for Tensorflow Research Cloud

Tensorflow Research Cloud (TFRC) is a program that allows you to use TPU for free for 1 month.

When you go to the link and apply, you will receive an email.

image

If you fill out and submit additional applications as required by the email, you will receive a reply as follows, and from that point, you will be able to use TPU in GCP:)

image

2. Upload data on Bucket

  • When using TPU, all input files must go through the Cloud storage bucket. (Related FAQ)

2.1. Create Bucket

  • Let's name the bucket test-for-electra.

  • Go to [Storage]-[Browser] on the left side of the GCP main page.

  • Click 'Create Bucket'.

  • It is recommended to make a bucket in the same region as the TPU to be used.

    image

2.2. File Upload

  • Upload prepared pretrain_tfrecords and vocab.txt to Bucket.

    image

3. Create GCP VM & TPU

  • Rather than making VM and TPU separately, it is recommended to open the cloud shell at the top right and enter the following command.

  • It doesn't matter if the VM instance is light because the storage is processed by the bucket and the operation is performed by the TPU.

$ ctpu up --zone=europe-west4-a --tf-version=1.15 \
          --tpu-size=v3-8 --machine-type=n1-standard-2 \
          --disk-size-gb=20 --name={$VM_NAME}

image

4. Now Pretrain your own ELECTRA

$ git clone /~https://github.com/google-research/electra
$ cd electra
$ python3 run_pretraining.py --data-dir gs://{$BUCKET_NAME} \
                             --model-name {$MODEL_NAME} \
                             --hparams {$CONFIG_PATH}

5. Delete Instance and Bucket after completing training

$ ctpu delete --zone=europe-west4-a --name={$VM_NAME}
$ gsutil rm -r gs://test-for-electra

Reference