Report of using TPU Pod to pre-train a Japanese NLP model

Cloud TPU offering by Google

I’ve pre-trained an XLNet Japanese model of the maximum sequence length 512 on a preemptible Cloud TPU v3–256 Pod kindly provided by Google TFRC (for free!), with the Japanese Wikipedia dataset. Although the training was still premature when I had used up all my TPU credits, I finetuned the model to classify Livedoor Japanese news. I got inspiration from the contribution of Yohei Kikuta in…