Byte-level models have been gaining prominence for a while now. The latest entrant is ByT5, a byte-level model by Xue et al., based on the T5 architecture with minimal modifications made to accommodate the larger sequence size of byte-level inputs. Byte-level…

GPT-J, a 6 billion parameter JAX based language model, is the latest entrant in the large language model space. Preliminary tests show that its performance is on par with the GPT-3 6 billion parameter version. Check out the JAX repo here as well as an introductory blog post by one of its…

Data augmentation is a technique critical to the success of applying NLP in low-resource settings. To summarize the current landscape of data augmentation techniques, Chen et al. have published an empirical survey, categorizing the techniques into token-level augmentation…

ML content on Twitch might be relatively hard to come by, but the channel ‘PyTorchTwitch’ is something to watch out for. Their most recent video features Autodesk’s experience in productionizing ensembles of Transformers.