Natural Language Processing

What is the difference between ELECTRA and BERT?

Photo by on

(Devlin et al., 2018) is the baseline of NLP tasks recently. There are a lot of new models released based on BERT architecture such as (Liu et al. 2019) and (Lan et al., 2019). Clark et al. released ELECTRA (Clark et al., 2020) which target to reduce…

Natural Language Processing

Data augmentation for NLP — generate synthetic data by back-translation in 4 lines of code

Photo by on

English is one of the languages which has lots of training data for translation while some language may not has enough data to train a machine translation model. Sennrich et al. used the back-translation method to generate more training data to improve translation model performance.

Given that we want to…

Natural Language Processing

An Introduction to Retrieval-Augmented Language Model Pre-Training

Photo by on

Since 2018, the transformer-based language model has been proven to achieve good performance in lots of NLP downstream tasks such as Open-domain Question Answer (Open-QA). To achieve better results, models intend to increase model parameters (e.g. more heads, larger dimensions) in order to stored world knowledge in the neural network.

Edward Ma

Focus in Natural Language Processing, Data Science Platform Architecture.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store