Using naive sentence embeddings from BERT or other transformer models leads to underperformance. To get around this, we can fine-tune BERT…