Retraining GPT-2 to write love letters in French

I used the model to write my wife a Valentine’s day love letter (in french). She said it was amazingly beautiful and would have been really moved by it had it not been written by an AI. I counter-argued that human curation of AI generated text was a perfectly valid expression of love. I lost :-|.

You can try it here (in french only): https://impactia.org/cyrano

The result

It generates results that are formatted like love letters with reasonably structured phrases. Verbatim reproduction of training content seems very low. But the sentences are often nonsensical and there’s not much context preservation apart from some repetition.

However it does generate some quite interesting original phrases:

“ Mon âme a moins de conversations que le désir “

“ C’est une joie bien trop violente pour moi “

“ vous [êtes] comme l’eau d’un pur amour “

These phrases can be cherry-picked based on the feeling you’re trying to convey and used to compose unique letters influenced by the best french writers in history.

Why we did it

• Mainly because we all thought it was a funny thing to do for Valentine’s day, but also
• It celebrates the one year anniversary of OpenAI releasing the first model (https://openai.com/blog/better-language-models/).
• It’s potentially a useful tool for the romantically challenged
• We were very curious to see how people would feel about receiving such letters (is it perceived as an act of love or an act of plagiarism?)
• It raises interesting questions about ownership of the result and intellectual property, particularly as some of the texts aren’t out of copyright and there’s a risk that verbatim parts could be reproduced.
• We wanted to see for ourselves how good text generation could be in french

Tech details

A small OpenAI GPT-2 model (117/124M parameters) trained on around 1Gb of french classical litterature was fine-tuned on 300Kb of french love poetry and love songs.

We used GPT-2-Simple https://github.com/minimaxir/gpt-2-simple and GPT-2-Cloud-Run https://github.com/minimaxir/gpt-2-cloud-run by Max Woolf.

The base french model (GPT2-French) was trained by William Jacques: https://github.com/aquadzn/gpt2-french. We used his “romans” model which is trained using text from https://www.bibebook.com/download, which looks like a compilation of project Gutenberg french novels. His model is trained over 5000 steps.

Fine-tuning was done using Max Woolf’s Colab notebook: https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce and lasted only 600 steps (less than 30 minutes). Less steps and the content wasn’t great, more and it started to reproduce training content verbatim.

What went right

• After even 100 steps the model adopts the love letter format
• Everything (GPT-2-Simple, GPT-2-Cloud-Run, GPT-2-French) worked out of the box
• We were astonished to get decent results with a 300Kb dataset (normally 1–10Mb is seen as reasonable)

What went wrong

• It was surprisingly hard to find love expressed in that kind of love letter format. A lot of literature you remember as romantic (e.g. “Les Liaisons Dangereuses”) isn’t in fact good love letter material.
• We were disappointed not to find a bigger GPT-2 model trained in french and on a larger corpus.
• There’s a big difference in quality between the small and XL versions of GPT-2 in english, and we were aiming for the XL level.
• However several people point out that a) the larger models are super hard to train due to memory constraints and b) you need large datasets to fine-tune the larger models.
• We tried using Facebook’s multi-lingual XLM model trained with Causal Text Modeling (CTM) which in theory should make good predictions but the results really weren’t great.
• We initially started using HuggingFace’s Transformer’s library but it was much simpler to use Max Woolf’s projects to get something running in production.

Similar projects that influenced us

• Twitter bot: https://minimaxir.com/2020/01/twitter-gpt2-bot/
• Cards against Humanity Challenge: https://www.cardsagainsthumanityaichallenge.com/

An example

The text below was co-created by cherry-picking phrases out of around 20 generated texts. The idea was to write something from the perspective of a couple going through a rough patch.

Notre relation n’est pas tous les jours facile à trouver. Je ne te demande ni amour, ni fidélité, mais seulement un peu de certitude que je pourrais te satisfaire un jour.

N’ayant rien à nous dire,

N’ayant rien à partager,

N’ayant rien à sentir,

N’ayant rien à mentir,

N’ayant rien à confier,

Hélas! j’aurai passé près de toi inaperçu. Le mal est sans espoir, aussi j’ai dû le taire. L’espoir me semble trop profond, je ne sais quelle mémoire ici-bas peut être l’océan des matins heureux?

Au fond de mon cœur je n’ai jamais su t’aimer mais j’ai pu te séduire. Il n’y a rien à comprendre. Le temps passe.

Mais tu peux ouvrir la porte à demi-close, parmi toutes ces balles imbéciles. Pas besoin de phrases ni de longs discours, mais de s’enivrer.

Il y a quelque chose en moi qui t’implore et te réclame, mon cœur sur l’amour est réveillé. Je te dis que tu me plais, mais j’ai peur de ta fierté que tu n’as pas besoin de moi. Quand je te promets tous mes plaisirs, quand je te promets mes faiblesses, quand je te promets mon âme perdue. Dans le désert je n’ai rien d’autre que tu m’aimes à oublier.

Tu as cessé de sourire et tu continues à me perdre. J’ai perdu le sommeil d’une heure. Je suis prêt à toi. Tu viendras longtemps marcher dans mes rêves.

Je t’aime à mourir.