Clickbait Rewriting on Artifact: Behind the Scenes
At Artifact, we harness the power of AI and combine it with creative product design to improve the news & content experience for our users. Lately, we’ve been looking at reducing clickbait titles in our feed, and we have something exciting to show you.
In May, we added a “Mark as Clickbait” button to our app. We learned that most of the articles we show don’t receive clickbait reports. However, for the few that do, they’re often reported by dozens or hundreds of people. For these highly-reported articles, we wanted to provide an AI-powered improvement on the experience.
With the assistance of OpenAI’s Chat API, which we previously integrated with for our Summarize feature, today we are launching clickbait title rewriting:
To rewrite a headline, tap “Mark as Clickbait Title”. The new title will only be visible to you by default. If enough people report the same article, a human reviewer may decide to make it the default title for all Artifact users.
Since this product improvement was in our sweet spot of harnessing AI & being thoughtful about product design, we wanted to share a bit more behind the scenes on how we built it.
Getting the Prompt Right
Integrating with Large Language Model (LLM) APIs like OpenAI’s ChatGPT AI involves having two parts: a system prompt and a user prompt. In our case, the system prompt is the instructions we provide to the LLM about its headline-writing task; the user prompt is context about the article itself.
Writing a good prompt is essential; there are GitHub repositories and educational materials dedicated to improving your prompting craft. In our case, we’d had some prior experience with prompting for our AI-powered Summarize feature. We carried over some lessons, and learned a few ones along the way:
- LLMs are good at role play: we had the best success with a prompt beginning with “I want you to act”:
I want you to act as an news editor (works at Techmeme or Business Insider).
I will provide you with a news article and you will create a headline for it.
Similarly, for our Summarize feature we have several “personalities” for the summarizer. For example, for our “explain like I’m five” voice, the role play looks like:
I want you to act as a teacher that can explain things clearly to a 5 year old.
2. Too much material might backfire: in early testing, we found that if we included anything similar to the original title in our user prompt, the LLM would often rewrite the headline to an almost identical headline as the original. Instead we feed it as much content as we can, but skip anything similar to the original title.
3. Set up a good test harness: we use JupyterLab extensively, for ML work as well as new feature development. For this project, our first step was to gather a set of input examples we wanted, and create a quick test environment where we could iterate on our system prompt. We added some additional instructions to help keep things succinct:
Be succinct and factual. Use fewer than 15 words. Do not wrap the response with quotations.
GPT-3.5 doesn’t always do a great job of counting, so some responses might still be longer than 15 words, but it gets close enough.
4. Responses will need cleanup: even after many iterations on our prompt, we still found the LLM would produce results with extraneous quotation marks, <code> tags, or other noise. After running many examples through the test harness, we created a set of filter rules for post-processing cleanup.
5. Keep a human in the loop: when we rewrite a title given a user request, that new title is initially only visible to the user. If enough people request a rewrite of a given title, the rewritten title will be escalated for human review and if it looks good, we’ll promote it the new default title for all users.
UI Feedback and Transparency
One of the principles we use when applying AI to features inside Artifact is to be as transparent as possible. For both Summarize and the clickbait title rewriter, we’ve used consistent iconography (the star-like shape) to denote that there’s AI involvement:
We’ve also used animation to help communicate that there’s work happening in the background while the user waits for a response. LLM models can take several seconds to reply, and in initial testing we found the lack of feedback led people to think something had gone wrong. We then prototyped a variety of treatments, ranging from ones that felt like “too much” to our eventual winner (on the right) which brought in the same animation language as we use in the Summarize feature:
Again, we opted for consistency with our other AI-powered products.
Summing Up
We think it’s an incredibly exciting time for product builders who can combine AI advancements with useful functionality in products. If you’d like to share your thoughts on this feature (or suggest other ones!) please leave a comment here or on this article in Artifact.
