This is the second in a series of blog posts on election experiments. To read more about Audience and our rationale for conducting these experiments, see blog 1 and check out our website (www.audienc3.ai).
Experimental method
First up, the Granite State! You may be wondering: why start here? After all, Iowa (traditionally) comes first, and New Hampshire only has 4 electoral votes. All that is true, but this year, New Hampshire was also a key part of candidate Nikki Haley’s strategy to beat incumbent (and far-and-away frontrunner) Donald Trump. In short, we picked New Hampshire because it was the first race that’s likely to be competitive, and predicting competitive races makes for more interesting blogs.
In this experiment, we started with two revs: (1) a “naive” model prompted with representative voter demographic profiles compiled from census data with our proprietary population builder algorithms and (2) the same model with what we call “thought activation” — fine-tuning with data from several, recent articles that NH voters have likely encountered in the last several days. In both cases, we’ll follow a statewide “polling” approach — ignoring district-by-district voting by simply selecting a sample that represents the demographics of the overall state. For the Bayesians out there, our prior is that Llama (the current base model we’re working with) will do a reasonably good job getting the outcome of the race right, but getting the margins right may require augmenting the demographically prompted participants with top-of-mind headlines, media, and data points pertinent to flesh and blood New Hampshire voters.
Ready? Here…we…go!
1/23/2024 Prediction
Compared to the actual results (Trump 54.3% / Haley 43.2% / Other 2.5%), our naive model performed remarkably well! The only obvious mistake was in our prompt, which only listed Trump and Haley as options. As a result, in our simulation, Trump picked up the collective long-tail of 2.5% that went to other candidates in the actual race (sorry Lizard Supreme🦎).
The “thought activated” model was another story. As a reminder, this experiment layered context from articles from Fox News, the Associated Press, the New Hampshire Union Leader, and New Hampshire Public Radio into the prompt when instantiating our simulated voters. Based on the outcome, it seems like the choice of articles was too pro-Haley, which makes sense given the NH Union Leader’s endorsement of her and general media buzz around the last serious Republican contender against Trump.
Next steps
Based on the output of this experiment, we want to try 3 new things in the next round:
- Allowing for “other” candidates besides the two frontrunners
- Picking a more representative corpus of news/media inputs for “thought activation”
- Evaluating the effectiveness of SLMs (small language models) built with Audience for voter prediction
That’s all for now. See you in Nevada!
