The problem of rating screenplays

What lessons are learnt when you read (most) of the Black List 2021 screenplays? A lot, along with a lot of problems.

This is a 2 part blog. This part will deal with the problem of ratings, the second will cover lessons learnt.

The Black List is the self-styled list of ‘top’ unproduced scripts, and while we don’t know the exact details, it is generally based on votes from agents, producers and other Hollywood insiders.

Or as the site itself puts it:

The Black List was compiled from the suggestions of more than 375 film executives, each of whom contributed the names of up to ten favourite feature film screenplays that were written in, or are somehow uniquely associated with, 2021 and will not have begun principal photography during this calendar year.

This year, scripts had to receive at least seven mentions to be included on the Black List.

The Black List is not a “best of” list. It is, at best, a “most liked” list.

Earlier this year I set out to read the entire 2021 Blacklist. I am most of the way through but it was only recently that I started publishing the reviews.

This was in part because I wanted to use ScriptReader.studio’s unique rating system and be able to easily analyse the data. So I spent a few weeks building that before I could come back to analyse everything and test it.

It is undergoing improvements and a version 2.0 but you’re not here for that.

Top 10 Black List 2021 scripts — and the problem of ratings

These are the top 10 scripts ranked the traditional way — due to the way we rank the scores can go as high as 100% as this is conversion from a multi-rank rating

The above table shows the traditional rating method of asking for a score at the end of reading. However, we have our own unique rating to track engagement over the course of a script that we can plot in this chart.

What’s the difference? Well for one the graph is a little messy so let’s focus on the highest rated and the lowest: Cauliflower, Mercury and Killer Instinct.

This shows an interesting result. All three started off with the same initial rating but Killer Instinct declined whereas Cauliflower rapidly gained interest, but had a rough patch, while Mercury took a little while to warm up but then rocketed up.

So what makes any of these scripts better than the others?

There are different ways we could go — highest average rating, biggest improvement, biggest change, most consistent. We measure a lot of things.

Let’s return to that traditional ratings table where only the overall rating is asked for at the end (one of the ratings we ask) and see what my top 10 is.

The problem as you can see is that the top 3 positions are tied as are the next, all jointly 1st and then jointly 4th. And are all those scripts really 10/10s, worthy of 100%?

Here are some other measures to create my own rank this time using the highest average page-by-page engagement. The ranking is created by this overall score, but I also include the score for the first 10 pages.

Why the first ten pages? because this is a traditional method in Hollywood of determining whether a script is worth pursuing.

It is not always precisely ten, it’s more a guideline than a rule, but I did find that I could often tell by the first 10–20 pages how likely I was to enjoy the script overall. Very few improved massively from the first ten pages.

Of course this is useful to writers who have a good script but want to know if their first ten pages help or hinder their prospects.

Can we say that one is better based on this? If we go for consistency then yes, but this is something we’ll be testing over the coming months.

As you’ve spotted, this is a table heavy analysis and along with the engagement chart earlier we are working on a more stripped down version we’ll release in the future.

Top ten Black List 2021 script reviews

If you’d like to read my reviews and see how I rated them visit the following blogs:

1. Cauliflower
2. See How they Run
3. Divorce Party
4. Killer Instinct
5. The Villain
6. Ultra
7. Wait List
8. Mercury
9. In The End
10. Mr Benihana

So if this is the quantitive take then what stands out for the qualitative? Check that out in part 2 for the main thing that was learnt from reading the scripts.