Major Changes: The Good, The Bad, And The Awesome (or The Ugly)

I’m happy/anxious/excited to announce some pretty big changes being made to the app.

4 years ago I started compiling ADP to help build a methodology around placing generic relative market values on players. It started off as something I did for my own teams, then became an Excel spreadsheet, and has now been 3 different web apps. I wanted to approach dynasty fantasy football as you would any marketplace, and to do so required being able to place some sort of value on players to be able to track how well i was maintaining or growing that value. Of course this alone isn’t enough to win at fantasy football, but I felt that by maximizing the amount of “value” I had on any of my teams, I was setting myself up for long term success — and by keeping these values in mind, I could avoid making bad decisions with my players and assets.

I used ADP because to me it represented the perfect snapshot of community “value” — a set of drafts by 12 random people who had to choose players from amongst the entire field of possible selections. If someone chose Dez Bryant over Julio Jones, it’s because they ultimately valued Dez Bryant more than Julio Jones. This didn’t necessarily mean Dez was going to be the better player (I think personal…or any rankings for that matter, are better for this), just that the market viewed Dez as the more valuable player. To me ADP was useful for establishing how the market felt about players, which I could then compare to how I felt about those players and try to expose those gaps. If someone was higher than they were on my list, I’d try to sell them for someone in that range closer in line with my own evaluations. If someone was lower, I’d try to buy them. These gaps represented opportunity, and while it was never perfect, I would always start my negotiations and target players using these.

This eventually evolved into a tool that combined relative values being exchanged to try and quantify in a very imperfect way the players changing hands — not to determine who “won” or to decide if I should make a move, but more to make sure that my moves were falling somewhat in line with my philosophy. It helped me make sure I wasn’t buying or selling players too far beyond their values. Obviously a lot of people reading this found a similar use for

However recently I found myself questioning the value of ADP for this. As I participated in mocks and spoke with people who often did the rounds on the big sites, I found that the data wasn’t exactly what I’d perceived it to be. In order to get so many mocks done in a month, many ended up reusing the same people over again — writers, twitter followers, and other reliable participants who could be trusted not to bail after round 8 in a 20 round mock draft. It’s not that the data is bad, just not quite as insulated from the opinions of a few as one would hope it would be.

For this reason and more I’ll get into below, I will be moving the site values away from ADP. For many that will be a deal breaker, and that is fine. This site has been a really fun project to flesh out and share my philosophy around dynasty fantasy football, and if your strategies and beliefs no longer coincide with my own shifting philosophy — that’s a perfectly acceptable scenario.

For those who haven’t already deleted my app from their bookmarks bar, I’ll explain exactly what my thought process is for this. I’ll explain what the site is moving towards. And I’ll explain why I’m really freaking excited about it.

ADP is extremely limiting, and it’s frustrating to work with

I already got into a bit about why I don’t believe ADP is as great of a market snapshot as I used to. That doesn’t mean it isn’t useful — I still use it as a tool to get a sense of where players are going in drafts. However it has a lot of drawbacks, and these combined with the realization above are why I’m moving on (at least as the main backbone of the tools).

Without the randomness of ADP, without the creeping in of the opinions of a handful of super users, it ultimately doesn’t become much better than aggregated rankings. Unlike aggregated rankings, ADP suffers from:

1. Being a huge pain in the ass to compile
2. Being a huge pain in the ass to find
3. Being a huge pain in the ass to keep up to date

Mock drafts are hard. Ask anyone who has been in a mock I’ve tried to run, keeping 12 people focused for 20 rounds on a draft that will yield them a team they will never use isn’t an easy task. They can take a really long time to run through, and ultimately its difficult to get more than 5–10 done in a single month (usually closer to the bottom of that range than the top). This makes the size of the datasets extremely limited. Because ADP by definition uses averages and not medians, it also means it ends up being more greatly affected by outliers and the opinions of a few recurring participants than I’d like.

Mostly because of the last point, the data is just hard to find. A few websites have it, but rarely do they have exactly the high quality data I require for my personal evaluations, and thus my app — year round, consistently released, large enough to capture a lot of players and frequent enough to provide sufficient data points. There are platforms out there that can generate mocks, but they go the other end of the extreme and don’t curate their data at all, leaving them even more open to outliers.

And finally, ADP by its very nature (and again thanks to point 1) is always stale. I’ve outline this before, but think about it — if Jimmy McRookieFootballer breaks out in Week 1, data from mock reflecting this typically won’t be available until the middle of October. In a game where the season is only ~4 months long, having data lagging behind that far makes it extremely difficult to justify building a concept of a “market” around it. Think about it — if Intel came out tomorrow and said they were pivoting and were going to start creating custom teddy bear outfits, do you think their stock would wait a month and a half before it started to reflect this change? Or would you expect their stock to plummet immediately (or rise…who knows!)? I know if I had two services that used two different methodologies, and one was delayed by a month and a half in giving me actionable information that I based my acquiring or dumping of assets on, I’d choose the one that was more agile.

The site became something it was never meant to be

Because of the problems I went over above, my code ended up finding only a handful of reliable sites to get data from. Over the past 2 years, I found that the data was really only being pulled from 2 (www.dynastynerds.com and www.dynastyleaguefootball.com). And after auditing the past year of my automatic data scraping, which had built in assessing of the state of the data it was pulling in (aka mostly when it was last updated), the vast majority of months ended up coming from one source — Dynasty League Football.

This is a problem, and when I realized this I knew this couldn’t continue.

I found too often on Reddit that, when people asked for ADP, my site would be shared. My site was never intended to be a place you could go to for ADP. It wasn’t so bad during the months when the app was actually aggregating ADP as I envisioned, and in this sense I thought the idea of aggregated ADP could be something people came for, but when I found out it was almost always just coming from DLF — guys who are awesome and put a ton of hard work into acquiring that data — the fact that people were able to essentially circumvent going to their site was something that I am 100% not OK with. Not the fact that people were doing it, but the fact that my system broke and was taking advantage of guys who have done a ton for the dynasty community.

There are a few reasons why I believe strongly in this.

• DLF ADP is behind a paywall. When it was part of their generous free content, I can see an argument that it wasn’t a big deal. Now that it is behind a paywall, I’m essentially unintentionally exposing their data and allowing people to not pay for paid content. It is stealing, and I appreciate everything they do too much for me to be ok with that. I do not make any money off of this app. In fact I lose literally thousands of dollars a year in server costs and development time. But this is not intended to be a free service that let’s users get around paying for quality content.
• DLF is starting to do more with the ADP. Originally I began developing my tools because I felt like there was a lot of opportunity in ADP. I wrote articles about valuation gaps, I created charts and graphs to compare trends and changes. My very first tool was basically just tracking ADP over time. This is something they’ve begun exploring on their own, and so I feel for me to use their premium data (and unfortunately because of the lack of the high quality data I require, it would have to be almost exclusively) doubles down on my above point.

Just to be clear about this, this is not DLF pointing this out. They’ve been the best source for content in this community for a really long time, and their philosophy of sharing and collaborating within the community has helped drive the growth we’ve been seeing in the format. This is 100% my choice and not anything coming from them — they’ve been extremely supportive of everything I’ve done since I started.

The site needs to lean even more on automation

Keeping up with ADP requires a ton of manual labor no matter how much I try to automate it. Some sites only have ADP part of the year, most sites don’t update theirs consistently, some sites will suddenly change up their naming/tables/some-other-thing-they-changed-for-who-knows-what-reason. Fitting this into the structured data that makes up my database is difficult, and making sure that these values pass through and don’t blow stuff up requires a lot of sifting through incoming data. In many ways I’m lucky ADP is only really ever available once a month, because manually going through the data to make sure Devante Parker/Parker, Devante/Parker, DeVante WR MIA 100 and any other variations all end up as the same player in my database is a lot of work. This is a one man operation, and this manual work in addition to the creation/maintenance of features can easily see me putting 20+ hours a week into the app.

So that’s half of the big news. Now the good news. Over the past few months I’ve been experimenting with tying values to other sources of data, and I’ve found one that I am really excited about. Fantasy Pros has been working to greatly expand their dynasty content, and as a result their dynasty rankings have become a pretty awesome resource. They are now up to 25 experts consistently updating their rankings from a bunch of well known dynasty sites including Dynasty League Football, Dynasty Nerds, Dynasty Trade Calculator, Dynasty Football Warehouse, and more. These are sites that I’ve found myself aggregating data from over the years, and the fact that the Expert Consensus Rankings (ECR on FantasyPros) now reflect an extremely varied set of opinions made me start to wonder if it would be a useful replacement for ADP. For those who don’t know, ECR is something FantasyPros uses to curate their data. It’s based on filtering out data that is no longer up to date, as well as only pulling from rankers they audit based on accuracy.

Over the past few months I’ve built a tool that basically does it, and have actually been using it for my own personal valuations on my teams. This is what I’ve found:

• The data is diverse enough to be valuable. When you have aggregated rankings from up to 25 different sources, this suddenly becomes a dataset large enough to provide a data point I am pretty confident in. If you compare it to ADP, you’re basically comparing 5ish mocks to, at the time of me writing this, 17+ different ranking sets on FantasyPros current ECR
• It’s pretty awesome having reactive data. I’ve found updating it weekly provides data points that can capture changes in player valuation pretty well. If Washington drafts a RB, it’s great being able to see Robert Kelley’s value quickly drop.
• The data is predictable and public. I don’t have to log in or pay to see their data (though their premium stuff is pretty cool because you can manually filter and change the rankings yourself), and so I am comfortable using it and attributing where its coming from. The raw numbers anyone can more or less see right on their site. Furthermore, their formats and ids don’t really change, which means I can set a script to simply pull data at any interval I want and be pretty confident things won’t blow up.
• Their data is pre-curated. I have a pretty high bar for quality when it comes to data I use, and I have the same for my app — its one of the reasons why I never used this before. However with a greatly increased number of rankers, combined with the built in curation they have, I am comfortable bringing in any datapoint that has 15+ rankers who’ve recently updated.

At this point the trade offs of moving from ADP to this seem to be worth the benefits, which include weekly value updates, more reactive data, and arguably larger data sets.

It also opens the doors for me to do cooler stuff with the data. Some of it you may have already seen in action.

Along with the average data point that current value is based on, you’ll be able to see the range that makes it up.

The player comparison tool now allows you to directly compare the ranges that make up the current value point for different players. You can see their average rank, their min and max spots, and a “confidence zone” made up of 1 standard deviation in both directions of the average.

One way you can see the above tools in action is if you look at a trade — Dez Bryant for Mike Williams.

In the calculator, it says pretty much what you’d expect — most people would agree this is not a fair trade.

However we know that the calculator is not the end all be all, and you really like Mike Williams…you just want to make sure you’re not crazy. So you put them both in the Player Compare Tool and see this graph:

This seems to tell a different story. While it still clearly shows a gap between where the two players are being valued, you can now see a lot more going on. On one hand, the community doesn’t seem to be too sure of what to think about Mike Williams based on his pretty wide range of positions. On the other, Dez’s current data point could be skewed by an outlier dragging his average down. If you limit the ranges to just their confidence zones — the points representing average position +/- standard deviation, you see there is still a gap. Or you could see that there are people taking Mike Williams before Dez entirely, and be completely OK with giving up Dez for him, or maybe lowering your asking price to make it happen.

The point is, you can use this data however you want. These are the kinds of tools I want to keep building, and this new way of applying value has already and will continue to unlock more awesome ways to build them.

Let me be clear. I am very confident in the new system and very excited to get it out there. This move is primarily being driven by me finding a better way to do this. Value will now be based on much larger data sets, data points will be much more frequent, value will be reflective weekly based on things happening (maybe even more frequently), and I simply have a lot more data to work with.

This is already out in the wild. The data is now pulling in weekly on Wednesdays in the middle of the night. Play around, take a look at what — if anything — has changed (you will have seen a drastic dip in your team values, but this is because of a change in relative values of players, not the new data source). I still have to update the wording around the site to make it clear this is no longer ADP, but the latest data is based on this new system.

I hope people are as excited about this as I am. For those who are inevitably disappointed, I welcome your comments and suggest you check out some of the sites I used to pull ADP from and start playing around with the raw data yourselves.

Thanks and best,

Nick