Are Credit Models Dead?

WEEL’s COVID Score and the future of credit modeling

For credit risk professionals like myself, COVID-19 has been an extremely humbling experience. Our job is to predict the future, and black swan events like this simply cannot be predicted. To make matters worse, governments around the globe quickly and erratically distributed trillions of dollars in aid to businesses and individuals undermining and distorting all of the financial and payment data that we rely on for our predictions.

Over the past few months, I’ve been on a mission to talk to my fellow credit risk professionals in almost every major world economy. These have been supportive but also challenging conversations. It almost feels like a support group for (formally) brilliant people that have had their heads bashed a few too many times recently. When I talk to colleagues about a clear vision for the future, I get a lot of blank stares.

Sifting Past The Nonsense

Articles by “industry experts” on this topic add to the collective confusion. Reading most of these articles make me feel like I want to vomit. This article from Ernst & Young, for example, drowns the reader in useless, generic statements like:

Most of the models were built on historical data from the last decade, which is not representative of the current environment.

And then the article continues on to squeeze in as many buzzwords as possible to basically promise a credit risk crystal ball.

EY’s experience suggests that we can apply a combination of macroeconomic approaches (general equilibrium and input-output) and pandemic susceptible, infected and recovered (SIR) models, as well as bottom-up sector and geographic recovery perspectives, in order to generate scenarios accounting for lockdown risk, sectoral impacts, policy responses and international risk transmission.

Are you kidding me? Are you going to start using your magic wand to predict the spread of a global pandemic, the recovery, and the specific impact by company or individual. C’mon EY! For credit professionals living this in the trenches, we can’t rely on a lot of theoretical approaches and buzzwords.

Embracing Reality

I’m going to try something bold both for the cathartic effect and to embolden others in my field on how to move forward. I’m going to be honest. WEEL’s pre-COVID credit models performed below expectations during COVID. That’s the truth for us, and I think after a few beers, most lenders around the world would admit to the same thing. A few leaders have even alluded to this publicly. Sudir Jha, senior vice president and head of Brighterion, the artificial intelligence (AI) company owned by Mastercard, told PYMNTS:

How up-to-date you are on the payments plays a very big role in the models … and that data may not be very useful.

Jha’s statement makes a lot of sense in a world where “zombie companies” are being held up by government loans and grants. All payment and non-payment data becomes distorted.

Yes, the truth hurts, but it’s the beginning of a new path forward.

Moving Forward

WEEL’s primary lending business is factoring. Our average receivable duration is only around 45 days, which means that we have already seen full performance data on multiple cohorts during COVID. So now for the good news. Our new models, incorporating our learnings during COVID, perform better than any models we have ever built!

What type of improvements have we seen? The Kolmogorov-Smirnov (KS) Statistic is one of the textbook methods for evaluating credit models. Models in the range from 20% to 70% are normal. We have been able to hit KS Statistics of 78.3% compared to a benchmark of 26% for Brazil’s credit bureau, Serasa.

We are currently working directly with Serasa to validate our new score on more data sets. We are also very open to peer review with other companies in our field. If you would like collaborate with us on a review of the WEEL COVID Score, please contact me on LinkedIn.

How did we reach this big milestone? I’m not going to give away all of the secrets, but I’ll point out a few interesting things that we learned along the way during the research.

In-Market Stress Tests

Stress tests are a very big topic in the credit risk world mostly thanks to the BASEL Accords. Stress testing includes a wide range of simulation techniques to test how a portfolio will perform in different financial stress scenarios. Stress tests are usually a theoretical exercise, but the early months of COVID’s spread provided us with an empirical stress test. We were able to see how borrowers performed in an environment where banks were cutting credit lines and buyers were delaying payment. Moreover, we had new data on borrower payment intent. We knew our borrowers had cash reserves, but during COVID we have been able to see how borrowers allocated their cash reserves in a real life stress scenario. This new information provided incredible insights for our new model.

Past Performance Was Not a Strong Feature

Following along the lines of the quote from, Masterscard’s Sudir Jha cited earlier, we also observed that payment data was no longer an effective way to evaluate the financial strength of companies. Non-payment from Company A might signal a distressed scenario, for Company B it could be associated with increasing cash reserves, and for Company C it could just be a big clerical error from confusion during the switch to home office. The myriad of new reasons for non-payment distorted this traditionally strong feature and forced us to exclude it from the model.

New Factors Indicating Company Instability

In the absence of strong payment data to drive our models, we set out searching for new features associated with financial instability. (A) Are we seeing a decrease in the number of employees associated with this company on LinkedIn? (B) Have we observed any erratic changes on the information on the company web site? (C) Has the company changed its ownership or board structure? (D) Do any key employees live in COVID hotspots? (E) Has the company’s supply chain faced disruption? As we asked ourselves more and more questions, we became more creative about finding new data sources and eventually found ways to innovate and succeed.

Congratulations

As a closing note, this is a great opportunity for me to congratulate my team on this big achievement. First and foremost, I want to congratulate Eliezer Schwarzberg for his role as lead data scientist on this model. This model is the culmination of over a year of research and hard work. Secondly, I would like to congratulate Hershel Safer for his role in developing many of the modules and techniques that we used for this model. Finally, I would like to thank Richard Iwai and Bruno Matos for their help in the model evaluation. Great work team! I can’t wait for the next version!