Macroeconomic predictions.

Uki D. Lucas
Nov 27, 2020 · 6 min read

Can we create AI models to predict macro-economic trends?

Image for post
Image for post

Not too long ago, the prediction of the weather was a hit or a miss. The joke was that the TV weather forecaster was carrying an umbrella on what was to be supposedly a nice day. Today, we can model the weather patterns very accurately for several days into the future.

Similarly, given enough relevant data, including market indicators, human behavioral patterns, and current events and the “black swan” events, we should be able to model the macro-economics.

Image for post
Image for post

In the words of someone very famous, the idea is to be “less wrong”.

Being consistently better than an average, even by a few percent in predicting the future, would have a big benefit.

For years, I have been discussing with my friends the concept of AI modeling based on thousands of macroeconomic data streams, and the opportunity it could give to the architect of such a system. Now, I feel that the data, the software and the hardware is within the reach of that goal.

It is very common to hear the business or the political pundits speak of what amounts to mono-thematic bias based on the very limited ability of a human mind to grasp only a few dimensions of a given topic. Here, I am trying to have AI combine thousands of complex data streams (think of each as a different spreadsheet metric) into a consistent and verifiable oracle.

At the start, I employed easily accessible market indications, data that influences the immediate macroeconomic changes such as manufacturing, unemployment, new housing permits, inventories, etc.

I made heavy use of averages as I realize that certain events build up to a change over periods of time, sometimes days, weeks, or months.

Of course, I realize that historical data is not enough.

I am interested in human behavior as well. Pundit and influencer opinions, even if wrong, definitely affect human behavior. We know for a fact that markets, or rather large groups of people, behave illogically. However, there are patterns in there in how an average Joe listens to the presidential speech, watches few pundits argue, reads an article or two, talks to friends at work, watches few videos titled “Imminent crash in [insert any year]”, hears that someone got laid off, and finally starts selling their stocks a few days later making a colossal investment mistake.

Such modeling is difficult, but it is achievable. We are already regularly deducting a human sentiment of Google searches, YouTube videos on the newest Apple or Tesla innovations, Twitter blasts, and blog posts. It is all known and proven technology.

This project is not meant to be a quick win, I do realize that every hedge fund is pouring millions in similar research. Rather, I want it to be a gradual improvement learning process, a labor of love, and means of recurrent income towards my retirement, I still have about 20 years to solve this complex problem.

The plan is that every few days I add new data sources and continue building the models until I get good at predicting something. I have used the word “something” on purpose. I am not sure if I will be able to predict the particular price of the S&P 500, but I am sure I will find patterns in some niche, and having a wizard-level insight might open doors, a path less taken.

This article, and the accompanying open-source code, is not meant to teach you about the market, nor to teach you how to code, but rather to share my progress and to receive positive feedback and cooperation.

Project Objectives

  • to gather and analyze as many market indicators as possible
  • to learn patterns and interactions in the data
  • to predict market trends 7, 30, or 90 days ahead
  • to predict a particular stock price
  • to produce a re-usable code and document the process
Image for post
Image for post

What are Market Indicators?

Market Indicators are collections of data points (think spreadsheets) reflecting the historical performance of a particular area of interest, for example:

  • “S&P 500” index shows how major 500 stocks as a whole are performing
  • “ISM Manufacturing” index shows how well the manufacturing industry is doing
  • “GDP” (Gross Domestic Product) index shows how the country is doing
  • etc., etc., etc.

There are hundreds, if not thousands, of such indicators. Each country, state, county, and community have sets of data that reflect some trend.

Why not using the spreadsheets?

The spreadsheets and their graphing capabilities are the bread and butter of an individual person’s market analysis. However, spreadsheets are good when comparing only a few indicators, when the data starts multiplying the inputs are overwhelming. Please remember that an average person can remember about 7 numbers, now try to imagine daily data for the last few decades coming for a few hundred indications, multiplied by a couple of hundred or regions, and add to it sentiments from hundreds of sources. We can easily talk about millions, if not trillions of inputs.

Today’s computers can process many “tera” operations per second.
Tera is 1,000,000,000,000. I think the point is clear.

Machine Learning approach

The overwhelming advantage of machine learning, or Artificial Intelligence (AI), is apparent when we try to find subtle patterns in thousands of data flows.

The human brain (using spreadsheets) fails to grasp the wealth of the information presented. Machine learning, on the other hand, can easily detect the patterns in massive datasets and derive a conclusion.

Why using Julia?

If I have chosen Python I would probably make my life easier. However, I was thinking about a long, multi-year, term investment, and wanted to select absolutely the best language.

Here are a few of the characteristics of Julia language:

  • Julia is modern, fast, elegant, multitasking, and does the math extremely well, of course, we are talking about Julia programming language
  • unlike C/C++, it is a pleasure to read and write Julia
  • it is designed in MIT specifically for scientific computing and machine learning
  • similarly to C, it is extremely fast
  • similar to Python, it is very easy to learn
  • it is designed for parallelism
  • it is designed for distributed computing, as one day I might decide to run hundreds of predictive models in parallel and not in sequence as I do now.

May the source be with you, Luke

I am making changes to the project’s code on a daily basis.

The logical approach and the code, will surely diverge from any examples I would be able to provide here, hence the pun on Obi-Wan Kenobi’s “Remember, Luke, use the source.”, in other words, study the actual code.

See the actual predictions

In my GitHub, I have provided a page where I actually post the results of my predictions, I hope you will find them interesting.

The rest of the project is documented at:

Please subscribe and share

If you enjoy my writing, please like, subscribe, and share this article.
Based on the statistics, I either get motivated to write more or get in touch with my innate introvert.

Contact me via LinkedIn.

Gain Access to Expert View — Subscribe to DDI Intel

Data Driven Investor

empower you with data, knowledge, and expertise

Sign up for DDIntel

By Data Driven Investor

In each issue we share the best stories from the Data-Driven Investor's expert community. Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Uki D. Lucas

Written by

System Architect, ADAS / AD @ Harman / Samsung.

Data Driven Investor

empower you with data, knowledge, and expertise

Uki D. Lucas

Written by

System Architect, ADAS / AD @ Harman / Samsung.

Data Driven Investor

empower you with data, knowledge, and expertise

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store