Predicting Electricity Spot Prices in Australia
with Amphora Data and Koz.ai
More information and source code for the project can be found here.
Electricity Spot Prices in South Australia
Predicting the price of electricity is increasingly important. The wholesale market of electricity is extremely variable and many companies and organisations are highly exposed. This is an issue in agribusiness, where many farmers are facing decreasing profit margins and price spikes can cause great difficulties. While farmers can benefit from periods of prices as low as $50 /MWh or even less, they also face prices of up to $14000 /MWh. There is a fundamental conflict — businesses want price certainty to plan their daily operations, but the Australian National Electricity Market (NEM) requires price variability to shift production along the supply/demand curve. Here, we present a small contribution to resolving this conflict with methods and tools for better predicting price fluctuations, which can help companies manage the use of their electricity accordingly.
What is the NEM?
The Australian National Electricity Market (NEM) is a wholesale electricity market in which generators sell electricity and retailers buy it to sell on to consumers. There are over 100 generators and retailers participating in the market, so it’s highly competitive and therefore an efficient way of maintaining relatively competitive electricity prices in the wholesale market.
All electricity sales are traded through the NEM. It is a wholesale market and prices fluctuate in response to supply and demand at any point in time. The NEM is split into 5 regions (QLD, NSW, VIC, TAS, SA), and prices are set per region.
In South Australia, electricity is predominantly sourced from renewable energy sources such as sun and wind. Non-renewables are generally more expensive than renewables, and so, as both wind speeds and sun insolation vary greatly, so can electricity prices. To model the electricity spot price spikes in South Australia, I used data from the Amphora Data agricultural data repository. I then used the Kozai data science platform for modelling and predictions. This resulted in a good predictor for price spikes based on a convolutional neural net.
What do the prices look like?
All plots and data below are available at Amphora Data. Amphora Data is currently in private beta. For access visit amphoradata.com/contact-us
Electricity prices are set every 5 minutes (the ‘dispatch price’) and 6 dispatch prices are averaged every half hour to generate the ‘spot price’. Below are the prices for an arbitrary week in South Australia. We observe that the price remains relatively close to $50, although occasionally dropping sharply and becoming negative. At these times, consumers of electricity are actually paid to consume.
Now compare the ‘normal’ week above, to the week that immediately followed it in December of 2019. Clearly things are very different. The ‘normal’ variability ~$50 is no longer visible, as the graph is dominated by a huge spike on December 19, when the price jumped to over $8,000, approximately two orders of magnitude larger than the maximum for the days either side. We also observe that the maximum scheduled generation has doubled compared with the previous week.
So what’s going on? A major driver of electricity demand (and therefore supply) is the weather. The plot of temperature below shows that Adelaide was experiencing a serious heatwave from December 17 to December 20, with the price spike occurring the evening before the hottest and last day. Keen observers will also note that the minimum overnight temperature was high at 28 degrees.
We’ve highlighted the same period for each day of the heatwave (7–9 pm), which coincides with the price spike on the last day. Temperatures are significant, because higher temperatures mean more air-conditioning (i.e. greater consumption) and often less efficient power plants. Wind speed is also significant because of the high use of wind energy in South Australia.
Counting the cost
To exemplify the situation, Imagine you had to pay wholesale prices for your factory that was regularly running at 7pm. If that that plant consumes ~250 kW of power, on a normal day you’d spend about $15/hour to run the plant. Compare that to the abnormal day on December 19. On that day, at 7pm, you’d spend $1,500 in a single hour!
Luckily for regular consumers, electricity retailers protect you from price variability by hedging against shortfalls in generation. So while, in general, there are no expensive days, consumers are paying for the price spikes, it’s just averaged out over the entire year.
The more accurately retailers can predict future demand and wholesale pricing, the more effective their retail pricing can be. Additionally, generators (such as gas turbines or hydro) will only want to enter the market when there is going to be sufficient price-demand to offset the cost of starting up.
So why would anyone want to be exposed to the variable price?
Remember that during the normal week of operations, the price actually went negative, meaning that anyone exposed and consuming electricity during that period was actually paid by the market. For our imaginary business, for the hour when price dropped to $-30, they would have actually been paid about $7 to run their plant — a $22 improvement on the typical situation.
You may also wish to be exposed if you have atypical usage patterns, because as long as you don’t consume during periods of high prices, your total electricity bill will be significantly smaller because you aren’t paying a retailer to hedge the market for you.
Into the Data Science
The project was started with some exploratory analysis of the downloaded spot prices and weather data from Amphora Data. As a proof-of-concept / baseline, a multiple linear regression model was then built to predict spot prices for electricity in South Australia (SA). These simple models often show a good performance in a wide range of situations and offer an indication of the importance of new features as they are added to the model. Then, more complex and involved models were developed and tested, measured against this baseline.
Better data, and more of it
Accurate predictions, especially in forecasting problems, require large amounts of data. Making a prediction on something not seen before by the model, will be difficult in most situations. Often, more of the same type of data is not sufficient and so augmenting it with other forms of data (e.g. weather, market data, etc) can help to improve model performance.
Exploratory Data Analysis (EDA)
As a start, as in most of my projects, data was checked for completeness, and if there had been any missing values, they would have been interpolated with splines. To capture the temporal nature of the data, new features were created consisting of the discrete (tᵤ)-(tᵤ-₁) time differences.
The correlations between the SA spot prices and all other variables were tested, yielding the result that the SA spot prices behave the most similar to the scheduledGeneration in Victoria (VIC) and SA. The next most correlated features were weather-related, namely those that influence solar cell and/or wind farm performance.
>>>corr_SA.price_SA.sort_values(ascending=False).head(10)
price_SA 1.000000
scheduledGeneration_VIC 0.388196
scheduledGeneration_SA 0.369061
d_price_SA 0.331827
rainProb_SA 0.225282
d_rainProb_SA 0.224557
d_windDirection_SA 0.215873
cloudCover_SA 0.207847
windDirection_SA 0.206184
d_cloudCover_SA 0.205422
Name: price_SA, dtype: float64Data from the other states was included for the SA price modelling effort, as their electricity grids are connected, thus influencing each other.
Similarly, the prices in NSW and QLD follow a similar pattern of mutual dependence. In an analysis as for SA above, the dominant features are the respective scheduledGeneration for NSW and QLD, as expected in connected electricity markets. The small influence between SA/VIC on the one side, and QLD/NSW on the other, lies in their geographic separation, and different electricity generation sources.
Adjusting the linear model to contain only these top nine features already improved the performance, showing an about 30% lower mean square error (mse):
>>>print('RMSE with correlation features:', rmse_corr, '$/MWh\n')
RMSE with correlation features: 207.58 $/MWh
>>>print('RMSE w/o correlation features: ', rmse_plain, '$/MWh')
RMSE w/o correlation features: 281.23 $/MWhFor both the random forest regressor, as well as the deep learning models tested for model selection, a similar improvement in performance manifested using only these highly correlated features, or a similar selection of a subset of features.
Distribution of data
Especially for classical machine learning methods the distribution of data matters, where normal distributed data leads to faster algorithm convergence and better predictions. In this case, fortunately, most data was fairly close to either a shifted normal, or log-normal distribution as can be seen in a pairplot. Correspondingly, no data transformation was undertaken, although it is a possible improvement factor to consider for future modelling efforts.
Feature engineering
The amount of data at the start of the project was limited in terms of typical forecasting applications, with less than 30 days worth of data available. As there was great variation in the performance of new features as fresh data came in daily, the creation of such features to help improve the modelling was not actively pursued to the later stages.
Model selection
The central aim for model selection was (1) prediction of spikes in spot prices for SA, and (2) describing all SA spot prices as accurately as possible. Given the goal of this project, (1) took precedence over (2).
Often, the central question for data science is the choice of the model, where several competing factors have to be considered: explainability, model performance, model inference time, and model train time. The latter two factors were not a concern for this specific project thanks to Kozai! Hence a variety of different models and architectures were tested, among them simple linear models (Lin), random forests (RFR), but also dense (NN_3), recurrent (RNN) and convolutional neural networks (CNN_2 and CNN_4). The linear model and RFR were chosen, as they often show good performance, especially so on small datasets, and allow for easy explainability of the models outputs. The deep learning models were chosen, because they either are explicitly geared towards this kind of problem, like the recurrent neural net and the causally padded convolutional neural net, or have well-performing historic use cases for similar problems.
Model training optimisation
To accelerate model training and obtain better results for the deep learning models, the Adam optimiser learning rate was fine tuned, by training the model for one epoch, and stepping through a nested interval optimisation, to find the optimal learning rate. For all, except the RNN, the best learning rate was a factor two larger than the standard setting of 1E-3, leading to slightly faster convergences.
Model error
For all these models, the optimisation criterion was the mean square error (mse). This criterion was chosen as large outliers are undesirable and the mse is easily interpretable, if its square root is considered, as an error in units of $/MWh. Each model was trained on features x, with the corresponding spot prices y lagged by 1 to 24h, to be able to predict 1 to 24 time steps into the future, with the results, listed below:
>>>print(mse)
Lin: 43090
RFR: 78459
RNN: 17158
CNN_2: 6350
CNN_4: 5832
NN_3: 369214The reason for these 24h prediction windows is rapidly decreasing forecast quality for longer periods, and a more accurate prediction of the windows between 4–9pm, and 2–9am, where most price spikes occur. With shorter prediction windows model accuracy, in general, improves but the usefulness of the model decreases.
The best performing model with the current amount of data (~45days) was a convolutional neural net (CNN_4) optimising by SA spot prices, as indicated by the overall minimal mse over the 24 prediction time steps. This model yielded a standard error of 76 $/MWh for the SA spot price predictions. The second most accurate model with very similar performance optimised for each state concurrently (CNN_2) with a standard error of 79 $/MWh.
Predictions
Predictions made by this model showed to be good for the prediction of large trends in spot price, like spikes, if they cross a certain threshold. For the precise determination of spot prices, model performance tended to be off between $ 10–30, as visible from the plots above in Figure 5a and below in Figure 5b.
On a Kozai single GPU instance, training the most accurate model was over 5.9x faster than on an 8 core CPU instance, ensuring a fast training and evaluation time.
More Data
In time, given a larger amount of data (~1–2 years worth), feature engineering and more fine tuned models are likely to improve predictions from spot price spike prediction to levels, where the actual price at any point in time can be predicted more accurately.
Focussing on the more regular day-to-day price variations, rebinning from 1h intervals into larger bins, e.g. 2, 4, 6h bins, can bring out further characteristics in the data and make it easier for the model to pick on different characteristics of the evolution and different parts of the signal, further improving model performance.
For a larger amount of data, rare events like spot price spikes will then have to be more carefully modelled for, to ensure the models correct predictions, in order not to drown out these rare events. For this end, training can be specifically focussed on these (up-/downsampling), or artificial data can be created to enrich the price spike data. In combination with both a larger dataset, and enriched data, other neural net architecture that have recently shown great performance in related tasks, like transformers, could become an interesting option.
Final Notes
- Predicting electricity spot prices using historical data is possible with a good degree of accuracy . Large price swings are made predictable, allowing for informed decision on when (not) to run energy hungry equipment ahead of time.
- GPU training took around 15 minutes, 8x CPUs took around an hour and half. This means that you could iterate on the GPU up to around 30 times in a single work day, compared with only 5 times with the CPU. This means faster feedback, and better use of your data scientist’s time.
- Amphora Data greatly reduced the effort of sourcing and cleaning up data. They provide well maintained, updated and catalogued datasets on the agriculture space — and an abundance thereof. Precisely what data scientists love!
- Kozai eliminated the need to set up data science infrastructure in the cloud and can scale well beyond what I can do with my laptop. Moreover, as it can be accessed through the browser, it allowed me to continue working and training my models even while commuting. So I can easily pick up from where I left off when I get home.
