Seams, ancillary services and congestion management: US versus EU Electricity Markets
Written by Ross Baldick (University of Texas at Austin)
Stitching seams with different threads: the US experience
Today, the majority of electric energy consumed in the US is served in so-called “organized” or “centralized” markets that are operated by Independent System Operators/Regional Transmission Organizations (ISOs/RTOs). These entities, established in the late 1990s/early 2000s, have significantly supplemented traditional bilateral trading between utilities with additional centralized trading options that include centrally dispatched “real-time” (RT) markets.
RT markets typically involve economic dispatch of offers for every five minute interval. Dispatch signals are sent to individual generation units, representing a target “base-point” to be reached at the end of the upcoming five minute interval. Transmission constraints are taken into account in the economic dispatch calculation and locational market prices are used to settle the market. To the best of my understanding, no European electricity market clears offers for energy at this timescale, and none of them represent physical transmission constraints at the nodal level nor use locational pricing.
Besides RT markets, US ISOs/RTOs also operate day-ahead (DA) markets, including unit commitment and locational pricing. These type of markets considers both energy and ancillary services, and is basically a forward market. That is, unlike in the EU markets, accepted generation offers in the US DA markets are not necessarily assumed to be accompanied by physical commitments to generate.
In the US, the RT market, which might be viewed as the “spot” market, is settled based on deviations from day-ahead positions, which in this context should be viewed as short-term forward financial commitments. The representation of transmission and other constraints is explicitly designed to match, where possible, the features represented in the DA market. That is, unlike EU markets, there is not the distinction between a market for trading and a “technical market” for, for example, dealing with transmission limitations. US DA and RT markets are, to the extent possible, designed to be consistent.
In addition to these options for trading through the ISO/RTO, bilateral trade options are available. These can be for financial performance (for example, as in a contract for differences), but market designs also allow for “physical” scheduling of generators. Bilateral trade options are available across seams between the various ISO/RTO markets too. In addition, several US exchanges trade monthly and longer term contracts (e.g., the Chicago Board of Trade and the Intercontinental Exchange).
ISOs/RTOs are also responsible for “reliability” and administer certain other markets, such as for transmission rights and, in some cases, longer term capacity.
Although the majority of the electricity in the US is now delivered through ISOs/RTOs, there is a large part of the country, particularly in the Western Interconnection outside of California, that is served by more traditional vertically integrated utilities with bilateral trading. However, over time, the organized markets have grown geographically. We might say that the “seams” between a particular ISO/RTO market and the geographically adjacent region outside that organized market have typically been “stitched” through growth of the organized market.
For example, as shown in Figure 1, the PJM RTO footprint grew from somewhat larger than the original states of Pennsylvania, New Jersey, and Maryland to currently include 14 states. Various entities joined PJM, thereby becoming part of the organized market. Similarly, the Midwest (now Midcontinent) ISO has evolved over time into a larger footprint, connecting entities from Canada to Louisiana.
The case of the California ISO (CAISO) is particularly interesting in this regard. CAISO has operated DA and RT nodal markets in California since 2009, and as with other ISOs/RTOs, it has also supported bilateral (hourly and longer term) trading with other entities, mostly in neighbouring states. As shown in Figure 2, the geographical scope of the CAISO RT market was greatly expanded in 2014 with the development of the “Energy Imbalance Market”, which now involves entities from many states in the Western Interconnection, offering into the CAISO RT market.
This “organic” growth of ISO/RTO markets has resulted in large areas operating under uniform real-time pricing rules, as shown in Figure 3. This has significant implications for the integration of renewable energy sources in the US. In particular, the wide geographical scale of current ISO/RTO RT markets often allows for the averaging of net load variability in a “single” large system. Such wide-scale balancing has clear advantages for dealing with variability in renewable output, since variations at the five minute to hourly range are likely to be uncorrelated across large geographical footprints.
Moreover, since the generator base-points are updated every five minutes, net load following capability at this time scale is being obtained for “free” due to the action of RT markets. That is, ancillary services are not used for following most intra-hour variations of net load, except for cases of extreme ramps. Therefore, ancillary services in the US primarily deal with intra-five minute variability and uncertainty.
Wide-scale RT markets have facilitated and will continue to facilitate the integration of renewables in the US markets. Texas is a good case in point. The part of the state managed by the Electric Reliability Council of Texas (ERCOT) has been able to manage relatively smoothly a significant penetration of wind energy in the past few years, surpassing the analogous figure for many Western European countries.
The US now has several large ISOs/RTOs. Those in the Eastern Interconnection, including PJM and MISO, are geographically adjacent and electrically interconnected. These entities will likely continue to exist and it is doubtful that one will supplant the other, because institutional barriers hamper any further comprehensive integration of adjacent RTOs. That is, some of the existing seams between those ISOs/RTOs will persist in the future.
Nevertheless, there is work ongoing to improve the efficiency of trading and the coordination of system operation across the borders of adjacent ISOs/RTOs. Particular effort is currently being made to facilitate the integration of inter-ISOs/RTOs trading in RT markets within the Eastern Interconnection, as it has already occurred through the Energy Imbalance Market in California. This focus on real-time is based on the argument that efficient trading in all forward markets stems from efficient real-time trade.
This distinction between the evolution of wide geographical scale trading in US markets and that in EU markets is key. The emphasis on real-time coordination has already facilitated large-scale integration of renewable resources in the US, with ERCOT as a very positive example of this success. I will argue that similar progress in the EU will be much more difficult, in part, because of the lack of large-scale real-time coordination.
Stitching seams with different threads: the EU experience
Due to the lack of large scale real-time markets, the evolution of the electricity industry in the EU is currently much less conducive to the integration of high levels of renewables than in the US. Recent and ongoing developments are in the right direction, but I will argue that the absence of a US-style real-time market will continue to hamper large-scale renewable integration.
Although my understanding of the European experience could be limited by the fact that I mostly focus my research on the US system, it is clear that electricity markets have been developed in the EU with a combination of day-ahead and intraday trading. In US parlance, European day-ahead and intraday trading is based on a “power exchange” model where intra-zonal transmission constraints are ignored. Moreover, this trading is conceptually separated from the balancing “market”, which is operated mostly at the country level by the relevant Transmission System Operator (TSO) and is aimed at dealing with what might be described as “technical” issues (intra-hour or intra-half hour balancing of short-term fluctuations of supply and demand, provision of contingency reserves, and handling of transmission constraints).
It is interesting to observe that such an arrangement has not been seen in the US after the California crisis, where this type of market design, together with other structural and design issues and particular circumstances, led to unacceptable outcomes in the early 2000s.
Actually, European balancing markets are not akin to the real-time markets developed in the US. Rather, they are a mechanism to deploy ancillary services such as frequency regulation, sometimes with an attempt to use merit order and typically assigning the costs to the entities deemed to be the cause of the need to deploy the ancillary service.
The finest temporal resolution of EU day-ahead and intraday markets varies from 15 to 60 minutes, with what I have referred to as ancillary services typically required to cope with supply-demand balance and contingencies that occur within these time intervals. This means that the ancillary services in the EU must cope with uncertainties over a longer duration between market adjustments than in the US. All else equal, this means that a greater quantity of such ancillary services is required in EU markets than in US markets. Consequently, the EU market structure poses greater difficulties for integrating large amounts of intermittent renewable resources (see Borggrefe and Neuhoff, 2011).
In short, the balancing market is not (as far as I can tell) the same as a US-style “real-time” spot market. On the contrary, the DA market, or the last intraday market, is viewed as the “final” market. Individual balancing market designs vary from country to country, but they have typically aimed at encouraging only limited trading after the DA and intraday markets’ closure. Crucially, transactions in existing intraday markets normally stop several hours before the actual delivery of energy, when the uncertainty about the future potential production from renewables is still relevant. This further complicates the integration of large amounts of intermittent renewables. A very positive sign is that recent rules for EU electricity markets are heading in the direction of later “gate closures” for intraday trading, towards being at most one hour before real-time.
As in the US, bilateral trading is possible between various entities in Europe too, and some power exchanges span seams between countries for day-ahead, and intraday trading. Additionally, “Price Coupling of Regions” has recently added day-ahead (and eventually intraday) pan-EU trading options through EUPHEMIA. Nevertheless, multi-country balancing markets or seams management are notably limited in the EU. In this context, a natural question is: why is there such a lack of coordination across wide geographical scales in European balancing markets? An obvious answer is the lack of consistently designed balancing markets, since each of them has been developed to provide balancing services for a specific country and reflects its historical characteristics.
The heterogeneity of balancing market designs presents difficulties for wide geographical scale balancing. Moreover, adjacent countries with different balancing models pose institutional barriers to consolidating EU TSOs, particularly given TSO responsibilities to individual countries. This is somewhat analogous to the situation with adjacent ISOs/RTOs in US. However, since the ISOs/RTOs in the US operate already at a large scale (of the order of 70 GW to over 150 GW peak load), and cover large geographical areas (on the order of the size of Western Europe), it might be argued that the additional benefits in the US of seams management and RT trading across seams is not hugely significant. It may also be true that sufficient geographical scale is inherent in the largest balancing regions in the EU, such as France. However, the lack of real-time markets in the EU prevents the realization of the full potential of such a large day-ahead or intraday trading area to average out short term variations in generation and demand. Although wide geographical scale balancing markets cannot provide all of the benefits of real-time markets, the good news in the medium term is that the EU is heading in the right direction in terms of harmonizing the balancing markets that will likely enable balancing over larger geographical scales. Moreover, the development of imbalance netting allows for shared utilization of balancing resources.
In both the US and the EU, bilateral contracting has always allowed trading across seams. However, the main effort to improve seams management in the US has focused on RT markets. In contrast, the EU version of centralized seams management focuses on day-ahead and eventually intraday trading, with some attention to balancing. It is hard to believe that the needs of electricity markets in the US and the EU are really so different that two polar opposite approaches to wide-scale coordination can both be optimal. The increasing amount of renewables, resulting in increasing uncertainties at the hourly and sub-hourly level, suggest that the greatest value would be in improving real-time coordination, not day-ahead or even intraday. Unfortunately, the various balancing market designs in the EU, and the fact that they are simply not real-time markets, make wide-scale balancing that utilizes all available resources difficult without significant redesign. This means that the inherent flexibility of transmission and dispatchable resources in the EU cannot currently be fully exploited for short time-scale variations in electricity generation and demand.
In conclusion, the threads used or proposed to bind the seams in the US and the EU are very different. In the US, there has been organic growth of the geographical scope of real-time markets with the elimination of many seams. In the EU, seams are being managed in the day-ahead and eventually intraday time-scale and with imbalance netting. As a general principle, renewable integration is facilitated by wide-scale, closer to real-time adjustment of dispatchable thermal and hydro generation. Current EU balancing markets are not as flexible in utilizing this dispatchability as are US real-time markets. EU markets currently even eschew the inherent flexibility of hydro resources and the utilization of continent-scale electrical interconnections because of the fractured rules for balancing and the lack of a true real-time market.
Despite a large and useful effort to develop day-ahead trading, I anticipate that the transition to a decarbonized electricity sector in Europe will require yet more market design changes including the development of coordinated real-time markets.
Why doesn’t the EU co-optimize the procurement of ancillary services with energy?
The US and the EU have different approaches to the procurement of ancillary services (AS), in particular frequency regulation and contingency reserves.
Broadly speaking, one of the main goals of AS is to maintain and restore the balance between supply and demand on timescales finer than the finest granularity of the market period. There are different types of AS. In the American jargon, frequency regulation reserves are needed to respond to continuous but relatively small scale fluctuations in demand and to variations in supply by intermittent renewables. Contingency reserves, which include spinning or responsive reserves and non-spinning reserves, are typically needed to manage sudden fluctuations in the system due to the loss of a large generator or, much less commonly, due to the loss of a large load center. More specifically, spinning reserves are deployed in the seconds to minutes timeframe after a contingency. Non-spinning reserves are deployed in a somewhat longer time frame after a contingency, with the goal to restore the spinning reserve capacity and be ready again for the next contingency.
In the EU jargon, at least as I understand it, the expression “balancing reserves” is used to refer both to frequency regulation and contingency reserves. According to a recent study for the European Commission, balancing reserves include “frequency containment reserves”, which I understand to be very similar in role to US frequency regulation, and “frequency restoration reserves”, which I understand to be similar in role to US contingency reserves.
However, beyond lexical distinctions, it is interesting to note that there are two crucial differences between the procurement of such AS in the US and the EU. First, in the US these AS are always procured in the day-ahead (DA) and in some cases in the real-time (RT) markets. Second, the procurement of the AS capacity is integrated into the energy market through what is referred to as “co-optimization”.
It is perhaps natural for Independent System Operators (ISOs) and Transmission System Operators (TSOs) to prefer to arrange for the provision of such reserves well in advance. In several countries in the EU, the norm seems to be that reserves are procured by TSOs monthly in advance or even through annual reserve procurement processes. The preference by TSOs to procure reserves in advance is likely a manifestation of concerns about making sure there are enough reserves “on the day”. However, ceteris paribus, the capacity for reserves will not change just because it is procured a week or a month in advance. Moreover, contracting monthly or yearly imposes a much greater risk that there will be outages between the time of contracting and the operating day.
Procuring reserves in advance has a related but further drawback. Decisions about reserve allocation will generally be better the more that they can be adapted to actual conditions, unless the state of nature is static and deterministic. While a static, deterministic world might have been a reasonable approximation in the (distant) past, in a world with increasing amounts of renewables, the state of nature is increasingly dynamic and stochastic. That is, a better estimate of the availability of reserves and the need for them is possible when the reserves are procured day-ahead instead of weekly, monthly, or yearly.
Moreover, if there is an opportunity to trade AS again even closer to real-time, this can further improve the allocative efficiency of AS. The Electric Reliability Council of Texas (ERCOT) still does not have trading of AS in the RT market, but this is a market improvement that might eventually be carried out. Several other US markets do allow for trading of AS in real-time. The good news is that the EU is apparently shifting towards procurement closer to DA, as specified in Article 32 of the drafted Electricity Balancing Guideline.
There is an additional important related issue in the procurement of AS. Even in countries like Italy where AS are procured DA, such procurement occurs separately from that of energy. The drawback associated with separated procurement of AS and energy stems from the fundamental fact that capacity for reserves (including capacity with particular ramp rate requirements as typically required for reserves) is still capacity, and therefore it is at least partially substitutable for capacity used to actually produce energy. Any such procurement has a constraint, whether explicitly enforced in a co-optimized US-style auction or implicit in the market rules for offering capacity, where it is required that the sum of the capacity for reserves plus the capacity for energy from a particular generator can be no more than the capacity of that generator.
Co-optimized markets allow for that generation capacity to be assigned to the production of energy or the provision of AS, depending on the circumstances and relative needs, enabling the most valuable use of that capacity. In contrast, making a decision to utilize specific generation capacity for AS well in advance of knowing the actual demand and market conditions always runs the risk that the allocation will be ex-post inefficient.
Lack of co-optimization also results in poor price formation. As is well known, and well understood as a reason to implement co-optimization, there is an opportunity cost for providing reserve capacity (capacity offered as reserve cannot be used to generate energy). This is manifested in the constraint on reserves plus energy produced per hour being no larger than capacity. Consequently, if energy prices are not determined simultaneously with the decision on reserve procurement, then reserve prices cannot reflect the actual opportunity cost, but must instead reflect an estimate of the opportunity cost as represented in the reserve offer prices. Even when the operating cost of providing reserves is negligible, the offer prices must be non-zero to reflect this opportunity cost.
Therefore, if AS are procured before energy, market participants must estimate the opportunity cost in their AS offer before making it. If AS are procured after energy, as in Italy, markets participants must still estimate the value of the reserves when they make their offers for energy. This estimate of opportunity costs can be expected to be particularly poor when the reserves are procured well in advance, resulting in both errors in price and misallocation of resources, i.e. reserves may be provided by units that are not the most efficient to do so. Besides, poor estimates of reserves value could be an additional barrier for new and smaller players entering the industry.
It is certainly true that I am advocating central procurement of energy and AS by a system operator. The experiences in the US of separating AS from energy, as for example in the earliest implementations of the California and ERCOT markets, show that such arrangements ultimately are not efficient. It is somewhat perplexing to me to see that these clear lessons from the US seem to have been ignored in designing the EU electricity markets. Ironically, those few markets in the EU that include some form of co-optimization, for example, Greece and Poland, are actually being encouraged in the opposite direction by the EU target model.
To summarize, EU style procurement of AS in markets that are separated from energy likely results in both poor alloccative decisions and poor price signals. Besides contributing to inefficiency of dispatch of existing generation capacity, such errors have the effect of stifling new entrants, who cannot see transparently the value of resources in the market.
The case for locational pricing
Day-ahead markets in the EU include only a coarse, zonal representation of physical transmission constraints. Sometimes this arrangement is justified on the basis that, for example, transmission networks within countries are highly meshed with high capacity, whereas links between countries are weaker. While I do not dispute that transmission systems within countries have been built out with higher capacity than inter-country links, that does not in itself justify thinking of individual countries as “copper plates”. Anecdotally, it appears that transmission limitations within countries are becoming more significant in some EU countries, particularly as new renewable build-outs occur in locations that did not previously have significant generation capacity (this is apparently the case of Germany). As such build-outs of renewables continue, and given the difficulty of building large-scale transmission links, the argument that individual countries resemble a copper plate becomes less realistic. On the contrary, locational values of energy may vary over regions even where there is apparently no internal binding transmission constraint because of the interaction with binding constraints in other regions in a meshed network.
In accordance with a recent piece of European legislation, a process of bidding zone review is ongoing in the EU. To me, locational marginal pricing (LMP) is the logical end point of processes that begin to sub-divide and adapt zones to the physical reality. Nevertheless, LMP does not seem to be on the agenda in the EU, at least for the moment. There apparently continues to be opposition to finer grain representation of transmission constraints into the commercial network model adopted for the day-ahead market. The reason for this choice seems to be the complexity of LMP and the problems that it may create for market participants. Such rationalizations are, to me, a déjà vu of the 1990s US arguments against LMP.
As an amusing caricature, you can look at the spoof logos of the ISOs that were drawn at the time (Figure 1). Tellingly, the spoof logo for the Electric Reliability Council of Texas (ERCOT) interconnection was “Exploring every alternative to LMP”.
This spoof sums up the type of opposition that occurred in Texas. It is the same that I also see today in the EU against reflecting physical realities into the market model. It took nearly a decade to first change the attitude towards LMP in ERCOT and then finally implement it in 2010. See “Shift factors in ERCOT congestion pricing”, for an analysis at the time that proved influential in the decision to change the market model. I would encourage any proponent of a zonal transmission model to perform an analogous study to elucidate the correspondence or otherwise between the incentives from a zonal model and efficient incentives.
Several changes were included along with the opening of the LMP market in 2010, including, for instance, the addition of a day-ahead market to what was previously only a real-time market; the dispatch of individual units instead of portfolio dispatch (necessary but not sufficient for the implementation of LMP); and the shortening of the dispatch interval from 15 minutes to five. In other words, the “big bang” of changing the fundamental market model enabled a number of improvements to be simultaneously implemented. Since then, several other changes and fine tuning have occurred.
ERCOT has seen the integration of the largest amount of renewables of any large interconnection in the world (around 15% of energy in recent years have been produced from wind, a share that is significantly higher than that recorded by Western Europe). Counterfactuals are hard to establish, but statistical analysis can elucidate some important results. Various market changes, most particularly the “big bang” change in 2010, have enabled ancillary services for frequency regulation to be utilized much more efficiently, allowing for many GW of additional wind and, amazingly, an overall reduction in the amount of regulation reserve to be procured. While the improvement in the utilization of frequency regulation is likely due primarily to the shortening of the dispatch interval, the shift to a LMP system allowed for a more effective utilization of transmission capacity. In turn, this has allowed for the integration of higher levels of wind generation than would have otherwise been possible. It has also fostered a highly competitive wholesale generation market that has successfully weathered fluctuations in natural gas prices, the challenges of wind integration, the retirement of significant coal capacity, and continues to accommodate one of the highest demand growth rates in North America.
While the EU review of existing bidding zones is a possible step in the right direction, the US experience tells you that once the fiction of copper plate zones is no longer accepted, the only logical end point becomes LMP. Claimed “complexities” of LMP markets in the US turned out to be far less complicated than the out-of-market actions (such as deploying balancing energy to handle transmission constraints) that are intrinsically necessitated by a zonal commercial network model. Particularly in the context of new generation additions and RES deployment in unexpected places, LMP aligns the incentives for production and consumption in wholesale electricity markets with the physical reality. While retail consumption can continue to be priced zonally in such a system, the underlying wholesale market will work most efficiently, and be able to integrate better large-scale renewables, if it is based on LMP.
This essay was originally published as a series of fully referenced blog posts on the FSR website which can be read here.
