The USS’s prudence: evidence-based or reverse-engineered?
Neil Davies, University of Bristol
The USS valuation is based on assumptions. By changing these assumptions the USS can radically change the outcomes of the valuation such as the deficit and the costs of pension in future. One of the most important assumptions is the level of prudence on investment returns. Pension trustees are legally obliged to be prudent in the assumptions that underpin fund valuations, and adequately justify any changes to these assumptions from one year to the next. The USS 2020 valuation has brought these obligations into sharp focus, as a huge deficit and steep proposed increase in contributions have led both the union and employers to question the calculations (see USSbriefs106). In this brief, I argue that USS’s justification for adopting a new definition of prudence falls woefully short of what is required, and meet no professional standards for financial modelling, providing further evidence that a change in governance is urgently required in order to restore faith in the fund and avoid further turmoil in HE.
How (not) to value a pension scheme
Defined benefit pension scheme valuations are governed by The Occupational Pension Schemes (Scheme Funding) Regulations 2005 section 5. This dictates that the economic and actuarial assumptions must be chosen prudently, taking into account either the yields on assets held by the scheme and/or the market redemption yields on government or other high-quality bonds. Any change in the method of assumptions used from the last valuation must be justified by a change of legal, demographic or economic circumstances.
Thus regulations require the USS Trustee to prudently choose assumptions for the valuation. The Trustee argues that prudence is a qualitative judgement that cannot be expressed as a single number [1]. However, to comply with the regulations above the Trustee must be able to demonstrate that the process it followed to decide the level of prudence in the valuation was reasonable and was based on reasonable evidence that was relevant to the scheme. Critically, it is imperative that the Trustee does not reverse engineer its assumptions to produce a certain prespecified outcome.
Unfortunately, the Trustee has not published any evidence to date to justify many of the changes it is looking to make to the valuation methodology, including one of the most consequential: changes to the “level of prudence” on investment returns. The USS’s rationale for choosing the confidence level on investment returns is incoherent and unclear. The most plausible explanation is that they have reverse-engineered the confidence level to get a pre-desired discount rate of gilts + 2%, then arbitrarily increased this to derive the proposed discount rates for different levels of guarantees (covenant).
Valid statistical analyses have essentially three steps:
- define assumptions and data;
- use the assumptions and data to estimate a model;
- and interpret results.
It is of course possible to reverse this process:
- define the results you want;
- estimate your model for a range of assumptions;
- and choose the assumptions that give you the result you want.
The latter approach — ‘reverse engineering’ — would normally be considered fraudulent for a scientific paper. Yet this appears to be the approach taken in the USS 2020 valuation, reverse engineering the discount rates by manipulating the input parameters used in the valuation — specifically the level of prudence on investment returns.
What is the “level of prudence” of investment returns?
The level of prudence in investment returns is the point in the distribution of returns that the valuation is conducted at. For example, consider Figure 1 below. These simulations use data from Jorda et al. (2019) to provide the mean and standard deviation of expected returns above inflation. The simulations are very simple, and can also be generated using a Google sheets toy example. For each year, the simulation draws a value of investment returns to bonds and equities using the distributions of investment returns estimated by Jorda et al, and the reference portfolio of the USS (a 64:36 equities:bonds ratio). The simulation draws 30 years of investment returns. This process is repeated 10,000 times to provide a distribution of annualised investment returns over 30 years.
The arithmetic mean annualised investment return is 4.7% and the median (50th centile or best estimate) return is 4.8% above inflation. While “prudence” is not defined in law, in the valuation, USS interprets the “level of prudence” in its investment forecasts as a point in the distribution of returns determining the discount rate to be used in the valuation. In the 2014 valuation, they used the 65th centile, i.e. the point such that 65 out of 100 simulated outcomes will give a higher return. In our simulation, the 65th centile has a return of 3.9% above inflation. So the discount rate based on this level of prudence is 3.9% above inflation. In 2017 and 2018, USS proposed using the 67th centile of investment returns; they increased the “level of prudence”. In our simulation, this gives a return of 3.8%, and thus a discount rate, of 3.8% above inflation. So as the level of prudence is increased, the discount rate falls, as we move to the left in Figure 1. In the 2020 valuation, the USS originally proposed using a range of different “levels of prudence” on the assets invested for pensions promised to members already drawing their pensions (73%) and active members still paying into the scheme (78% or 85% depending on the commitment of employers to the scheme). In our simulation, this equates to a discount rate of between 3.4% and 2.5% above inflation for active members. Thus, even though our assumptions about future returns have remained unchanged (the simulations are identical) USS can manipulate the discount rate by changing the level of prudence, or where in the distribution of returns it is choosing for its calculation.
As the expected discount rate decreases, the costs of providing benefits increase, very sharply. Moving from the 67th centile of expected returns to between 73rd to and 85th centile is estimated to cost the scheme, and ultimately members between c£7.4bn and c£9.5bn.
What evidence supports the USS’s proposed changes to the level of prudence?
We have recently submitted a complaint to the USS, signed by almost 4000 colleagues, which included two simple questions:
- What process did the Trustee use to decide to recommend increasing the level of prudence from 67% used in 2018 to between 73% and 85% for the 2020 valuation?
- What evidence did the Trustee use to inform its recommendation to increase the level of prudence for 2020 valuation?
The USS has recently published a note which attempts to explain and justify the changes they are looking to make to the level of prudence used in the valuation. They argue that comparing the “level of prudence” across valuations is challenging and that the “overall level of prudence” in the 2020 valuation is not obviously higher or lower than in 2018. However, as we have seen above, the level of prudence on investment returns is simple to compare and has been increased very substantially, at great cost to members.
The USS argues “Looking at prudence purely in terms of the assumptions for life expectancy and discount rates focuses only on valuation inputs. It is more instructive to view prudence through a wider set of lenses that also considers its impact on valuation outputs.” In the Pension Regulator’s (TPR) letter in response to the Section 76.1 report, they are also explicit that discount rates should be selected by choosing valuation input assumptions “However, in his initial advice on the discount rate assumptions (report date 24 May 2020), the Scheme Actuary advised the Trustee to limit the discount rates derived ‘mechanically’ by applying confidence levels on the FBB returns.”
However, this is precisely what the Trustee is not supposed to do — to reverse-engineer the assumptions and inputs for its models by considering the outputs (i.e. discount rates, the deficit, and contribution rates). Thus, they provide no external evidence to justify the changes in the level of prudence they are looking to make.
What is the impact of using 20 years expected returns rather than 30 year expected returns?
The documents the USS has provided for the 2020 valuation fall very fall short of the standards expected of academics and scientists, and are provided on a non-reliance basis, meeting no professional standards. For example, in the original proposal published by the USS in September 2020, no mention was made of a major change in their analytic models — specifically that they had changed from modelling 10 to 20 year returns to modelling 30 year returns. We only know they have made this change because of the documents they have published to justify how they are modelling prudence. They make a number of very curious claims about their modelling of investment returns:
“The 2017 and 2018 valuations used a combination of 10-year and 20-year return distributions, allowing for the time profile of expected returns relative to CPI, and the weighting of the return in each future period by the size of the corresponding benefit.
By contrast, the 2020 valuation uses a 30-year distribution of investment returns. Using a similar approach to the 2017 and 2018 valuations would result in confidence levels for both the discount rates for the 2020 valuation that were 73% — 79% (some 7% — 9% lower than confidence level for the pre-retirement discount rate indicated in Table 2) depending on the investment strategy.”
We can investigate the implications of this change using the simulations above. In Figure 2, the simulations have been rerun, but this time they include the distribution of returns at 15 years.
The average 15 and 30-year returns are very similar, around 5%. However, the distributions are very different, and the 15-year returns are much more variable. This is because, over time, year on year variation tends to get smoothed out, thus the variance of 30-year returns is much lower than 15-year returns. Therefore, the estimated returns at the 67th centile are 2.7% and 3.3% for the 15 and 30-year returns respectively. However, this creates a problem for the USS — the investment returns at a given level of prudence are higher for 30 year than 15-year investment returns, because 30 year returns are less variable than 15 year returns. To resolve this, USS argues we should keep the discount rate constant, and vary the level of prudence (i.e. which point in the distribution of returns should be used to derive the discount rate for the valuation). For example, in our simulations, the 73rd centile of returns in the 30-year returns is the same as the 67% centile of the 15-year returns. However, in doing this, they have fundamentally increased the level of prudence in the valuation — the expected investment returns at 15 and 30 years can only be equated if the level of prudence in 30-year investment returns is substantially and quantifiably higher. Yet, the USS has provided no evidence or justification for this increased level of prudence.
The USS also notes LCP’s “approximate analysis”, which has not been published or any meaningful details provided. Until the USS publish the details of this analysis, it must be disregarded.
Is 65% or 67% sufficiently prudent?
The USS used a 65% level of prudence in their 2014 valuation, and TPR did not challenge the valuation as being non-compliant with the legislation. In 2017 and 2018, USS increased the level of prudence to 67%, although no evidence was provided to justify this change. Therefore 65% and 67% level of prudence on investment returns were not challenged by TPR in 2014 and 2017 and 2018 respectively. Since then has there been any legal, demographic or economic circumstances that have changed sufficiently to justify this change in the valuation methodology? To date, the USS has provided no evidence to support this change. The most plausible explanation for why USS changed the level of prudence is that this is the level of prudence which gives a discount rate they, and TPR want. However, the discount rate is an outcome from the statistical model used in the valuation. Adjusting the input assumptions, such as the level of prudence, to achieve a desired discount rate is reverse engineering. This reverse engineering invalidates the valuation, as manipulating the level of prudence allows the Trustee to set the discount rate, deficit and future service costs to be whatever they want, irrespective of evidence.
USS lenses and contexts for comparing prudence
Despite arguing that the level of prudence is a qualitative judgement that cannot be expressed as a single number, USS prudence note details no less than five quantitative “lenses” and three “contexts” to justify their proposal to increase the level of prudence on investment returns for the 2020 valuation. I go through each of the arguments in the appendix. The key question is “do any of the arguments provided by USS provide quantitative evidence to justify increasing the level of prudence on investment returns in the 2020 valuation? The short answer is no. The lenses either suggest that prudence has been substantially increased (lens 1), or are outputs of the USS valuation model (lenses 2–5), or are qualitative statements about other factors in the valuation that are not justification for increasing the level of prudence on investment returns (e.g. the economic conditions at 30th March 2020, the strength of the covenant, and the size of the self-sufficiency deficit). None of these lenses or contexts are evidence to support increasing the level of prudence on investment returns, and suggests that the USS has reverse engineered assumptions of the valuation. See the appendix for full details.
Considering “the level of prudence” in the round
USS concludes by arguing that they do not use any single metric to define prudence, instead taking a holistic qualitative judgement on prudence. Prudence is widely defined as the excess over best estimate — it is the basis of all capital adequacy regimes for financial institutions. But equally, none of the comparisons they present, with the exception of lens 1, provides any meaningful quantitative comparison of the levels of prudence used in the valuation.
Das ist nicht nur nicht richtig; es ist nicht einmal falsch
It would be enormously helpful for stakeholders if the USS could provide a single overall measure of prudence used in the valuation, rather than making ad hoc changes to individual assumptions. The USS proposal to tweak the level of prudence on investment returns to make up for perceived underestimation of prudence elsewhere in the valuation is risible. There are many approaches for integrating statistical estimates into a single overall model to provide an overall indication of uncertainty or “prudence”. For example, multi-parameter evidence synthesis, meta-analysis, etc. But despite repeated criticism of its ad hoc approach to modelling, the USS has not been able to develop a coherent valuation methodology.
As a result, for the last six years, every valuation published by the USS has been met with incredulity from its stakeholders. I am 36 years old. I have no interest in seeing my pension underfunded, and would be happy for contributions to increase if necessary. Yet the USS has provided no acceptable reasoning for increased contributions, ludicrously basing its decisions based on qualitative judgements and arguments with scant theoretical or empirical justification.
None of the above will come as a surprise to members. Highly qualified USS members have been raising serious concerns about the valuation methodology for years. UUK have recently raised a series of similar concerns, only to be fobbed off with a contemptuous response from the Trustee.
How can we fix this mess?
All of this is indicative of a major ongoing governance crisis at the scheme. The JEP2 report made a number of simple recommendations to improve the scheme’s governance and valuation process. Naturally the USS has done nothing to integrate these recommendations into the scheme rules. I outline the problems with USS’s governance here and include some recommendations about how oversight and accountability of the Trustee must be urgently improved.
If USS governance is not radically improved, the USS Trustee stands little chance of recovering its members’ trust and will continue to fail in its fiduciary duties.
This paper represents the views of the author only. The author believes all information to be reliable and accurate; if any errors are found please contact us so that we can correct them. We welcome discussion of the points raised and suggest that discussants use Twitter with the hashtag #USSbriefs108; the author will try to respond as appropriate. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
[1] This is incorrect. Many statistical models synthesise errors from multiple parameters to provide an overall indication of uncertainty or “prudence”. Widely used examples include meta-analysis and Multi-Parameter Evidence Synthesis (MPES). Indeed, USS also argue “Prudence in the valuation means that there is much greater than a 50% confidence level that our funding target, the technical provisions (TP) liability, will be achieved” i.e. prudence is the probability of meeting the funding target in excess of 50%.
CORRECTION: On April 12th, the value in this sentence was corrected from 3.5%: “So the discount rate based on this level of prudence is 3.9% above inflation.”
