Making regulators accountable
September 23rd, 2007 by Jon Stern, City UniversityThere are now over 100 independent electricity and/or energy regulatory agencies around the world and all EU member states have them. In 2005, the budgetary cost of the EU energy regulators was over €200 million and to this must be added the costs to energy supply companies and others. So, how well do they perform? Can we identify any net benefits from their existence? And how should we go about trying to establish the impact of these agencies?
It is worth remembering that most energy (and telecom) regulators are young institutions. Although the US and a few developing countries such as Costa Rica and the Philippines have had independent regulators for over 50 years, most have only been established in the last 10-15 years. Indeed of the current EU member states, most energy National Regulatory Authorities (hereafter, NRAs) were only established within the last 10 years. The UK regulator is the only decision making regulator approaching 20 years of operation, followed by Italy and Spain at 11-12 years. At the other end of the spectrum, the German energy NRA was only established in 2005. In consequence, the median age of EU energy NRAs is 7-8 years.
EU energy NRAs vary considerably in their composition, funding and powers. In particular, in countries where regulation of retail prices still continues, this is a competence that is sometimes maintained by the government rather than being handled by the regulator – as in Belgium and France. However, since the 2003 EU Electricity and Gas Directives, most of the energy NRAs have a core set of responsibilities that include transmission and distribution access and pricing rules as well as wholesale market issues. (See the CEER 2005 Regulatory Benchmarking Report for a detailed survey.)
Evaluating the performance of regulatory agencies is not easy and relating this back to the characteristics of the regulators is harder still. However, there are now a series of studies that are beginning to provide a basis for evaluating their performance as well as providing some evidence on the its quality. These studies include both academic studies as well as policy audit reports such as those of the UK National Audit Office.
Econometric studies in this area have focused primarily on developing countries. This is because developing countries have a clear excess demand for infrastructure industry capacity and supply, including electricity so that we can identify supply-side effects without worrying about periods of excess capacity. For electricity, a 2006 study by Cubbin and Stern estimated the impact of 4 regulatory variables on 28 developing countries again using a fixed effects model, including GDP per capita and other control variables. They found that three of the four regulatory characteristics – having a regulatory law, autonomy of the regulator and funding by licence fee rather than from a central budget – were all significantly associated with higher generation capacity per head. A maximum score on the index was associated with 15-25% higher generation capacity in the long-run, although the effect took a long time to build up. There was no perceptible effect of the regulator for 3 years and it was at least 5 years on before one-half of the long-run effect was achieved. This study also found that majority privatisation of the incumbent electricity company was also associated with 8-10% higher generation capacity levels in the long-run. Although country governance variables were significantly associated with generation capacity, the estimated impact was small and, surprisingly, there was no evidence that the quality of country governance (i.e. measures of the rule of law, the degree of corruption or political risk measures).affected the impact of electricity regulators.
Econometric studies like these are useful but are far from sufficient to provide a robust evaluation of the impact of electricity and energy regulators. First, the methods cannot be applied to EU and other developed countries that have (or have had) periods of excess capacity. Second and more importantly, however powerful the statistical associations, they cannot provide information on how or why regulatory agencies have their apparent impact. That requires a policy audit approach (e.g., structured case studies based on careful examination of key decisions).
As with all ex post evaluations, the fundamental issue in evaluating regulatory agencies is the comparator base-line – the ‘counter-factual’. The key practical problem is to how establish a reliable counter-factual that sets out what would have been expected to occur in the absence of the intervention. For most public expenditure projects, this is reasonably straightforward. For instance, when a new urban by-pass road is constructed, the obvious counter factual is to compare traffic loads and congestion with the new road relative to projections on pre-existing roads without the new by-pass.
For energy and other regulators, it is much harder to construct an appropriate counter-factual. They make sequences of decisions, most of them relatively minor, but a small number very important and most decisions dependent on previous decisions. Assuming a counter-factual of no regulator makes no sense. What would the energy market outcomes have been like in the absence of Ofgem for the last 15-plus years? They would have been very different – and in ways that we cannot measure or predict.
The only effective answer to this is to carry out structured case-studies which investigate the outcome of particular significant regulatory agency decisions and place these in the context of government policy and previous regulatory practice. The outcomes can be compared to:
(a) what was happening in previous years;
(b) what was predicted to happen by the regulator (and/or others) with the change – or, if unavailable a constructed prediction of the outcome; and
(c) what were the results of similar changes in comparable countries (where available).
The most effective evaluations are those where there were sufficiently precise predictions made by the regulator or others and there are comparable changes in other countries. However, in all cases, this is an extensive exercise which requires large amounts of data, preferably for at least 2-3 years after the change and 7-10 years before the change.
In 2006, the World Bank published a Handbook ‘Evaluating Infrastructure Regulatory Systems’ by A. Brown, J.Stern and B.Tenenbaum. The Handbook sets out and explains in some detail an approach of this type and set out various evaluation tools. The Handbook approach was subsequently used to evaluate the performance of the Jamaican OUR (Office of Utility Regulation).
The evaluation of the OUR showed major benefits from regulatory actions to liberalise telecoms. However, for electricity, the major decision examined was the privatisation of JPS, the incumbent power company with no vertical separation – not even accounting separation. The result was that there had been no observable improvements on operating costs, maintenance expenditure or investment; indeed, in several of these performance had arguably worsened post-privatisation. The key problem identified in the evaluation was that the OUR has very little regulatory purchase by which to monitor and take action to improve the various activities of JPS. This was confirmed by a comparison with Trinidad & Tobago (T&T) where the incumbent was split in 1994 between a network company and a generating company. In T&T, system losses are significantly lower than in Jamaica and, unlike Jamaica, have been falling. In addition, T& T non-fuel generation costs were found to be around half the level of those in Jamaica.
The UK National Audit Office (NAO) has carried out a number of evaluations of major regulatory decisions using this type of approach but in a less formalised way, as has the US GAO and others.
The NAO reports on Ofgem activities include:
• a 2001 assessment of the impact of introducing full retail competition in energy;
• a 2003 assessment of the results of introducing NETA (the New Electricity Trading Arrangements) in England and Wales;
• a 2004 assessment of policies on social plans (pre-payment meters, fuel poverty, etc) and energy efficiency; and
• a 2006 assessment of Ofgem’s role in the sale of regional gas networks by National Grid.
In general, the NAO reports on Ofgem have been favourable. Those on NETA and retail competition pointed out some problems and make recommendations for the future but endorsed the change. The 2006 report on Ofgem’s handling of the sale of the gas networks was particularly supportive (e.g., in the transparency of the cost-benefit estimates and the use of sensitivity analysis) to the extent that it was suggested as a model for other regulators. Of the reports listed above, only the 2004 report on social plans and energy efficiency included any noticeable criticisms and recommendations for additional action.
NAO reports are not always favourable. Ofcom (the telecoms and broadcasting regulator) attracted considerable criticism in an evaluation by NAO in 2005 for its decision (and its implementation) of the mandatory introduction of competition into the directory enquiries service. In the energy area, the 2005 NAO report on renewable energy support made substantive criticisms of the DTI (Department of Trade and Industry) policies in this area. The report pointed out that the programme was significantly failing to meet its targets, the cost of CO2 reductions from supporting renewables was significantly higher than from other methods and that some lower cost renewable technologies were receiving support in excess of that necessary to support investment.
This discussion shows how ex post evaluation can, in practical terms, set up a learning process by which regulators can build in good practice as well as identify and learn from their mistakes. They significantly build in real accountability and should be a part of all energy regulatory frameworks.