Choice of an Index Method for Consumer Energy Prices

Recently, detailed customer data of utility companies were made available to Statistics Netherlands (CBS). These data include prices and numbers of consumer energy contracts provided by the largest utility companies in the Netherlands. CBS has obtained data from 2020 onwards that also contain some important product characteristics such as contract duration and contract type. The characteristics of these data are very similar to transaction data, although they do not exactly fit the definition¹⁾. Treating these comprehensive data as transaction data enables the use of more sophisticated index methods than the method that is currently in use.

This report describes the method chosen by CBS to compute the energy price index with this new data source. Underlying considerations are set out and several methods from the literature are evaluated against a number of selection criteria.

In terms of both index-theoretical properties and feasibility of implementation, CBS concludes that, in the current situation and given the available data, a multilateral Quality Adjusted (QU) Geary-Khamis index method with a Fixed Base Monthly Expanding Window (FBEW) is a good choice to compile a consumer price index for energy.

The data

The transaction data used originate from a number of utility companies that cover about three-quarters of the consumer market. First the data selectivity was studied. Apart from these data sources, the Netherlands Authority for Consumers and Markets (ACM) collects information from all utility companies on prices of new energy contracts available to consumers. Using the ACM data, CBS has examined whether the prices and trends of the utility companies in the transaction data resemble the prices and trends of the other companies that are not present in these data. It appeared to be neither possible nor strictly necessary to correct for selectivity of transaction data. It was concluded that the data sent by the utility companies could be used as representative for the energy contracts and prices for all consumers in the Netherlands.

The product definition

An important aspect in the implementation of any index method is product definition. Too narrowly defined products lead to great discontinuity, i.e. many new and disappearing products. Too broadly defined products lead to inhomogeneity, which negatively affects the accuracy of the index. The so-called MARS method²⁾ can be used to balance homogeneity and continuity. Based on this method, products can best be defined by utility company and duration of contract.

Selection criteria

To choose an appropriate index method it is important to have relevant selection criteria. Here, the following selection criteria were considered:

1. simplicity,
2. compliance with international standards,
3. usability in the context of transaction data,
4. flexibility:
a. The method should be able to deal with new and disappearing products in real time.
b. Data from new utility companies can be added or removed in real time.
c. A high-quality index can still be calculated even if a data provider is absent for one or several months.
d. The method can be easily adjusted, for instance if a correction for selective data collection turns out to be necessary.
e. New government measures such as the price cap can immediately be taken into account, for instance an imposed maximum price.
f. The method reflects without delay any changes in market share.
g. A so-called constant-tax index series can be derived. This is a variant of the regular price index produced for Eurostat and national purposes, in which the tax rates are kept constant throughout a year.
h. Use of available product information: the availability of price-determining characteristics (brand, contract type, etc.) allows for a more advanced and refined method than when only individual product codes are available.

5. theoretical properties:
a. No drift: the index reaches the correct level in the long term. The so-called "multi-period identity test" is an important indicator for a drift-free index.
b. Transitivity: the outcome does not depend on the starting period of an index series.
c. Characteristicity: Price changes in the published index are in line with the price changes of the underlying products.
d. Method corrects for quality differences in case of migration, for instance consumers that replace one contract by another (cf Eurostat HICP Manual (2018), p.154).

Given the criteria above and given the detailed data provided to CBS, a so-called multilateral index method is the most appropriate. 

This class of methods also has the additional benefit that it aligns with choices previously made by CBS for index computations from transaction data. Within the class of multilateral index methods Geary-Khamis was deemed the best. At CBS, Geary- Khamis has been chosen as the standard index method for transaction data. In principle, unless there are specific good reasons, we should not deviate from a stanard method. As we did not find any motivation against Geary-Khamis, the focus in the motivation is on this method.

Below we will present the motivation for choosing the Geary-Khamis method. The first choice was between bilateral and multilateral, and second the choice for a specific bilateral or multilateral method.

Motivation

Bilateral versus multilateral
Regarding index methods, a distinction can be made between bilateral and multilateral methods. Bilateral methods raise some serious concerns in the case of transaction data. A significant drawback of many bilateral methods is that they do not deal in a flexible manner with product dynamics: new products that enter the market or old products that cease to exist. A drawback of many bilateral methods with a fixed base month is that they are not transitive: the choice of the base periods matters. A drawback of so-called chaining methods is they are known to suffer from chain drift.

Multilateral methods do not have the above drawbacks: they are transitive, not subject to drift (at least within the calculation window) and suitable for large amounts of data with a lot of product dynamics. Multilateral methods are being applied by an increasing number of statistical agencies. At CBS and elsewhere, they are often seen as the standard for transaction data. In a multilateral method, the price change between two months not only depends on the prices in those two months but also on the prices in other months. This makes the index more complex, results can be less straightforward to interpret and it complicates validation. Another drawback is a lower score on characteristicity than the more traditional methods. This is a downside of transitivity: because the index change between two months also depends on other months, it can give a less clear picture of the price change of the underlying products.

So in short, in this situation multilateral index methods in general score better than the other methods. There are some drawbacks with respect to simplicity and characteristicity, however the interpretation of results can be facilitated by using existing tools developed for other applications, such as impact and contribution calculations.

Multilateral index methods
In the literature, different multilateral methods are described. The most well-known are GEKS in combination with a bilateral method such as Walsh or Fisher. Other multilateral methods are: Time Product Dummy (TPD), and Geary-Khamis.

The Geary-Khamis method can be seen as an extension of a Unit Value method with a quality correction. Migration between contracts is taken into account, but with a quality correction. For example, if cheap contracts are exchanged for expensive ones, the price index does not necessarily increase, because of correction for quality change. Such a quality correction is in line with international regulations.

The Geary-Khamis method is very flexible in terms of product dynamics. New contract types and new data providers can be included when they are first sold.

Also, the index calculation is technically not disturbed when certain contract types are missing in one or more periods. However, some kind of imputation might be necessary to obtain reliable results.

Multilateral index methods compute an index for a certain number of periods, e.g. 13 months. The indices produced by such a method need to be combined into one long time series. This can be done using a so-called extension method. A logical way is to apply the Geary-Khamis method in one-year periods. In this way, the average consumption of electricity and gas within each year can be kept constant. Therefore, changes in consumption cannot influence the price index. This meets the criterion that a price index must measure pure price effects. The calculation of annual index series is practically appropriate in relation to the available data because many tax rates are adjusted annually, and data on yearly energy consumption are also provided on an annual basis.

An extension method for a multilateral index such as Geary-Khamis that fits best with annual series is Fixed Based Expanding Window.

Also, like other index methods, Geary-Khamis can handle government measures such as the price cap.
Also, constant-tax index series can be derived.

Unit value as a reference method
Because a unit value index has some complementary characteristics compared to the multilateral Geary-Khamis method, it can be used for evaluation of results.

A unit value method is easier than Geary-Khamis. The drawback of its simplicity is that a unit value index does not take into account quality differences. The method does not only measure pure price changes, but is also affected by shifts to products with different qualities. Because quality changes should not be reflected in a price index, the method is discouraged for many applications, unless it is clear that the product groups are homogeneous. For energy contracts this is debatable, for it is highly questionable whether long-term contracts that were offered in the past can directly be compared to flexible contracts that are offered now. For these reasons, a simple unit value method was not chosen as the preferred method.