Choice of an Index Method for Consumer Energy Prices

Recently, detailed customer data of utility companies were made available to Statistics Netherlands (CBS). These data include prices and numbers of consumer energy contracts provided by the largest utility companies in the Netherlands. CBS has obtained data from 2020 onwards that also contain some important product characteristics such as contract duration and contract type. The characteristics of these data are very similar to transaction data, although they do not exactly fit the definition. Treating these comprehensive data as transaction data enables the use of moresophisticated index methods than the method that is currently in use.

This report describes the method chosen by CBS to compute the energy price index with this new data source. Underlying considerations are set out and several methods from the literature are evaluated against a number of selection criteria.

In terms of both index-theoretical properties and feasibility of implementation, CBS concludes that, in the current situation and given the available data, a multilateral Quality Adjusted (QU) Geary-Khamis index method with a Fixed Base Monthly Expanding Window (FBEW) is a good choice to compile a consumer price index for energy.

The data

The transaction data used originate from a number of utility companies that cover about three-quarters of the consumer market. First the data selectivity was studied. Apart from these data sources, the Netherlands Authority for Consumers and Markets (ACM) collects information from all utility companies on prices of new energy contracts available to consumers. Using the ACM data, CBS has examined whether the prices and trends of the utility companies in the transaction data resemble the prices and trends of the other companies that are not present in these data. It appeared to be neither possible nor strictly necessary to correct for selectivity of transaction data. It was concluded that the data sent by the utility companies could be used as representative for the energy contracts and prices for all consumers in the Netherlands.

The product definition

An important aspect in the implementation of any index method is product definition. Too narrowly defined products may lead to product discontinuity, i.e. many new and disappearing products. Too broadly defined products lead to inhomogeneity, which negatively affects the accuracy of the index. The so-called MARS method can be used to balance homogeneity and continuity. Based on this method, products can best be defined by utility company and duration of contract.

Selection criteria

To choose an appropriate index method it is important to have relevant selection criteria. Here, the following selection criteria were considered:

  1. simplicity,
  2. compliance with international standards,
  3. usability in the context of transaction data,
  4. flexibility:
    a. The method should be able to deal with new and disappearing products in real time.
    b. Data from new utility companies can be added or removed in real time.
    c. A high-quality index can still be calculated even if a data provider is absent for one or several months.
    d. The method can be easily adjusted, for instance if a correction for selective data collection turns out to be necessary.
    e. New government measures such as the price cap can immediately be taken into account, for instance an imposed maximum price.
    f. The method reflects without delay any changes in market share.
    g. A so-called constant-tax index series can be derived. This is a variant of the regular price index produced for Eurostat and national purposes, in which the tax rates are kept constant throughout a year.
    h. Use of available product information: the availability of price-determining characteristics (brand, contract type, etc.) allows for a more advanced and refined method than when only individual product codes are available.
  5. theoretical properties:
    a. No drift: the index reaches the correct level in the long term. The so-called "multi-period identity test" is an important indicator for a drift-free index.
    b. Transitivity: the outcome does not depend on the starting period of an index series.
    c. Characteristicity: Price changes in the published index are in line with the price changes of the underlying products.
    d. Method corrects for quality differences in case of migration, for instance consumers that replace one contract by another (cf Eurostat HICP Manual (2018), p.154).

Given the criteria above and given the detailed data provided to CBS, a so-called multilateral index method is the most appropriate.

This class of methods also has the additional benefit that it aligns with choices previously made by CBS for index computations from transaction data. Within the class of multilateral index methods the Geary-Khamis was chosen at CBS as the standard index method for transaction data. In principle, unless there are specific good reasons, we should not deviate from a standard method. As we did not find any motivation against Geary-Khamis, the focus in the motivation is on this method.

Below we will present the motivation for choosing the Geary-Khamis method. The first choice was between bilateral and multilateral, and second the choice for a specific bilateral or multilateral method. Additional reference is available in Chessa.


Bilateral versus multilateral

Regarding index methods, a first distinction can be made between bilateral and multilateral methods. Bilateral methods raise some serious concerns in the case of transaction data. A further distinction is between fixed base methods and methods that combine shorter period to period changes into a so-called chained index. Another important distinction is between methods that use equal or varying product weights based on expenditure share.

Index formulas with equal product weights, like the Jevons index, are transitive for fixed baskets of products but these methods do not capture shifts in expenditure and consumer preferences over time. A significant drawback of bilateral fixed base methods with varying product weights is that they do not deal in a flexible manner with product dynamics: new products that enter the market or old products that cease to exist. Another drawback of these methods is that they are not transitive: the choice of the base periods matters. Chained bilateral methods with varying weights are able to capture product dynamics but suffer from chain drift.

Multilateral methods do not have the above drawbacks: they are transitive, not subject to drift within the calculation window, and suitable for large amounts of data with any degree of product dynamics. Although the Jevons index is still used by a respectable number of statistical agencies for processing supermarket transaction data in their CPI, a growing number of statistical agencies are switching to multilateral methods. It is also important to emphasise that the aforementioned advantages come at a certain cost. In a multilateral method, the price change between two months not only depends on the prices in those two months but also on the prices in other months. This makes the index more complex, results can be less straightforward to interpret and it complicates validation. Another drawback is a lower score on characteristicity than the more traditional methods. This is a downside of transitivity: because the index change between two months also depends on other months, it can give a less clear picture of the price change of the underlying products.

So in short, in this situation multilateral index methods in general score better than the other methods. There are some drawbacks with respect to simplicity and characteristicity, however the interpretation of results can be facilitated by using existing tools developed for other applications, such as product impact and contribution calculations.

Multilateral index methods

In the literature, different multilateral methods are described. The most well-known are GEKS in combination with a bilateral method such as Walsh or Fisher. Other multilateral methods are: Time Product Dummy (TPD), and Geary-Khamis.

The Geary-Khamis method can be seen as an extension of a Unit Value method with a quality correction. Migration between contracts is taken into account, but with a quality correction. For example, if cheap contracts are exchanged for expensive ones, the price index does not necessarily increase, because of correction for quality change. Such a quality correction is in line with international regulations.

The Geary-Khamis method is very flexible in terms of product dynamics. New contract types and new data providers can be included when they are first sold.
Also, the index calculation is technically not impeded by missing contract types in one or more periods. However, some kind of imputation might be necessary to obtain reliable results.
Multilateral index methods compute an index for a certain number of periods, e.g. 13 months. The indices produced by such a method need to be combined into one long time series. This can be done using a so-called extension method. A natural choice for the energy data is to apply the Geary-Khamis method in one-year periods. The calculation of annual index series is practically appropriate in relation to the available data because many tax rates are adjusted annually, and data on yearly energy consumption are also provided on an annual basis. An extension method for a multilateral index such as Geary-Khamis that fits best with annual series is Fixed Based Expanding Window.

Also, like other index methods, Geary-Khamis can handle government measures such as the price cap, and constant-tax index series can be derived.

Unit value as a reference method

Because a unit value index has some complementary characteristics compared to the multilateral Geary-Khamis method, it can be used for comparison and evaluation of results.
A unit value method is easier to understand than Geary-Khamis. The drawback of its simplicity is that a unit value index does not take into account quality differences. For this reason, and because it is affected by shifts to products with different qualities, the method does not measure pure price changes. Because quality changes should not be reflected in a price index, the method is discouraged for many applications, unless it is clear that the product groups are homogeneous. For energy contracts this is debatable, for it is highly questionable whether long-term contracts that were offered in the past can directly be compared to flexible contracts that are offered now. For these reasons, a simple unit value method was not chosen as the preferred method.