4. Evaluation of the HSMR 2024 model

This chapter presents and evaluates the HSMR model results. Summary outcomes of the 158 logistic regressions are presented, with in-hospital mortality as the dependent variable and the variables mentioned in section 3.4 as explanatory variables. More detailed results are presented in the tables “Statistical significance of covariates HSMR 2024 model” and “Coefficients HSMR 2024”. These tables are published together with this report.

4.1 Target population and data set

All hospitals that register complete records of inpatient admissions and prolonged observations without overnight stay in the LBZ are included in the HSMR model. In 2021-2024 all general hospitals and university hospitals were included in the model, as well as three short-stay specialised hospitals (two cancer hospitals and an eye hospital). From 2022 onwards a fourth short-stay specialised (orthopaedic) hospital also participated in the LBZ and was included in the model as well. All of the hospitals had completely registered all admissions in the LBZ. However, admissions from one hospital in the first 9 months of 2024 were omitted from the HSMR model because the urgency of the admissions had not been registered correctly.

The total number of hospitals included in the HSMR model of 2021-2024 is 75 and includes 64 general hospitals, 7 university hospitals and 4 short stay specialised hospitals. These numbers are based on the number of hospital units in 2024, counting hospitals that merged in the period 2021-2024 as one unit for all years.

Table 4.1.1 lists some characteristics of the admissions included in the HSMR 2024 model per year. Admissions not meeting the criteria of the Dutch Healthcare Authority, admissions of non-residents and admissions of healthy persons were excluded. In addition, admissions at rehabilitation or psychiatry units were excluded for the years 2021-2024. For the year 2021 only, admissions with COVID-19 as the principal diagnosis were also excluded. The number of admissions included in 2024 (1,490,942) was comparable to the number of admissions in 2023 (1,489,171). The total number of admissions in the 2021-2024 model (5,867,796) however, is 1.6% higher than the total number of admissions in the previous model (2020-2023; 5,774,810). This increase is most likely due to the fact that the year 2020 is no longer part of the model, a year in which the number of admissions was substantially lower compared to more recent years due to the start of the COVID-19 pandemic.

Crude mortality of the admissions included in the HSMR 2024 model was increased in 2022 (2.4%) compared to mortality of previous years (on average 2.0%). This was partly due to the inclusion of COVID-19 admissions. In 2023 and 2024 crude mortality was 2.3%.

4.2 Hospital exclusion

Hospitals were only provided with (H)SMR outcomes if the data fulfilled the criteria stated in section 3.5. In order to qualify for a three-year report (2022-2024) hospitals had to fulfil these criteria for the three consecutive years.

Of the 75 hospitals included in the model, all had registered (valid) data over 2024. The four short stay specialised hospitals and one general hospital had not been asked to grant authorization for providing HSMR numbers because their case mix was very different from that of the other hospitals. In fact, all of these hospitals had participated in the LBZ but the data of these hospitals did not meet one or more of the previously stated criteria, such as a minimum number of 60 registered deaths or on average a minimum number of 1.5 comorbidities per admission. All of the other 70 hospitals that had granted authorization fulfilled the criteria and were provided with a HSMR figure for 2024. For these 70 hospitals the data of 2022 and 2023 was additionally investigated in order to determine if a three-year report could be provided. The data of all 70 hospitals met the criteria in all years considered and so all hospitals were also provided with three-year HSMR figures.

4.3 Impact of the covariates on mortality and HSMR

The table “Statistical significance of covariates HSMR 2024 model” published together with this report shows which covariates have a statistically significant (95 percent confidence) impact on in-hospital mortality for each of the 158 diagnosis groups: “1” indicates (statistical) significance, and “0” non-significance, while a dash (-) means that the covariate has been dropped as the number of admissions is smaller than 50 (or as there are no deaths) for all but one category of a covariate; see section 3.6.2. The last row of the table “Statistical significance of covariates HSMR 2024 model” published together with this report gives the numbers of significant results across the diagnosis groups for each covariate. These values are presented again in table 4.3.1, as a summary, but ordered by the number of times a covariate is significant.

Age, urgency of the admission, and severity of the main diagnosis are significant for a large majority of the diagnosis groups. This is also true for several of the comorbidity groups, especially for group 2 (Congestive heart failure and cardiomyopathy). Comorbidity 15 (HIV) is only rarely registered as a comorbidity; most diagnosis groups had fewer than 50 admissions with HIV comorbidity. In only one of the models comorbidity 15 is statistically significant.

Overall, the number of significant covariates in the HSMR 2024 model (1,587) is comparable to that of the HSMR 2023 model (1,586). For most covariates, the number of times they are significant in the separate models fluctuates over the years (CBS, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025). For a few covariates however, a gradual increase or decrease of the number of significances was observed over time. Comorbidity 1 (Myocardial infarction) has shown the most profound decrease: from a significant effect in 64 models in the HSMR 2017 model to 36 significances in the HSMR 2024 model. To a lesser extent this is also true for Comorbidity 14 (Cancer): from 82 (HSMR 2017 model) to 62 (HSMR 2024 model) significances. In contrast, the number of models in which socio-economic status was significant, showed an increase over time: 15 versus 27 significances in the HSMR 2017 and 2024 model respectively. Finally, the number of times year of discharge was significant, was increased in the 2022 and 2023 models compared to previous years, which might be due to the inclusion of both pre-pandemic and pandemic years in those models. In the 2024 model the number of times year of discharge was significant decreased.

The significance only partially reflects the effect of the covariates on mortality. This is better measured using the Wald statistic. Table 4.3.2 presents the total of all Wald statistics summed over the diagnosis groups with the respective sum of the degrees of freedom, for each of the covariates ordered by value. It shows that severity of the main diagnosis has the highest explanatory power. Age and urgency of admission are also important variables. Of the comorbidities in the model, comorbidity groups 2, 16, and 13 are the groups with the most impact on mortality. Compared to the outcome of the 2023 model (CBS, 2024), the order of the covariates with regard to explanatory power is almost identical. Note that the values of the Wald statistics themselves cannot be compared directly between years as these values depend on the number of admissions used in the models.

The explanatory power of year of discharge has increased since the HSMR 2021 model, with the largest increase from 2021 to 2022 (13%) and further small increases of 3% in the HSMR 2023 and 2024 models. This implies that the differences in mortality between the years in the model (corrected for differences in patient characteristics) has slightly increased. The impact of comorbidity 1 (Myocardial infarction) has decreased by 45% since the HSMR 2017 model. A decreased impact of a comorbidity could reflect a decreased effect of the comorbidity (e.g. the likelihood of dying in hospital when having this condition as comorbidity has decreased), and/or a decreased number of patients with this comorbidity resulting in less accurate estimates of the effect of this comorbidity (which also decreases the Wald statistic). The opposite applies in the case of an increased impact of a variable.

As was mentioned in section 3.6.2, when the hospitals differ little on a covariate, the effect of this covariate on the HSMR can still be small even if this covariate is a strong predictor for mortality. Table 4.3.3 shows the impact of each covariate on the HSMR by showing the shift in HSMR by in- or exclusion of the covariates, as measured by the formula in paragraph 3.6.2, for the hospitals for which HSMRs are calculated. The comorbidities, which are considered here as one group (17 comorbidities together), have the largest effect on the HSMR. This is caused by differences in case mixes between hospitals, but possibly also by differences in coding practices (see Van der Laan, 2013). However, the effect of the comorbidities has decreased over the years by 30% when compared to the HSMR 2017 model. Deleting sex or month of admission as covariates hardly has an impact on the HSMRs, whereas SES has a reasonable impact on the HSMR. This is probably because hospitals differ more in terms of SES categories of the postal areas in their vicinity than in terms of the sex distribution of their patients. In general, the magnitudes of the effect on the HSMR of the covariates shown in table 4.3.3 are about the same as in the previous model.

4.4 Model evaluation for the 158 regression analyses

Table 4.4.1 presents numbers of admissions and deaths, and C-statistics for the 158 diagnosis groups. The C-statistic is explained in section 3.6.2. Overall the C-statistics have changed little compared to the previous model. Only one C-statistic showed a moderate increase: from 0.82 to 0.87 for “Fever of other and unknown origin” (149). All other increases are smaller than 0.03, with most below 0.02. For 79 diagnosis groups the C-statistic did not change compared to last year.

The C-statistics in two groups showed a moderate decrease compared to the 2023 model: from 0.88 to 0.84 for “Other psychoses” (50) and from 0.91 to 0.86 for “Complications of pregnancy childbirth and the puerperium; liveborn” (118). The decrease of the C-statistic in the group “Other psychoses” might be due to the exclusion of admissions at the psychiatry unit which is new in the 2024 model: as a result the number of admissions in this diagnosis group decreased from 6040 in the 2023 model to 5258 in the 2024 model. For several years in a row, only two of the 158 diagnosis groups have a C-statistic below 0.70: “Congestive heart failure, nonhypertensive” (70), and “Aspiration pneumonitis; food/vomitus” (84). For the diagnosis groups with a C-statistic below 0.70, the model’s ability to explain patient mortality is less than ‘good’, which may indicate that the model does not completely correct for population differences between the hospitals. For the highest scoring diagnosis groups (0.90 and above) the covariates strongly reduce the uncertainty in predicting patient mortality. In 2024, 64 diagnosis groups had a C-statistic above or equal to 0.90, the same number compared to 2023.

When comparing the outcome of the C-statistics per diagnosis group over several years, most C-statistics are rather stable or vary only slightly by year. However, the C-statistic of ‘Intracranial injury’ (138) has decreased considerably: from 0.87 in 2017 to 0.75 in 2024. Over the years, crude mortality in this diagnosis group has increased from 4.6% in 2017 to 7.2% in 2024. The decreasing C-statistic might be due to an increasing inability to completely correct for differences between patients with intracranial injury.

As mentioned in chapter 2 and section 3.4, the covariate ‘month of admission’ of the COVID-19 model was changed from a monthly to a bimonthly variable, in order to increase the similarity to the other 157 models. The C-statistic of the COVID-19 model has not changed in 2024 when compared to 2023 (both: 0.74), indicating that this change has not decreased the predictive value of the model.

4.5 Regression coefficients

The file “coefficients HSMR 2024” contains the estimated regression coefficients (columns “Estimate”), also called “log-odds”, for each of the 158 logistic regressions, as well as their standard errors (columns “Std. Err.”). The estimated regression coefficients are the elements of the estimate of vector 𝛽_d in the formula for  p̂_dhi (see section 3.6.1), for each diagnosis d. Notice that a β-coefficient has to be interpreted as the difference in log-odds between the category in question and the reference category (first category of the same covariate). For the sake of clarity, the reference categories are given in the first row of the corresponding covariates and these have by definition a zero coefficient for each regression. 

In many cases categories are collapsed (see section 3.6.2). This results in equal coefficients for the collapsed categories. If all categories were collapsed into one category for a certain variable and for a certain diagnosis group (i.e. if there was only one category with ≥50 admissions and ≥1 death), the variable was dropped from the model and all associated coefficients were set to zero. Therefore, the coefficients in the file “coefficients HSMR 2024” can be used directly to calculate mortality probabilities, with the exception of two of the Charlson comorbidities (Comorbidity 17 and Comorbidity 11). If Charlson comorbidity 17 (Severe liver disease) contains <50 admissions or no mortality, it is collapsed with Charlson comorbidity 9 (Liver disease). In this case the coefficient of Comorbidity 17 is set to zero. When a patient has both comorbidities, it counts as only one comorbidity. Therefore, when the coefficient of Comorbidity 17 is zero in the coefficients file, one should first recode all Charlson 17 comorbidities to Comorbidity 9 and use the coefficient of Comorbidity 9. The same holds for Charlson 11 (Diabetes complications) when it is collapsed with Charlson 10 (Diabetes).