6. Demand for workers with AI skills

In order to understand the role of AI in today’s labour market, it is important to have a clear picture of the demand for workers with AI skills. If the supply of workers with AI skills does not meet the demand, this could lead to a number of challenges, including a delay in the adoption of AI technologies by businesses and agencies. One way of measuring this demand is to survey the number of AI vacancies in the Netherlands. When counting job openings, standard classifications – such as the ISCO classification of occupations – are often used to distinguish job openings for specific occupational groups. However, conventional classifications do not include a specific AI category. For that reason, this chapter examines the effectiveness and feasibility of an alternative method that – based on job postings and modelling – distinguishes AI vacancies from non-AI vacancies. To this end, a conceptual delineation of the term AI vacancy is first provided in Section 6.1. Next, different methods of identifying AI vacancies are described and compared in Section 6.2. Using those methods, we assess the feasibility of identifying AI vacancies based on vacancy text data. The final section, 6.3, shows the results of the model that performed the best; we will also discuss the characteristics of the AI vacancies identified.

6.1 Conceptual delineation of AI vacancies

In order to identify AI vacancies, it is important to create a conceptual delineation. We must first define the terms ‘AI system’ and ‘AI vacancy’ and operationalise them.

6.1.1 Definition of AI system

As in chapter 3, we have opted to use the most recent definition of an AI system as established by the Organisation for Economic Cooperation and Development (OECD) in 2023. Statistics Netherlands is thus consistent with the definition of an AI system in the European AI Regulation (2024/1689), which came into effect on 1 August 2024. According to this definition, an AI system is ‘A machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.’³⁵⁾ A non-exhaustive set of examples using this definition includes: autonomous robots, self-driving cars, machine-learning models used for data analysis, AI-driven image analysis, and generative AI models that can produce text and/or images based on prompts.

6.1.2 Definition of AI vacancies

An AI vacancy is defined as follows: 'A vacancy for a job in which the use or production of AI systems (according to the OECD definition) is a core component. This requires that the job, as described in the job posting, can only be performed by a person with (in-depth) knowledge of AI systems.’ This means that a central aspect of the job is the production or use of AI systems. Production of AI systems can include writing code to develop, maintain, or improve AI systems; testing already existing AI systems; or contributing content to the development process of AI systems. Using AI systems can include data analysis with machine learning (ML) models or doing research using AI systems.

Vacancies for employees who are involved in the production of AI systems, but who do not require knowledge of AI systems – such as managers, IT specialists, or employees who make professional contributions in the scope of the AI system – are not considered AI vacancies. The same applies to job openings for workers who do have (in-depth) knowledge of AI systems, but who are not directly involved in the production or use of AI systems, such as ethicists, legal professionals, and journalists specialising in AI systems.

6.1.3 Operationalisation of AI vacancies

The following operationalisation was used in the manual classification of vacancies: 'The vacancy text and title demonstrate that the vacancy meets the definition of an AI vacancy.' Using manual assessment, it is impossible to avoid at least a degree of subjectivity. This mainly applies to determining the following: whether the job posting indicates that AI systems are central to a job; whether AI systems make up a large part of the job description; and whether (in-depth) knowledge of AI systems is a requirement for the position.

6.1.4 Data source: differences between online job ads and vacancies

This study used a sample of more than 7.5 million online job ad texts from the company Textkernel. These texts were posted on the internet between Q1 of 2018 and Q2 of 2024. The model-based approach being developed to identify AI vacancies (Section 6.2) makes direct use of these online job ad texts. A weighted model was applied to the number of online job ad texts found in order in order to arrive at nationally representative vacancy figures (Section 6.3). A detailed description of the weighted model can be found here (only available in Dutch). A vacancy is defined as a job opening for which employees are being sought outside a company or institution, and who can be put to work as soon as possible. Online job ads (henceforth: ads) are announcements of open positions posted online by employers. A single vacancy can have one, multiple, or no ads.

6.2 Methodology of identification of online AI job ads

This section presents the two methods tested: an ML classification model and a search terms model. The results of these models are compared with each other as well as a method previously developed by TNO and Jobdigger. Based on this study, an assessment is made of the reliability and feasibility of model-based identification of AI vacancies. For this study, we decided to investigate only those ads with text in Dutch or English. Internships (paid or unpaid) were not included in the population.

6.2.1 Machine learning model

Several machine learning (ML) classification models have been developed to identify AI vacancies. This section provides a brief overview of the best performing ML model. For further technical details, as well as an overview of other ML models tested, Technical Annex to Chaper 6 (only available in Dutch).

Encoding

Because ML classification models use a numeric input, the job texts are first converted to numeric values using an encoding method. Of the various methods tested for this purpose, term frequency - inverse document frequency (TF-IDF) encoding was part of the best performing combination of encoding method and ML classification model. This approach involves counting how often relevant words appear in the job posting. That number is then weighted based on the frequency with which ads normally contain these words. In addition to the standard TF-IDF encoding, additional information about ad characteristics and word embedding vectors have been added. These vectors map the interrelationship between words within the job posting.

Classification Model

The best performing classification model is a combination of regularised logit models. A regularised logit model is a standard logit model that selects the most relevant features in the dataset, based on an algorithm. In the case of TF-IDF encoding, individual features correspond to words. This means that the model first selects relevant words and then determines the extent to which the presence of these words in the job posting is associated with an increased probability that the ad meets our definition of an AI ad. The logit model then assigns each input  a probability score between 0 and 1. The higher this score, the more likely it is that the job posting concerns an AI vacancy, according to the model. As in Chapter 3, we chose to work with a combination of models trained on different data sets. The average probability score of four models is calculated; if this score exceeds 0.5, the ensemble classifies the ad as an AI ad.

Selection process

We compiled a manually labelled dataset to select the best performing combination of encoding method and ML classification model. The dataset consisted of 807 ads, 235 of which were AI ads. Of this dataset, 20 percent was used as test data and 80 percent as training data. The test data was used to measure the performance of the models and remained unchanged throughout the process. The training data was used to train the classification models; it was supplemented several times during the process.

Based on the initial manually labelled dataset, multiple combinations of encoding methods and ML classification models were tested. The best performing combinations were then applied to new unlabelled data from the full dataset. The ads that emerged as potential AI ads using this method were labelled manually. In this way, the manually labelled set expanded twice: first to 958 with 253 AI ads, then to 1,092 with 274 AI ads. These iterative steps were carried out to supplement the training dataset with ads whose status was not immediately clear. The inclusion of such hard-to-classify vacancies is valuable in order to improve the performance of the model. The best performing combination of encoding method and ML classification model was chosen based on all the datasets collected.

6.2.2 Search Terms Model

We also researched a simpler search term model that only looks to see if certain AI-related words appear in the job posting. An initial list of AI terms was compiled for this purpose. Several sub-selections were then made by taking a random number of AI terms from the initial list. Simultaneously, the minimum number of required words from a sub-selection that must appear in the job posting to classify it as an AI job posting was determined. We then determined the best combination of the chosen AI-related words and required word count by comparing the performance of each combination tested on the manually labelled dataset. The initial glossary and the final selected glossary can be found in Appendix 2.

6.2.3 TNO Search Terms Model

Previously, Statistics Netherlands participated in a study on AI vacancies. In that study, TNO and Jobdigger identified AI vacancies (using Jobdigger vacancy data) and Statistics Netherlands provided a statistical description of the companies with those vacancies. The study compiled a list of ads that met one of the following two conditions: 1) At least one of the words used in the job ad appeared in a pre-established glossary (see glossary B2.2.1 in Appendix 2); or 2) at least one of the skills in the ad specified by Jobdigger appeared in a pre-established list of AI skills. It then looked at the ISCO (International Standard Classification of Occupations) code of each ad in the list. When this ISCO code occurred in a predetermined selection of AI-related ISCO codes (see B2.2.2 in Appendix 2), the ad was identified as an AI ad.

In order to evaluate whether the models developed in this project outperform the method developed by TNO and Jobdigger, this approach was also applied to the Textkernel data used in the current project. However, because the skills specified by Jobdigger are not part of the Textkernel data, the approach taken by TNO and Jobdigger cannot be fully replicated. The comparison below will present two versions of this method: one with and one without the ISCO code requirement.

6.2.4 Results of feasibility study

Comparison tested model

The different methods were compared based on their balanced accuracy score³⁵⁾ on the test dataset. We also looked at the number of positive ads identified by applying the model to the entire dataset. These results are shown in table 6.2.4.1.

The ML model is the best performing method. This model scored the highest on balanced accuracy, and would thus appear to result in the most accurate selection. Interestingly, this method also identified the smallest number of AI vacancies, suggesting that the other models resulted in an overestimate. While the search term model performs poorly in comparison, it may be preferable in certain situations due to its speed and simplicity.

Both versions of TNO’s search terms model identified a much higher number of AI vacancies than the new models. The main reason for this is that they used an extensive glossary of possible AI terms. It includes several terms that can also be used outside an AI context, such as 'statistics,' ‘cloud’, and 'data cleaning'. As a result, these models are expected to overestimate the number of AI ads significantly. The model with the ISCO filter did identify fewer AI ads, but it also achieved a significantly lower balanced accuracy score compared to the unfiltered model. The reason for this is that the list of ISCO codes appears to be too limited. Many AI ads use ISCO codes outside the pre-established filter list, e.g. ISCO code 3314. As can be seen in section 6.3.2, that was one of the most common ISCO codes for AI vacancies found with the ML classification model.

Performance of the best-tested model

The percentage of false positives (i.e. ads falsely classified as AI vacancies) in the ML classification model was evaluated using a sample of 150 AI ads marked by the model. A manual review showed that thirteen ads (9 percent) had been incorrectly classified as AI ads. Eleven of those thirteen ads were AI-related but did not meet the specific operationalisation of AI vacancies. For example: there was a vacancy for a data engineer position, which involved developing a data pipeline in close cooperation with AI specialists. Although in this case the data engineer would contribute to an AI system, the job itself does not require AI expertise.

Due to the relative scarcity of AI vacancies, it is difficult to determine the percentage of false negatives (vacancies incorrectly marked as non-AI) using similar sampling methods. However, an evaluation of all manually-labelled AI ads showed that 15 percent of positive cases were not identified as such by the model. This is further compounded by the AI ads that were overlooked due to their lack of representation in the training data. We conclude that it is likely that the model-based identification slightly underestimates the actual number of AI ads.

6.2.5 Points of improvement

This project provided us with a reliable ML classification model for identifying AI vacancies. However, the modelling approach still has room for improvement in the future. This section identifies several areas that warrant revision or further scrutiny.

Context in encoding

The TF-IDF encoding method counts how often words appear in job texts and then weights them accordingly. Consequently, this encoding method does not take context into account. That context can be expressed at three different levels: word level, sentence level, and paragraph level. Words can have multiple meanings, and sentences and paragraphs contain information that may or may not relate to the candidate sought. A lot of useful information is lost due to the lack of context in the current method. It would therefore be relevant for follow-up research to look at whether model performance improves when an encoding method is used that takes context into account. We did not do this in the current study, as it was impossible to do for the entire dataset in a realistic timeframe, given the computing power available to Statistics Netherlands.

Dependence on training data 

In TF-IDF encoding, a set of words is determined from the training data, which are then used as features in the classification model. The only words included in the final input for the classification model are those that appear in more than m texts in the entire training dataset; the parameter m is defined beforehand. This method ensures that infrequent or less relevant words do not disrupt the classification procedure. However, one disadvantage is that the results are even more dependent on the training data used. The composition of the training dataset directly affects which words are considered relevant for classification. One potential issue with this is that the model might fail to recognise AI ads if the language differs from the training data. Again, a possible solution would be to apply a more complex encoding method: one that encodes based on the overall use of text in ad texts, rather than individual words alone.

Concept drift

One issue related to the point described above is known as ‘concept drift’. AI technologies are developing rapidly and the number of possible applications is expanding constantly. That also means that future AI ads will look different compared to those of the period 2018-2024. This can be observed in the terms used in the job posting, the type of jobs advertised, and the type of employers. It is therefore important that, if the model is to be used over the coming years, the training data should be supplemented with relevant new (AI) ads. Additionally, it may be necessary to remove older, no longer relevant ads from the training data. The dataset used in this project has already demonstrated some limited degree of concept drift. Data scientist vacancies in recent years often have machine learning as a requirement, which was much less common in previous years (2018/2019). The relationship between the concepts of data scientist and AI is changing over time.

6.2.6 Conclusion

The best performing ML classification model in this study is good enough at identifying AI vacancies to produce reliable figures on the number of AI vacancies in the Netherlands. However, the model does still produce errors: false positives as well as false negatives. It is expected that the model can be improved in the future to further reduce the percentage of false positive and false negative results. Once more computational power becomes available, it is expected that the biggest improvement can be made by employing a more complex encoding method; one that takes context into account and looks at broader use of language, rather than individual words. If other encoding methods are tested in a follow-up study, it is important to retest which ML classification model best fits these new methods, too.

6.3 Statistical description of AI vacancies

This section provides a description of the AI vacancies identified using the ML classification model. The figures provide a picture of some possible applications of the AI vacancy identification explored in this chapter. These are provisional figures, as the modelling process could still be improved. Figures are rounded to the nearest five. The numbers presented relate to vacancies in the Netherlands, and were obtained by weighting the labelled results of online vacancies using the aforementioned weighting method. After applying the weighting model, the 9,430 online job ads found correspond to a total of 8,725 vacancies.

6.3.1 Number of AI vacancies over time

Figure 6.3.1.1 shows the number of AI vacancies from the first quarter of 2018 to the second quarter of 2024. An increase in AI vacancies can be seen during the period 2018 to 2022, peaking at the beginning of 2022. After this, the number drops before hovering around a value of 430. Over the time period considered, the proportion of identified AI vacancies ranged from 0.05 to 0.13 percent of the total number of vacancies in the Netherlands.

6.3.2 Characteristics of AI vacancies

Figure 6.3.2.1 reveals that, from the first quarter of 2018 to the second quarter of 2024, the total number of AI vacancies was greatest in Noord-Holland (2770), Zuid-Holland (1435) and Noord-Brabant (1205). Zeeland, Flevoland, Drenthe and Friesland had the lowest number of AI vacancies (less than 100).

Figure 6.3.2.2 shows the total number of AI vacancies per occupational group according to the International Standard Classification of Occupations (ISCO). Most AI vacancies fall within a limited number of occupational groups. The five most common groups are: 2511 Systems Analysts; 3314 Statistical, Mathematical and Related Associate Professionals; 2512 Software Developers; 2310 University and Higher Education Teachers; and 1330 Information and Communications Technology Service Managers. The top five occupational groups account for more than 70 percent of all AI vacancies found, with the remaining 30 percent distributed among 192 other ISCO codes.

Roughly 75 percent of AI vacancies were written in English, while the remaining 25 percent were written in Dutch. This distribution differs significantly from the language distribution across all vacancies, where 84 percent were Dutch and only 16 percent were English. Figure 6.3.2.3 shows that more than 72 percent of AI vacancies fall under occupational skill level 4, which is defined as ‘highly complex, specialised tasks requiring a higher or academic education level.’ About 23 percent of AI vacancies fall under skill level 3, which is defined as ‘complex tasks, requiring secondary education or higher.’ Thus, AI vacancies are almost entirely geared toward individuals with higher levels of education.

6.3.3 Characteristics of companies with AI vacancies

Figure 6.3.3.1 shows the number of AI vacancies, organised by sector of the companies posting those vacancies. The sectors with the most AI vacancies are P Education, J Information and communications, M Specialised business services, G Trade and C Manufacturing. Together, these five groups account for 71 percent of the total number of AI vacancies. In Education, the largest sector, 91 percent of AI vacancies fall into the subcategory Tertiary Education.

Comparing figures 6.3.3.1 and 5.3.2 (Chapter 5) shows that the number of students leaving AI education and the number of AI vacancies are similarly distributed across sectors. It is notable that sector P Education is the largest sector when it comes to AI vacancies, while it is smaller in terms of the number of students leaving AI education. One possible explanation is that this category includes predominately positions for PhDs, postdocs and professors. These positions are regularly filled by international candidates and not only by students leaving Dutch AI education. Chapter 5 showed that a relatively high number of AI students end up in public administration or public services (figure 5.3.2), but the number of AI vacancies in this industry is relatively small (figure 6.3.3.1). This could be explained by the fact that, while AI students are desired for jobs in the public sector, the government does not specifically ask for AI skills.

Of all AI vacancies, 86 percent could be assigned to a company or organisation. The ten organisations with the most AI vacancies to their name collectively sent out 2,000 AI vacancies over the period studied. Of these top ten organisations, seven are Dutch universities. As could be seen in figure 6.3.3.1, many AI vacancies are associated with university research and education.

Figure 6.3.3.2 shows the distribution of AI vacancies among the corresponding companies and organisations. The x-axis corresponds to the percentage of the total enterprise population and the y-axis to the percentage of the total number of AI vacancies. The figure shows that 20 percent of companies send out 80 percent of AI vacancies. Thus, the number of AI vacancies is not distributed evenly among companies and organisations. There is a long tail of companies with two or fewer AI vacancies, starting at around 40 percent.