Should you make a career change to AI & Machine Learning? Some thoughts about the next 5 years.

This post is my opinion. I’ve based my conclusions on data and my extrapolation of trends in that data. As with all things on the internet, take the ensuing conclusions with a pinch of salt.

Whenever I tell people I work in AI, their response is invariably some variant of, ‘That’s the future! You’re in the right field; you’ll have no trouble finding a job.’ There’s no doubt that AI is going to be be a major industry by and of itself and impact other industries in coming decades, but I wonder whether it really is true there’ll be ‘no trouble finding a job’. In this post, I’m going to speculate about the AI job market over the next 5 years. I hope it’s useful to help you decide if you should make a career change to this field.

A Note on Terminology

I’m using AI as a catch-all term to refer to machine learning, deep learning, and reinforcement learning.

Type of Employers and Types of Positions

AI employers come in 3 flavours: technology companies (e.g., Google), non-technology companies (e.g., HSBC or RBC), and start-ups (e.g., Xanadu from Toronto). These three categories of employers hire people trained in AI for different reasons. For technology companies, AI is becoming their raison d’etre; For example, at Google, AI powers their primary products such as search, Nest, Translate, and so on. For large non-technology companies, AI is used to build features to improve customer experience. For example, banks use machine learning to build probabilistic models to flag potentially fraudulent transactions. Finally, AI start-ups exist to provide some service to provide some service built upon AI. For instance, the start-up Scale provides service to manage AI data sets.

There are also 3 primary roles for which people skilled in AI: research scientist, research engineer, and machine learning engineer. Research scientists, as the name implies, develop novel solutions to both applied problems (e.g., Deepmind’s project on minimising data centre energy consumption) and push the boundary on long-standing research aspirations such as building Artificial General Intelligence. Research engineers fill a relatively new position that was created by big technology companies to address the current shortage of researchers in AI. They support research scientists with a focus on on programming. Finally, machine learning engineers may be thought as software engineers specialised in development for AI and machine learning.

The Job Market Today

Presently, there is a significant shortage in all three types of roles for workers in AI. Overall, demand for workers in this field is growing because all three types of companies previously identified (i.e., technology companies, large non-technology companies, and start-ups) are actively trying to build and incorporate some kind of AI into their products. Amazon’s Alexa, for example, exists because of recent research breakthroughs in in AI. Furthermore, this demand is amplified by the prevailing hype around AI. Venture capital firms are investing into AI and machine learning start-ups who are in turn hiring for roles such as machine learning engineer. On the flip-side, supply side factors contribute to this shortage as well. Specifically there is a shortage of research scientists because many leading academics who would otherwise be training the next generation of PhDs are currently employed by technology companies. Similarly, training research engineers also requires senior academics who are in short supply. Lastly, many classical software engineers are re-tooling for machine learning engineer positions, but this process requires time and effort due to the non-trivial mathematics and statistics background required.

Trends

Research Scientist

Over the next 5 years, demand for research scientists will persist. This is because research scientists are a necessity to large technology companies in order to gain and/or sustain a competitive edge. To illustrate this point, imagine a scenario where Amazon stops investing in research scientists while other firms such as Google continue their investment into research scientists. In a short time, the Android voice assistant will become superior to Alexa. In turn, the superior user experience provided by Google’s voice assistant will enable Google to capitalise on market share at the expense of Amazon. More generally, in order just to remain competitive, technology companies will have to continue their investment into research scientists. Given that most demand for research scientists is by technology firms rather than large non-technology companies or start-ups, we can safely ignore whatever trends there might be for research scientists by those employer categories.

Supply of research scientists will moderately increase over the next 5 years. In recent years while interest in graduate studies in AI has increased, enrolment in Ph.D. programs has not been able to keep up with this interest due to departure of senior academics to industry. Moreover, while many new faculty positions are being created for researchers in AI and machine learning, it will take more than 5 years for the Ph.D. students trained by new faculty to reach the job market. Therefore, overall, the job market for research scientists will be robust over the next 5 years, and indeed for the foreseeable future.

Research Engineer

As afore-mentioned, the position of research engineer is relatively new and was created to support research scientists in AI and machine learning. As the production of research scientists increases over the next 5 years, the requirement for research engineers may diminish. That is, research scientists may not need support of research engineers. Supply of people qualified as research engineers will also increase much faster than supply of research scientists. Many universities now offer some kind of master’s program in machine learning, and as this takes only 2 years to complete, many people qualified to be research engineers will be produced over the next 5 years.

As a consequence of the reduction in demand and increase in supply, job prospects for research engineers will become increasingly scarce over the next 5 years. The cohorts of students graduating over the next few years with just a master’s degree may find it difficult to find jobs as research engineers.

Machine Learning Engineer

Demand for machine learning engineers will (largely) maintain over the next 5 years. As all three categories of employers integrate AI and machine learning into their products, engineers will be needed to build, maintain, and improve these products. The trend with supply, however, suggests an increase in supply of machine learning engineers over the next 5 years. One factor contributing to this is the current high wages being paid to machine learning engineers. This is attracting people into the field who may have otherwise gone into other technical fields such as engineering or science. Another factor is the maturation of tools for building machine learning models which has lowered the barrier to entry for people who wish to join the field. For instance, compare the learning curve of early versions of Tensorflow to Keras.

The combination of unchanged demand but increased supply will likely result in loosening on the job market for machine learning engineers in 5 years time. That is, machine learning engineers may not receive a premium in salary over software engineers who do not specialise in machine learning. However, note this does not mean that machine learning engineers will not be sought after, merely that demand will come down from its current highs.

Conclusions

In conclusion, the job market for people working in AI and machine learning will almost certainly cool off over the next 5 years. While demand will remain robust at technology companies, hiring at large non-technology companies and start-ups will subside. Non-technology companies will have incorporated AI and machine learning features into their products, and therefore will only need sufficient employees to maintain and incrementally improve their products rather than having to actively hire in order to build AI and machine learning features. For start-ups, the waning of the hype sorrounding AI and machine learning will curtail venture capital and accordingly trim hiring. On the supply side of the market, the current trend of people training in AI and machine learning will bear out over the next few years resulting in an increase in supply, particularly of machine learning engineers and research engineers. The attenuation of demand and swelling of supply will combine to cool off the job market.

Returning to the title of this post, should you make a career change into AI & machine learning? If you’re planning on becoming a research scientist, unequivocally yes. If you wish to only become a machine learning engineer or research engineer, still yes, but be aware that you won’t be hot property for whom a job is guaranteed. In a few years time, being a machine learning engineer or research engineer will be about the same as being any other type of software engineer.