Despite huge adoption of AI and machine learning (ML), many organisations are still struggling to get ML models into production at scale.

The result is AI projects that stall, don’t deliver ROI for years, and potentially fail altogether. Gartner Group estimates that only half of ML models ever make it out of trials into production. 

Why is this happening? One of the biggest issues is that companies develop successful ML prototype models, but these models aren’t equipped to be deployed at scale into a complex enterprise IT infrastructure. 

All of this slows down AI development. Software company Algorithmia recently reported that most companies spend between one and three months deploying a new ML model, while one in five companies took more than three months. Additionally, 38% of data scientists’ time is typically spent on deployment rather than developing new models. 

Algorithmia found that these delays were often due to unforeseen operational issues. Organisations are deploying models only to find they lack vital functionality, don’t meet governance or security requirements, or need modification to provide appropriate tracking and reporting. 

How MLOps can help 

Enter MLOps. While MLOps leverages DevOps’ focus on compliance, security, and management of IT resources, MLOps add much more emphasis on the consistent development, deployment, and scalability of models. 

Organisations can accelerate AI adoption and solve some of their AI challenges by adopting MLOps. Algorithmia found that where organisations were using MLOps, data scientists were able to reduce the time spent on model deployment by 22%, and the average time taken to put a trained model into production fell by 31%. 

That’s because MLOps provides a standard template for ML model development and deployment, along with a clear history and version control. This means processes don’t need to be reinvented for each new model, and standardised processes can be created to specify how all models should meet key functional requirements, along with privacy, security and governance policies. 

With MLOps, data teams can be confident that new code and models will meet architecture and API requirements for production usage and testing. By removing the need to create essential features or code from scratch, new models are faster to build, test, train and deploy. 

MLOps is being widely used for tasks such as automation of ML pipelines, monitoring, lifecycle management and governance. MLOps can be used to monitor models to sense any fall in performance or data drifts that suggest models might need to be updated or retrained.

Having a consistent view of ML models throughout the lifecycle in turns allows teams to easily see which models are live, which are in development, and which require maintenance or updates. These can be scheduled more easily with a clear overview of the ML landscape. 

Within MLOps, organisations can also build feature stores, where code and data can be re-used from prior work, further speeding up the development and deployment of new models. 

Learn more about MLOps 

Our new playbook, Operationalising Machine Learning, provides guidance on how to create a consistent approach to monitoring and auditing ML models. Creating a single approach to these tasks allows organisations to create dashboards that provide a single view of all models in development and production, with automated alerts in case of issues such as data drift or unexpected performance issues.

If you’re struggling to realise the full potential of machine learning in your organisation, the good news is that you’re not alone. According to industry analysts VentureBeat, 87% of AI projects will never make it into production.

MLOps emerged to address this widespread challenge. By blending AI and DevOps practices, MLOps promised smooth, scalable development of ML applications.

The bad news is that MLOps isn’t an immediate fix for all AI projects. Operationalsing any AI or machine learning solution  will present its own challenges, which must be addressed to realise the potential these technologies offer. Below we’ve outlined five of the biggest MLOps challenges in 2022, and some guidance on solving these issues in your organisation.

You can read about these ideas in more detail in our new MLOps playbook, “Operationalising Machine Learning”, which provides comprehensive guidance for operations and AI teams in adopting best practice around MLOps. 

Challenge 1: Lack of user engagement

Failing to help end users understand how a machine learning model works or what algorithm is providing an insight is a common pitfall. After all, this is a complex subject, requiring time and expertise to understand. If users don’t understand a model, they are less likely to trust it, and to engage with the insights it provides.

Organisations can avoid this problem by engaging with users early in the process, by asking what problem they need the model to solve. Demonstrate and explain model results to users regularly and allow users to provide feedback during iteration of the model. Later in the process, it may be helpful to allow end users to view monitoring/performance data so that you can build trust in new models. If end users trust ML models, they are likely to engage with them, and to feel a sense of ownership and involvement in that process.

Challenge 2: Relying on notebooks

Like many people we have a love/hate relationship with notebooks such as Jupyter. Notebooks can be invaluable when you are creating visualisations and pivoting between modelling approaches.

However, notebooks contain both code and outputs, along with important business and personal data, meaning it’s easy to inadvertently pass data to where it shouldn’t be. Notebooks don’t lend themselves easily to testing, and cells that can run out of order means that different results can be created by the same notebook based on the order that cells are run in.

In most cases, we recommend moving to standard modular code after creating an initial prototype, rather than using notebooks. This results in a model that is more testable and easier to move into production, with the added benefit of speeding up algorithm development.

Challenge 3: Poor security practice 

There are a number of common security pitfalls in MLOps that should be avoided, and it’s important that organisations have appropriate practices in place to ensure secure development protocols.

For example, it’s surprisingly common for model endpoints and data pipelines to be publicly accessible, potentially exposing sensitive metadata to third parties. Endpoints must be secured to the same standard as any development to avoid cost management and security problems caused by uncontrolled access.

Challenge 4:  Using Machine Learning inappropriately

Despite the hype, ML shouldn’t always be the default way to solve a problem. AI and ML are essentially tools that help to understand complex problems like natural language processing and machine vision.

Applying AI to real-world problems that aren’t like this is unnecessary, and leads to too much complexity, unpredictably and increased costs. You could build an AI model to predict whether a number is even or odd – but you shouldn’t.

When addressing a new problem, we advise businesses to try a non-ML solution first. In many cases, a simple, rule-based system will be sufficient.

Challenge 5: Forgetting the downstream application of a new model   

Achieving ROI from machine learning requires the ML model to be integrated into business systems, with due attention to usability, security and performance.

This process becomes even longer if models are not technically compatible with business systems, or do not deliver the expected level of accuracy. These issues must be considered at the start of the ML process, to avoid delays and disappointment.

A common ML model might be used to predict ‘propensity to buy’ – identifying internet users who are likely to buy a product. If this downstream application isn’t considered when the model is built, there is no guarantee that the data output will be in a form that can be used by the business API. A great way to avoid this is by creating a walking skeleton or steel thread (see our Playbook for advice on how to do this).

Find out more about these challenges and more in our new Operationalising Machine Learning Playbook, which is available to read here.

Building a predictive model to forecast the future from historical data is standard practice for today’s businesses. But deploying, scaling and managing these models is far from simple.

Each ML solution depends on an algorithm (code) and a set of data used to develop and train the algorithm. For this reason, building ML solutions is different to other types of software development. 

Enter MLOps, or machine learning operations, a set of processes that help organisations to develop, deploy and monitor ML models at scale by applying best practices to infrastructure, code and data. 

MLOps is a relatively new idea but one that has been adopted by many organisations – the market for MLOps solutions is expected to reach $4 billion by 2025. At Equal Experts, we have been involved in developing and deploying AI and ML for a number of applications including to: 

• Assess cyber risk 
• Evaluate financial risk 
• Improve search recommendations for retail websites 
• Improve logistics and supply chains 

Key Terms used in MLOps

If you’re new to MLOps there are several important terms to be aware of:

• Machine learning (ML) – a subset of AI that involves training algorithms with data rather than developing hand-crafted algorithms. A machine learning solution uses  a data set to train an algorithm, typically training a classifier that says what type of thing  this data is (e.g. this picture is of a dog ); a regressor, which estimates  a value (e.g. the price of this house is £400,000.) or an unsupervised  model, such as generative ones  which can be used to write novel text (such as song lyrics).  
• Model – In machine learning a model is the result of training an algorithm with data, which maps a defined set of  inputs to outputs.  
• Algorithm – we use this term more or less interchangeably with model. (There are some subtle differences, but they’re not important and using the term ‘algorithm’ prevents confusion with the standard software engineering use of the term ‘data model’ – which is a definition of the data entities, fields, relationships etc  for a given domain, that is used to define database structures among other things.)
• Ground-truth data – a machine-learning solution usually needs a data set that contains the input data (e.g. pictures) along with the associated answers (e.g. this picture is of a dog, this one is of a cat)  – this is the  ‘ground-truth’.
• Labelled data – means the same as ground-truth data. 

How does MLOps work? 

We talk about MLOps as a set of processes that help data scientists to develop consistent, scalable ML models, and monitor their performance. To create and use these algorithms, you will usually follow these steps: 

Initial development of the algorithm – Developing a model is the first step in machine learning. Data scientists will identify or create ‘ground truth’ data sets and explore them. They will build and evaluate prototypes of the models, trying out different core algorithms and data transformations  until they arrive at  one which meets the business need.

Integrate/deploy the model – once the model has been built, it must be integrated into the business. This can be done in various ways depending on the consuming service. In modern architecture, models are commonly implemented as a standalone microservice and models are deployed by copying an approved version of the model into an operational environment. 

Monitor performance – All ML models need to be monitored to ensure they’re running and meeting demand, but also that the results of the model are accurate and reliable.

Update model – over time, models must be retrained to reflect new data, or improvements to the model. In this case, it’s important to maintain version control and to direct downstream services to the new model.  

Operationalising Machine Learning 

Our MLOps playbook, brings together our experiences working with algorithm developers to build ML solutions. It provides a comprehensive overview of what you need to consider when providing the architecture, tools and infrastructure to support data scientists and to integrate their outputs into the business.

Download the playbook for expert guidance on how your organisation can attain  the promised business value from algorithms by providing engineering to support algorithm development, and by integrating ML more effectively into your business processes. You’ll find helpful advice on how to:

• Collect data that drives machine learning, and make that available to data scientists 
• Integrate algorithms into your everyday business 
• Configuration control, deploy and monitor deployed algorithms 
• Test and monitor the algorithms  

View our online version or download a pdf here.  

 

 

The inconvenient truth is that most big data projects fail to deliver the expected return on investment. In fact, Gartner predicts that only 15% of data projects utilising AI in 2021-2022 will be successful.

Companies are spending more than ever on data and analytics projects, often using cutting-edge AI and machine learning tech – but many of them don’t generate the ROI that the business expected. In fact, a recent ESI ThoughtLab study of 1,200 organisations found that companies are generating an average ROI of just 1.3% from AI data projects, while 40% don’t generate a profit at all.

There are lots of reasons why this happens. Sometimes, the expectations of data projects are too high. But more often, companies embark on data projects without a clear strategy and without appropriate skills and resource to replicate the benefits of a pilot project at scale. AI projects require time, expertise and scale to deliver a decent ROI.

This might come as a surprise to some early project teams. Building a proof-of-concept AI data project can be relatively easy – if you have a team of skilled data scientists, a small project could be ready to test in a few months. The challenge comes when organisations try to scale up those prototypes to work in an enterprise setting.

If your data scientists don’t have the appropriate software development skills, then you could end up with a machine learning model that works in principle but isn’t fully integrated into workflows and enterprise operations – meaning it’s not collecting, sharing or analysing the intended data.

Enterprises need to ensure that they have the skills needed to make machine learning models work within their business. This might mean creating an app or integrating machine models with existing sales platforms.  

When a global online home retailer developed a machine learning model to improve the efficiency of logistics, they soon realised that this was only the first step. Data scientists had created a model that was able to predict which warehouse and logistics carrier would be the most efficient for individual projects based on the product size and likelihood of sale in a particular region.

Our development team was able to help  take the project to the next step, by creating ways to integrate this model into existing systems and automate the data collection process. The result is a system that can advise the business which proportion of a product to store in a particular warehouse, and which carrier to use to cut 5% from shipping costs, for example.

To increase your chances of creating positive ROI from data-enabled AI projects, organisations need to ensure they have the right skills in project teams – in addition to data scientists, you will need engineers, process owners and strong DevOps.

Second, ensure that you are measuring ROI over an appropriate timescale. The upfront costs involved in scaling data projects can result in flat ROI in the short-term. Data preparation, technology costs and people development are substantial expenses, and it takes an average of 17 months to show ROI, with firms surveyed by ESI showing a return of 4.3% at this stage.

Third, are you measuring the right things to accurately measure ROI? Capturing the cost savings from automated processes and data availability only tells half the story. By incorporating machine learning into the transformation of enterprise supply chains, logistics and product development, companies can drive increased revenue, market share, reduced time-to-market and higher shareholder value.

To find out more about how you can realise higher ROI from data investment, download our free Playbook here.

<tl;dr>

Haystack is a question-answering framework – a tool to answer natural language questions from a text corpus which uses AI deep learning techniques to do the natural language processing.  If you give it a bunch of wikipedia articles on Game of Thrones and ask “Who is the father of Arya Stark,” it will tell you “Lord Eddard Stark,” and “Ned“, and give you the text which supports it. I made a quick subjective evaluation to see what it can do. It worked pretty much out of the box and has a useful set of tutorials and code samples. Results are pretty good but not perfect, and  you can improve them by changing the complexity of the model, or refining the model using your own data. 

What is Haystack?

Haystack is a question-answering framework – a tool to answer natural language questions from a text corpus. It can handle the typical ways of storing documents – PDF, doc, txt etc., and uses deep learning technologies (specifically transformer networks), to improve on traditional pattern-matching or NER techniques.

Does it work?

We have several clients who need to search large document corpuses so I decided to have a look at Haystack to see what it can do. I started at the Get Started page in the documentation  and tried it out. Quick tip: I needed to increase the memory available to my  Docker environment  before it would run correctly, but apart from that it worked using the instructions provided.

The demo comes with about 2,500 documents about the  Game of Thrones series hosted on an ElasticSearch instance. A sample document is:

”A Man Without Honor” is the seventh episode of the second season of HBO’s medieval fantasy television series ”Game of Thrones.” The episode is written by series co-creators David Benioff and D. B. Weiss, and directed, for the second time in this season, by David Nutter. It premiered on May 13, 2012. The name of the episode comes from Catelyn Stark’s assessment of Ser Jaime Lannister: “You are a man without honor,” after he kills a member of his own family to attempt escape.

I tried the following questions on the corpus.

Question 1: Who is the father of Arya Stark?

This is the suggested test query. The first result is great. The second and third are wrong (although their scores are much lower).

Question 2: Who is littlefinger?

The first two results are great (curiously the UI does not show them in relevance order). The last is wrong.

Q3: Who is little finger?

This is the same question as Q2 but with a space in the key term. None of these results are correct.

How does it work?

It’s straightforward to create a question answering method from documents in a corpus. This code snippet asks a question about documents stored in an ElasticSearch document store.

document_store = ElasticsearchDocumentStore(host=”localhost”, username=””, password=””, index=”document”)

retriever = ElasticsearchRetriever(document_store=document_store)

reader = FARMReader(model_name_or_path=”deepset/roberta-base-squad2″, use_gpu=True)

pipe = ExtractiveQAPipeline(reader, retriever)

## Voilà! Ask a question!

prediction = pipe.run(query=”Who is the father of Arya Stark?”, top_k_retriever=10, top_k_reader=5)

print_answers(prediction, details=”minimal”)

The most important stages in the pipeline are:

• Retriever – does an initial filter of the documents to find  ones which might have the answer. In most cases this uses a simple TF-IDF or similar approach. 
• Reader – looks at the documents returned by the retriever and extracts the best answers.  The readers use deep learning transformer networks (see here for a quick overview of transformers.) In the example code above it is using a RoBERTa model. You can use models from Huggingface or similar. Different models allow you to trade off between accuracy, speed and available processing power. 

Can you improve the results?

The results fundamentally depend on the model utilised in the reader stage and can be improved by changing the model. I tried changing it to the  bert-large-uncased-whole-word-masking-squad2 model which is bigger (1.34GB compared to 1.4MB), but ran fine on my MacBook Pro. 

The results were quite a bit better.

Question 1: Who is the father of Arya Stark?

It gets the right answer most of the time and finds a variety of names for him.

Question 2: Who is littlefinger?

This gives spot-on answers – 100% correct.

Question 3: Who is little finger?

The mistyping with the space between the words still leads to some incorrect results. The first result is correct but the rest are wrong. All in all it’s definitely an improvement.

You can also fine-tune a model. You can collect examples of questions and answers and use them to update the model. Haystack provide an annotation tool to help with this (The manual is here.) Once you have your data it seems straightforward to refine the model:

reader = FARMReader(model_name_or_path=”distilbert-base-uncased-distilled-squad”, use_gpu=True)

train_data = “data/squad20”

reader.train(data_dir=train_data, train_filename=”dev-v2.0.json”, use_gpu=True, n_epochs=1, save_dir=”my_model”)

I have not had time to test how well this works.

What else does Haystack do?

Apart from improved search you can also use Haystack to:

• Return a novel answer composed to a question (Using Generators)
• Summarize all the answers into a single response
• Translate between languages
• Question answers on data stored in knowledge graphs (instead of as documents). Sadly it does not yet generate a knowledge-graph from text 🙁

The repo is here. I found this article to be a useful introduction to Haystack.

If you’re a senior IT leader,  I’d like to make a prediction. You have faced a key data governance challenge at some time. Probably quite recently. In fact, there is a good chance that you’re facing one right now. I know this to be true, because clients approach us frequently with this exact issue. 

However, it’s not a single issue. In fact, over time we have come to realise that data is a slippery term that means different things for different people. Which is why we felt that deeper investigation into the subject was needed, to gain clarity and understanding around this overloaded term and to establish how we can talk to clients who see data governance as a challenge. 

So, what is data governance? And what motivates an organisation to be interested in it?

Through a series of surveys, discussions and our own experiences, we have come to the conclusion that client interest in data governance is motivated by the following wide range of reasons.

1. Data Security/Privacy

I want to be confident that I know the right measures are in place to secure my data assets and that we have the right protections in place.

2. Compliance – To meet industry requirements

I have specific regulations to meet (e.g. health, insurance, finance) such as:

• Storage – I need to store specific data items for specified periods of time (or I can only store for specific periods of time).
• Audit – I need to provide access to specified data for audit purposes.
• Data lineage/traceability – I have to be able to show where my data came from or why a decision was reached.
• Non-repudiation – I have to be able to demonstrate that the data has not been tampered with.

3. Data quality

My data is often of poor quality, it is missing data points, the values are often wrong, or out of date and now no-one trusts it. This is often seen in the context of central data teams charged with providing data to business functions such as operations, marketing etc. Sometimes data stewardship is mentioned as a means of addressing this.

4. Master/Reference Data Management

When I look at data about the same entities in different systems I get different answers.

5. Preparing my data for AI and automation

I am using machine learning and/or AI and I need to know why decisions are being made (as regulations around the use of AI and ML mature this is becoming more pressing – see for example https://ico.org.uk/for-organisations/guide-to-data-protection/key-data-protection-themes/explaining-decisions-made-with-ai/).

6. Data Access/Discovery

I want to make it easier for people to find data or re-use data – it’s difficult for our people to find and/or access data which would improve our business. I want to overcome my data silos. I want data consumers to be able to query data catalogues to find what they need.

7. Data Management

I want to know what data we have e.g. by compiling data dictionaries. I want more consistency about how we name data items. I want to employ schema management and versioning.

8. Data Strategy

I want to know what strategy I should take so my organisation can make better decisions using data. And how do I quantify the benefits?

9. Creating a data-driven organisation

I want to create an operating model so that my business can manage and gain value from its data.

I think it’s clear from this that there are many concerns covered by the term data governance. You probably recognise one, or maybe even several, as your own. So what do you need to do to overcome these? Well, now we understand the variety of concerns, we can start to address the approach to a solution. 

Understanding Lean Data Governance

Whilst it can be tempting for clients to look for an off-the-shelf solution to meet their needs, in reality, they are too varied to be met by a single product. Especially as many of the concerns are integral to the data architecture. Take data lineage and quality as examples that need to be considered as you implement your data pipelines – you can’t easily bolt them on as an afterthought.

Here at Equal Experts, we advocate taking a lean approach to data governance – identify what you are trying to achieve and implement the measures needed to meet them. 

The truth is, a large proportion of the concerns raised above can be met by following good practices when constructing and operating data architectures – the sorts of practices that are outlined in our Data Pipeline and Secure Delivery playbooks.  

We have found that good data governance emerges by applying these practices as part of delivery. For example:

• Most Data security concerns can be met by proven approaches – taking care during environment provisioning, implementing role-based access control, implementing access monitoring and alerts and following the principles that security is continuous and collaborative.
• Many Data Quality issues can be addressed by implementing the right measures in your data pipelines – incorporating observability through the pipelines – enabling you to detect when changes happen in data flows; and/or pragmatically applying master and reference data so that there is consistency in data outputs. 
• Challenges with data access and overcoming data silos are improved by constructing data pipelines with an architecture that supports wider access. For example our reference architecture includes data warehouses for storing curated data as well as landing zones which can be opened up to enable self-service for power data users. Many data warehouses include data cataloguing or data discovery tools to improve sharing.
• Compliance challenges are often primarily about data access and security (which we have just addressed above) or data retention which depends on your pipelines. 

Of course, it is important that implementing these practices is given sufficient priority during the delivery. And it is critical that product owners and delivery leads ensure that they remain in focus. The tasks that lead to good Data Governance can get lost when faced with excessive demands for additional user features. In our experience this is a mistake, as deprioritising governance activities will lead to drops in data quality, resulting in a loss of trust in the data and in the end will significantly affect the user experience.

Is Data Governance the same as Information Governance?

Sometimes we also hear the term Information Governance. Information Governance usually refers to the legal framework around data. It defines what data needs to be protected and any processes (e.g. data audits), compliance activities or organisational structures that need to be in place. GDPR is an Information Government requirement – it specifies what everyone’s legal obligations are in respect of the data they hold, but it does not specify how to meet those obligations. Equal Experts does not create information governance policies, although we work with client information governance teams to design and implement the means to meet them.

The field of data governance is inherently complex, but I hope through this article you’ve been able to glean insights and understand some of the core tenets driving our approach. 

These insights and much more are in our Data Pipeline and Secure Delivery playbooks. And, of course, we are keen to hear what you think Data Governance means. So please feel free to get in touch with your questions, comments or additions on the form below.

 

With the global pandemic of 2020 and the depression that followed, came the realisation that our economic system was hugely vulnerable in the face of disruptive events. Companies inevitably rushed to automation more than ever before, and the emerging AI & Robotics business sector played a pivotal role in this transition.

Why the past tense?

Because, regardless of whether it’s right or wrong, this is what will happen. This blog post does not look at whether Robotics is the future, but who is best placed to succeed. As a new economy emerges from the other side of this pandemic, businesses will be forced to question the previously unquestioned: are global supply chains optimal? Is ‘Just in Time’ manufacturing robust enough to survive future shocks? Do we need offices any more? 

As we adapt to lockdown, industry is struggling to cope with the sudden absence of people from essential processes, and automation is forefront in their minds. Previously, many saw automation as a way to remove human fallibility from well defined, repetitive processes in order to improve quality and productivity. Now it will focus much more on removing people completely from the process in order to remove a point of failure.

Automation of production lines has long been a thing, and although people play an important part in these lines, they are treated more as a cost/benefit equation than as the actual human beings they are. If (and it is a big if) we see a new economy emerging over the coming months – one that incorporates elements such as a more than minimum living wage or a universal basic income – then that cost/benefit equation will swing even more towards automation, adding to the sense of vulnerability businesses are now feeling. 

Given the extent of the damage caused to businesses by the lockdown, companies will now look for solutions throughout the supply chain, not just in the factories. They will accelerate the development and introduction of self-driving vehicles; they will roll-out Amazon Go – style self-service retail stores; they will copy Ocado and Amazon and replace people with robots in their warehouses.

This is a dangerous strategy, as it represents a swing of the pendulum to another extreme, and as discussed in Part 3, specialisation leads to fragility in the face of disruption. Imagine, if you will, what happens to this automated world in the face of a virus of the electronic variety.

Back to the Map

Rightly, or wrongly, it will happen, and so the more important question relates to which industries are best placed to capitalise on this trend. And so we return to the Map, and another “sea” waiting to become a landmass (business sector) in its own right. We’ve labelled it AI & Robotics. 

As you can see, the Map shows that the neighbouring territories are Computer Consultants, Software Development, Electronics, Electrical & Mechanical Manufacturing, and Motor Vehicle Manufacturing. Companies in each of these sectors are well placed to enter the field of robotics, and some are already doing so. In each case, the entry point is different and so it’s worth looking at a couple of them in more detail by way of explanation.

Step forward Dr. Susan Calvin

In Isaac Asimov’s body of work, there are a significant number of stories that centre around intelligent machines. Many of these stories feature Dr. Susan Calvin, who Asimov refers to as a robopsychologist working at US Robots and Mechanical Men, Inc. He postulates that this profession would be a combination of advanced mathematics and traditional psychology, but in reality the need is more likely to revolve around the training, utilisation and integration of Robots into the business world.

This is a reasonable role for companies operating in the Computer Consultants business sector to take on and thus start the migration into the AI & Robotics business sector. Following the disruption caused by the COVID-19 outbreak, and the resulting demand for greater automation, there is a clear opportunity for these businesses to promote their skills in this area and start the journey.

The first areas of greatest demand are likely to be in the production line, and in the warehouse element of the supply chain, where “dumb” robots already play a major role. There will now be a push to further automate the more complex activities currently undertaken by people, and this will lead to a demand for consultants with experience in introducing technology to organisations. Computer Consultants are ideally placed to benefit from this demand, especially if they include software development capability in their offering or partner with companies that do.

Management consultants are less well equipped to help as the level of technical expertise required to understand the art of the possible, and design solutions is far outside their skill set. They will, of course, have a go, but the Map confirms that they are not well placed to enter this sector.

Here in my car

The other area of the supply chain that will, no doubt see renewed demand for automation is that of transportation. This will accelerate development of self-driving technologies coupled with increased pressure from business to make changes to the road transport system to make introduction of such technologies less challenging. We can expect to see proposals for “freight only” lanes, and dedicated telemetry systems to lower some of the barriers to entry.

The companies best placed to occupy this part of the robotics landscape are the Motor Vehicle Manufacturers. Much work has already been done into self-driving vehicles, but most of the focus has been on cars. It is likely that attention will now move onto the larger freight vehicles. Despite their size, these vehicles actually present an easier route into this sector as they generally follow more predictable routes, and travel between a smaller set of end points. 

Transportation companies such as Uber have also tried to make inroads into this area, but the Map predicts a less successful outcome for them, as they are a significant distance from the new area of AI & Robotics. Remember, on the Map proximity indicates similarity of skills and mindsets – companies located in other areas take much longer to develop the required attributes than organisations on the immediate borders. Uber are making the classic mistake of assuming that being a consumer or seller of a product somehow positions you to become a producer in your own right.

Stuck in the middle

So, that covers the types of business that will benefit from the inevitable demand for automation, but what about the demand itself? At the start of this post, (and in previous posts) we’ve discussed the dangers of specialisation and the increased resilience that comes with diversification. It is for this reason that a headlong rush to “automate all the things” could create as many problems as it might solve. It would also lead to an unmanageable portfolio of change that could cripple an organisation during what will inevitably be an extended recession.

One of the more difficult decisions for most companies is where technologies such as AI and machine learning can and should be effectively deployed. There is much talk of AI as the answer to everything, but there are places where it is most appropriate and places where it is less useful. There is also the confusing matter of machine learning algorithms versus “true” AI in the form of neural networks. The same question arises – which to use and where.

The problem is complex, but as a starting point here is a simple 2×2 grid (because we all love a 2×2 grid): 

The horizontal axis represents the sophistication of the problem being solved ranging from highly complex (multiple variables and multiple outcomes), and the vertical axis represents the nature of the decision to be made ranging from fully objective (where there is little or no doubt) to highly subjective (where the outcome is open to interpretation and opinion).

For highly complex problems involving a significant amount of subjective judgement, people are by far the best suited to this type of activity. At the other extreme, simple problems with highly objective outcomes can easily be automated using traditional and well understood hard coded solutions. 

As we remove subjectivity from a problem best suited to people, or add complexity to a problem currently solved using traditional code, machine learning algorithms come into their own. These are complex, knowledge based solutions that take broad sets of inputs to make a decision in a predictable and traceable way. The automation of the NHS 111 service is a good example of a problem well suited to machine learning.

Heading in the other direction, if we can take some complexity out of the decisions currently made by people, or there are simple problems that were previously not automatable using traditional coding techniques due to the desired level of subjectivity, we now have AI as a solution. Familiar examples involve identifying the subject matter of documents, interpreting medical scans or identifying people or behaviours in CCTV footage.

The same grid can be applied to physical robotics. In the bottom left square we have the type of machines we’re all familiar with on car assembly lines. In the bottom right, (Complex/Objective) space we have the potential for automating surgical procedures. In the top left AI opens the door for semi-autonomous machines such as exploration vehicles. Self-driving cars sit on the boundary between the top left and right squares, and this is why the problem has proved so difficult to crack. Deliberate simplification of the problem by altering the highway environment (or reducing the scope as described for freight vehicles) could accelerate the introduction of such vehicles faster than advancements in the current level of AI might achieve.

And let’s not forget that automation does not have to mean less people; far from it. History has shown that as machines take over in one area of human endeavour, this opens up areas previously ignored. If social distancing has taught us anything, it has told us that personal contact is essential to our wellbeing and to the success of our businesses. Instead of replacing people with robots, think instead of using technology to do the mechanical things, and free up people to be more human.

And so that brings to a close our quick visit to the new landmass that is the AI and Robotics business sector. In part 5, we’ll look more broadly at the Map and how things might unfold as we move out of lockdown and into a time of financial uncertainty. We’ll look at the challenges, but more importantly we’ll seek out potential green shoots and identify where they could emerge.

Part 1 – Dealing with disruption
Part 2 – A fascinating journey, explained 
Part 3 – The Rise of the Avatar
Part 4 – Domo Arigato Mr Roboto – (you’re here)