Machine learning and AI are everywhere. More and more people are discovering the advantages that learning algorithms can provide – decreasing costs, improving efficiency, eliminating menial tasks, etc. Knowing at least a little bit about how these algorithms work is quickly becoming a necessity.

While the actual ML model construction, training, and evaluation is a topic of numerous courses, talks, and tutorials, one crucial area of working with ML is much less discussed – bringing the trained models to production (also known as operationalization).

There are many ways of accomplishing this, depending on your technological stack of choice. A lot of workflows are currently moving to cloud and it’s no surprise machine learning is one of them – having on-demand compute makes scaling demanding machine learning models a breeze.

Microsoft Azure is currently in the process of testing a new set of capabilities for bringing models to production. Seeing this in action at the WeAreDevelopers conference in Vienna, I wanted to get some hands-on experience with it before it goes to general availability. This set of articles is about what I learned along the way.

The scenario

A customer I’m working for is currently exploring a way of finding and extracting a piece of information from a photo of an official document. I created a small convolutional neural network for this purpose and reached a very good level of accuracy (error below 3px for the localization of the information).

The model was written using python, Keras and Google Tensorflow and was trained in-house (not in the cloud) on a single GPU. When saved to disk, the model is about 1.2 MB.

I wanted to operationalize this model using Azure to see how would the performance be in the wild.

Enter Azure Machine Learning Workbench

Azure ML Workbench (AMLW), as the name suggests, is a set of tools and libraries for creating, testing, and deploying Azure-based ML models. Since our model is already created and trained, we don’t really need the experimentation features and the actual workbench application – for us, the most important part of the toolset is the Azure ML CLI.

This CLI contains the necessary commands and integrations that will allow us to take our pre-trained model, expose it via a web service, package the whole thing into a Docker image, and either test it locally or run it in an Azure Kubernetes cluster for production-grade deployment at scale.

Quite a few steps are required to go from a local model to a Kubernetes-deployed model cluster. I will go over each step in detail and explain the how and why of it.

Getting the required tools

First things first – in order to follow the steps, we need to install the necessary software. If you make use of Azure Data science virtual machines, all these tools should already be available there so you can skip to the next section.

Since we are going to use Azure, first thing you need is an Azure account. A free one should be sufficient, but if you want to do the Kubernetes deployment, a paid subscription works much better.

In the Azure portal, we create a new Machine Learning Experimentation resource, including a Model Management account. We will need to provide names for various resources – keep these at hand as we will need them later. There is a free tier of the Model Management service that should serve well for testing the process.

Next, download and install the Azure Machine Learning Workbench CLI (you need to have python installed for this work):
pip install -r

After the installation finished, install Docker. Since the final output of the whole process is a docker image, having Docker locally will be help with debugging and testing.

Next, we need to log into the Azure account:

az login

This will display a link for you to follow and enter a registration code displayed in the command line.

Setting up the CLI environment

We now need to bind the CLI to the Azure account and resource groups:

az provider register -n Microsoft.MachineLearningCompute
 az provider register -n Microsoft.ContainerRegistry
 az provider register -n Microsoft.ContainerService

Registering those three providers might take a minute or two, we can monitor the progress using

az provider show -n [provider name]

Once all three providers show as “Registered”, we can proceed to the Azure environment setup:

az ml env setup -l westeurope -n amlwenv -g AMLW

If you have more than one Azure subscription available, the tool will ask which one to use. The example command line above will create a new environment called “amlwenv” in the West Europe Azure region and within a resource group called AMLW (it’s the same resource group we used for the Experimentation resource above). Provisioning a new environment will take a minute or two.

Once the provisioning succeeds, we need to activate the environment:

az ml env set -n amlwenv -g amlw

You should get a message similar to this one:

Setup phase complete!


In this initial part of a series of articles I attempted to introduce the Azure Machine Learning Workbench as a new way of bringing machine learning to production.

We saw how to setup an Azure Machine Learning Experimentation resource and its Model Management account. We also installed the required CLI tools and created a new ML environment we will use for testing the model’s service locally and activated it.

In the next part of the series, we will see how to package, describe, containerize, and test the ML service, complete with code examples.


Read next article: Operationalizing machine learning models 2/3 – From model to service

News from ERNI

In our newsroom, you find all our articles, blogs and series entries in one place.

  • 22.11.2023.

    Recognising trends: An insight into regression analysis

    Data plays a very important role in every area of a company. When it comes to data, a distinction is made primarily between operational data and dispositive data. Operational data play an important role, especially in day-to-day business. However, they are not nearly as relevant as dispositive data. This is because these data are collected over a longer period of time and provide an initial insight into the history or the past.

  • 08.11.2023.

    Why do we need digital transformation for medical devices?

    For hospitals, it is not up for discussion as to whether they want to digitalise. The increasing age of the population in western countries and the progressive shortage of medical professionals mean that without digitalisation, the healthcare system will not be able to provide the quality that patients want in the future.

  • 25.10.2023.

    Mastering the challenges of mobile app testing: Strategies for efficient quality assurance

    Discover the unique challenges faced in testing mobile applications and learn how to overcome them effectively. From selecting suitable devices and operating systems to leveraging cloud-based test platforms, test automation and emulators, this article provides seven essential strategies for optimising your mobile app testing process.

  • 11.10.2023.

    Incorporating classical requirements engineering methods in agile software development for a laboratory automation system

    Traditional agile methodologies can sometimes struggle to accommodate the complexity and regulatory requirements of laboratory automation systems, leading to misalignment with stakeholder needs, scope creep, and potential delays. The lack of comprehensive requirements documentation can result in ambiguous expectations and hinder effective communication among cross-functional teams.

  • 27.09.2023.

    Unveiling the power of data: Part III – Navigating challenges and harnessing insights in data-driven projects

    Transforming an idea into a successful machine learning (ML)-based product involves navigating various challenges. In this final part of our series, we delve into two crucial aspects: ensuring 24/7 operation of the product and prioritising user experience (UX).

  • 13.09.2023.

    Exploring Language Models: An overview of LLMs and their practical implementation

    Generative AI models have recently amazed with unprecedented outputs, such as hyper-realistic images, diverse music, coherent texts, and synthetic videos, sparking excitement. Despite this progress, addressing ethical and societal concerns is crucial for responsible and beneficial utilization, guarding against issues like misinformation and manipulation in this AI-powered creative era.

  • 01.09.2023.

    Peter Zuber becomes the new Managing Director of ERNI Switzerland

    ERNI is setting an agenda for growth and innovation with the appointment of Peter Zuber as Managing Director of the Swiss business unit. With his previous experience and expertise, he will further expand the positioning of ERNI Switzerland, as a leading consulting firm for software development and digital innovation.

  • data230.08.2023.

    Unveiling the power of data: Part II – Navigating challenges and harnessing insights in data-driven projects

    The second article from the series on data-driven projects, explores common challenges that arise during their execution. To illustrate these concepts, we will focus on one of ERNI’s latest project called GeoML. This second article focuses on the second part of the GeoML project: Idea2Proof.