Building A Mature High-Performing ML team

We’ve spent a lot of time thinking about the paths that software companies take toward ML maturity. As a result, we created a framework that you can follow as you adopt ML and then mature as an organization.

The framework covers every aspect of building a team: product, process, technical and organizational readiness. Your team’s success will ultimately be judged on sales. That’s why the framework also recognizes the importance of cross-functional expertise and process improvements in bringing AI-driven products to market.

While the knowledge and skills of your hires can drive individual project success, good processes drive the success of a portfolio of projects and the team overall. When creating processes, work with your team and focus on taking small steps and quick iteration of ideas, not major breakthroughs.

You’ll also need to select the best tech stack to support your objectives. Technology should not be a substitute for good processes and the right team. Instead, as you put processes in place, look for technology to help drive efficiency.

When creating processes, work with your team and focus on taking small steps and quick iteration of ideas, not major breakthroughs.

How Does the Maturity Framework Work?

It’s no easy task to build an ML team. Our ML Framework models the evolving process of building a team and provides guidance on where to focus as you mature.

There are five levels.

At each level, your organization will have to tackle a unique set of challenges, be it setting a strategy, adding new hires, or putting more mature processes into place.

Getting Started

First, develop a strategy and application focus for ML at your organization. At this stage, consider the following questions with your leadership team:

• What are the business processes your solution enables?
• Who are your key customers and audiences?
• How can you address their daily pain points or improve
  their workflow?
• What key insights are they looking for?
• Where do they currently experience friction?
• How can automation improve processes and job roles? Do your users regularly make decisions based on data?

Talk to product management, customer success, support, sales and your clients themselves to gather information needed to answer these questions.

Use the information gathered to brainstorm opportunities for automation in your business, always keeping in mind that the end goal is to make your customer’s life better. Think about where you could drive the most valuable insights with your unique data for your customers. You can prioritize these opportunities using a value/cost framework as in the opportunity matrix below. This allows you to build a priority list before you hire anyone and to know why you’re hiring.

Next, conduct a data audit to understand how your data will support the identified opportunities. Think about how you might collect any missing data or retool your processes to collect additional data for future use. Keep in mind that some key ML initiatives cannot start until the data required is in place.

With a vision and strategy in place, you’re ready to make your first hires to bring it all to life.

Exploring

Make your first hires count! At this early stage, think of your ML team as a startup within a startup. It’s often better to look for generalist candidates who have broad skill sets and are used to working across the machine learning development process. Your ideal candidates will have worked in a similar startup environment—folks who can get creative with limited resources. This is because in enterprises, roles are more specialized and teams have greater resources to deploy to solve problems. Your early hires will need to do everything until you’re ready to expand the team.

First, hire or assign a product manager who can help to validate market problems, create product requirements and manage the roadmap. If you hire a PM, firsthand ML experience is a must. You may also be able to find a senior scientist who can take on the PM role temporarily.

Making an internal assignment has its own pros and cons. They can help the rest of the team understand the domain and context quickly, but may need an external mentor or training resources to guide them through their first ML project.

The next roles to hire are an ML scientist to run experiments and an ML engineer who can support the experimentation, establish a data pipeline and integrate predictions and models into the core application. Hire the most senior scientist you can attract to set your team up for long-term success.

A senior hire can:

• help build and attract a first-class data science team
• accelerate the R&D by selecting the most feasible avenues to explore
• drive the creation of IP and develop ML models that are close to production-ready
• set up tooling and processing efficiently and with scale in mind

Ideally, the scientist you hire should also have data engineering skills and be comfortable taking on some of this work until the team grows. Starting the team with at least two technical members drives more productive brainstorming and balances the research/implementation trade-off.

Leverage domain expertise from elsewhere in the organization to guide the team. The scientist and engineer can then learn in parallel about your industry and the problem they’re solving.

Choosing good communicators will yield results. They’ll be able to ask the right questions to understand your business and report back on their findings with clarity to the leadership team. Once your hires are settled, provide a briefing on what to expect to a cross-functional team including product, engineering, marketing and customer success.

Getting the first project over the line

In our experience, it’s better not to worry about process and tooling for the first project. Let the team figure it out and do what they need to get the first product through the door. Make sure the ML team has the flexibility to select the computation resources they need. Compared to software engineering, the computation requirements (cores, RAM, GPU, etc) changes much faster, which makes it challenging to pre-order the machines needed. Cloud platforms usually are more time- and cost-efficient.

With your first project, you want to build momentum  and quickly show the ROI of investing in ML.

Look for projects that:

• Focus on one use case
• Can be completed with a high chance of success
• Can be completed within a relatively short timeframe

This approach also provides some extra breathing room for your team to understand the business before tackling more complex projects.

Though it’s tempting to choose a project that could provide higher returns, this is risky. Projects that are too ambitious can take longer to bring to market or even fail. There are many causes for delays: your data might be messier than thought, you might uncover additional requirements, or you might not have a process for deployment. Delays cause a loss of confidence and support over time and could be terminal for an embryonic ML team.

Building

After a full cycle of ML product development, your next priorities are to build go-to-market muscle and repeatable processes for other opportunities.

Revisit your priorities and ensure that the roadmap still makes sense. Validate that you are solving market problems as you would with any other product. Early adopters or an advisory board can help. Then, build your go-to-market plan, maintaining focus on how you are solving your buyer’s problems. Create buyer personas for everyone involved in a buying decision. Think about what they’ll need to see to move forward. Build a sales enablement and marketing program to deliver the right messages along the buyer journey.

As you start to iterate and bring more models into production, it’s time to scale and hire more specialized roles on the team. For example, you can bring on:

• A dedicated product manager, if it was a part-time role previously, to establish a full product strategy and long-term roadmap
• A data engineer to build a catalog of all available data as well as architect and build data pipelines
• More data scientists to scale up the model building throughput
• An ML architect to oversee the overall technology vision and implementation
• A QA engineer

As the team grows, concentrate on processes to reduce friction, increase predictability and enable cross functional collaboration. You can use Agile as a starting point, but it needs to be adapted to ML. Keep in mind here that the data science work is not as predictable as software development and requires faster iteration.

If the whole team is working on one project at a time, Kanban could speed up iteration without a lengthy planning cycle. You can use team deep dives to encourage knowledge sharing, overcome technical challenges.

Now’s the time to settle on a machine learning tech stack. The ecosystem is growing fast and new offerings sprout up all the time. The major choices are: language (Python or R), frameworks (Tensorflow or PyTorch), cloud provider, ETL and deployment solutions.

Scaling

As the team matures and data gaps are addressed, new opportunities will open up. Other product lines or business units might want to leverage ML. To harness this momentum, you’ll want to build new ML teams.

First, though, think about the org structure. It’s common to spin up small pods consisting of a couple of scientists, one or two engineers and a product person. Each of these pods takes responsibility for a product line or an internal business function. Where possible, look for people with the specialist skills to match these areas of your product. Ask the team where the bottlenecks are and adjust to create a nice balance between scientists and engineers.

There are three common models for larger data science organizations.

• Decentralized model: In the decentralized model, individual data science teams report to their corresponding product or business unit. This allows close collaboration between the data science team and their business stakeholders but risks isolating the data science team.
• Centralized model: In the centralized model, all data science teams report to CTO or Head of Data Science. This approach fosters collaboration among peers, helps with hiring and knowledge sharing. If you adopt this model, pay extra attention to your alignment with business goals.
• Hybrid model: The third approach, which has been successfully adopted in many cases, is a hybrid, where the data science team is embedded with their business partners but still reports centrally.

See this article for a deep discussion on this topic.

Every growing team will face inefficiencies. As you grow, set aside some time outside the hustle and bustle of your day-to-day to work on longer term projects. For example, democratize data access through a well-annotated data catalogue, data warehouse and ETL solutions. Seek ways to systematically automate manual steps in the workflow and increase the usability of others, such as pipeline orchestration and data/model quality monitoring.

As you put more and more models into production, you’ll need to monitor each version of your model and compare the performance.

For some considerations on how you should track the performance of your machine learning products, see principle 5 of our Principles of Applied AI, Understand Performance Objectives and principle 9 of our Principles of Trust, Make Trust Measurable.

Advanced

At the Advanced stage, you’re a mature ML organization with an established ML team, closely aligned with Product and Engineering teams. Your processes and tooling allow you to manage many ML models in production. Your team and company are recognized as thought leaders in your space, and you are able to use this to clearly differentiate from your competitors.

Add a lead engineer and a lead product manager at this point to drive best practices across the sub-teams that are led by the scientists. Consider appointing a head of data science to drive best practices and consistency between sub-teams.

As an advanced team, consider the emerging roles in the section below, depending on your industry and use case.

• As the team and product portfolio expands, add more tools to maintain productivity:
• Use a centralized and searchable notebook repository to enable knowledge discoverability.
• Add a discoverable feature/model store to enable reusability.
• Use metadata-driven orchestration to track data/model dependencies and manage changes.
• Build unified ETL and training infrastructure management to reduce computation costs.

Companies at the bleeding edge may be developing their own ML frameworks to bridge gaps in open source and commercial tools and are likely deploying the latest ML research into production to build a strong, differentiated product.

Why Use the Maturity Framework?

Our maturity framework allows companies that are on the path towards AI/ML maturity to perform an honest assessment of where they are today and understand what they need to do to reach the next stage.

We’d love to hear from organizations about their thoughts on the framework and where they think they currently sit. What’s the main block on your path to maturity? How did you get started? What resources were most useful to you? How did you map out the most valuable opportunities available to you? How did you set about hiring a team? How did you balance building an AI/ML organization versus other roadmap priorities? Who is the champion in your organization? What benefits have you seen since starting on your journey?