Topple old mindsets and thrive in AI projects: P.I

 

After working in AI space in small and big organisations, traversing the cyberspace on the same subjects -- I found that it is a subtle difference of organisational mindsets what defines the project's success.

Very roughly, software development for the most part follows the linear intuition s = Vt: (1) there are requirements what system needs to do [distance s]; (2) which are translatable into some formal programming language/3rd party systems with the work of the team [velocity V]; (3) what takes amount of time t.
Over time software projects gained practical management learnings -- like how to pick "distance chunks" to cover with team's work (80/20 rule, pick smallest chunk providing most of the value); or like choices of architectures / ways of working / frameworks to use to increase team's velocity and reduce reworks and refactors. Notably, those software wisdoms are true and important -- but they still hold linearity of problem solving as a basic assumption.

Data Science on the other hand starts with something Freudian -- the difficulty to admit that we as humans with human brains are having a difficulty to be specific enough to define the problem in formal terms in the first place. The Freudian part is that somehow we associate inability to define the problem with low self worth. As the result, organisations often use a spoken language sounding right, the facade trrying to compensate for an empty house. Once this psychological tendenency is taken into account, we find that there is nothing shameful in not knowing something at all.
For example "clients want a self-driving car" might sound as a specific requirement on a company meeting, but the roads of the actual world are not formally specific. While all roads adhere to some rules/requirements, there are deviations which make each meter of the road unique on the globe.
Another way to look into it is that: things which we could specifically define -- we already automated with software. And we have our first observation -- most of the ML/AI projects deal with an infinite or 'practically' infinite problem scope. Unlike the software projects, dealing with 'practically' finate scope.

The ML models themselves on the other hand, can be seen as auto-generated programs satisfying certain statistical criteria. The way I look into it, is that Data Science automates jobs of both: of the one who was creating requirements for the software projects and of software developer.
Indeed, before we had a human in the loop who would contact clients, research market and part with data part with guts define requirements for the software. Now, instead of that person we have accuracy and other metrics -- and based on those metrics we "generate programs" to continiously be up to date...
...You've noticed it, right? Each model retraining is pretty much a phoenix, burned to ashes to appear again. Imagine a software being re-written from scratch for every single deployment -- and you get a glimpse of what data science is about. This is our second observation on the big difference of what iterations mean for software and for the data science.

In the article below, we are going to look into those subtlties and what does it mean for personas working in/next-to an AI project.

Investment

Software. Investment into software projects today became pretty predictable. Investors need to check that people starting the project are up to the task, that the strategy reasonably matches data from the reality. See to it that the technical strategy is feasible and in place, with risk-reward being managed by the technology choices (widely accepted Java/Spring or similar technologies to get a relatively low-cost development but slowly; more niche technologies to solve some aspects faster with the risk of more dependency on right skill availability). Check that metrics are in place and adjust velocity/quality/tech-debt/infrastructure-cost accordingly to your strategy. As the problem is finate, the date of completion is pretty much the matter of time and the roll of a dice. Easy peasy.

Data Science. It should be just like the above, but somehow it doesn't work.
The first problem was already mentioned -- the problem space (requirements) is not known and to make it worse, it is practically changes every day. To some degree it is true to software as well, but the scale of the influence of this factor becomes overwhelming in data science. The main consequence is that nobody can tell what would it take and how long it would take to solve a particular problem. Some problems might be still technically unsolvable at the moment of the project start (insufficient amount of globally existing quality digital data for the subject).
The second problem is that of an uncanny valley -- as AI resembles human behavior, we have emotional response to their outputs. As AI develops and becomes more and more human-like, we find it cute, promising etc. until the point of annoyance. The annoyance follows from our dissonance, that for a short period we trusted it just to discover a moment later that it is still a stupid machine. Early developments of systems like Alexa and Siri can be used as a reference. It means that if clients of our project are unaware of that valley (a very good default assumption) -- their early excitement about an AI product for the field is very likely to become a dissapointment on the later stage of development of the project. Even more frustratingly -- for most of the project ideas, people are happy to use them if its performance has crossed the uncanny valley but not before (the expectation of human-like performance and mistakes of an AI model). Which means people a ready to pay only in the very end.
The combination of those two problems implies that AI projects require much higher rates of investment to have a chance to succeed. And a more difficult part -- it requires leaders who truly understand the nature of the beast.
Last but not least, most models require continious updates to state relevant. Unlike the software, where we can built an MVP and roll it to different markets while pausing feature development -- models for existing clients need to be maintained all the time. It does not mean the full time, rather that the more models we host, the more people we need to maintain it. The same is true for the software, but the coefficient for ML models is much greater.

What can be done. There are some things which can be put in place to manage risks however. Lets look into them one by one:

• If you are an investor, make sure that it is not a sleak corporate tech mumble what you hear. Put some of your time to understand the essence of the technological space and its implications for the feasibility of a proposed solution for a given field. Ensure that professional data science processes are in place (more on them below). And don't mistake a POC for an MVP.
• If you are a manager speaking to investors, make sure that they understand the specificity of the AI space and that they do not apply software mindsets to the field. Good gesture is to remember that the project needs to stop once we know that finding solution is unfeasible. Communicate problems of much higher cognitive load in AI teams, the uncanny valley problem. Simply that for the project to take off, high and prolonged investment rates are required. The development team needs to be aware of that unusual for the IT strain of trust and to be as transparent (metrics, tech choices logging and strategy wise) as possible. On the technical side, complete mlops life-cycle setup is the starting foundation to steadily built from.
  
• If you are one of the development roles in the AI team, communicate to you manager and team-mates the problems of old-ways bias and their negative impact on the project. Things we are speaking about are common sense, there is no need to convince anybody.

Project approach

In the beginning of an earlier article I was talking about a dialectic approach to understand what would it take to solve the problem in the ML/AI field. This approach is typically a 'slower' counterpart of the 'greedy' proof-of-concept strategy. Lets have a look in more detail into the contrast between the two and why the latter strategy more often than not crashes for AI projects.

Greedy Proof-of-Concept approach 

The POC strategy is typical for new software features/products. We build something what solves the happy path, show it to clients, calibrate, cement it into MVP. For software projects it means that we chose technology to build and iterate POC fast (Node.js raised because of that factor), as we don't know what features we shall require in the end. We want our throw away discovery work to be as cheap as possible. Once somebody is ready to pay for what we shown in POC, we make it into MVP (extra layers of testing, extra deployment processes, more solid platform management). We iterate/refactor MVP later, as we get money from our clients to support the product. Many of the known giants had their projects started in PHP to be re-written few years after the boom in more solid and quality-controlled languages.

If you followed me carefully, you should be able to spot the caveat, hidden assumption in this strategy. The assumption is that the distance between the POC state of the project and an MVP state of the project is predictable (the linearity of software problems). Now, for the  Machine Learning, most Computer Science students have a project in college recognising the written text which they need to grade on. Those projects do work, the concept of visual recognition is proven -- but somehow there are very few competitive text recognition systems out there.

You might be surprised to hear how many IT startups try to do exactly that, with a small difference that they hire somebody with expirience in Data Science instead of a student. However the problem is not in individual velocity (student vs expert), but in the distance not be known. If we look again into the physics model s = Vt as the reference to a old-ways bias: for software projects we know that the distance s between POC and MVP stages is finite, so we can regulate delivery velocity V with the level of expertise of team-members to match delivery deadlines t. Hower for AI projects, s is not defined. I can make a car going in and out by itself from my garage, but it does not mean that I can say would I need 1, 10 or 100 years to make it into self-driving car crusing safely through the city.

What would it take approach

What would it take approach or a scientific approach is a step-by-step method of creating a map of what we know. The map naturally highlights the boundary of what we don't know, or concepts we cannot distinguish between. And just like in the Science we cannot be sure of what fruits a particular research approach would bring, we cannot give guarantees on the chosen approach success for ML/AI.

But we obviously want to track our problems, build our "theory" (ml solution) over time. So we need a way to have tooling/processes to provide certainity that we don't repeat ourselves. In the actual scientific field the system is build around publications to conferences and journals. Conferences are often a way to broadcast to community what we work on and some primary highlights, journals serve as a deeper and more detailed research tracking mechanism.

As models are our artifacts in the ML/AI project space, we naturally need to track how well what model, with what parameters, trained on which data perfmormed. And by this we mean not just some formal tick-boxing, that model v3 trained on data corpus-3 had accuracy 0.87. The team can iterate the model only in the presence of practical understanding of differences between those versions. And this depends on a problem strongly, but usually it means that the team has the tooling to reflect on the difference (like corpus-3 has 3 new files and one deleted, those files had this and this is concrete granular data aggregated for my analysis). This leads us to the neccessity of a complete MLops lifecycle us a very basic requirement for ML/AI projects to count themselves as eligible for an MVP status. If it is not implemented, it means that we do not really know if we improve or decay. We cannot tell how much we can improve or how long would it take. We cannot tell how accurate we are on what matters for the business and how we can maintain/improve those levels.

Another way to look into the same reasoning is from the client/investor perspective. Let's assume they are on board with us on the non-linearity of AI problems. How can they trust our team that we are progressing and how they can be sure that if there will be an opportunity -- we have the capacity to take it? If I would be them -- I would expect transparency. Knowing how often models are retrained, what is the volume and the quality of the data. How the data management and model monitoring systems adopt to team's growing understanding. How long it takes to deploy a model, how much time data scientists spend on the actual research and improvement and how much they waste on trying to recall the line they accidentally deleted in Jupyter.

So those are actually our good news -- there is an estimatable requiried work for ML/AI project, which is the implementation of MLops life-cycle scaled to the project needs. Good mlops combined with good team composition should allow to check if the problem is feasible to be solved/solve it as soon as it is humanly possible and as fast/cheap as it gets.

Hybrid approach

Practically, we obviously don't want to head deep into the ML/AI investment without knowing what clients actually need. Fortunately we don't need to build AI system to find in what form clients would like to consume its output. And more often than not we can start working on the principal ML/AI project solving a problem which we know needs to be solved, while a separate team can work with mockups to find in what form clients would like to consume the output.

In the next part we shall talk about the day-to-day setup of Data Science work, similarities and differences from software development.