Turbo Charge Agile Processes with Deep Learning
There is an inordinate amount of contribution from the software development field to the way we do work and the way we run businesses.
In the early 2000’s the “Agile Manifesto” was created in response to our collective inability to effectively manage complex software development. The tenets of the manifesto are as follows:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a planThat is, while there is value in the items on
the right, we value the items on the left more.
Software development is fundamentally a knowledge creation activity. The factory floor processes that are so effective in the physical world simply don’t apply in the virtual world. The agile manifesto was created to challenge our conventionally wisdom of processes that was prevalent at that time.
Agile development has evolved since then and what we have seen is practice of employing automation where needed. The key advantage of agile is its ability to respond quickly to change. If you can automate sections of that feedback loop without compromising, then you have a tighter loop and therefore a more nimble process.
The concepts of “continuous integration” and “devops” are enabled because of automation. In other words, agile processes are effective because there is a clear understanding of where automation can be best leveraged to enhance the human workflow. The correct prescription for introducing automation in one’s processes is to begin with a process that is centered around humans and then to augment that with automation.
If this is done in the other direction. That is beginning with a factory floor model (i.e. waterfall) and then automating the parts, then you are left with an even more inflexible processes and a lot of unhappy workers. The mechanization of work may have worked in an earlier era that lacked the coordination and communication capabilities of computers. In a world were computation and devices are abundant and pervasive, the only efficient workflow model that one should consider is that one based on adaptive principles.
Ideas coming from the agile development have lead to further usage in the world of startups. The startup world is a world were agility is a critical capability for survival. Startups are buying into the agile philosophy of prioritizing feedback with potential customers. The purpose is to create a learning organization by effectively iterating through many business plans. For startups with a lack of vision, this kind of “The Lean Startup” process is better than having no process at all. How can we learn more quickly what works, and discard what doesn’t? That’s the essence of the approach.
The key to leveraging Deep Learning, or more broadly AI, in the workplace is to understand where it fits within an agile development environment. ( It is of course obvious that Deep Learning development and operations should work within a continuous integration and devops environment. We will discuss this issue in another post.) The application of Deep Learning to one’s own processes requires a keen understanding of our human workflow processes and seeking opportunities to enhance, not a rigid process but one that is adaptive, through automation. The very notion of a complete replacement of a human worker through automation only makes sense when the original business process has always been mechanized.
What are the strengths of DL relative to classical AI technologies? The key strength of DL is that is that it is able to function in an almost biological and adaptive manner. That plays very well in the application of DL as a conduit to interaction with humans. We already see this DL applications in speech recognition and in gesture understanding. DL is essentially the new UI. A UI that is ambient, allowing its users to easily summon its capabilities for the task at hand.
Most business perceive DL or more broadly Machine Learning or AI solely as a prediction tool. That is a tool employed by Data Scientists to gather insight about the business. However, prediction is just part of the solution. Prediction requires decision making and subsequent action. Paco Nathan explains this most eloquently:
Judging by tech talks and case studies from Silicon Valley start-up circa mid–2010s, one might believe that machine learning as pattern recognition drives business. It does not. In particular, ML does little to act upon insights gained. Let’s consider Uber as an example. ML use cases may help detect patterns: traffic, drivers, commuters, etc. Those can help indicate where value could be “harvested” within the system: opportunities for action. Even so, the dispatcher at the center of Uber works to schedule and optimize rides. It operates as a control system at the heart of the business. Those kinds of control systems may leverage ML to detect patterns, etc., but there’s much more involved. Notably, determining which offers to sell, scheduling resources to deliver on those customer promises, handling contingencies etc. Manipulating the supply chain is where a business earns profit. Patterns only play minor parts, while control is center stage.
We can classify DL capabilities into 2 kinds: Assistive and Generative. These kinds are differentiated by how a human interacts with the DL system. In both kinds, the human is rarely out of the loop, rather the human is in constant collaboration with the AI in their work.
We are very familiar with automation with assistive capabilities. We experience it every day when we type message on our cell phone. The phone is able to spell correct and provide suggestion to words that we type. We also find this capability in the cameras we use. We now take for granted the auto-focus capabilities of our cameras. Newer DL driven cameras like that found in the Google’s Pixel are even able to blur out image backgrounds. A capability only available on DSLRs with much larger lenses. Assistive capabilities allows us to work at a quicker pace, enhancing on the fly our content creation towards even greater quality.
We see many of these assistive technologies deployed in the software development space. IDEs are enable with many capabilities that assist and correct developers on the fly. We have style checkers, bug checkers, security vulnerability checkers, code metrics, code coverage etc. applied to source code as we write code. This kind of automated feedback via continuous integration is a best-practice in software development. There are tools that not only correct, but perform tedious tasks that require precision like refactoring.
As we can see assistive capabilities can happen in real time as well as in the backend. DL allows us to perform repetitive and time consuming tasks such as sorting and categorizing our photo collections. We are also continually overloading with information. There are certain professions where the ability to curate and analyze information is extremely valuable. We can enhance these curation capabilities and analysis capabilities by reducing the deluge of information into smaller chunks that are more quickly digestable.
Generative capabilities are a new kind of capability that we is just become more pervasive. By now, we’ve all experienced the capabilities of mobile app Prisma, that is able to re-render photographs into the style of different artists:
This is an example of a generative application. We can use DL to exhaustive explore a design space to serve as a “brainstorm” of ideas. This mode also works in the realm of planning and execution. DL can quickly explore many scenarios and present workers with a Chinese menu of promising options. We’ve written previously about some interesting examples of generative design: “The Alien Style of Deep Learning Generative Design”. We have only seen the tip of the iceberg here, there is so much more to explore in this space. One application of DL in the space of “Game Theoretic” design is enough to scare the bejeezus out of anyone.
