Article

Being Resilient: Agile-Powered Data Intelligence - Reinventing Business

By Dr. N.R. Srinivasa Raghavan, Harry Keir Hughes

10 Aug, 2020
10 min read

General Motors Co., the global automaker, has responded to the COVID-19 pandemic surprisingly well. Instead of producing cars, it loaned its talent to the U.S. government and manufactured ventilators. It could do this because it had intricate, real-time insights into its supplier and distributor chain. Using its data-as-a-service platform, Maxis, within days it found the cheapest, highest-quality ventilator manufacturer to enable its pivot to this market.

Walmart is another firm using data well during this challenging time. This retailer collects 2.5 petabytes of data from customers every hour. It uses optimization and simulation algorithms to price its products in real time, leveraging insights from trend, weather and economic data.²

In the past two months, this data also has been used to uncover peak traffic channels (online and in-store) and shopping patterns across the U.S., which has helped the grocery behemoth deliver additional support to customers most in need of help. It also uses digital systems and analytics to analyze inventory across the supply chain, reducing the chance that popular products will be out of stock.³

It’s clear that data-driven, digital-first companies can be more agile and change distribution channels quickly when disaster strikes. They can re-engineer flagging legacy processes quickly, utilize infrastructure more cleverly and understand how to spin up a new product, or even a business model, on the fly.

The difficulty with data

Using data and the intelligence it provides is challenging for many firms. Data is often unstructured (video, images, text, etc.) and comes from many different sources, with data repositories both on premises and in the cloud or sometimes in outdated databases. Data processing is fragmented and often not powerful enough to run prescriptive artificial intelligence models that are needed to generate the required intelligence. With legacy systems in place, siloed business units don’t know how to get data in and out of preferred AI models.

Moreover, as we highlighted in The Four Pillars of an Enterprise-wide Data Strategy, a data-driven culture within the firm is also hard to come by. Many firms lack top-down decision-making, and the decision-making itself is slow and cumbersome due to waterfall project methodologies. By the time new products and services are released, it’s often too late.

The agile DDI platform

A data-driven DDI (data, digital, intelligence) system can be built to improve decision-making in the organization, solving the above problems in concert by giving the organization advanced “eyes and ears.” In this paradigm, data is ingested from disparate sources, manipulated, and used to create insights and new modes of operating that positively impact the bottom line. Because insights are close at hand, the whole firm is able to derive higher levels of intelligence in real time, developing products and business models on the fly.

A data-driven DDI system improves decision making by giving the organization advanced eyes and ears

Some companies already have these systems in place. Netflix uses a DDI platform to determine what niche content to promote in different parts of the world, while giving decision-makers a bounty of intelligence on which new content to syndicate or invest in.⁴ Lyft, the ride-sharing platform, was able to quickly partner with Amazon to offer the digital giant its drivers as warehouse workers, delivery people or grocery shoppers during the pandemic. In this case, its DDI system provided real-time intelligence on the drivers’ location, availability and preferred job description.⁵

So what are the components of this platform, and how can one be built? The platform proposed here is made up of three layers — an agile “data” layer, an agile “insights” layer and a process layer that works on top. We will look at the function of each layer in turn.

Agile data layer: This layer of the platform pulls in structured and unstructured data from numerous sources, including the firm’s own internal system of record (enterprise resource planning). The platform must be able to ingest data rapidly to keep up with market upheaval and competitor behavior. GM, for instance, has a flagship data mart that corresponds closely with what we’re proposing here, ingesting more than 30 billion records per day from applications, “internet of things” sensors, and partners and other entities that make up GM’s supply chain.⁶

This data mart for analytics is composed of various elements, including the following:⁷

An ingestion framework to quickly onboard data.
Agile data pipelines to both establish and manipulate the data repository.
Orchestration and automation processes to make sure data is “live.”
The ability to scale using a common data model.
The ability to onboard new data sets dynamically.
A framework for highly confidential data.
Agile AI and data science pipelines to help create and deploy new math models on an as-needed basis.
A framework to support continuous integration/continuous deployment between operations and the process layer (see below).

Agile insights layer: This layer of the platform is used to assemble insights quickly, enabling a business to adapt quickly to new market information, competitor strategy and customer sentiment. Derived insights also can birth new modes of operation. This platform layer does so by capturing data flowing in from transactions and interactions, and then connecting the dots to produce insights in real time.

For instance, Google used customer sentiment analysis to work out the perfect shade of blue for advertising links. The insight was as simple as “Because more people click on a deep-purple shade than on a green shade, advertising links on Gmail should be more purple in shade.” This insight actually made the business an extra $200 million a year in ad revenue.⁸

Therefore, this platform layer is where job functions for true data scientists emerge, such as working out what customers think and what business opportunities new partnerships provide. A self-service portal is also important here. Different stakeholders will want to make data-driven decisions on the fly, with insights ready at hand when the C-suite is brought in to approve business strategy. Maxis, GM’s insights portal, again offers us an example. The system makes up-to-date information easily accessible not just to business owners but also to every person with access to the system. The portal features a Google-like search interface for insights, including sales forecasting and problem detection, along with advice for purchasing departments to get the best deals from vendors.⁹

Agile process layer: The future-proof enterprise also should look at internal business processes in a new way. Up close, these processes can be thought of as highly configurable digital components that can be wired together in an extremely agile manner. To achieve this, AI-assisted or AI-automated codebases should be developed that can be set up to run a new product line (or business model) quickly. This state-of-the-art platform layer also requires a playbook for specific knowledge — such as what specific components need to be tied together and what data sources need to be included.

In the future-proof enterprise, processes are highly configurable digital components that are wired together in an agile manner

Additionally, this layer can be used in plug-and-play mode so the business can use it as and when it’s needed. One possible use case could be an Uber-like model for retail stores. The layer could be turned on during a pandemic, allowing delivery volunteers close by a store to pick up and drop off groceries for vulnerable people. This would help the business reduce inventory while doing good for society.

To quickly get a leg up on delivering this vision, firms will need to work with startups that may have bits and pieces of the solution they are looking for. The platform should be built in a piecemeal manner, moving up and down the stack rather than across it. Project teams will need to work with regulatory bodies and the government too so that new policies around privacy and security are built into the framework.

“It’s been said that agile isn’t something you do; it’s something you become,” said Oliver Ratzesberger and Mohanbir Sawhney, authors of “The Sentient Enterprise.” Building on this data-driven platform architecture, the real North Star for any organization is one that is able to sense and respond immediately to all events of material importance. We call this the “live enterprise,” where decisions are made by applications and systems autonomously. Here, AI is used to detect subtle trends in data and to position the workforce so it can share insights and respond in next-to-real time to black swan events. Such a vision is both exciting and challenging. Building an agile DDI system is a step on that journey, one that will serve the enterprise well in the not-too-distant future.