Article

DevOps Success, Lessons Learned and Insights for The Future

By Alok Uniyal, Harry Keir Hughes

15 Aug, 2019
13 min read

A few years ago, one leading global bank’s South American IT operations were not performing at desired levels. To keep up with digital giants such as PayPal and other fintech upstarts on their doorstep, the bank’s program team created and implemented a new operating model and tooling methodology that had DevOps at its core.

The move from siloed development and operations teams to agile, cross-functional units that work better, faster and less expensively has been the subject of the preceding five articles in this series.

This final article in this case study reflects on the success and lessons learned from this DevOps transformation and concludes by looking ahead, providing business and technical leaders insights for their own DevOps programs.

Introduction

A few years ago, one leading global bank’s South American IT operations were not performing at desired levels. If a developer needed an estimate for production capacity, they logged a ticket. If they needed to deploy code to production, a labyrinth of processes would slow them down. Operations, on the other hand, hardly ever saw the people who were creating the code, and the culture was full of finger pointing and passive resentment.

The bank compared its progress with other firms like Netflix and PayPal. Where these multibillion-dollar enterprises were deploying code every hour (or nearly every second)¹ and meeting customer expectations, the bank struggled to deploy code every month. Unresolved issues and customer complaints were the norm.

The bank transformed its software development process, using an external partner (Infosys) as a catalyst to assist in the agile DevOps journey and help them become self-sustaining in the new model. Ultimately, the bank’s program team created and implemented a new operating model and tooling methodology that had DevOps at its core.

This article reflects on the success and lessons learned from this DevOps transformation and concludes by looking ahead, providing business and technical leaders insights for their own DevOps programs.

Success

After 13 months, the firm had adopted a fail-fast, data-led and customer-facing enterprise delivery architecture, putting itself on par with other big tech firms

With the introduction of the agile way of working, business and technical IT people were brought together, with the product owner as single point of contact, creating deeper relationships between business and IT. After some behavioral change, this eventually fueled more knowledge sharing and accountability throughout sprint cycles; everyone became adept at working toward a single goal, taking as their mantra “What we build, we run!”

DevOps accomplished three goals that the bank envisioned as markers of success:

Significant cost optimization through direct savings and absorbing headwinds
Sizable improvement in IT quality and service stability
Greater than 10% improvement in key measures and KPIs

But the journey was not easy. Mistakes were made, and lessons were learned along the way.

Lessons learned

With new agile teams in place and working in a new operating model, the bank developed highly relevant applications quickly, significantly reducing turnaround times.

Some of the lessons gleaned from the bank’s sources and its partner (Infosys) team experts follow.

Focus on the people

Initially, more emphasis was placed on improving processes and technology than on the human element. However, the executive team managing the transformation found that by not immediately focusing a lot of effort on the people, progress around processes and tools hit a ceiling or even started to regress.

“Progress began to plateau and fall back,” says Joseph Kaplan, principal consultant at Infosys, who was part of the behavioral change journey. “A focus on people was introduced to then reignite and further advance the transformation process. As we expand this transformation to other regions of the world, executives are making sure that people, process and technology are all focused together in parallel and in an integrated manner from day one.”

Shift the culture from top to bottom

DevOps is not just about tools and processes. Its success depends on innovative team members working together in a cohesive culture where fear of failure is mitigated.

By initiating team-building activities or “DevOps days,” the bank’s partner (Infosys) provided a venue for team members to spread the word about best practices while learning more about one another’s roles. This reduced anxiety and fostered collaboration among agile team members.

After 18 months, there was still uncertainty around the commitment level for leaders within agile teams. Many of the coordinating tasks that midlevel managers traditionally handled ceased to exist, and servant leadership was a new concept that was met with resistance. Education, enablement and ongoing experience with DevOps helped smooth the transition.

Empower and train team members

After the first phase of the transformation journey, individuals within the teams felt that they were not empowered to carry out their work effectively. Lack of relevant skills was also a problem area for these individuals. These problems were attributed to a pattern within agile teams of unclear roles and responsibilities and insufficient skills and tools training.

“Being able to operate from a customer mindset and building the confidence to speak up and make their voices heard in team decisions were problems from the outset,” says Kaplan. “These individuals had to gain knowledge to have open dialogue and constructively criticize decisions in an effective manner.”

These problems were resolved by updating agile team roles for increased relevance and accuracy and implementing specialized tools training sessions.

Collaboration and communication are the linchpins of success

DevOps brought the development and operations sides of IT together in cross-functional teams that had end-to-end responsibility for the full software life cycle. Agile practices emphasized collaboration and communication, since everyone had a stake in the software that was developed. Eventually, this helped produce superior code.

Make team members accountable

By introducing the DevOps life cycle, developers were made accountable for every feature or application they built, resulting in better-quality code. When bugs occurred, they were more easily traced — because of automation — but also because developers were working with smaller chunks of code.

Hire coders with broad knowledge and strong collaboration skills

During the transformation, developers realized that they were highly accountable for the quality of their code. This was a shock to many who were previously used to focusing just on their area of expertise. By employing coders with operating system basics and strong collaboration skills, team members were able to see the bigger picture and communicate their needs in the larger context of a project.

Design rewards to reinforce agile practices

Output was improved by introducing new rewards and recognition for working differently. As Kaplan says, “In any transformation such as this, an alignment of recognition and rewards to the new ways of working is critical. The rewards and recognition scheme we set up had to be updated to recognize, reinforce and create consequences for working differently and getting different results.”

Standardize tools, processes and practices to increase efficiency

A method to standardize tools, processes and practices was a crucial part of the transformation. Standardized approaches and the use of practices instead of ad hoc solutions, reduced errors and improved knowledge sharing. They were a prerequisite to implement automation. Standardization also reduced the overall number of tools that were eventually implemented in the transformation, increasing efficiency and reducing costs.

Faster code times, fewer security risks

The window of opportunity for hackers to target vulnerabilities in new releases was reduced through ongoing and continuous release of new code, patches and updates. The threat is still there, but faster development times mean organizations can be quicker to identify and respond.

Insights for the future

It is a long-held ideal that IT should be a powerful driver of value, and move beyond a cost center to enabling if not driving revenue.

For the leading global bank that is the focus of this case study, DevOps is a fundamental step toward achieving this goal.

Since deploying this new approach, the bank can now compete successfully for market share against fintech pure players, with increased service quality and product speed to market.

What comes next? Like a ripple in a lake that quickly radiates outward and is felt sometime later on the farthest shore, the agile DevOps program is creating positive ripples of its own.

With the inevitable sharing of knowledge that happens when things can be done quicker, faster and cheaper, we expect big things to happen both inside this organization and further out into unexplored territory. Not only can the enterprise replicate the success of its Latin America DevOps transformation in other parts of its IT landscape, but it can spawn copycat initiatives in other departments and units worldwide.

— Mohit Joshi, Infosys President

Utilizing the full power of DevOps and agile teams is not a one-time transformation, as this study has attempted to emphasize.

Industry leaders must continually refine and update their people, processes and technology if they are to stay ahead.

Benefits accrue to those companies that make bold moves early in their transformation life cycle and then persist to completion.

“Organizations need to make fundamental changes in the way their teams are structured and organized,” says Alok Uniyal, vice president and head IT process consulting at Infosys Technologies Limited. “Integrated teams — comprising software delivery and operations, with strong engagement and collaboration with business — become the new normal. In the future, we may not see separate development and operations teams, especially in the case of greenfield implementations. New engagement models, commercial constructs and SLA frameworks will evolve to foster a highly collaborative, agile and outcome-centric working model.”

Broad, systemic change that significantly alters dialogue between all software-delivery teams, IT operations staffers and business stakeholders is the goal for DevOps transformation. This is a highly complex undertaking² and requires a balance of boldness and care to mitigate risk.

This bank is in a prime position to achieve this objective, with a high level of diversity, experience and ability to create small, focused DevOps teams across the international landscape. One only has to look at how Microsoft has succeeded to feel inspired and confident with the sense of possibility; with product-focused units such as Windows, Office, and Azure DevOps, supported by a unified enterprise-level vision and transformation strategy, they continue to disrupt their own business model and compete with much smaller agile tech startups³.

How should this play out for the enterprise in regions next in line for transformation?

There are several insights, but perhaps the most important one is to ensure that IT leaders concentrate their time and energy on areas of the business where speed is at a premium and exceptional customer experience is critical. Game-changing applications and new capabilities spring to mind; those parts of the IT stack where development, security and operations need to be automated and continuously integrated are also ripe for transformation.

The ripple effect of DevOps will mean that other teams and departments will soon find themselves transforming in a similar manner, taking advantage of improvements in service management and deployment, along with reduced waiting times and an easier approval process. Scaling DevOps across the enterprise is an expected end and an exciting prospect.

As the firm matures on this journey, it will likely find that successive iterations of the techniques discussed in these articles become easier; business goals will be reached faster, less money will be invested and more value will be delivered to customers.

Still, DevOps is tough.

Leaders must ensure that they don’t lose heart early in the process.

Gartner predicts that through 2022, 75% of DevOps initiatives will fail to meet expectations due to issues around organizational learning and change⁴; management necessarily pushes back as they find themselves losing a huge deal of executive power.

One way to counter this is to establish a community of practice that fosters innovation, encourages leaders to develop a servant leadership mode of operation, and puts teams in close touch with those who are ultimately accountable for their daily work.

To close, we re-emphasize the importance of concentrating on the people aspect of DevOps. Empowerment of team members means value-driven roles where upskilling happens on the job. This in time leads to a productive and collaborative atmosphere where products are built faster, better and less expensively. As research by Puppet has shown⁵, when people can get work done with minimal handoffs, approvals and wait time, they’re happier and more productive. These are some of the lesser-lauded aspects of a DevOps transformation, but they make all the difference in the long run.

Once DevOps reaches a threshold level of maturity within an enterprise, with people, processes and technology working together in self-reinforcing concert, other teams start to think and act differently too. This new way of working permeates the organization, transforming the business into a digital visionary that beats the competition. IT then undergoes a metamorphosis from a cost center to supporting revenues, becoming a more valued and valuable part of the business.