Cloudification as the Agility booster for Value Streams

We live in an era of accelerated digital disruption, where the aggregate effect of rolling waves of new market entrants have already pushed many incumbents out of business and has taken away a large and growing portion of the profit from others.

It is no longer a question of “if” an organization should embrace the needed - dual transformation in process and technology - to stay competitive and relevant in the quickly shifting market landscape, but “how” to implement it properly to maximize the ROI on the fundamental and organization-wide change that is needed at the DNA level of its entire structure.

Agile Scaling and Agile Transformation

The Agile transformation – or more recently, Agile Scaling – at enterprise level had been underway over the past 1.5 decades at a progressive rate across all market sectors with many transformation stories to tell.

The need for Agile Scaling is due the fact that Agile frameworks (such as Scrum or Kanban) are designed for running single teams and cannot automatically scale up when the number of teams grow. They have no inherent inter-team synchronization and collaboration mechanism towards joint efforts towards larger solution deliveries. They also do not have any solution for aligning the aggregate efforts of the Agile teams with the strategic steering of the organizations.

As a growing number of companies successfully tested working with individual Agile teams and enjoyed their new ability to quickly deliver changes in response to market movements, they kept adding more Agile teams and shutting down their traditional development shops, and ended up facing the inevitable rising complexity of the growing Agile practice and now had no choice but to start the Agile Scaling transformation to stay functional and also benefit from enterprise Agility as a result of the new working model.  

Peter Drucker, the legendary management consultant once said, “culture eats strategy for breakfast”, since then it seems that any enterprise transformation plan is also included in that breakfast buffet’s menu.

As a result of that flaw, many organizations have been quite busy “customizing” and “tailoring” their Agile Scaling transformation to “what works” for them, which in many cases meant over-trimming and converting too many key elements of the process to a level that it would maintain some flavor of Agility with a lot of after taste of traditional, wastefulness.

But that is a topic for another article!

Digital Transformation

As Agile Scaling was ramping up, a new technological and business paradigm shift grew into a vital survival and thriving factor for organizations that wanted to stay in the market and even push for supremacy: The Digital transformation, which refers to “Digitization” of products and services and “Digitalization” of service offerings and value delivery models in an organization.

Organization soon learned that in order to survive the waves of disruption they should change how they function internally and evolve they way they do business externally.

They needed to find ways to provide their products and services through digital platform and continuously try to add to the spectrum of what they can offer. That “Digitization” also needed to cover their own internal ways and tools that were in play, to modernize and speed up the flow of work that leads to product and service creation, deployment, operationalization, and maintenance.

They realised that they had to reshape their perspective on how the new digital market should be served to maximize their impact, expand their market share, and grow their customer base at global level. That “Digitalization” included a number of revolutionary ways in conducting business in the digital world.

They learned to look at their clients in a new light of loosely coupled partnerships as dynamic networks of influencers – with a variety of levels of depth - which could cascade their good views and satisfaction of the products and services they receive, down to their tribes and generate more revenue by expanding the sales deep into their personal downstreams.

They moved away from focusing on the finished product – traditionally driven by their elite internal experts, best guessing what market would demand – towards the pragmatic rapid experimentation and confirming the incremental success of their products – and fine tuning them - through short feedback cycles and engaging customers through the entire idea-to-market flow.

They upgraded their value propositioning model through continuously challenging their own status quo against the quickly changing market landscape, putting themselves ahead of the game through changing before they absolutely had to. This would require a deep cultural change in ramping up people’s skills and expertise to match what is needed in the digital world as well as establishing the needed mindset behind relentless pursuit of improvements through continuous learning and exploration.

They started digging into the true value of the “big” data that was now pouring in, from internal and external sources, and could be transformed, processed, and turned into valuable information to generate a wide range of predictive and prescriptive actionable insights (from hyper-customization of offerings to mass market targeting).

They blurred the borderlines between partnerships and rivalries with their “former” competitors in the market, leading to creation of joint platforms where the old ways of “Zero Sum Games” would fade into the new ways of partnership in networking effects and sharing the rewards, especially in collaboration with the former partners in their supply chains which were now stepping out to join the paradigm of direct product offering to customers.

They turned their products and services into platforms where their first level customers could use to establish their own businesses and create and deliver value to their own clients, thus expanding their revenue by amplification through enabling customers as business owners.

Cloudified Transformation

The Digital transformation coincided with the accelerated growth in Cloudification of technology platforms in the enterprises.

This was due to the key driving factors of lower cost, higher availability and service capacity elasticity, which overtime had received industries’ seal of approval and widespread adoption as it strengthened with proven security and resiliency of public cloud services, out-matching any on-premises options for growth.

Public cloud lowered the barrier of entry even further for the new digitally disrupting market entrants through its high speed in resource provisioning and pay-as-you-go costing model, which not only allows them to quickly hit the markets with their products and service at a global reach, but also to benefit from the agility and low cost of market exploration through rapid setup of production pipelines and short feed back cycles.

Cloud adoption received a massive boost when pandemic hit at global level in late 2019, forcing lock-downs and turning the remote working capability into a “do or close business” throughout the 2020 and beyond.

The speed and ease of allocating resources to the teams and sharing it safely across geographic areas not only provided a workable solution to that dilemma, but also indirectly evolved the way work was done.

The forced remote working model have also brought the organizations with significant savings on the annual cost of maintaining their commercial space footprints and office running costs.

It also allows them to expand their workforces’ geo-dispersion beyond the daily commute limits of their areas, into any location where the time-difference would still make it feasible for their teams geo-dispersed teams to function seamlessly.  

Of course, that opportunity extends both ways and we are seeing a rising number of resignations – dubbed as “The Great  Resignation” - from employees who want to stay in the remote working model and seek employment with organizations that can accommodate that, or are taking this option and seek better employment outside their commute area.

As if all of these benefits were not enough, a second boost to cloud adoption came from a rather less expected angle: The disrupted supply chain due to the pandemic impact and a newly brewing technological competition between economic leaders of the west and the east, created a long lasting chip shortage that affected the availability and cost of digital hardware everywhere. From smart fridges and passenger cars to on-premises server stacks, it became even harder to expand owned data centers in-time and / or within budget, which created a stronger pull towards the services provided by public cloud hyperscalers.

Cloud adoption has also gone through its own evolution over the past few years and shifted from mere IT infrastructure on a pay-per-use model, towards leveraging the expanding tools and solutions provided by the main public cloud service providers in optimization of the business process, enriching it with AI’s insight generation and predictive and prescriptive power, and tapping into the aggregated intelligence and capabilities previously used internally by the cloud hyperscalers.

According to IBM, three archetypes exist in the way cloud adoption is driving enterprises transformations: Optimizers, which use cloud to raise their operational efficiencies and provide a better customer experience; Innovators, which are looking for new ways to engage customers and expand their market share, and Disruptors, who try to uncover and fulfill the unmet customer needs and creating new market demands and evolving their new value delivery ecosystems.

Gartner forecasts that public cloud service revenue will surpass $330 billion by 2022 (an 80% growth since 2018). A study done by McKinsey shows that about 75% of the workloads of the organizations participating in the research are expected to run on cloud platforms by end of 2022.

Boosting the Agility of Value Streams

A Value Stream is the chain of people, processes and tools that together form a flow of value creation and delivery to the customers.

It has a starting point (which can be a trigger in the form of an idea or request or any other kind of starter) and the set of steps, people and systems that would work to manifest and implement the valuable product or service as the work proceeds through the stream, and deliver that to the target recipient.

The early versions of diagrams visualizing the flow of material and value creation go as far back as 1918, which were invented partially due to the dire need to visualize how materials are spent in production at a time of heavy strain on supply chains caused by the World War One.

The concept was developed further by industrial pioneers through the first half of the 20^th century and received a second major boost from the  World War Two and the renewed need for higher efficiencies in value production amidst the massive global disruption of supply chains and the vital need to reduce wastefulness that could no longer be afforded by any nation nor any market sector.

Value Stream Mapping and its power of visualization got recognized by the IT industry at the end of the 20^th century and was incorporated into a variety of modern software development approaches, especially Kanban – which itself was adopted from lean manufacturing practices.

Scaled Agile Framework (SAFe®), which is the most prominent and successful framework used in Agile Scaling, recognized its value from the early onset of developing into the software industry and positioned it at its core of the transformation practice. According to SAFe® there are two main types of Value Streams:

• Operational Value Streams: Responsible for delivering products and services to customers (the entire product/service  offering, sales, delivery, and maintenance).
• Development Value Streams: Responsible for converting a business hypothesis into a technology enabled solution that will be presented by the Operation Value Streams and delivered to customers. They are the people, processes and systems that are covering the end-to-end flow from ideation through design, development, deployment and operationalization of products and services.

As hard to believe as it may sound, most of the organizations starting with the Agile Scaling transformation do not have a readily available map of their Value Streams. That generally means, they don’t have a good visualization of all the steps, people, processes, and tools that are currently working in the company to produce and deliver their products to the world outside. This also means they do not have a good measure of the efficiency of their delivery pipelines and most likely are wasting away precious time and money in bottlenecks and queues and hand-offs.

As SAFe® would advocate, these organizations need to start with running a enterprise-wide Value Stream Mapping practice in order to visualize and identify their Operational Value Steams – and they can also do any process improvement and optimization as needed – and then use the maps to identify the Development Value Streams serving them. As you can imagine, Value Streams are living and breathing entities that need attention, maintenance and continuous improvement to serve the customers receiving their outcomes.

Value Stream Mapping is a lean manufacturing technique that has been adopted by the IT world and has received wide acceptance in the industry due to its great value in process optimization and value delivery flow management.

SAFe® goes further and recommends that organizations should team up the identified people serving the Development Value Stream into Agile teams and then encapsulate their joint activity through a collaborative and synchronized increment where they plan and coordinate their development efforts in a bundled set of iterations, in alignment with the strategic steering of the organization.

This way the organization is wrapping the Agile teams around the Value Streams they serve, and maximize their contribution to the value delivery.

Value Streams served by their dedicated Agile teams – which SAFe® would call their group an Agile Release Train (ART) – are share an Architectural Runway, which is the existing code, components, guidelines, and technical architecture needed to implement the products and services.

According to SAFe®, Architectural Runways support the continuous flow of value through the continuous delivery pipelines, providing the needed technical foundation for developing business initiatives and implementing new features and capabilities.

This is where the Cloudification of the technology platform and Cloud-Native architectural design comes in contact with the Value Stream.

Business Agility, which is the aggregate Agility of an organization’s Value Streams, is the ability of the enterprise to adapt rapidly and cost efficiently to changes in the market, which in today’s accelerated shifting business environment can make or break everything for them.

Value Streams’ Agility is their ability to adapt and respond rapidly to “their share” of market changes, and the opportunity to innovate and disrupt their internal processes effectively, which is served by a culture of relentless push for improvement and optimization.

So, let’s have a look at how Cloudification of the Architectural Runway acts as the booster to Value Streams’ agility:

• Competitive Time-to-Market Lead Times– cloud platforms help the Agile teams serving Value Streams to significantly reduce their lead time in provisioning and releasing resources (service elasticity) as needed throughout the pipeline, without incurring delays waiting for resources to free up or being purchased and commissioned during the spikes or routine workloads. Cloud service elasticity, using well architected design patterns, provides high availability and auto-scaling resources that can easily shoulder any needed additional capacity without crashing the pipeline and having to roll-back the failed deployments. This Agility factor enables Value Streams to benefit from the short feedback cycles that are created through rapid product / service releases and collecting data on customers experience.
• Cheaper and Faster Explorations– cloud services provide cheap and instant access to the needed computing and storage resources on a pay-as-you model, enabling Agile teams to run exploratory initiatives and proof-of-concepts in search of better value delivery options or finding better and more optimized ways of delivering their products and services, following the SAFe® Continuous Exploration (CE) cycle.
• Global Reach with Millisecond Latency– the mainstream cloud service providers have globally distributed data centers connected through optic fibre networks, making it possible for the Agile teams to deploy their products and services easily and quickly at global level, close to their customers’ geographical locations and expand their reach from local and national to international.
• Cheaper and Faster Growth– the cost optimized cloud service model allows organizations to expand their teams and add Value Streams to seize market opportunities, much faster than traditional approaches and for a fraction of the cost of needed infrastructure.
• Easier Automation– cloud service providers have a wide range of fully managed, integrated cloud-native services that can provide and support the automation of the pipelines, establishing true DevSecOps’ Continuous Integration (CI) / Continuous Deployment (CD) flow, which now can be integrated with Continuous Exploration (CE) and create a triple loop, where Value Streams can innovate, design, implement and deploy their ideas and market re-targeting attempts as part of one pipeline.
• Easier Service Performance Monitoring– cloud service providers have a set of sophisticated service performance and usage logging and tracking on all engaged resources and can provide near real-time visualization on the flow of work in the Value Streams and the service provision in the product servicing pipelines. These tools can use ML algorithms to learn what constitutes the normal and expected behavior of each service or functional domain and can provide actionable predictive insights and a variety of alarms and notifications on the required activities to maintain the service quality within the acceptable range.
• Stronger Focus on Customer Value Delivery-  cloud service provider’s large array of fully managed services take away the burden of administration and maintenance of the computing and storage units and free up the Agile Teams serving the Value Streams to invest their capacity in creating more value for the customers. This is a great win for cloudified Value Streams as they can optimize their human capacity in serving the flow without having to add to the number of teams.
• Democratization of Smart Technical Power– cloud service providers have a large toolbox of powerful AI/ML based analytical and inferencing services that can easily be integrated into the cloud-native solutions that Value Streams create and enrich the pipeline at many levels (e.g., assisting developers in creating better code faster  using autogenerated components, running predictive models on testing scenarios to optimize testing accuracy and depth, post-deployment customer behavior tracking and insight generation, hyper-customization of offerings, product recommendation to customers,  and a wide array of other predictive and prescriptive capabilities).  
• Easier Governance and Compliance Management– organizations can easily setup landing zones in their cloud environment and create and enforce policies in alignment with their corporate governance and internal and external compliance requirements across all accessible services to their Agile teams. Cloud’s managed services are also natively compliance eligible under all main international mandates.
• Participatory Lean Budgeting – organizations using cloud services can easily monitor and manage the spending by each Value Stream and its Agile teams and delegate some of the financial responsibilities back to them to closely watch and forecast their spending and compare it with their budgetary guardrails. Cloud service providers have wide ranges of cost optimization advisory and budget management tools which enable Agile teams and the organization to watch over the spending and receive recommendation on right-sizing of the allocated services to continuously improve their ROI.

Challenges in Transformation

There are usual existing obstacles in incumbents who were established long before the digital era, and there are new, less identified yet very negatively impactful oversights that can hamper the efforts in transformation and cause a lot of headaches down the road.

To name a few of the usual existing obstacles, we can look at:

• Hard to crack mash-ups of obsolete legacy and quasi-modern to state-of-the-art systems (ranging from monolith client-server software all the way to service-based, containerized solutions) stitched together and semi-manually maintained by an army of tech teams.
• Missing the needed automation, compensated for by a patchwork of hand-offs between a growing number of teams trying to stay afloat under the expanding workload which is also giving raise to the number of production defects, in any effort to response to market changes.
• Lack of smart and relevant metrics that instead of focusing on flow quality of Value Streams and the Business Outcomes are giving value to the volume of activities done by the teams, incentivising wastefulness under the disguise of being more active (instead of becoming more productive).
• Siloed Roles and Responsibilities that have casted the flow of value between the walls of siloes, limiting their functions and domains and under-utilizing the organizations’ true capacity to establish sophisticated, efficient and enriched Value Delivery Ecosystems.
• Overgrown Operations and under-allocated Value Engineering, which has amassed a growing army of teams in the Operational Value Streams, leaving a skeleton structure of engineering teams left to bridge the end-to-end flow from value delivery ideation to deployment. This is mostly due to the dropping service level quality and the growing number of production issues that is syphoning out the good resources from Development Value Streams to join the non-stop – and exacerbating – firefighting in the Operational Value Streams.
• Talent shortage and good hire scarcity, which is mostly due to the low attractiveness of opportunities advertised by the incumbent (since the available market talents would like to refrain enlisting themselves into the firefighting teams, which usually come with un-attractive compensation offers from these employers).

One critical oversight in planning for transformation of organizations that are embracing both Agile Scaling and Cloudification is to miss their much needed alignment and integration throughout the dual plan.

As the Value Streams are mapped being part of the Scaled Agile transformation and the Development Value Stream and their grouped Agile teams are formed into an Agile Release Trains, it is vital to identify their Architectural Runway as a Minimum Viable Cluster for the Cloudified transformation initiative.

This approach ensures that Value Streams are properly Cloudified and can maximize the cloud benefit by establishing an underlying platform and tooling that forms a cohesive, managed, automated and integrated pipeline that can empower the Agile teams to follow the CE/CI/CD model, and establishes competitive time-to-market and short feedback cycles.

Since the Development Value Streams are providing the products and services and also the tools to the Operational Value Streams to enable and support their function, the cloudification benefits are inherently transformed to them as well.

Organizations which underestimate the depth of the interdependencies between Agile Scaling and Cloudification, end up running the two transformation separately, ignoring the Value Streams that are mapped by the first and setting up their Cloudification roadmap based on the pre-transformation development and operations layout.

This misalignment will cause partial cloudification of Value Streams’ Architectural Runways, creating a new array of hand-offs and integration problems between platforms and opens the door to a new rising wave of production defects and delays.

Conclusion

Organizations around the world and in all market sectors have already identified the need for higher agility through Digital Transformation, in responding to the ever changing market conditions and customer’s interest which are living in an 24/7 globally connected network.

They also understand that for any chance of success in their Digital transformation journey, they should scale up their Agile practice and empower their Value Streams through adopting cloud services, which are the best technology platform that can help them raise their agility and competitiveness.

While the push for process and technology modernization should be done in parallel, it is vital that organization understand the interdependencies and required integration between the two transformations to not only avoid introducing chaos to their teams, but to maximize the amplifying agility gains from the intertwined power of the two.