Hybrid (Waterfall/Agile), A band-aid fix or an inevitable eco-system?

The beginning

Coming from a long track of project management since late 1990s to this date (the end of the second decade of 21st century), I had the chance to witness the trail of attempts in finding better ways to manage solution delivery across a number of market sectors.

While large enterprises were busy perfecting their Waterfall-based PMO, establishing Gates and Checkpoint and Documentations and Status Reporting, smaller companies – and mostly start ups – were experimenting with new ideas and approaches in faster response to changes in customer interest and hitting the market earlier with their projects.

Many new ideas came, failed to deliver and went to dust over the past 30+ years, till finally Agile methodology and its frameworks made the transition from a successful history in manufacturing, into the world of Software Development.

Among Agile methodologies, Scrum and later Kanban managed to better succeed and quickly penetrate the delivery pipelines of small and medium size businesses.

As would be expected, large enterprises were still not ready to accept some new framework which would seem to be loosely managed, outside PMO’s marching machine, to disturb their perfected version of Waterfall.

I joined the Agile movement in 2006 when I got certified – among the first wave of Canadians to do so – as a Certified Scrum Master. 

I was working as a consulting project manager for large Canadian financial institutions at the time and finding value in what Scrum (and later Kanban) could provide, started advocating and promoting them alongside my practice.

When Jack met Jane for the first time

In 2008, I was managing a very large and complex project for two major divisions of one of Canada’s largest banks (with over 60 people and $10+ million in budget) and just a few months into the initially planned 2-year project, we received an executive mandate to shrink our delivery date back by 6 months. This order came while we had already reached the understanding that our scope was too big to be delivered in 24 months!

Finding this to be a great opportunity to introduce Agile to this bank, I provided a presentation to the Steering Committee on how I believed this can be done by using Scrum’s ability to deliver high speed changes and updates to every main release that Waterfall team would send out.

Scrum teams would be working in parallel to our Waterfall team structure and would take ownership of each main release and become responsible for all release changes and market response updates for that release, while the Waterfall team would continue into the next major delivery of new scope.

This was the first time in Canada any large organization would brave carving out Agile teams out of their Waterfall structure, have them trained, coached and brought to maturity through running them in parallel and conjunction to Waterfall delivery pipelines.

They accepted the proposal – as it was impossible to do it with our Waterfall method – and we worked hard and trained and practiced and revise and realigned our effort forward. 

To everyone’s delight – esp. the bank - we not only survived the odd marriage between the two worlds, but also delivered the entire scope in 17 months (1 months sooner than what we were asked to deliver).

This later triggered establishment of their first Agile Center of Excellence and Scrum started developing into the other development pipelines in the bank.

Enter the Digital Revolution

The era of Digital Disruption that soon followed enabled small businesses to step into the lifelines of large enterprises and take away their profits with an ever-increasing impact.

Running on Agile methodologies and armed with ease of experimenting and scaling their processing power through Cloud computing, small businesses and start-ups were soon identified as real threats to survival of big players in the field.

This was a turning point for adoption of Agile methodologies by the large organizations and each started investing heavily in forming Agile departments and Scrum and Kanban teams and learned – or at least tried - to provide them the needed autonomy to be able to match the Digital Disruptors in the market.

The early experience of severe and ever growing conflicts right after the introduction of Agile delivery pipelines into Waterfall environments made it obvious to large organizations that there is no natural compatibility between Agile and Waterfall that would allow them to co-exist – or fall into place - nicely without a well-designed cross-framework collaboration model.

As it proved to be, successfully establishing Agile teams is far more complex than simply hiring Scrum teams and letting them do their jobs: 

1. The first and most prominent issue in doing so in each and every enterprise has been the lack of clear understanding of the differences between the two schools.
2. The second key factor, right after the gap in understanding, has been lack of corporate cultural adaptability that is needed to embrace this new working model and having the needed flexibility to initiate the needed changes across all lines of business and delivery pipelines to establish the needed collaborative model

This change needs to become a cultural shift at all levels within the entire organization with executive support and their heaving investment in training and forming a corporate-wide understanding on:

• What Agile is and can be expected to do, 
• How they should interact with Agile teams, 
• What they should not do when working with Agile teams,
• How to establish a sustainable, efficient relationship between the pipelines.

A Tale of Two Schools

Waterfall (From Stone Age [!] to Present)

Waterfall approach is the oldest known approach in doing any work. It starts by some idea and moves forward by completing an entire stage before pushing the output of that stage to the next one for further processing, until it reaches the end of the line and pushes out the final outcome (i.e. product).

In modern days, it takes a long, detailed, gated path from the Ideation to Delivery. In this approach, the project starts after a detailed Business Case is approved and a funding (based on a rough-order-of-magnitude estimation) is provided to get the work started till a more detailed funding request can be made. 

The next key step is to make the work official, with a Project Charter to define the high-level understanding of the scope, rough schedule, funding that is needed, risks, dependencies and more. Before the actual work can begin, a few other artifacts are created including the Project Schedule, Project Plan, Task Breakdown Structure and Detailed Scope Documentation. 

Once all of the wordy documents are prepared and approved, then the Design stage needs to take over and be completed to create more artifacts in explaining the Detailed Design and System Requirements.

Next, the Developmnt team can start their work on the approved artifacts and once they are done coding the Testing team can start their work on the code and do the needed back-and-forth with Development team until all code is considered good and can compile into the build.

The next stage, once the Testing signoffs are received would be to give the build to the release team to deploy it to the Production environment to release to customers (based on whatever plan for that is in play).

The traditional - key problems - with this approach has been the fact that:

1. It takes a significant amount of time to finish. A mobile app that is needed in market today would come out 10 months from now (in a market of digital era which keeps changing all the time and would be quite assuredly be following new trends and seek solutions for different needs when we go to Production 10 months later).
2. It works based on the assumption that our understanding of what needs to be implemented, has always been correct from the beginning and will still be valid when we release the product several months later.
3. It needs too many hand-offs from one stage to another (and one team to another) which always carry the human error in understanding what was meant by the previous team. The cascading effect of these hand-to-hand misunderstandings can lead to significant deviations from our initial scope. (i.e. it is difficult to even stay on the path to deliver our own initial simplistic assumptions).
4. It forces early decision making on all key aspects of scope (and what would customers like), way before customers see and interact with the Product for the first time!
5. It discovers any potential functionality issues with the Product late in the process when all the stages in requirements, design and development are done and we have arrived in testing and now would need a trail of changes and repeat approvals on all artifacts that need to be updated to allow for the revised functionalities.
6. It is unable to adjust itself quickly (and with minimum impact) to any major market demand (or regulatory) changes as any major change would need to redo of all the stages from the beginning to that point.

Agile [Scrum and Kanban in this example] (from late 1980s to Present)

The pain associated with process inefficiencies of Waterfall approach was felt centuries ago. 

A long trail of Trial-and-Error in finding better ways for production and managing the work exists in history with records as early as late 1700s (Benjamin Franklin and his efforts in cost reduction in production process through using smaller batches and removal of wasteful activities, through Henry Ford’s creation of parallel functioning assembly lines in the early 20th century, further down to Toyota in implementing Kanban and Kaizen method in the first and second half of 20th century, and even the successful innovative approaches used in managing the construction of the Empire State building in the 1930s and the famous Polaris Nuclear Submarine in the 1950s) are some of the examples.

Finally, in late 1980s and early 1990s groups of senior software developers and process experts started to jointly work on adaptation of modern manufacturing approaches into the software production and this gave birth to Scrum (and later to Kanban, as used in IT today).

The immediately visible benefits with Agile approaches were:

1. They can engage the customer in the design and testing process early on by quickly creating prototype level products (called MVP, Minimum Viable Product) and creating a feedback loop to be used to gather information on the most recent customer needs and market position, to adjust the scope and re-align it to what would deliver the most value in the next release.
2. They do this through using short delivery cycles. Scrum uses Sprints which are timeboxed durations from one week to one month (teams choose the lengths and stay with that length). Kanban teams do not have a timebox and can take on new scope item on a daily basis.
3. To be able to deliver the MVP during the short delivery cycle, they break down the scope for the product into a number of smaller deliverables that would fit into a cycle and then by the end of the cycle they will have an increment of the final product that has the features that would be interesting (and valuable) to customers, that can be released if the business decides so.
4. The incremental build up of the product would allow for market experimentation with minimum investment in design and development and will continue as far as we receive a positive feedback from the customers. Otherwise it will either result in correcting our path or stopping further development on that path and starting to experiment with other ideas and assumptions.

The composition of teams and how they are managed are quite different:

1. There is no such thing as a Project Manager in an Agile delivery pipeline. The responsibilities of Project Manager are distributed between the 3 main roles of Scrum: Scrum Master, Product Owner and the Development Team.
2. Scrum Master (SM) is a Server / Leader role that is tasked to assisting the Dev Team doing their work, through removing impediments to their day to day activities and protecting them from distractions and interruptions of the environment around them. SM is also responsible for the Scrum process through hosting the Scrum Ceremonies (which replicate Project events such as Planning, Estimation, Scheduling, Status Reporting and Lesson’s Learned, only they repeat it for every cycle). SM also leads the team in practicing Agile values and Scrum Process and adoption of best practices and helps them implement the needed mindset and understanding in their activities.
3. Product Owner (PO) is responsible for the scope and the release. PO works with the client (be it a line of business in a large organization or marketing / sales team in a smaller company) to understand what would provide their customers the best value they can deliver. PO then breaks down that work into small items (aka Stories) that would be small in size and “described enough” to the level that Dev Team members can bring in during Planning and develop and test during the delivery cycle (aka Sprint). PO is also responsible for accepting or rejecting the completed work and owns the release as well.
4. The Development Teams (which points to the entire design, development and testing group working as a Scrum Team) are expected to manage their work (aka practice self-organization). They are trained and helped by SM to reach the maturity level needed to enable them to take responsibility and ownership of their work, prioritize its development and testing during the cycle and deliver it at the end of the cycle as they get ready to plan for the work they will do in the next cycle.
5. The work is demoed at the end of each cycle and feedback is collected from stakeholders to be used as corrective action for lining up the best items for delivery through the next cycle (Sprint).

As simple and light this approach may look like, due to its fundamental differences with the Waterfall approach, cannot easily fit or co-exist without establishment of a cross-framework (Hybrid) collaboration model.

The Marriage of Old and New (aka the inevitable eco-system)

It is not hard to imagine that large enterprises cannot be expected to achieve a comprehensive shift from Waterfall to Agile quickly, and many of them would never be able to completely evolve into an All-Agile organization.

Considering the internal shifts they are going through were ignited by their attempts in responding to the threat from smaller businesses and start-ups surfing the wave of Digital Disruption in combination with Agile delivery approaches, the inevitable solution would need to establish a hybrid eco-system to allow for fast market engagement / change respones in collaboration with the rest of the organization running on Waterfall process.

In order to establish this much needed cross-framework collaboration between these two approaches, the following practices are recommended:

1. Early Validation of the Assumptions: we should first start with “Validation of Assumptions” as early in our engagement as we possible can. This would enable us to leverage Agile’s early market engagement abilities in helping Waterfall to get a better sense of what would work and what would fail in the market and take corrective action as early in their process as possible. This can be done through:
   1. Joint Requirements and Design Workshops: This will reduce the risk or wrong ideation and bad requirements by avoiding Waterfall’s traditional Upfront Requirements and Design approach. This would need a collaborative cultural shift that needs Executive support to happen as it would be a very new and scary experience for the Waterfall teams:
      1. Before they meet, both sides will do their own homework internally, trying to establish the holistic view of the entire product from their perspective. 
      2. Each team will identify and capture dependencies they can fish out and try to build as much understanding and context as possible, before the meeting. 
      3. During the joint meeting, teams use whiteboards, walls, and flipcharts abundantly to allow for easy capture of ideas and topics and concerns.
      4. The best approach would be to follow “Specification by Example” to compose concrete system behaviors and remove confusion. For each idea or suggestion, they would try to answer the question “Give me an example”. They would also try to envision the final product and what features they believe it Should have on its release date. They would then move to identification of dependencies and planning for joint integration points in the future and any needed technology coordination.
   2. Prototyping and Early Market Engagement: Wireframes and Visual Mocks allow the teams to test for a market reaction early in the process. It can be as simple as a few sample screens – sometimes with a fake functionality – just to test customers’ interest in the potential product. Prototypes and Mocks also facilitate understanding of technology requirements and integration points between the two sides. It does not matter which side makes the mock-ups as long as they are going to review and revise them together. It would always be a good idea to create the mock-ups by using the same tools and programming languages that team will use to implement them later.
2. Cross-Framework Visualization: It would be highly beneficial to provide access to our Kanban (or Sprint) board for the Waterfall Dev / QA team and their Project Managers. This would create the transparency that is vital to the success of our coordinated efforts. Their Project Managers should also be encouraged to use a Kanban board to provide a high-level visualization on their larger blocks to assist the Product Owners on the Agile side to get a sense of their progress and priority.
3. Scrum of Scrum Masters and Project Managers (SoSaP): It is highly recommended to establish recurring weekly or bi-weekly session with Scrum Master and Project Managers attending for verbal updates on the progress of work, impediments, risks and concerns. These sessions may need to have follow-up meetings with relevant parties to brainstorm and discuss raised issues and find solutions and establish action items for the.
4. Architecting for Cross-Collaboration: Waterfall teams which are advanced in their architectural skills have been using design patterns in their solutions. Design patterns are important because of their ability to address inherent complexity associated with software development and difficulty in predicting requirements or designs. Design Patterns provide design approaches that are less fragile against changes and provide better refactoring capabilities. Design patterns such as Bridge, Adapter, Decorator, and Chain of Responsibility support separation of domain concerns. Factories are based on use of mock-ups, testability, and early integration as the instantiation of the entities and separate them from their use. Facade pattern supports early validation of integration with legacy systems.
5. Frequent Integration Points between Waterfall and Agile pipelines: Having recurring Integration between the two worlds would facilitate coordination of development and testing between the two sides. Two types of integration would be needed:
1. Soft Integration: Before the start of each Sprint for Scrum teams, we would need to have a Soft (Logical) integration which is essentially a synchronization of upcoming development and testing activities needed to ensure we all arrive at the release date with completed scope of delivery, ready to go. This integration would not need any code merge or joint testing sessions, but creates proper visualization on what each side is working on, supposed to be working on, and what would be the predictability for each side to make it in time.
2. Hard Integration: These are Physical integration points between the Waterfall and Agile team when code merge and joint testing is done to make sure what we planned for during the Soft Integration sessions, is ready and can come together to produce the outcome we are going to release.

Happily, Ever After

The cross-framework eco-system that we reviewed would foster establishment of effective, functional and sustainable hybrid delivery pipelines, based on continued support of executive management in the organization.

Executives’ support and promotion of the needed cultural changes through training and encouragement of best practices within the enterprise is vital to any Scaled up agile practice or cross-framework extended one. 

Such support would help the teams build up maturity - internally and cross-teams - and would enable a large organization to compete against the ultra-flexible Start-ups and Digital Disruptors through using their Agile delivery pipeline as the beating heart of the release machine, supported by a Waterfall “terra ferma” as the standing platform to rely on.