The Story of Evolution - The Thousand-year Journey from Waterfall to Agile

When in March 2006 - out of pure personal curiosity - I completed a two-day training and certification workshop as a Certified Scrum Master, the majority of the industry had not even heard the word Agile in their lives ... let alone Scrum ... or Scrum Master.

I soon took on the challenge of becoming an advocate of Agile approach and Scrum through many enterprises and helping with the adoption of this brilliant way of thinking by the market.

In those days, Waterfall ruled the arena and I was working as a PMP certified Project Manager for large financial institutions.

According to these enterprises, they had just finished the painful journey of perfecting their own implementation Waterfall process and their strong PMO (Project Management Office) teams and would have no appetite to start a new shift in their - just settled - paradigms!

It took some significant, prolonged effort by me and my peers across multiple industries to slowly convince these giants to brave the idea of experimenting with Scrum and carve out a few people from the Waterfall teams to form up as Scrum teams to check this new thing as well.

The positive results, in the shape of higher productivity, lower quality issues and faster time-to-market were so strong that accelerated the adoption process and Agile (and its most popular framework: Scrum) became the new modus operandi.

Even though Agile Manifesto was signed in 2001, and despite the fact  that Scrum as a framework was introduced by Jeff Sutherland and Ken Schwaber in 1995 (Yes! Agile Manifesto was signed 6 years later but Agile as a concept was discussed in academic environments for a couple of decades before that.), the actual search for better ways of managing the work of a team goes way back in history!

When did the search for Find Better Ways to Manage Projects start?

The prehistoric human had to cover all the roles by him/herself:

Manager,

Idea generator,

designer,

builder,

tester and the

End-User!

Humans learned it would be much more efficient if they distributed responsibilities among the team members, so each one can become better at what they are doing (Specialization).

Now a New Problem had emerged:

How to get a team to work together effectively?

This led to invention of the Traditional Approaches

Through a few centuries of trials and errors, humans finally established the plan-based (sequential) traditional project management methods.

Using them, they even managed the work to build the world’s wonders and magnificent monuments – many of which - still standing today.

And from the project management point of view

They all had a Serious Problem: None of them would finish in Time!

Tombs commissioned by the Pharaohs (the famous pyramids), would finish a long time after their deaths!

It took 140 years to finish Florence’s most famous Cathedral (Santa Maria del Fiore);  They spent 100 years of it building the dome!

The famous Wall of China, designed to protect an empire from invaders, took 200 years to complete!

Even with today’s life expectancies, it would them generations after generations to finish any large project.

Same was the issue with any small or large manufacturing efforts, that would take a very long time (several generations) for any improvements in design and build (for example, clock as a measure of time, took 4,000 years to become what we use today!)

The projects would also never finish within their initially anticipated Budget.

The efforts for improvement of traditional approaches in tracking the progress against the schedule and budget continued for centuries.

More accuracy in tracking the efforts and progress was achieved through heavier in-process stages and gating (checkpoints). They also grew heavy in paperwork and documentation.

The better Time and Budget Management allowed projects to finish during the lifespan of the original project team. (No more generational hand-offs!)

But the extra documentation and multiple gating approach had a heavy price tag!

It made them even slower than before to react to any changes in the market!

The Industrial Revolution arrived and:

• Manufacturing became more and more competitive
• Local markets grew into global trading
• Local competition had changed into an international race for customers’ money.

More than ever before - to survive and thrive - businesses needed to:

• Make sure the Product is exactly as customers needed it
• Sell it to them before the competition does it, or their demands change

But the Traditional Approaches kept missing the mark

… Let’s take a closer look …

The most famous traditional approach is the Waterfall approach.

The reason for this name is its close resemblance to a waterfall.

Once one step is completed, the outcome pours into the next level down the path and next until it reaches the end. The steps are in sequence and the movement is only in one direction.

Now imagine someone comes up with a Great Idea …

and they want to implement that Great Idea into a Product and send it to the Market!

Using Waterfall, the will start from the Requirements step, then once the Requirements work is done, the output (Requirements Document) is given to the Design Step, where the Design Documents are created, and once that is done, Development starts based on the Design Documents, and after that is done it goes to the Testing and when passed goes to the Release and there you enter the market with your great idea ...

Awesome! Right? ...... Well ....

History has shown that: This approach would ONLY SUCCEED …

IF We very accurately know what customers want from the beginning, and …

IF There will be no surprises along the timeline of the project, and …

The timeline is very short so the Market (i.e. What Customer is looking for) won’t change.

Which would be far from a Real World Scenario in Any Market!

In reality, what we deliver will always be –to some degrees – different from what customers need at that point in time!

How Bad has it been with the IT Projects?

Historically close to 70% of Waterfall IT Projects have failed to accurately deliver what market needed at the time of their release.

Waterfall is slow to deliver (as it needs to finish one step completely and pass the gate setup right after that step, before it can go to the next step. The amount of work that needs to be put into place before the project qualifies to move forward sums to a long period of time, typically months - even years - before anything can come to the release date and get delivered to customers).

Waterfall is not designed to quickly respond to changes in the market (as it needs to do one full round of steps before it can test the product in the market, if we decide, in the middle of the project, to make a change, we need to re-do all of the steps we completed and put them through gating again. This way, by the time we have completed a response, such a long time has passed, that we most probably need another change to re-align with the market).

Even if we go a great job of capturing exactly what the market needs when we start our Requirements step, while we are following our straight path from that step towards the Release, market would drift away from that captured idea and go down a different path.

As time goes by this diversion becomes wider and the market expectation for an ideal product would change even further away from our starting Requirements.

That means by the time we deliver what we captured at the beginning; we have landed far away from where market needs us!

We need to also consider the fact that: Value of an Idea drops over time!

Because:

• Competitors may find a similar orbetter idea and enter the market earlier.
• Customers may start following adifferent trend or
• Customers may slow down onspending money due to a newly tightening economy.
• Customers may demand new features that they didn’t want before.

What else?

• We may learn – too late in the project – that we do not have the skill sets for the project.
• We may learn – too late in the project – that it is not feasible, technically or financially, to implement.
• We may learn – too late in the project –that this was not a very good idea to begin with!

When were the modern ways invented?

As you can imagine, The problems with the Traditional Approaches were noticed long before the 20th Century!

Benjamin Franklin

One of the earliest traces of non-traditional ways of thinking and lean approaches in manufacturing and raising efficiency in using time and material goes back to the 1700s and to Benjamin Franklin (The American scientist and Politician).

He was a pioneer in understanding that reducing cost by choosing less wasteful approaches and higher speed in production are the best ways to succeed in the market.

He used to publish these ideas in his famous annual publication called: Poor Richard’s Almanac.

Henry Ford

Henry Ford, the American Industrialist and a great pioneer of Assembly Lines in production (from 1908 to 1940s),  adopted and improved the idea of reducing wasteful processes and improving the speed of production (The Concept of Agility).

Just before the Second World War, and based on Ford’s great success in using the modern approach, the British and the Japanese started using and expanding on Light (Lean) and Agile approaches in manufacturing and maintenance works.  

The Royal British Airforce

The British used an early version of Lean approach in assembly lines of Spitfire fighter planes (circa 1936) to improve speed and quality of the production line.

Their approach – called The Two-Bins approach - was later adopted by the Japanese researchers (at Toyota) and further developed and titled as KANBAN (which means “Visualization”)

Toyota

The Japanese inventor and Industrialist Sakichi Toyoda and his son Kiichiro Toyoda (who established the world’s largest automaker, Toyota, adopted the Ford’s and the British lean models and invented a series of methods in Lean / Agile manufacturing and maintenance works (Pull system, Kaizen, Kanban, Jidoka …)

The severe shortage of raw materials immediately after WWII in Japan and the high urgency in rebuilding the production capacity became a key driver for innovations in process optimization and development of Lean/Agile approach.

So for decades in the 20th century, Innovative & Pioneering Architects and Project managers were using Agile and Lean concepts to Finish their Projects Faster and Better Match Customers’ Needs without an official recognition of their approaches

The Secret Rebellion within the Waterfall ranks 

Waterfall dictates a sequence of mutually exclusive phases, separated by gates where one phase must be completed before the next one can start.

But the reality of what is actually happening is very different!

Over the past several decades, the managers and leaders of production teams - looking for ways to improve their teams’ performance - had discovered that they can raise their team’s productivity through secretly allowing the work to slip through the gates and to let the team start working on the next phase before the previous phase is completed, just going ahead with a bare minimum output (Lean) from the work still-in-progress at the previous gate! 

This became an un-written agreement among the team managers and the upper management to pretend it is not happening. 

The team managers won’t talk about it and the upper management won’t ask about it.

The empirical success of these “lean” and “Agile” forward thinking process manipulations, triggered the ideas behind official creation of the modern approaches of today.

And this incremental growth of Agile (Lean) mindset within the Waterfall ranks developed into the foundation we needed in the industry for official recognition of Agile as an approach and its further development into frameworks such as Scrum, as we enjoy practicing today.

What is Agile?

In one sentence:

A new approach to Deliver Value to Customers Faster and Minimize Paperwork”

Why Agile?

Because we have seen that this approach can generate more happy customers, which would lead to more income for the business.

What do we mean by Value?

A Product or Service that Customers find useful (can actually use) and would Pay for!

What sits at the core of the Agile approach?

In Agile we value:

1. Individualsand Interactions  over Processes and Tools.
2. Working Product over Comprehensive Documentation.
3. Customer Collaboration over Contract Negotiation.
4. Responding to Change over Following a Plan.

That is, while there is value in the items on the right, we value the items on the left more! Because at the end of the day, for a business to survive and thrive, nothing is more important than delivering value to the customer, matching what they need, as fast as possible.

Agile frameworks (like Scrum, XP, Kanban etc) do that by using short cycles that do a set of design / development / deployment / feedback collection which would allow them to engage the customers fast, get them to check the product or service and collect their feedback and use it to better align their work with what customers want, which has empirically proven to be the most efficient way of production in our ever changing market conditions and trends.

Agile, at principle believes that

• The highest priority is to satisfy customers through early and continuous delivery of valuable software (and services).
• We should welcome change in requirements, even late in development of our product, because Agile Process, harnesses change for the customers’ competitive advantage!
• Our plan should be to delivering working software (and/or services) frequently, as often as possible.
• Business and Development should work together, on a daily basis, during the Production.
• Working Product is our primary measure of success.
• We should adhere to Simplicity: The art of maximizing the amount of work that is not needed to be done!
• And that the Development team, should regularly reflect on their current quality of work and on how to become better.

In today’s market, in all sectors, Agile framework have been taking over all other methodologies, by storm.

They are no longer considered as exclusive frameworks for Software Development as their considerable success at all levels, from personal to organizational have proven their worth in every aspect of work.

Cheers