Rocky Beginnings

Mammoth Insurance’s first attempt at Agile came during January of 2015, when it hired an external consultant to deliver training and follow up coaching. Neither the courses or later support were well received; the consultant was soon fired and the Sunrise program leadership were left with a very poor first impression of Agile.

ScrumDo was selected to continue the adoption of Agile by Sunrise, which quickly ran into limitations of Mammoth Insurance’s technical practices. Specifically Mammoth Insurance’s approach to testing software relied on traditional manual QA. Such test-last practices are widely proven to increase lead times, batch sizes, and increase the cost and time to remedy defects. ScrumDo presented Agile testing approaches, which were adopted as a strategy. In a decision that would become a hallmark of Mammoth Insurance's reaction to encountering challenges that may have been too ambitious to solve, an external vendor was retained to create a testing strategy. The resulting 130 page text document found its way onto the program Sharepoint site, where its recommendations could sit unfollowed.

As the program’s immense complexity continued to be revealed, documentation grew as well; a first sign of the Agile value of working software over comprehensive documentation being inverted. The Lean Startup concept of Minimal Viable Product is a powerful strategy to mitigate risk by identifying and as quickly as possible delivering for real customer feedback a scaled back solution. While MVP has become a buzzword in most software development circles, a deep understanding of how and why to define one often remains elusive. Certainly within Sunrise, decision makers continued to build out massive process flows to capture “system requirements” defined by system functionality, rather than valuable user features. On a side note, by the time the implementation phase of the project launched, the small discovery team needed an entire 5 days’ time to present their process map flows, which was termed a knowledge transfer; looking around the room, where attendance dropped noticeably day to day; thus it was obvious the intended knowledge was not being transferred.

In parallel to Mammoth Insurance’s internal subject matter experts mapping extensive business processes, the discovery phase sought to determine the degree of required functionality already existing in the selected vendor’s product. In another powerful example of working software over comprehensive documentation being ignored, the assessment was carried out exclusively through documentation artifacts. At no time was the software itself demonstrated, or even basic tasks attempted. While those making these decisions could not know the severe implications, as Agile Coaches, ScrumDo attempted to raise these issues as they were detected. However, not only were the coaches prevented from interacting directly with senior decision-makers, they were not able to participate in what feedback loops did exist. The overall effect was of an echo chamber, where only information that fit the assumptions was gathered.

As the discovery phase continued, ScrumDo was able to establish an alliance relationship with senior executives who could influence the Sunrise program. ​

As the year end approached and the Sunrise program prepared to transition from discovery to implementation, it was becoming clearer to decision makers that the size of the program, being substantially larger than any in Mammoth Insurance’s history, might be problematic. With ongoing relationship building, the ScrumDo lead Agile coach was able to exert influence toward some powerful wins: creation of Agile contracts with vendors, which allowed for a learning phase and affiliated option to exit the contract. Senior leadership at this time also participated in a Kanban training and were exposed to critical ideas including managing for flow efficiency rather than people utilization, and the importance of limiting work in progress to enable flow through a system.

To set the stage for success, program leadership wanted to embrace the common Agile practice of cross-functional teams capable of delivering work with minimal dependencies. It was decided that there would be four Scrum teams consisting of Mammoth Insurance employees drawing from two locations in the US, vendor employees working as team members in the States, and vendor employees based in India. To support these “vertical” teams, there would be a collection of “horizontal” teams with specific domain expertise such as architecture and release management that would be needed in all the teams. In a planning document, this arrangement made sense, though like many decisions was based on assumptions and risks that had not been verified.

The last stage of the year was to add two additional Agile Coaches to begin designing immersive training sessions for project Scrum teams, with an emphasis on product backlog creation and refinement. User Story Mapping was identified as a tactic fit for the context and a 5-day session split between training and backlog refinement was decided on.

With the beginning of the new year, the Sunrise program shifted from discovery to implementation. Like many organizations, Mammoth Insurance operates on an annual fiscal model, and from an accounting perspective requires its efforts to fit into this cycle. Knowledge work at the scale and complexity of Sunrise, however, has a tendency to have little regard for calendars and even master-planned schedules.

Kick Off

The Sunrise kickoff lasted 5 full days. Mammoth Insurance employees at two locations sat in the largest rooms on each campus with a video link, joined by nearly 100 remote employees. The first day included presentations and introductions by the CEO and other C-suite executives, along with various levels of program management and stakeholders. While the steadily declining attendance through the week spoke to the efficacy of the perceived value of such a “knowledge transfer” the initial message by the organization’s top leader was powerful, energizing, and persuasive, serving as a resonant touchstone for months to follow.

With the program ramping up from Discovery to Implementation, a substantial number of people were being brought in, including two full-time ScrumDo Agile Coaches. Within hours they began having concerns about the program. What and where were the goals? Who was the customer and what needs would be solved? What kinds of milestones would mark progress? What might a definition of Minimal Viable Product be? What business goals were there?

Instead of answers, the following days brought end-to-end hours of lecture, process flows, technical information, and more declining attendance. While clarity in complexity is dangerously elusive, the initial impression shared by the coaches was that one of the primary Agile values (working software over comprehensive documentation) was being robustly inverted. At no point in the 5 days was there a single demonstration of any vendors’ current state, or even prototypes or proof of concepts. The entire Sunrise program was being built on the simple assumption that functionality expressed in presentations, check lists, verbal conversations, and contracts would fulfill Mammoth Insurance’s needs. They were quickly to learn just how off target these guesses were.

Training

In the month leading to the kickoff, two of the Agile Coaches had been creating a 5-day training immersion for the newly-formed Sunrise Scrum teams. The four teams were to be the heart of the program; divided into two functional specialties, each pair of teams would further focus on a specific area of the process of creating and selling an insurance policy. Initial inquiries into what the teams needed revealed a need to break large scopes of work into smaller ones, as well as introductions to Scrum and Agile principles, values, and practices the teams could use on their own.

An immersion training was designed around the practice of User Story Mapping, with an initial two days of experiential workshops before the teams attempted mapping one of their own epic-sized stories. The sessions received rave reviews from participants with some reporting it the best training they’d ever had. As powerful and positive as the trainings were, they also further illuminated an utter lack of customer focus in any of the strategic planning in Discovery. All planning documents had a firm system/functional focus. The result was a series of User Stories focused not on users and their needs, though specific processing steps of the system. This discovery created powerful opportunities for deepening the user-centric tenets of design thinking and Agile practice, though also allowed the teams to realize the limits of their knowledge and some of the real impediments they would face as a Scrum team.

​

Where's the Big Picture?

Upon completing training, the coaches joined the Sunrise Program collaboration space, which had been newly renovated with fresh paint, custom artwork and inspirational quotes on the walls, and brand-new half-wall cubicles, many still vacant. Having spent the past several weeks designing and delivering User Story Mapping workshops, and also having begun to comprehend the massive complexity Horizons represented, they wondered where in the space an overall program story map could be built. The map would be visible from every place on the floor, and have visual cues for each team so they could see where other teams depended on them. It was also suggested that each team conduct their daily standups at the map to ensure it would be maintained. While there was a question of distributed teams and members, the coaches believed that once the program realized the incredible value of seeing all of the work visualized in one place, everyone would collaborate to solve for distribution.

None of this was to be. The freshly-painted walls could not be marred by hanging white boards, blue tape, or post-its. The empty cubes had people assigned to move into them, and those assignments turned out to be difficult and costly to change. Also, the floor space needed to access the only suitable wall was deemed too valuable for the 3 work stations occupying it. Extremely budget-conscious, the program management could only see the dollar costs to make changes, rather than the enormous benefit of gaining perspective and potentially dropping the cost of coordinating teams. The tactics advocated by the coaches were intended to create a program-scale information radiator that would be always on and up to date, and that program teams would be interacting with on a daily basis. Mammoth Insurance’s lack of experience with large programs combined with a reductionist approach to problem solving created a disconnect where management simply could not conceive not only how valuable a visual big picture could be, though how critical it would become as individual teams dug deeply into their respective work.

Although managers were quick to point out reasons why they would or could not have a visual radiator of the entire program, they were accordingly baffled at how they themselves would gain such a perspective, or be able to communicate one to senior decision makers. In the past, status report documents had been the tool of choice, and soon PowerPoint templates were being designed that each team would need to fill in on a weekly basis to be rolled together by program PMO. The stage was set for further difficult learning.

Additionally, as teams worked on their backlogs, they began to use slightly different approaches and tactics. The common starting point that had been established through all teams participating in the same set of training quickly bifurcated. While each team gained a deepening understanding of its work, there lacked a common approach or visual language. It became difficult to see inter-team dependencies; communicating these fell to the ScrumMasters, who relied on the program management hierarchy as the conduit for messages. Messages sent along multiple handoffs lost integrity, caused delays, and created a sense of isolation among the teams.

ScrumDo Feature Highlight

With portfolio structures, ScrumDo users get a single, simple, real-time view of the truth about the state of work in your organization using the Big Picture view.​​

Gaps

As the teams moved from kick off and training to their work of implementing their respective sections of the process, they quickly began discovering the extent of their greatest obstacle. An external software vendor had been chosen based on a document-driven evaluation. At no point was their software demonstrated, only screenshots and lists of functionality. By choosing this particular vendor, Mammoth Insurance was looking to execute on two high-risk strategies: buying an incomplete application, which would be finished through a partnership of Mammoth Insurance’s industry knowledge and the vendor’s technical expertise; and working with an outside vendor to operationalize core business processes. Either of these would be exceptionally difficult on their own. In combination with multiple other major strategies, they would prove insurmountable.

While the application was known to be incomplete, based on document-driven analysis it was believed to be roughly 80% complete, with functional gaps consisting of only about 20% of needed features. As teams decomposed their Epic User Stories, they were blocked over and over by gaps in even basic functionality. Within a few weeks of their work starting, a picture was emerging that was drastically different from what Mammoth Insurance planned would be happening.

The information was so concerning that Sunrise management decided, not for the last time, to cease implementation work and instead seek a fuller understanding of the overall effort by having teams break all assigned Epic User Stories down into Feature User Stories, along with rough estimates of effort on their features. While coaches warily supported this effort, it was with several conditions, including a strong warning about the lack of empirical basis for estimates as a predictive measure, and a tendency for organizations to hold teams accountable for estimates.

ScrumDo Feature Highlight

The aim of Program Implement Planning is to create agreement on a plan for deliverable objectives for the next program implement in a Portfolio. It is performed on a regular cadence, usually quarterly, and face-to-face.

​

Replanning, Part I

The next several weeks brought a concerted Story Mapping effort which sought to understand the actual scope of the Sunrise Program initial effort. Each team was given priority for a room and began working through the list of Epics they had been assigned. Notably, the effort was completely siloed so that while each step in the overall system process became deeply understood, there was a distinct lack of focus on integration, or possible identification of Minimal Viable Product. As the teams worked, a proto-Kanban board emerged high on a wall on the main project area floor to track their progress. Each work item was an Epic that was being decomposed into Features and had been estimated as a t-shirt size. Each team had a different color post-it note, and a swim lane on the board. Each time an Epic was complete a team member would update the board. For the first time, program leadership and the teams themselves could see a real-time visualization of effort and know at a glance the progress of effort and state of work. Despite this win, there was still an unwillingness to translate the benefit to a larger-scale perspective; slideware status updates continued instead.

Within a couple weeks, a picture emerged for the first time of the full scope of work required to deliver the Sunrise platform. Mammoth Insurance’s management had allocated 12 months of timeline and budget for implementation of the first phase. The consolidated estimates reflected 4 years’ worth of effort. Even with the demonstrated unreliability of relative estimates as predictive tools, it was clear there was a massive disconnect between the meticulously-created library of documentation and what would be required to achieve valuable working software. Despite over a year’s worth of presentations, meetings, process flows, technical strategies, and architectural plans, the efforts of a small group of teams working against an actual application revealed a picture of reality that was far outside any forecasted worst case scenario. In many organizations, such information would have been buried as long as possible. To the program leadership’s credit, they accepted this data and began to investigate alternatives, specifically to answer the question: is it even possible to, within the required timeline, to create a platform that will support new products in the following year?

Much of what drove the massive expansion of work was teams discovering that the vendor application, believed to be 80% fit for purpose, was actually full of functional gaps. Not only did these gaps expand the scope of required work by several fold, it blocked the teams from being able to demonstrate any meaningful value of what small amount of functionality did exist. These discoveries were like a falling line of dominos. The lack of functionality exposed a severe lack of technical practices and quality in the vendor, an inability to discern this lack, inability by the vendor to quickly turn work around either of bugs or new functionality, and a lack of integration between people working in India and the US. On paper, the Scrum teams were cross-functional groups that included the capability to deliver software. In reality, while there were very senior Mammoth Insurance developers on the teams, they did not have access to the vendor source code and so were constrained that all changes had to be made in India, by employees of a different company, with different managers, a substantial time zone difference, and IP agreements that prevented source code from being shared with Mammoth Insurance. Any of these would have been difficult scenarios on their own, when combined they were simply a bigger mess than could be untangled.

Following the decomposition work The teams returned to their implementation work, at least for a few weeks. During this time, unknown to the teams, executive leadership was wrestling with the revelation that the program could not be successful as planned. The lack of a big picture perspective now became an acute and pressing need. Program leadership scrambled to pull together enough information to satisfy their stakeholders’ needs. To do so required putting implementation work on hold for a second time for teams to “baseline;” an attempt to squeeze further prediction out of the initial decomposed estimates to inform the somewhat desperate conversations about project scope. It also began to emerge that one of the teams was unable to make any progress at all due to the gaps they had identified in the vendor’s application. Another two teams were severely limited in what they could accomplish; the fourth team was able to get quite a bit done, though it was largely configuration work; the most straightforward and least complex of the entire effort.

Replanning, Part II

Shortly thereafter, it was announced that once again, the teams would be engaged in another planning effort. Program leadership and representatives from an external consultant had combed through the estimates trying to identify a scope of work that might possibly deliver some value in the calendar year. Key business stakeholders were acutely concerned that any further reduction of scope would create a system incapable of actually doing the job required of it. With so much of the needed functionality tied up in functional gaps that were exclusively the vendor’s responsibility, program leadership scrambled to find work that the Sunrise teams could complete that would progress toward program goals. The focus they chose was “architectural significance,” meaning work that had to do with integrations, infrastructure, or the other limited domains both within Mammoth Insurance’s sphere of control.

Replanning, Part III

Once a scope of work with an emphasis on architectural significance had been mapped out by leadership, the teams were once again called to cease implementation work to focus on applying a new estimation pattern, which was provided by the new vendor. ScrumDo coaches expressed substantial concerns with the approach, though they were told by program leadership to “get behind it.” Meetings with the vendor resulted in a conditional compromise: the vendor would make explicit the long list of assumptions and conditions that needed to be true for their approach to work. While they agreed to do so, there was never a clear communication to this end. Rather the teams were once again busied with pulling together their best guesses based on what little data they had been able to gather in the few and infrequent short bursts of implementation work.

In a climactic session, program leadership, middle management, vendor representatives, and Agile Coaches were gathered into a conference room with a small representative cohort from each Scrum team. The team representatives one by one were invited by the external consultants to present their estimates, and were questioned about how they arrived at them, and ultimately, the confidence in their accuracy.

While this exercise did result in surfacing the final set of data that informed the final decision to dramatically restructure the program, the quality of the evidence was shaky at best. Instead, Mammoth Insurance’s ravenous appetite for data and documentation drove creation of polished slide decks. It is telling of the massive amount of undiscovered work that even these extremely low fidelity approaches could clearly show how bad the Sunrise plan miscalculated and mitigated risks. In the language of Nassim Taleb in his book The Black Swan, this would be a “grey swan;” an event of paradigm-shifting proportions that could have been known, though simply wasn’t looked for.

The Final Steps

Unknown to the Agile Coaches or teams, program leadership was presenting all of this information to company executives, who were asking about alternatives. While the timeline of the Sunrise program had always seemed aggressive, program leadership started talking about a projected revenue loss of $500 million the following year if the technological capability to bring new products to market was not realized by October 1st, a mere four months away.

Around this time, the team of ScrumDo coaches created a presentation to make explicit and transparent the substantial risks they perceived as being unseen or outright ignored. The deck was delivered to top executive leadership, and combined with the undeniable flow of data showing the sheer volume of work to be impossible to complete, Mammoth Insurance’s top leadership decided to essentially cancel the program. While many team members felt a deep frustration, leadership and the coaches praised Agile’s feedback and transparency to enable such a decisive action so quickly. While there was an extremely short window for Mammoth Insurance to implement an alternative to Sunrise, a window did exist, and to their credit, the organization moved very quickly to reorganize a much smaller program around this opportunity. The Sunrise program, scaled back almost beyond recognition, was also given the opportunity to continue on the remaining strategies of co-creating a new platform with an external vendor. However, without the high-pressure of date-driven market deliverables.

..​​

.​

ScrumDo Feature Highlight

​​ScrumDo can be effectively leveraged to help individuals and teams by integrating risk-based visualizations into their workflow. In the Risk View, the severity and probability is also captured along with various dimensions of Risk. This visualization becomes a point for discussion for those interested in managing the risks, informing decisions on how to deal with it and when to start work on mitigating the risks.​​

​

Conclusion

One hallmark of complexity is that we can never know for sure how we’ll get from point A to point B. Sometimes, in hindsight we can piece together the steps and path, though we can never know for sure the entire story or picture.

Though even looking back, much of Sunrise cannot be known, there are some powerful lessons for any organization:

1. Seek real things. The Agile Manifesto codifies this as working software over comprehensive documentation. The Kanban Method expresses this through tracking work items themselves over time, rather than proxy measures such as story points or other estimates. Had Mammoth Insurance embraced evidence early on, it could have made more informed decisions much earlier, avoiding spending multiple millions of dollars and exposure to substantial risks.

2. Focus. The strategy guiding Sunrise was fuzzy at worst and impossibly ambitious at best. Starting with known wins and building incrementally with clear focus is more likely to enable successful execution. Overloading a program with multiple, substantial new directions is challenging and likely to result in possibly serious unintended consequences.

​3. Humility. Leadership acted as if they were correct even when evidence clearly showed otherwise. It is important to keep in mind the perspective of others, even when you are sure of your opinion.

​4. Be Ready To Change...Everything. Established organizations with bureaucratic layers of decision making will be challenged to respond and adapt as needed. Things as simple as where people sit and how physical space can or cannot be used need to be negotiable, if not outright the teams’ choice.

5. People Together Work Together. While co-locating an entire program may not always be feasible, it is vastly advantageous for feedback loops and healthy relationships.​​

​​