Graebel Agile SOA

Note: This is part of a new and continuing series which is collected under the Agile SOAcategory.

Here at Graebel, we believe that an Agile methodology provides great benefits to our development teams and our customers. Similarly, the decision to implement a service-oriented architecture has the potential to provide numerous benefits to the enterprise and our users. The question then becomes: Can an Agile methodology and SOA complement one another?   

In many ways there are strong synergies between the two disciplines as well as potential conflicts. First, we’ll examine areas where SOA and Agile are complimentary. Secondly, we’ll discuss specific areas where potential conflicts may arise.

SOA and Agile are complimentary

SOA is architecture. SOA stresses that businesses must be able to respond to the market and that by building services, we can get rid of the duplication and get closer to the elusive goal of reuse. By building services instead of applications, teams can leverage work by others from within the enterprise as well as externally.

Agile is a methodology. Agile stresses that everything changes and that software development teams must be able to recognize and respond to change frequently. By introducing both technical and non-technical practices, our development teams are able to help Graebel become an agile enterprise.

Architecture and methodology can be used together. They are complementary by nature. Moreover, SOA and Agile share the same broad goals. They both recognize that change is inevitable and that organizations need to effectively cope with that change. So we would expect that Agile is by default the methodology of choice when building SOAs and vice versa – right?

SOA and Agile may conflict

You would think that with shared goals the overlap of practices and architecture implied by these two techniques would be in agreement and not in conflict. At this point in time there is little in the SOA community that is agile and vice versa. Why is this?

One of the main reasons is that the SOA community approaches the agile methodology discussion with its own perception. Agile is historically grass-roots and small-project based, although throughout the past years the community has gained experience and learned to adapt the principles of the Agile Manifesto to large projects. SOA is a newer initiative and is top-down in nature, taking an incremental approach to the development of loosely-coupled services, processes and messaging components.

Two distinct areas where the SOA and Agile approaches are at odds:

• SOA encourages that architecture be upfront while Agile has a derogative term for this approach – coined BDUF (Big Design Up Front).
• SOA encourages teams to split along functional lines while Agile encourages cross-functional teams. 

Graebel’s Approach

To enable the best aspects of SOA and Agile, Graebel uses a leave-and-layer strategy in which the application platforms are incrementally transformed from point-to-point solutions to service-enabled interfaces. This transformation provides reliable access to the right data and processes at the right time, all while continuing to run the business in a transformational manner. This unique approach includes the best aspects of an Agile methodology to plan and manage the transformation and a Separation of Concerns process to build each layer of the SOA stack for each application in the enterprise. 

Graebel’s SOA Leave-and-Layer Methodology

This methodology, which we refer to as Incremental / Leave-and-Layer provides a wise alternative to the BDUF approach. Additionally, we are able to accommodate a cross-functional teaming strategy meaning that our SOA Program team is able to join existing application development teams for 2-week release cycles as each application is added to the integration architecture.

Graebel's SOA Program

The second day… Establishing Agile @ Graebel

Note: This is part of a new and continuing series which is collected under the Agile Journey category.

Prelude

In the last post Let there be light! I described how a past project failure and some of its circumstances served as fertile ground upon which the seeds of agile could be sown and nurtured. I examined how the 4 core elements of the Agile Manifesto were relevant to the circumstances experienced on that project. Yet it wasn’t until I joined Graebel in 2005 as the Director of Web Technologies that I was in a position to apply agile software development methodologies within a development context.

Note: We partnered with Pete Behrens of Trail Ridge Consulting to mentor us through our agile adoption and implementation. His expertise and efforts were invaluable throughout the fits & starts we experienced during our pilot project.

Introduction

Throughout the interview process at Graebel I positioned my engineering mindset with agile development methodologies rather than the traditional waterfall mindset. Specifically I was a fan of scrum as I believed the practice was best suited to be applied to the full workload of a corporate enterprise IT department. My long-term plan was to take the scrum approach, apply it not only to new software development but also to support, maintenance and application enhancement development activities. Yet before I could achieve that mission I needed a pilot project where scrum could be applied and to serve as the catalyst for change within my newly acquired development team. We embarked upon that pilot project at the end of 2005 and what follows is a quick description of several aspects of our initial scrum implementation.

Team space

We took down cube walls within a section of cubes to create a six person bullpen. We relocated most of the members of the team into this bullpen to better facilitate face-to-face communication and collaboration.

Note: We were unable to move everyone into this space as we had significant resistance to moving to such an open environment from several “legacy” members of the team. However today we restructure our floor plan at least quarterly according to projects and their teams.

Sprint Planning

Initially we rolled with 30 day sprints. However progress was elusive for the team and planning was extremely difficult because our backlog was very shallow. We adjusted to one week sprints and as momentum built we expanded to two week sprints.

Note: Today we still plan two week sprints. We found that one week sprints were too chaotic and sprints longer than two weeks didn’t facilitate enough agility for the business.

Stand-ups

I have struggled with the need for stand-ups from the beginning. I remember meetings where we would go around the horn asking people “What did you do? What do you plan to do? Do you have any roadblocks?” People would just look to the task board and regurgitate their tasks without adding any valuable context. After they stopped talking, we still had no clue as to their progress towards the sprint goals. The stand-up was not meant to be a status report but that is essentially what it became.

Note: Even today we struggle with stand-ups and are working to transform them to contain focused communication that enables the evaluation of progress towards the sprint’s goal. My thought is that if I need a status, I can just run a sprint report and spare everyone the meeting. Yet the meeting is a key collaboration point and discipline so instead of getting rid of it we just need to transform it.

Story Boards and Task boards

We wrote our user stories out initially on index cards and then moved to yellow stickies. These stories were either taped or placed on a whiteboard that had four sections: To Do, In Development, Testing and Done.

Task cards were the result of the user story being broken down into logical development tasks. We initially used index cards, progressed to yellow stickies and then proceeded to have our own custom stickies printed. These task cards were placed on a whiteboard with five sections: To Do, In Progress, Awaiting Verification, Verified and Complete. These cards and their progression across the whiteboards were instrumental in keeping the sprint and its progress in front of the team at all times. This same effect could not be provided via an electronic tool.

Initial Task Board Concept

Note: We eventually got rid of the manual cards in favor of leveraging online tools to track their state, progress and particulars across our process. Yet today we are in the process of bringing back a manual Kanban system to run in parallel with our online tools. Relying completely on our online tool resulted in a significant loss of visibility and awareness across the team, within management and with the customer.

Product Owner

Initially I served as the product owner within the team. I was the one who developed the technology vision and strategy for our pilot project and thus was best positioned to serve in the role. Plus at the time there was not a business analyst on the team. If there had been a business analyst, they would have been the most logical choice to serve in this role.

Note: Today we have a team of business analysts that we actually call product managers. These individuals are collocated with the team and own the release, product backlog and product roadmap.

Looking Ahead

This posting is the starting point for a new and continuing series which will be collected under the Development Process category. Nate Kresse, Graebel’s Manager of Software Engineering, will lead this series and examine the processes, disciplines, tools and vision for Graebel Agile.

Let there be light!

Back in the mid ‘90’s I had the opportunity to establish and lead a small software development company named Strategic Innovations. Initially Strategic Innovations focused on application development via Visual Basic. But as the ‘90s progressed we naturally expanded our expertise to include Internet-based application development via Java and ASP technologies. In 2000 I sold Strategic Innovations to a Midwestern company, Meritage Technologies, who was rolling up small Microsoft-focused software consulting companies in key markets in hopes of eventually taking the company public.

One of the first projects we landed after being absorbed into Meritage Technologies was a job to develop a Java application to enable Fortune 500 companies to manage their global Six Sigma efforts. This application was truly innovative in concept but the project was a nightmare and by most measures a failure. A solution was ultimately delivered but it was over budget, under scope and late.

When I eventually had the luxury of assessing the project I identified several key circumstances that when taken in isolation did not cause the project’s failure but when combined together they created a ‘perfect storm’. Some of those circumstances were:

• The application was 100% conceptual and there was not a point of reference to draw upon within the client’s business or even the general marketplace.
• A monumental effort was undertaken by the business analyst at the beginning of the project to elicit and define the envisioned application’s full set of functional and technical requirements.
• The requirements were well documented and placed immediately under change control.
• Development was “outsourced” to a sister office in another state at the request of corporate management.
• The business analyst was the sole interface and intermediary between the client and the remote development team.
• A significant amount of requirements churn eventually stymied and all but paralyzed the development team and its progress.
• Once the requirements began to churn the project became a bureaucratic mess due to a rigid change request process and an overarching focus on the project plan.

It was not until a few years after this project that I was able to start to correlate these circumstances together and begin to comprehend their association and collective contribution to the project’s failure. It was also at this time that agile methods started to gain traction in the development community as an emerging development mindset and alternative to the typical waterfall approach. I can still remember reading the Agile Manifesto and having all kinds of bells and whistles going off in my head as I reflected upon the principles behind the manifesto. It was an epiphany and a key watershed moment in my professional career.

Working software over comprehensive documentation

Since the project was truly a ‘greenfield’ development effort the expectation that the client definitively define the application’s requirements at the outset was completely and utterly unrealistic. It isn’t that the client didn’t have a clue about their business but rather they had no point of reference to draw insight or knowledge as to exactly what needs the application was to fulfill. Thus the requirements that were initially defined and well documented essentially became worthless once development got underway. The only thing that would have enabled the client to more fully understand their needs and the application’s requirements would have been working software that they could have touched, felt and played with.

Customer collaboration over contract negotiation

The primary reason why such a huge effort was undertaken to capture all of the application’s requirements upfront was because both parties “needed” a defined scope to build a contractual agreement around. Once the contract was in place a rigid change management process was enacted to ensure the contract’s integrity as well as to facilitate mutually agreed upon scope changes to that contract. Yet as pointed out previously the client had no frame of reference to draw upon when defining the application’s requirements and thus the contract was, for the most part, based upon an ill-defined scope and a premature set of requirements.

The result was that collaboration with the customer was essentially squelched. Why this happened was because as the application became more tangible in the client’s eyes, the more clarity that was realized about the application’s true requirements. Yet that clarity necessitated endless change orders as well as addendum to the contract. Eventually whatever collaboration and trust that had existed disintegrated into an ongoing negotiation and renegotiation exercise.

Individuals and interactions over processes and tools

Outsourcing development to a sister office was a corporate mandate in an effort to help offset the economic downturn experienced by this sister office as a result of 9/11. This decision decimated any semblance of a team as well as effectively prevented meaningful collaboration and cooperation between the client and development team. This separation eventually contributed to mistrust between the business analyst and development team as delivery dates slipped and confusion surrounding requirements prevailed. Combine these circumstances with the previously discussed challenges and it can be seen that close, daily cooperation between the business and technology team is essential.

Responding to change over following a plan

Given the nature of this project as described above it can be plainly seen that change was inevitable on this project. But in every facet of the project life cycle, change was considered the antithesis to the project’s success. It was to be eliminated and avoided at all costs. The reality is that change should have been fully embraced and enabled throughout the life cycle and within the disciplines of the project. Yet both the client and development team were incented both contractually and monetarily to keep to the plan regardless of whether responding to change would have helped ensure the success of the project.

Conclusion

It was this project experience and resulting soul searching that prepared me to fully embrace agile when I took the reins of software engineering at Graebel back in 2005. Yet over the last few years we have not only embraced agile for our new development efforts but have expanded our agile mindset and disciplines to encompass all of our software engineering activities from maintenance, support, minor enhancements, major enhancements, architecture, infrastructure and large capital projects. We have also incorporated business analysis and product management disciplines with the goal of quickly expanding into program and portfolio management disciplines.

Future posts will bring insight into our current processes and disciplines with a look to the future and how we plan to continually improve, reduce waste and journey further into what we call enterprise agile.