Tag Archives: lean

Lean Development Applied To SOA

Share the article!

I’ve been doing a little bit of musing (“Is SOA an Agile Enterprise Framework?“) about a development framework to support SOA. However, maybe Lean Production/Thinking would be better fit for SOA. In an earlier entry I mused about “How Web 2.0 supports Lean Production“. Let’s turn this question around and ask the question “How can Lean development be used in support SOA development?”.

Lean focuses on the elimination of waste in processes. Agile in comparison is tuned toward practices that adapt efficiently to change. There are though some commonalities between Lean and Agile Development. These specifically are:

  • People centric approach
  • Empowered teams
  • Adaptive planning
  • Continuous improvement

The last two bullet points align directly with the SOA Manifesto. For reference:

  • Business value over technical strategy
  • Strategic goals over project-specific benefits
  • Intrinsic interoperability over custom integration
  • Shared services over specific-purpose implementations
  • Flexibility over optimization
  • Evolutionary refinement over pursuit of initial perfection

Lean differs from Agile in that Lean is purported to be designed to scale (see: “Set-based concurrent engineering” and Scaling Lean and Agile Development ):

One of the ideas in lean product development is the notion of set-based concurrent engineering: considering a solution as the intersection of a number of feasible parts, rather than iterating on a bunch of individual “point-based” solutions. This lets several groups work at the same time, as they converge on a solution.

In contrast, agile methods were meant for smaller more nimble development teams and projects. One would therefore think that in the context of enterprise wide SOA activities, Lean principles may offer greater value than the Agile practices. Well, let’s see if we can convince ourselves of this by exploring this in more elaborate detail.

Lean Software Development is defined by a set of “seven lean principles” for software development:

  1. Eliminate Waste – Spend time only on what adds real customer value.
  2. Create Knowledge – When you have tough problems, increase feedback.
  3. Defer Commitment – Keep your options open as long as practical, but no longer.
  4. Deliver Fast – Deliver value to customers as soon as they ask for it.
  5. Respect People – Let the people who add value use their full potential.
  6. Build Quality In – Don’t try to tack on quality after the fact – build it in.
  7. Optimize the Whole – Beware of the temptation to optimize parts at the expense of the whole.

Can we leverage these principles as a guide for a better approach to SOA development?

Where can we find waste in the context of software development? Poppendieck has the following list:

  • Overproduction = Extra Features.
  • In Process Inventory = Partially Done Work.
  • Extra Processing – Relearning.
  • Motion = Finding Information.
  • Defects = Defects Not Caught by Tests.
  • Waiting = Delays.
  • Transport = Handoffs.

What steps in SOA development can we take to eliminate waste? Here’s a proposed table:

Waste in Software Development Lean SOA
Extra Features If there isn’t a clear and present economic need for a Service then it should not be developed.
Partially Done Work Move to a integrated, tested, documented and deployable service rapidly.
Relearning Reuse Services. Employ a Pattern Language. Employ Social Networking techniques to enhance Organizational Learning.
Finding Information Have all SOA contracts documented and human testable on a shared CMS. Manage Service evolution.
Defects Not Caught by Tests Design Testable Service interfaces. Test driven integration.
Delays Development is usually not the bottleneck. Map the value stream to identify real organizational bottlenecks.
Handoffs Service developers work directly with Service consumers (i.e. developers, sysadmins, help desk)

The customers for a Service are similar to the customers for an API. I wrote years ago about how the design and management of APIs leads to the development of damn good software. The same principles can be applied with the lean development of services. Taking some wisdom from that, here are some recommended practices for Lean SOA:

  • Designing Services is a human factors problem.
  • Design Services to support an Architecture of Participation. Focus on “Organizational Learning
  • Focus on what a user of your Service will experience. Simplicity is the #1 objective. Only when this has been accomplished (at least on paper) do we talk about implementation details.
  • Services arenxe2x80x99t included in a release until theyxe2x80x99re very simple to use.
  • In tradeoff situations, ease of use and quality win over feature count.
  • Useful subsets of standards are OK in the short term, but should be fully implemented in the longer term.
  • t

  • Continuous Integration – Fully automated build process and tests.
  • Always Beta – Each build is a release candidate; we expect it to work.
  • t

  • Community Involvement – Community needs to know what is going on to participate. Requires transparency.
  • Continuous Testing.
  • Collective Ownership.
  • Preserve Architectural Integrity –
    Deliver on time, every time, but must preserve architectural integrity. Deliver quality with continuity.
  • Services First – When defining a new Service, there must be at least one client involved, preferably more.

t

Finally, there is one last principle in Lean development “Decide as Late as Possible” that is of high importance. The ability to decide as late as possible is enabled by modularity. The absence of modularity makes composing new solutions and therefore new integrations extremely cumbersome. The key however is not to become a Cargo Cult and practice Lean SOA without understanding how one achieves modularity (or to use another phrase “intrinsic interoperability”). The key is to understand how to achieve that. That of course is the subject of the Design Language that I am in the process of formulating.

In conclusion, Lean SOA (A mashup of Lean and SOA) follow these principles:

  • Eliminate Waste – Spend time only on what adds business value.
  • Create Knowledge – Disseminate and share Service knowledge with the organization and its partners.
  • Defer Commitment – Be flexible before you optimize.
  • Deliver Fast – Deliver value quickly and incrementally. Don’t try to boil the ocean.
  • Respect People – Let the people who add value use their full potential.
  • Build Quality In – Don’t try to tack on quality after the fact – build it in.
  • Optimize the Whole – Strategic goals over project-specific benefits.
  • Build Interoperability In – Services should be designed to be modular.

  • This is a simple as it gets. The devil of course is in the details.

    Notes: You can find an earlier and different take on this here: SOA Agility.


    Share the article!

    The Lean Nature of Google’s Development Practices

    Share the article!

    First, a bit of disclaimer, I’m not privy to the details of Google development practices. I’m gathering this information from multiple public sources such as the a set of notes recorded by Evelyn Rodriguez on the subject, Joe Beda’s blog and an Newsweek article written by CEO Eric Schmidt.

    For the uninitiated, Lean Production is a set of princples and tools first conceptualized by Toyota in the 1980s (a.k.a. Just-In-Time, Kanban system) to support its auto manufacturing business. Mary and Tom Poppendieck subsequently leveraged the ideas and defined a set of seven lean principles for software development:

    1. Eliminate Waste – Spend time only on what adds real customer value.
    2. Amplify Learning – When you have tough problems, increase feedback.
    3. Empower the Team – Let the people who add value use their full potential.
    4. Deliver as Fast as Possible – Deliver value to customers as soon as they ask for it.
    5. See the Whole – Beware of the temptation to optimize parts at the expense of the whole.
    6. Build Integrity In – Don’t try to tack on integrity after the fact – build it in.
    7. Decide as Late as Possible – Keep your options open as long as practical, but no longer.

    The question I would like to pose is “how close does Google’s development practices match Lean software development?”. In addition, what does Google do that goes beyond Lean Software Development? So let me then go through each lean principle and see what Google has to say about it.

    Eliminate Waste – Google teams are small, agile engineering teams
    3-person units, that don’t have departments and is co-located and sit next to each other and also with Project Manager. Documentation is very sparse and only what is needed is a Product Requirements Document. Here we see that Google strives to elimate waste by reducing paperwork, the need to wait for things to happen and the motion required to find an answer. That is when you “pack them in”, you make communication eaiser, there is “no telephone tag, no e-mail delay, no waiting for a reply” says Schmidt. Schmidt adds that “sitting next to a knowledgeable employee was an incredibly effective educational experience”. Schmidt subsrcibes to Druker’s “strip away everything that gets in their way.”

    Amplify Learning – Google applies both statistical analysis and feedback on the iterative delivery of their products. The attempt is to make good products better over time. Information sharing in Google is the norm, according to Joe Beda:

    “The intranet in Google is super transparent. Teams are actively encouraged to share the most intimate details of their projects with the rest of the company. This happens through tech talks, design docs, lunch table conversations, etc. There is, by and large, only one code base at Google.

    Schmidt gives some light on how Google development enables synchronization:

    Make coordination easy. Because all members of a team are within a few feet of one another, it is relatively easy to coordinate projects. In addition to physical proximity, each Googler e-mails a snippet once a week to his work group describing what he has done in the last week. This gives everyone an easy way to track what everyone else is up to, making it much easier to monitor progress and synchronize work flow.

    Empower the Team – Every member of the team is empowered in that they can spend up to 20 percent of their time on a project of their choice. The company maintains “an ideas mailing list: a companywide suggestion box where people can post ideas ranging from parking procedures to the next killer app.” The ideas mailing list also provides “the ability for everyone to comment on and rate ideas, permitting the best ideas to percolate to the top”. In addition there is a brainstorming session every week. They maintain collaborative workspaces (ex. Wiki) on the web with editable pages for further exploartion of new ideas.

    It is distinctly unique in Google practice that decision making is done via consensus. According to Schmidt:

    Modern corporate mythology has the unique decision maker as hero. We adhere to the view that the ‘many are smarter than the few,’ and solicit a broad base of views before reaching any decision. At Google, the role of the manager is that of an aggregator of viewpoints, not the dictator of decisions. Building a consensus sometimes takes longer, but always produces a more committed team and better decisions.

    Deliver as Fast as Possible – The prefer delivering a good solution quickly rather the best solution ater. It is not atypical for Google to release services in Beta to quickly test out feedback from users.

    See the Whole – According to Joe Beda:

    When two teams are doing similar things, people start with the assumption that they must have their reasons and that the situation will be worked out in time. There isn’t a huge push to over optimize and have only one solution for each problem. This means that there isn’t an adversarial relationship between teams that can lead to long standing animosities and information hiding.

    Google uses data to drive decision making. Schmidt writes:

    At Google, almost every decision is based on quantitative analysis. We’ve built systems to manage information, not only on the Internet at large, but also internally. We have dozens of analysts who plow through the data, analyze performance metrics and plot trends to keep us as up to date as possible. We have a raft of online “dashboards” for every business we work in that provide up-to-the-minute snapshots of where we are.

    Build Integrity In – Google’s process is based on User-centered design. It is focused on quality and asks: “What does user really care about?”. Through experimentation via public sites they can assess if users respond well. In fact, Larry Page has an interesting philosophy “There is xc2x93no such thing as a successful failure; if it is useful to people, later we can make revenue from it in a logical way.” In short there is an emphasis on providing true value to a user first, and sometime along the line a monetization team comes along to figure out the details.

    Decide as Late as Possible – Eric Schmidt describes Google’s process as “a late binding decision-making process”. The fact that for Google projects, the issue of monetization is addressed extremely late in the game starkly differentiates itself from just about everyone else. This late binding strategy and its $100+ market cap should just give just about everyone else in the industry the shivers.

    Google’s software development process from the perspective of Lean Software development principles show nothing that is out of the ordinary. What is not ordinary is the massive scale that they are applying these principles. What Google apparently seems to understand that monetizing the internet is in the realm of “Unforseen Uncertainty”. Just as Google has leverage a massive number of low-cost off the shelf servers to power their search engine, they are equally leveraging a massive number of lean software development efforts in support of their search for monetization opportunities.

    Web 2.0 companies like Flickr have shown how little it takes to build a business, Google replicates this effort to who knows how many yet-to-be-announced concurrent efforts. The late binding strategy to monetization is what makes the process truly unique. Unlike the venture funds which base their funding on potential monetary gain, google bases their funding on a collective decision that does not necesarilly involve profit in the equation. This puts the venture community in a conundrum. Web 2.0 business are at a scale that do not need venture funding. Google is investing heavily in such efforts that appear to have no-monetary upside. These are clearly blind spots that one can’t afford the risk of not seeing.

    Eric Schmidt summarizes it best when he said:

    Our goal is to get more at bats per unit of time and effort than anyone else in the world.


    Share the article!

    How Web 2.0 supports Lean Production

    Share the article!

    The cognoscenti are abuzz about migrating Web 2.0 into the enterprise. It’s certainly nice to speculate about considering that it could be quite lucrative for the first mover. Transitioning almost anything from the mass consumer market to the enterprise market surely would mint a couple of new millionaires. However, I’ve got this nagging need for more precise details on why it may work.

    Andrew McAfee attempts to make his case, however he struggles with the argument that many enterprises tend to squelch collaboration. He writes:

    The third reason to be pessimistic about Enterprise 2.0, however, is also culture, especially as itxe2x80x99s defined and shaped over time by business leaders. If these leaders signal that they really donxe2x80x99t want open, freeform, and emergent collaboration, they really wonxe2x80x99t get it. I predict that the diffusion of these tools is going to sharpen differences among companies as some work to foster the new styles, modes, and practices of collaboration and others work (subtly or overtly) to squelch them.

    Ross Mayfield side steps the argument entirely in most of his piece, and submits a strategy on how to bring acceptance of social sofware into the enterprise. However he is forced to re-address the issue when he writes:

    Adoption isn’t a goal in and of itself. Lots of people use email an awful lot, but that doesn’t mean that it’s being used well. Think about what your ultimate aims are; make them discrete, measurable and attainable. Go for ‘reducing occupational spam’, for example, rather than ‘improve communications’. Measure your email usage before you start, monitor it whilst you adopt, and report back regularly so that people can see the progress that they are collectively making.

    Which in my mind is simply not convincing enough. ‘Reducing occupational spam’ really means reducing waste. That’s why the ideas of Lean Production serve as a perfect framework and spring-board for defining the rational for migrating Web 2.0 in the enterprise. McAfee and Mayfield both however are in consensus in that ultimately to make it work one needs to redefine the culture.

    To get a proper context we need a rough definition of Web 2.0. Sure, there are a multitude of competing definitions. But, I’m not going to split hairs and endlessly debate it here, so I’ll just rely on a trusted source. Don Hinchcliffe who devotes his entire blog on Web 2.0, provides this set of key aspects of Web 2.0:

    1. The Web and all its connected devices as one global platform of reusable services and data
    2. Data consumption and remixing from all sources, particularly user generated data
    3. Continuous and seamless update of software and data, often very rapidly
    4. Rich and interactive user interfaces
    5. Architecture of participation that encourages user contribution

    What do I mean by Lean Production? Lean Production is a set of princples and tools first conceptualized by Toyota in the 1980s (a.k.a. Just-In-Time, Kanban system) to support its auto manufacturing business. Mary and Tom Poppendieck subsequently leveraged the ideas and defined a set of seven lean principles for software development:

    1. Eliminate Waste – Spend time only on what adds real customer value.
    2. Amplify Learning – When you have tough problems, increase feedback.
    3. Decide as Late as Possible – Keep your options open as long as practical, but no longer.
    4. Deliver as Fast as Possible – Deliver value to customers as soon as they ask for it.
    5. Empower the Team – Let the people who add value use their full potential.
    6. Build Integrity In – Don’t try to tack on integrity after the fact – build it in.
    7. See the Whole – Beware of the temptation to optimize parts at the expense of the whole.

    What one learns from Lean Software Development (i.e. LSD) can be easily mapped to other knowledge creation activities. LSD not only provides 7 guiding principles but goes further be providing some 22 recommendations, called tools, on how to achieve them. Rather than exhaustively going through the 22 tools, I’ll just skip ahead to show how Web 2.0 tools support the 7 Lean principles. A more extensive treatment may possibly be an entire book in itself!

    Eliminate Waste – The common web platform eliminates the waste of maintaining heterogenous architectures. It removes the waste of re-inventing services by exploiting what’s already available in the Web. It reduces the waste in training costs by providing simple intuitive user interfaces. It reduces motion my providing more accesibility and findability of creators and their knowledge. Its asynchronous collaboration capabilies reducing wait required by synchronization.

    Amplify Learning – Wikis improve feedback by allowing members to directly edit knowledge as it evolves. Rapid iterations are the norm when knowledge is instantly made available at the time of edit, rather than going through a long release and approval process. Synchronization happens all the time, as wikis support collaborative editing and merging.

    Decide as Late as Possible – Wikis encourage a breadth first approach to collaboration by making it easy to create and refactor pages. Concurrent development is encouraged as all members of a team share each others partial and ongoing work. The Architecture of Participation in general amplifies the ability to do concurrent development.

    Deliver as Fast as Possible – The continuous and seamless update of software and data supports this principle directly. To achieve fast delivery of complex products, self organization is necessary and methods for distributed signaling and commiment are necesarry for effective coordination. Web 2.0 based calendaring applications, lightweight “to do” applications, issue trackers, trouble ticketing apps are some examples to support these distributed coordination requirements. More sophisticated Web 2.0 will need to be developed to address this.

    Empower the Team – Wikis, Blogs, Collaborative search, Voting systems are all tools to empower the individual. Architectures of Participation provides the mechanism for contribution. Rather than muffle voices by virtue of organization hierarchy, Web 2.0 applications encourages the widest range of contribution.

    Build Integrity In – Blogs provide a natural form of ensuring integrity. Web 2.0 peer review capabilites ensure that knowledge errors are quickly caught and fixed. Bilateral communication deemphasizes hierachy and diminishes the weight of assigned to knowledge as a virtue of position. The fluid nature of documents allow for easier refactoring and open opportunities for simpler, clearer, less repititous and suitable communication.

    See the Whole – Mashups, remixes and the open nature of data allows the easy creation of effective visualizations of key statistics and their correlations. RSS feeds allow for visibility of a project from the time perspective.

    This is my inital attempt at aligning Web 2.0 and Lean Production. It almost certainly matches like a glove, but there are indeed some slight gaps. These gaps however are glaring opportunities that need to be filled. Devising innovative solutions to these will lead to an effective suite of tools that would be extremely useful for knowledge based enterprises in general, and lean enterprises in particular. I hope this piece spurs further discussion and can lead to some innovative ideas.


    Share the article!