The Long Tail of Recombinant Components

Joe Kraus has a new blog entry on the " Long tail of Software. Millions of Markets of Dozens". The analogy is conceptually and visually compelling, however, the billion dollar question is how? I wrote earlier about Joe Kraus's approach to this problem in the conceptual underpinnings of JotSpot. It's a first stab at trying to get some kind of glimpse of understanding.

One key to a solution is the need for a recombinant architecture. That is a framework that will allow rapid assembly of components like we have never seen before. After all, if such a system exists today, then obviously then we wouldn't have situation of millions of unserved and disenfranchised markets.

So what are the elements of rapid assembly or rather recombinant computing? Fortunately, there's been some research on this matter. The Obje project from Xerox Parc has defined a minimal set of architectural attributes that enable dynamic interoperability (i.e. interoperability outside of static standards). They list these "Meta -Interfaces" as follows:

Aggregation/Discovery

One of the basic interfaces allows a component to represent a collection if other components. For example, a file system component is an aggregation of the components that are its directories and files. The aggregation interface is also used for discovery, i.e., to find other components on the network.

Once discovered, another basic agreement is how to negotiate a connection. The discovery component includes the ability for third party Obje components to serve as bridges. This means that two devices can interoperate without direct physical connectivity if they can find a device with connectivity to both or a chain of devices that can connect them.

Data transfer

The Obje platform's basic view of interoperation is (1) send data, (2) do something to data, and (3) sometimes, return a result. It allows data transfer between components to occur in ways that are not only independent of the underlying wire protocol, but also independent of the format of the data being transferred.

In the Obje platform, components that are capable of sending data provide "custom objects" that extend the behavior of potential receivers in specific ways. While the Obje platform defines the interface common to all custom objects, the implementations of these objects can leverage mobile code to provide dynamic, runtime extensibility to new wire protocols and data formats.

User Interface/Control

Critical to successful interoperation is that the "something" that is done to the data depends on what component it is sent to. The control meta-interface allows the user to provide application semantics by directing the interoperation.

Instead of having different agreements for showing a picture on a display or printing a page on a printer, the Obje platform needs only the data transfer agreement. The user making the connection provides the semantics by sending the data, and adds additional information such as size of print or number of copies. This is key to allowing end-user assembly of solutions.

Metadata/Context

The final Obje component interface returns a custom object that provides contextual metadata about a component. The context custom object returned by this interface provides access to a set of attributes describing the component that returns it: name, location, administrative domain, and so on. The goal of this interface is primarily to allow sense making by users, and secondarily to allow programs to use metadata in their interactions.

At first glance one would recognize the attributes enumerated above as being similar to technologies that have come before it (i.e. JINI, uPnP etc). However the key distinguishing feature of the work is the empahsis on humans being able to define the semantics:

Put another way, the programmatic interfaces define the syntax of interaction, while leaving it to humans to impose semantics. It is the stabilization of the programmatic interfaces that enable network effects; humans can, presumably, “understand” new components in their environment, and when and whether to use them, much more easily than machines can (and, it should be noted, without the need to be recompiled). This semantic arbitration happens at the user level, rather than at the developer level as in traditional component frameworks.

John Udell has an interesting perspective that relates this with Rich GUI interfaces:

In one crucial way, the rich GUI is tragically disadvantaged with respect to its poor browser cousin. Trying to sort out a permissions problem with IIS 6, I clicked a Help button and landed on a Web page. The page could only describe the tree-navigation procedure required to find the tabbed dialog box where I could address the problem. It could not link to that dialog box. This is nuts when you stop and think about it. Documentation of GUI software needs pages of screenshots and text to describe procedures that, on the Web, are encapsulated in links that can be published, bookmarked, and e-mailed. A GUI that doesn't embrace linking can never be truly rich.

In otherwords, the components of the interface should be usable in contexts outside of its original intent. To build a Recombinant framework requires that user interfaces embed semantic components that allow composition outside their container. That would therefore imply that software composition is done not at the server, but rather at the client.

I wrote about this in "Integration at the Glass and the 80/20 point". One recurring pattern of Loose Coupling is that of late binding. Binding the solution at the client (i.e. Browser) is a late a binding as you can possibly achieve. The important rule of thumb then is to "always allow composition at the client side". The Achilles Heel of GUIs is that the various components are frozen in code, allowing users to make only but the most basic of customizations.

The last sentence also hints to another requirement for recombinant computing. That is you customize something that already exists, its pretty basic. One cannot expect users to build applications from scratch, one should start from an existing feature rich application and permit users the capabilities to make customizations easily. This is one of the suggestions I think the Java community should take seriously. The Eclipse architecture takes a step in this direction, by providing an architecture for participating it has spawned hundreds of plugins. However, Eclipse's success is not only based on its modular architecture, but because it provides impressive functionality to begin with.

Therefore, a web based framework without extensive functionality to customize from is simply DOA. A web application without public extension points can't be improved by the community. Finally, an application without easy user customization requires too much of an invesment to be used for situated softare (that's where you find the long tail). So, when Yahoo introduced its search API, it had already a rich platform to begin with. Now rather than choosing a customization methodology for software developers, it chose a methodology that a reasonably skilled web user could exploit. That is , API usage is via URL construction. Something you should ponder on for your next customization API.