CTO Articles
Published in IT World
E-Business in the Enterprise – July 13, 2004
Publish and subscribe - an imminent overnight success after twenty years
By Sean Mc Grath
My first exposure to networked computer systems was in financial trading rooms in the city of London in the height of Margaret Thatcher's Britain in the late Eighties. Never since, have I witnessed such full-on contempt for technologists combined with such total reliance on technology. Those were interesting times.
In those days, I spent a lot of time crawling under trading desks strapping Sun machines together onto Ethernet LANs. The company I worked for developed applications that processed financial data from a number of data feeds and disseminated it to a whole trading room of Sun workstations, all at the same time.
The critical piece here is the phrase "at the same time". Not "one workstation after another as quickly as possible". That would not do. All workstations had to get the data at exactly the same time within the limits of the technology. Some of the most outrageous Prima Donna moments[1] I witnessed in those years came about because one twenty-something trader thought another twenty-something trader was getting feed data every-so-slightly faster than the other.
The systems we implemented used a technique known as publish/subscribe. We had a set of financial data available split into "topics". Any trader interested in receiving a particular topic would subscribe to that topic. Thereafter, we used a combination of a TCP/IP data transmission method known as "IP multicast" and a message type known ad "UDP", to send the data around to the traders.
Fast forward 20 years. Everything has changed and nothing has changed. TCP/IP itself has not changed, Ethernet has not changed. These days, we have a world-wide network based on this technology called the Internet. Local LANs, such as those used in trading rooms, have changed their names to Intranets but otherwise remained fundamentally the same from a communications perspective.
If I look at the current Internet from a vantage point under the trading desk of a US T-Bond trader in the City of London in 1988, one software level change is very striking. The publish/subscribe methods we used back then are not the dominant techniques on todays networks. Instead, most of the great new things such as the Web, are based on a point-to-point communications model.
Clearly, point-to-point and multicast techniques both have their place. Distributing the price of frozen concentrated orange juice to a set of arbitrage traders is a very different activity from distributing e-mail from person to person. Having said that, it appears that more and more compelling use cases for multicast and publish/subscribe are popping up with every passing day.
Information publishing, particularly content syndication technologies such as RSS/Atom would appear to be ripe for a publish/subscribe treatment. Then there is the burgeoning world of broadband and multimedia broadcasting. A clearer case for a multicast approach is hard to imagine.
There are signs that such use cases are refueling innovation in the use of the publish/subscribe/multicast capabilities that have been in the TCP/IP stack now for decades. At a communications level a multicast Internet known as the MBONE[2] is now 10 years old. (I have a rule of thumb that major technology innovations take about 10 years to go mainstream[3]. Based on this rule, the time is right for MBONE.) Then there is the Internet2 project which seeks to make multicast a first class member of the new Internet.[4]
Further up towards software, lots of exciting developments are afoot in the publish/subscribe/multicast space. mod-pubsub[5] is a clever mechanism for doing publish/subscribe within a browser that uses a Javascript library to provide developers with subscription services.
In the Web Services world there are (surprise!) multiple proposals vying for attention to standardize the basics of publish/subscribe web services[6]. I take this as a sure sign that something significant is in the offing in this whole area.
Watch that space.
[1] http://www.comics.com/comics/dilbert/archive/dilbert-20040611.html
[2] http://searchnetworking.techtarget.com/sDefinition/0,,sid7_gci212533,00.html
[3] http://www.itworld.com/nl/xml_prac/10252001/
[4] http://multicast.internet2.edu/
[5] http://www.mod-pubsub.org/
[6] WS-Eventing http://xml.coverpages.org/ni2004-01-07-a.html
WS-Notification http://www-106.ibm.com/developerworks/library/specification/ws-notification/