CTO Articles
Published in IT World
January 16, 2007
Chaos and order in information systems
The word 'chaos' is an interesting example of an old word that is having its meaning reshaped by the, um, chaotic forces of change that surround us all.
In speech and in informal writing we generally use the term in the negative sense of an orderless bedlam of some sort. Few IT personnel would be happy to hear their systems described as chaotic. However, to a physicist, a chaotic system is not senseless bedlam at all. On the contrary, chaotic systems can be very simple systems that just happen to look as if they are utterly senseless when observed casually. The apparent disorder disappears once you look at the (often simple) feedback-based equations that drive the system.
Thankfully, it often turns out that extremely intricate patterns can be explained with very simple underlying models. Furthermore, extreme diversity of behavior through time can often be explained by nothing more complicated than extreme sensitivity to the initial conditions that launched the system. Therein lies the physics/math interpretation of the word 'chaos'[1].
I try to keep this underlying simplicity in mind when, on a day to day basis, I fight the forces of Bedlam that besiege the information systems I'm involved in. Chaos - in the colloquial sense - often appears to be the normal state of affairs. Order - if it exists - is rarely apparent and tracking it down requires faith and determination in equal measure.
Take information for example. For a long time, our concept of order in information has been tied to the physical world. We speak negatively about information that is 'all over the place' or 'scattered'. We have a centuries old tradition of bringing order to information by physically 'gathering' the stuff into the same physical place: books and libraries being classic examples. We also have a centuries old tradition of adding value to information by the simple expedient of gathering disparate pieces and putting them in the same place. Reference works being the classic example.
Alongside this centuries-old tradition there is a shorter tradition of using computer systems to bring order to information. For a long time, this tradition mimicked the physical world view by gathering disparate information resources in to some central place. For example, putting them together on a hard disk or together on a CD-ROM.
That centuries old tradition is dying fast. In this area at least, there can be no doubt that the impact of the Internet is very dramatic. On the Internet, it is not bad that information is 'all over the place' or 'scattered'. The magic of search engines means that information can be gathered together in a much more powerful way than is possible in the physical world, without bringing all the information together physically. Search engines bring order to the apparent chaos of distributed information. If we want to use the word 'chaotic' in its colloquial sense when speaking about information, we need to apply it to aspects other than the physical location of the information...
So what is that next frontier of apparent chaos that needs to be knocked down and replaced by a powerful new order? Some say it is the Semantic Web - making the meaning of the information explicit. Some say that once we do that, search engines can take us to a whole new level of productivity.
It sounds great. It really does. However, I'm not so sure. I have difficulties with the simple sounding notion of making the meaning of information explicit but let us leave that aside for the moment.
What was the big false start of electronic information management? I would argue it was the notion that replicating the physical world by gathering electronic information into the same place - disk/CDROM etc. - was the way to go. History has shown that the search engine approach blows the old model clean out of the water. There are many ways of dealing with apparent chaos and sometimes the less obvious ones are the most powerful.
Will something utterly different from the search engine concept come along for the next big reassessment of the relationship between information chaos and information order? Probably.
[1] http://www.onelook.com/?w=chaos&ls=a