Dealing with complexity

Ok folks here goes. The next big thing according to those in the know (analysts) is CEP and ESP on an ESB or SOA for real-time business intelligence or BAM. I thought I would try and cram as many three letter acronyms into one sentence as possible to show how ridiculous analysts' obsession with three letter acronyms has become.

By now you are probably thinking here we go again. First it was ESB (enterprise service bus),then SOA (service-oriented architecture), now its CEP (complex event processing). As one journalist from Information Age quipped recently at a Progress Software press event about event processing (an umbrella term used to talk about CEP and ESP-event stream processing),'Everyone is bored of SOA, we've heard it all before,' and by the way is anyone actually doing it? So is CEP or ESP just another three letter acronym destined for the same fate?

Well, unless you design trading algorithms or are a logistics company tracking goods throughout the supply chain, you probably have not heard of CEP or ESP, therefore your boredom threshold is unlikely to have been maxed out yet. And as for whether people are actually doing it, well, the short answer, very few. According to Mark Palmer, general manager, Apama division, Progress Software, the Event Driven Architecture market is currently worth $30 to $50 million, small fry by software standards.

A lot of these three letter acronymns tend to be the 'love child' of computer boffins who spend most of their lives in laboratories dreaming up great whizz bang technologies that rarely find it into every day application. You could say event processing is one of these technologies having been pioneered by boffins at Stanford and Cambridge universities. However, Phil Howard, research director at Bloor Research, believes that event processing will be widely used, but that adoption will be gradual. After all, event processing has yet to reach the peak of its hype cycle on Gartner's adoption curve before it descends into the 'trough of disillusionment'.

Having said that, Dr Giles Nelson, director of technology, Progress Software and co-founder of Apama, a Cambridge UK startup (bought by Progress) that developed one of the first CEP engines, did a good job of explaining why you may want to think about adopting event processing some time in the not too distant future, particularly if you are a business that needs to make rapid decisions on streaming data (tick prices, for example) that is "constantly changing". According to Dr Giles, putting the complexities of the technology itself aside, the nuts and bolts of event processing "is about being able to understand information in real time."

This information could be coming from multiple sources both within and outside the company, RFID tags for example on cases of goods. But Nelson made the clear distinction between business intelligence software, which tends to be based on historical data and doesn't allow someone to act on that data in real time, and ESP. Unlike conventional data management models, where data is indexed and stored and then request/response queries are made on it, if a company needs to act on information in real time, Nelson says there is no time to index data. "That is why SQL is unsuitable for this," he adds. Also vendors like Progress Software tend to favour object oriented databases as opposed to relational for ESP.

In a nutshell, ESP says Nelson, is about storing queries and then flowing data (both historical and real-time) across them. Great you say, but what would I use it for? Well, it has long been used by algorithmic traders who want to test out VWAP and other trading strategies on real-time and historical data. It could also find application under regulations such as MiFID in Europe and RegNMS in the US, where the emphasis is on achieving best price for clients and smart order routing to the cheapest execution venue. As best price is a constantly changing variable, according to Progress it is suited to ESP.