I was teaching my seminar on Business Integrated Insight last Thursday, 15 April, in Rome when the ash cloud descended over Europe. I was supposed to fly to Dublin on Saturday, but by Monday morning I had decided to set out by trains, car and ferries to get home. And I did - on Thursday evening, 22 April. Four days travel Rome to Dublin would probably have looked good on the Victorian railways and steamers! In any case, it turned out to be a very nice trip with some built-in thinking time...

In the coverage of the unfolding chaos, the word that seemed to spring most frequently to the mouths of people responsible for managing any aspect of the situation was "unprecedented". A great word if you want to suggest that you shouldn't be blamed in any way for anything that ensued. After all, if it's unprecedented, you have no basis of information from the past to make decisions about what to do now. Or do you?

The truth of the matter is that there were probably enough precedents of most separate aspects of the event to allow reasonable judgments to be made. The problem was that no-one was able to consolidate enough of the disparate information to really make a difference. Focusing just on the issue of getting hordes of stranded passengers across Europe to every point of the compass: Which trains go where? How do they connect? How to connect from a train to a ferry, or a bus to a train? Minimize travel time or cost? Not to mention hotel rooms?

Could the airlines have minimized their regulatory compensation costs if they could work this out? For sure. Could surface travel companies maximize profits by fully utilizing spare capacity (as opposed to raising prices to exorbitant levels!)? Absolutely. Could groups of enterprising travelers get together to make the best plan to get home? Probably. So, there's lots of incentive to make it work. But none of this happened.

Don't get me wrong. I'm not complaining. I'm just pointing out much of the underlying information to answer the above questions not only exists, but is often accessible on the internet. Every stranded traveler with web access spent hours checking options, trying to make online bookings (usually at severely overloaded sites) and then starting all over again as one link in the chain broke. Some succeeded, while others went and queued for hours at ticket offices. 

Operational BI was probably used by some of the more advanced travel companies to track what was going on. Some even managed to schedule additional services to carry extra passengers. Others, such as the Calais-Dover ferries, just stopped taking bookings and went back to the "just turn up at the pier and we'll try to get you onboard as soon as possible" model.

But the really interesting question is this: given that all that information was out there on the web in all its various forms and gory details, how would one go about integrating it in a way that allowed it to be used in an end-to-end travel discovery and booking process?

I'm not expecting the IT industry to have a complete solution any time soon, for a wide variety of political and financial reasons. But a little thought, and none of it very new, suggests we'd need: a common model spanning the information of multiple companies, the ability to link hard and soft information together in a meaningful way, services that act in a fully plug-and-play manner with well-defined interfaces and the ability to mashup a dashboard joining the different steps of the journey together. What I really needed was Business Integrated Insight to get me home!  

Preparing materials for a seminar really forces you to think!  I just finished the slides for my two-day class in Rome next week, and after I got over my need for a strong drink (a celebration, of course), I got to reflect on some of what I had discovered.

Perhaps the most interesting was the amazing changes in the database area that have been happening over the past couple of years.  A combination of hardware advances and software innovations have come together with a recognition that data is no longer what it once was to pose some fundamental questions about how databases should be constructed.

Let's start on the business side - always a good place to start.  Users now think that their internal IT systems should behave like a combination of Google, Facebook and Twitter.  Want an answer to the CEO's question on plummeting sales?  Just do a "search", maybe "call a friend", join it all together and voila!  We have the answer. 

From an information viewpoint, this brings up some very challenging questions about the intersection of soft (aka unstructured) information and hard (structured) data and how one ensures consistency and quality in that set.  IT's problem is no longer just combining hard data from different sources; it's about parsing and qualifying soft information as well.  This is not a truly new problem.  Data modelers have struggled with it for years.  It's the speed with which it needs to be done that causes the problem.

So, what has this got to do with new software and hardware for databases?  Well, the key point is that database thinking has suddenly moved on from strict adherence to the relational paradigm.  The relational model is an extraordinarily structured view of data.  Relational algebra is a very precise tool for querying data.  You need to have a strong understanding of both to make valid queries, but do you really want your users to think that way?  Should you necessarily store the information physically in that model?  When you free yourself of these assumptions, you can begin to think in new ways.  Store the data in columns instead of rows?  Perfect!  A mix of row- and column-oriented data, and maybe some in memory only?  Yes, can do!  And then there's mixing searching (a soft information concept) with querying (a hard data thought) to create a hybrid result.  That's easy too!

And on the edges of the field, there are even more fundamental questions being asked.  Do we need always need consistency in our databases?  Can we do databases without going to disk for the data?  Could we do away with physically modeling the data and just let the computer look after it?  The answers to these questions and more like them are not what you might expect if you've been around the database world for 20 years.  And with those different answers, the overall architecture of your IT systems is suddenly open to dramatic change.

Believe me, the first businesses to adopt some of these approaches are going to gain some extraordinary competitive advantages.  Watch this space!

Business users of information have increasingly high expectations these days.  Not only do they want relevant information, irrespective of source and consistent across multiple sources, but they also want it up to the minute.  Such demands require a new approach to Enterprise IT Architecture, and nothing less!

And yet, the question of how to create such a consistent, integrated information resource almost begs a simplistic answer.  If that's what you want, you must stop creating duplicates of existing information that have to be managed to consistency in ever shorter time windows, and you must eliminate--or, at the very least, substantially reduce--existing data duplication.

The original data warehouse architecture from 1988 showed the way. It proposed a logically single data store--the Business Data Warehouse--modeled at the enterprise level as the consistent and integrated source of all information for decision making. This simplicity was ultimately lost with the emergence of the layered architecture (with multiple data marts fed from an enterprise data warehouse), due to a combination of database performance and enterprise modeling issues.

Nonetheless, the approach remains valid for the current much-expanded needs for integration. First, model all the information according an enterprise-level model and then implement as far as possible in alignment to that model. This is the approach proposed in a new architecture, Business Integrated Insight (BI2), which for the first time gathers all the information of the enterprise, hard and soft; operational, informational and collaborative into a single component called the Business Information Resource...

Read more in my article just published and if you're in the vicinity, come to my two-day seminar in Rome on 15-16 April :-)