Business Intelligence Network - The Vision for BI and Beyond

Almost everything about the business intelligence industry has changed in the past two decades.

Moore’s Law has driven quantum leaps in the processing power of software and hardware systems. Organizations have become larger and more complex. The demands for up-to-the-minute access to data have intensified. And vendors have refined their business intelligence (BI) solutions to account for the constantly evolving needs of their customers.

Despite these evolutionary shifts, what has not changed is the underlying goal of the business intelligence community: to facilitate decision-making by turning unstructured data into usable data.

The IBM scientist Hans Peter Luhn, in an October 1958 IBM Journal paper that is acknowledged as coining the term “business intelligence,” described the concept in terms that ring true nearly 50 years later. “Ideally, an automatic system is needed which can accept information in its original form, disseminate the data promptly to the proper places and furnish information on demand,” wrote Luhn.

He noted that the objective of a BI system “is to supply suitable information to support specific activities carried out by individuals, groups, departments, divisions, or even larger units. To this end the system concerns itself with the admission or acquisition of new information, its dissemination, storage, retrieval and transmittal to the action points it serves.”

The need for systems that can tackle these complex tasks is especially urgent today, as organizations increasingly rely on multilayered data analysis as a vital element of their decision-making process. ERP, SCM, CRM, and EPM applications automatically gather massive volumes of information about both internal and external business processes. The data pool has also increased through the use of websites, e-mail, blogs, XML, and enterprise storage systems.

But rather than illuminate the state of our businesses, this barrage of data often confuses the picture by delivering too much raw information too fast. It’s as if the data comes at us in a fire hose, and our biggest challenge is to channel this seemingly overwhelming flow into productive energy.

A Visual Approach

In my view, the most effective way to tame the data beast is through interactive visualization. We are clearly at the limits of spreadsheets and tabular reports. Utilizing visual metaphors gives context to many dimensions of the data – it provides a “narrative” to the data.

By empowering knowledge workers with visual tools and hands-on access to data, we allow them to find patterns, distributions, correlations, or anomalies across multiple data types. Users can select data elements, filters, highlighting, and display options to change data perspectives – from high-level overviews down to the lowest levels of detail. The visual cues inherent in the software enable a deep exploration and understanding of the dataset at hand.

The challenge that software developers face is to graphically represent data so that key people within an organization have the right amount of detail at the right time. While it’s important for IT leaders to have oversight of a company’s data, a truly holistic BI solution puts knowledge and analytic resources into the hands of those who are in the best position to use it. These employees are not necessarily statisticians or programmers, but everyday managers and executives who need to generate their own data analysis to do their jobs.

To deliver maximum value to an organization, visualization tools need to scale with the needs of individual users and enhance productivity across functional teams. This is achievable, but only if the developer has a crystal-clear understanding of the total data schema and understand how it will show up in the views of employees with different deadlines and priorities. So, if a user sees that something is delayed, the next logical questions might be: How late is it? How long has it been delayed? Why is it delayed?

I like to refer to this visual approach as “the last 18 inches” – that is, the distance between the computer screen and the human brain. Focusing on this last and most vital link of the data chain is essential to getting value out of the massive investments in IT infrastructure that companies have made. After all, if we can’t get information to our brains in a meaningful way, then we’re just drowning in numbers.

A Multidimensional View

We also need to move beyond a two-dimensional plane, which is where most data management solutions tend to get stuck. Look at it this way: if a company’s issues are invariably multidimensional, shouldn’t the technological solution that addresses them reflect this reality and provide more nuance than a spreadsheet with X and Y axes?

The volume and complexity of business data has outpaced simplistic charts and graphs. More than ever before, business owners need intelligent systems that combine data from multiple sources into pictures that update dynamically as users apply filters or add new information. These systems need to work across the entire enterprise, providing a comprehensive view of business operations.

In order to build such systems, developers must assess their clients’ business needs and map out an all-encompassing approach to data management.

All too often, this design burden falls on the IT and software development teams, which usually excel at data management but are seldom in a position to gauge the company’s needs for business process analysis, reporting requirements and visual interaction. Conversely, functional teams may see the part of the picture that’s relevant to them without necessarily appreciating how it relates to the broader scope. In other words, they may see the trees but not the forest.

Only by pooling the collective wisdom of a company’s usage, IT, and development experts can a third-party developer conceptualize a solution that enhances decision-making and produces results.

To frame it in an architectural metaphor – something I’m admittedly prone to doing, since I was trained as a building architect – we need to formulate a clear picture of what a structure will look like before laying the foundation and starting construction. The dimensions, perspectives, elevations, and overall topography must be embodied in a blueprint of the end-users’ experience.

Harnessing Brain Power + Machine Power

The beauty of a visually oriented data system is that it harnesses the strengths of the human brain and the processing power of computers to realize a solution that is greater than the sum of its parts. Think about the things that humans are particularly good at. For example, the brain can easily recognize faces and patterns, but computers have a lot of trouble doing this. On the other hand, computers have data processing and memory storage capacities way beyond that of the human brain.

By combining those pieces, we begin to envision an information ecosystem that delivers continuity between the question and the answer – almost as if we were facilitating a conversation with a database. And, if we succeed in making this conversation fluid, we reduce the number of ad-hoc queries and collapse the cycles so that users get an instant picture of what’s going on in their business.

Along with a visual approach to business intelligence systems, we also need to emphasize transparency at the place where data and analytics intersect. Whenever there is data to analyze, there is almost always a visual corollary. For example, the equation “distance = rate x time” is usually drawn as a triangle.

With more complex data sets, the calculations are more involved and the visualizations more challenging to achieve. Yet, it is precisely in these cases that advanced visual tools can make the biggest difference. Most business users might not be able to write the algorithm that pinpoints outliers in a certain set of data, but they can identify the outlier from a visualization. An intelligent system enables this more intuitive connection between data and insight.

Interactivity

Another imperative of the next generation of business intelligence software is interactivity. Today’s workers expect not only to have access to data, but also to be able to manipulate it. People have become accustomed to Web tools that enable mashups, like Yahoo Widgets or Google maps that show where all the coffee shops are relative to one’s route to work. We need to apply these concepts to enterprise-level data by enabling a sort of “enterprise mashup.” The feedback loop between the user and the data system needs to be direct and immediate, so that the system “listens” to the user and responds right away.

In most companies, the speed of the decision-making cycle is directly related to the availability of information. And yet, most business tools lack the visual feedback to deliver insight into the data on which decisions depend.

The technologies that enable this visual, interactive approach are now within reach of any user with a Web browser, thanks to faster desktop PCs and software applications like Adobe Flash and AJAX scripting. Companies that successfully deploy these tools in the context of a visually intuitive data management system will give business users the visibility they need to make better decisions.

Picture a scenario in which executives at a pharmaceutical company are meeting to decide on a multimillion dollar marketing campaign for a product that is nearing the end of its R&D cycle. Typically, the meeting would produce a set of proposals that each executive would later test against available data, using the resources of the IT department to run queries and reports. Days or weeks later, the same group might reconvene to refine their proposals based on the data analysis that each executive separately brought to the table. This process could easily go on for several iterations until a consensus is reached.

As a way to circumvent this time-consuming process, employees often resort to other dysfunctional strategies like working around the technical infrastructure or flying blind without the benefit of data analysis. While these are common tactics in today’s business world, they lead in the opposite direction of an enlightened solution.

Imagine, on the other hand, that each executive in the pharmaceutical company meeting has access to a business intelligence system that provides detailed, customizable, and easy-to-comprehend views into their business. As soon as an idea is put forward, the team can run quick models and test their suppositions and hypotheses on the spot. The system gives them the “rollup” overview but also allows them to drill down to details that give confidence to decision-making. Best of all, because the feedback is visual, it can be easily shared by everyone in the room without the need to translate, explain, rationalize, or contextualize the information. It’s all right there in front of everyone’s eyes.

This kind of interactivity goes far beyond the “report-based” architecture that so many current BI solutions employ. Given the technological strides the software industry has made in recent years, report-driven methodologies are clunky and disruptive to a company’s response time. Today’s fast-moving world screams for a more dynamic and interactive approach – one that visually and intuitively closes the gap between end user and database, between knowledge worker and IT professional, between input and output, between the problem and the solution.

The poet T.S. Eliot articulated this dilemma with a pair of provocative questions that are relevant to the business intelligence community: “Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?”

If we are to succeed in helping our business customers make more informed and timely decisions, we need to provide them with systems that turn information into knowledge and knowledge into wisdom.