<-- Back to full color view

Technical Trends Shaping Business Intelligence

Originally published March 10, 2009

There is no better example of the power of technology to shape behavior than business intelligence (BI). The use of business intelligence to inform, analyze and enable has been the driving force behind the use and expansion of this powerful tool for managing information. Technology, however, never stands still, and there are many interesting trends, at all layers of the BI technology stack, that will shape business intelligence in the near future.

Trends in Data Management

The demands on data have always been affected by its physical storage. Capacity, speed, availability and structure are all factors that affect performance and have demanded the attention of developers, both for applications and for business intelligence. However, there are several trends that may alter the BI landscape:
  • Forms of data organization: The relational model of data organization has evolved for decades and has come to dominate the DBMS market. New forms of data organization are emerging, spurred forward by demands to provide high performance access to high volumes of data in military and search engine applications, that alter relational concepts, particularly its focus on table rows.

    Through the process of data normalization, data elements are grouped as columns in a table where each row of data element instances is identified by a key. Row-oriented thinking works well for transaction processing but connecting several rows from multiple tables for reporting and analysis can create performance problems.

    Dimensional data structures like star and snowflake schemas were developed to address the problem of data retrieval and analytics. Columnar data structures were also developed to make data retrieval easier. All of these optimize data retrieval for reporting and analytics over the performance of transaction processing. In fact, retrieval-based data organization is not suited for transaction processing.

    All of these data organization forms are what I call “architecture based.” They require a process of data structure design, which is called data architecture, before reports or analytics can be developed. Adding new data elements, analytics, reporting needs, data hierarchies and other elements cause the architecture and data structure to be altered before they can be made available that incurs a time delay between definition and implementation.

    Self-adapting data organization is next. Data is used much more than it is “transacted” (I think of data as transacted once, read over and over) and how data is used continually changes. Data structures derived from the use of data and analytics in reports, queries and screens don’t require data design, are “architecture-free” and support all forms of usage including transaction processing. Such usage-based data specification can incorporate new data elements, analytics, reporting needs, data hierarchies and other elements without requiring changes to existing programs or data structures.

  • Forms of data storage: Data can be stored on media or in memory. While data storage media continues to improve its capacity and speed, it always pales in data retrieval performance compared to data held in memory. Media will always be useful for archiving data; but as memory gets cheaper, it will no longer be storage for data used on a day-to-day basis.

    The continued increase in RAM chip capacity and associated cost reductions will make memory-based data storage a reality. Memory-based data storage is a technology already deployed in several BI platforms and planned for many more. A memory-based DBMS is under development as a research project to investigate distributed main memory transaction processing. A memory-based DBMS can offer simplicity as well as speed – existing DBMS systems provide performance optimization and data control that may be simplified because of the speed of in-memory data. While not yet commercially available, research was the source of a popular, commercially available columnar DBMS.
A self-adapting data organization combined with a memory-based DBMS will create a data management foundation that will alter the nature of analytics and information management across the enterprise.

Trends in Analytics

Analytic technology is well-rounded and has a wide range of uses. These range from statistical and mathematical processing to database functions to BI metrics and key performance indicators (KPIs). This wide range of analytic capabilities reflects the broad need for analytics in organizations today. Unfortunately, it also reflects dispersal of analytics into applications, stored procedures, analytic software, including BI, and spreadsheets. This dispersal breeds lack of a directory of defined analytics, difficulty in accessing analytics used in the organization and inconsistency in analytic results.

While BI vendors have components for managing analytics, these apply to a proprietary BI environment and often are added to, and require additional investment in, the core BI product. The add-on and proprietary nature of BI analytics management does not solve the problems of managing analytics across the organization. Building upon the data management foundation described above is how these problems will be resolved with improved:
  • Metric definition: Metrics are defined within BI environments, but outside these environments they remain unknown and undocumented. Usage-based data specification requires definitions as a part of data management for both analytic and transaction processing uses.

    Creating the definition as a specification ensures that the metric is understood, can be created and can be reused where required. This is the basis for a directory of defined, consistent, and reusable analytics across the organization.

  • Process management: Analytics are a critical component in process management. The trend here continues to emphasize the integration of business intelligence process definition and measurement. This integration has been left to the organization when “best-of-breed” products are chosen for process management and business intelligence.

    Aligning data usage and the analytics, reports, queries and screens with the process(es) and process step(s) where they are used creates a powerful analytic management capability. If the process management capability robustly maps the logical process definition to its real-world or physical implementation, the power and use of analytics will be extended significantly.

  • Metric identification: When analytics are combined with process management, they create an opportunity for automatically identifying additional, previously undefined metrics. Processes begin and end. They have triggering events and final dispositions. They have elements that can be defined as basic performance measures for a process and for process steps. These measures, while basic, can be defined automatically.

    Additionally, the binding of data to a process and a representation (report, screen and so forth) creates opportunities for automated data mining. A data crawler, analogous to a web crawler, could scour data using these bindings to identify correlations and their metrics. With a memory-based DBMS, a data crawler can detect real-time changes and trends in these correlations and metrics.

    These suggestions for automated metric identification do not displace the need for skilled practitioners to perform sophisticated analyses. They will provide basic metrics allowing practitioners to focus on more challenging problems.
While organizations have made progress with their basic analytics, control of metric definitions and uses, process management, and data crawlers generally are under- or non-utilized.

Trends in Information Management

In order to leverage self-adapting data organization, a memory-based DBMS, metric definitions and control, process management and data crawlers, organizations will need to develop a strong capability in information management. In considering the above discussion, it is hard to imagine being able to take advantage of any of these trends without an information management capability focused on:
  • Definitions and rules: The nature of a self-adapting data organization, a memory-based DBMS, metric control, process management and data crawlers rely on consistent definitions and rules. A fundamental change will be making definitions and rules the basis for information, its storage, and its use. In other words, definitions and rules will be the basis for software elements, not their documentation.

  • Centralized management: Tying definitions and rules to processes means that they are cross-functional and cross-departmental. Ensuring consistency in definition, use and presentation requires centralized responsibility and management. This also requires a greater integration of data, processes and applications than exists today.
These capabilities need to integrate with usage-based data specification, process management, and all definitions and rules.

How likely is it that self-adapting data organization, a memory-based DBMS, metric definitions and control, process management, data crawlers and centralized information management will come into existence? I expect to see these commercially available within three to five years.

SOURCE: Technical Trends Shaping Business Intelligence

  • Richard SkriletzRichard Skriletz

    Richard is a manager and management consultant with more than 35 years experience working in large corporate and start-up environments. His professional focus is on the strategic application of information technology to improving operational performance, managing organizational and technical change, and optimizing business effectiveness. Richard is a Global Managing Principal with RCG Global Services and CEO of InfoNovus Technologies. He can be reached via email at Richard.Skriletz@rcggs.com.

    Editor's Note: You will find more articles and resources, and Richard's blog in his BeyeNETWORK Expert Channel. Be sure to visit today!

Recent articles by Richard Skriletz

 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!

 

Copyright 2004 — 2019. Powell Media, LLC. All rights reserved.
BeyeNETWORK™ is a trademark of Powell Media, LLC