There is no better example of the power of technology to shape behavior than business intelligence (BI). The use of business intelligence to inform, analyze and enable has been the driving force
behind the use and expansion of this powerful tool for managing information. Technology, however, never stands still, and there are many interesting trends, at all layers of the BI technology stack,
that will shape business intelligence in the near future.
Trends in Data Management
The demands on data have always been affected by its physical storage. Capacity, speed, availability and structure are all factors that affect performance and have demanded the attention of
developers, both for applications and for business intelligence. However, there are several trends that may alter the BI landscape:
Forms of data organization: The relational model of data organization has evolved for decades and has come to dominate the DBMS market. New forms of data
organization are emerging, spurred forward by demands to provide high performance access to high volumes of data in military and search engine applications, that alter relational concepts,
particularly its focus on table rows.
Through the process of data normalization, data elements are grouped as columns in a table where each row of data element instances is identified by a key. Row-oriented thinking works well for
transaction processing but connecting several rows from multiple tables for reporting and analysis can create performance problems.
Dimensional data structures like star and snowflake schemas were developed to address the problem of data retrieval and analytics. Columnar data structures were also developed to make data
retrieval easier. All of these optimize data retrieval for reporting and analytics over the performance of transaction processing. In fact, retrieval-based data organization is not suited for
All of these data organization forms are what I call “architecture based.” They require a process of data structure design, which is called data architecture, before reports or
analytics can be developed. Adding new data elements, analytics, reporting needs, data hierarchies and other elements cause the architecture and data structure to be altered before they can be
made available that incurs a time delay between definition and implementation.
Self-adapting data organization is next. Data is used much more than it is “transacted” (I think of data as transacted once, read over and over) and how data is used continually
changes. Data structures derived from the use of data and analytics in reports, queries and screens don’t require data design, are “architecture-free” and support all forms of
usage including transaction processing. Such usage-based data specification can incorporate new data elements, analytics, reporting needs, data hierarchies and other elements without requiring
changes to existing programs or data structures.
Forms of data storage: Data can be stored on media or in memory. While data storage media continues to improve its capacity and speed, it always pales in
data retrieval performance compared to data held in memory. Media will always be useful for archiving data; but as memory gets cheaper, it will no longer be storage for data used on a day-to-day
The continued increase in RAM chip capacity and associated cost reductions will make memory-based data storage a reality. Memory-based data storage is a technology already deployed in several BI
platforms and planned for many more. A memory-based DBMS is under development as a research project to investigate distributed main
memory transaction processing. A memory-based DBMS can offer simplicity as well as speed – existing DBMS systems provide performance optimization and data control that may be simplified
because of the speed of in-memory data. While not yet commercially available, research was the source of a popular, commercially available columnar DBMS.
A self-adapting data organization combined with a memory-based DBMS will create a data management foundation that will alter the nature of analytics and information management across the
Trends in Analytics
Analytic technology is well-rounded and has a wide range of uses. These range from statistical and mathematical processing to database functions to BI metrics and key performance indicators
(KPIs). This wide range of analytic capabilities reflects the broad need for analytics in organizations today. Unfortunately, it also reflects dispersal of analytics into applications, stored
procedures, analytic software, including BI, and spreadsheets. This dispersal breeds lack of a directory of defined analytics, difficulty in accessing analytics used in the organization and
inconsistency in analytic results.
While BI vendors have components for managing analytics, these apply to a proprietary BI environment and often are added to, and require additional investment in, the core BI product. The add-on and
proprietary nature of BI analytics management does not solve the problems of managing analytics across the organization. Building upon the data management foundation described above is how these
problems will be resolved with improved:
Metric definition: Metrics are defined within BI environments, but outside these environments they remain unknown and undocumented. Usage-based data
specification requires definitions as a part of data management for both analytic and transaction processing uses.
Creating the definition as a specification ensures that the metric is understood, can be created and can be reused where required. This is the basis for a directory of defined, consistent, and
reusable analytics across the organization.
Process management: Analytics are a critical component in process management. The trend here continues to emphasize the integration of business intelligence
process definition and measurement. This integration has been left to the organization when “best-of-breed” products are chosen for process management and business intelligence.
Aligning data usage and the analytics, reports, queries and screens with the process(es) and process step(s) where they are used creates a powerful analytic management capability. If the process
management capability robustly maps the logical process definition to its real-world or physical implementation, the power and use of analytics will be extended significantly.
Metric identification: When analytics are combined with process management, they create an opportunity for automatically identifying additional, previously
undefined metrics. Processes begin and end. They have triggering events and final dispositions. They have elements that can be defined as basic performance measures for a process and for process
steps. These measures, while basic, can be defined automatically.
Additionally, the binding of data to a process and a representation (report, screen and so forth) creates opportunities for automated data mining. A data crawler, analogous to a web crawler,
could scour data using these bindings to identify correlations and their metrics. With a memory-based DBMS, a data crawler can detect real-time changes and trends in these correlations and
These suggestions for automated metric identification do not displace the need for skilled practitioners to perform sophisticated analyses. They will provide basic metrics allowing practitioners
to focus on more challenging problems.
While organizations have made progress with their basic analytics, control of metric definitions and uses, process management, and data crawlers generally are under- or non-utilized.
Trends in Information Management
In order to leverage self-adapting data organization, a memory-based DBMS, metric definitions and control, process management and data crawlers, organizations will need to develop a strong
capability in information management. In considering the above discussion, it is hard to imagine being able to take advantage of any of these trends without an information management capability
Definitions and rules: The nature of a self-adapting data organization, a memory-based DBMS, metric control, process management and data crawlers rely on
consistent definitions and rules. A fundamental change will be making definitions and rules the basis for information, its storage, and its use. In other words, definitions and rules will be the
basis for software elements, not their documentation.
Centralized management: Tying definitions and rules to processes means that they are cross-functional and cross-departmental. Ensuring consistency in
definition, use and presentation requires centralized responsibility and management. This also requires a greater integration of data, processes and applications than exists today.
These capabilities need to integrate with usage-based data specification, process management, and all definitions and rules.
How likely is it that self-adapting data organization, a memory-based DBMS, metric definitions and control, process management, data crawlers and centralized information management will come into
existence? I expect to see these commercially available within three to five years.
SOURCE: Technical Trends Shaping Business Intelligence
Recent articles by Richard Skriletz