We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

New BI Demands Pushing Data Architecture Limits

Originally published August 8, 2012

We all know change is constant, but when we are responding to demands to extract value from new forms of the most important asset in modern competitive business – information – and are busily managing the increased volumes, change is both constant and fast-paced. Don't get left behind: With "big data" increasingly taking hold, the  five-year future is going to bring major transformation in the way information systems are built to support business intelligence (BI) and analytics applications.

It’s going to be more about the net additive effect of the possibilities than about dismantling current technology investments. It’s going to be about seriously using information for BI and analytics through the exploitation of all possible data, including data that is larger than anything else you’ve experienced to date, providing less value “per byte” and requiring new data management methods. It’s going to be about exploiting information sooner in its lifecycle – as soon as possible.

Maturity in business intelligence is correlated to maturity and success in business. BI maturity is a worthy progression to understand, and working to increase maturity levels is a recommended strategy for every BI team.

Most companies begin their BI journeys because of a need for reporting. In many cases, reporting requirements can overwhelm, either technically or politically, the operational systems executing the business transactions. That often leads to the development of additional structures – data warehouses and marts – to house transactional data for reporting and analysis. Integrating that data, regardless of where it comes from, becomes critical, as does improving the overall quality of the information.

Developing a More Expansive View of BI

Eventually, monthly, weekly and even day-after reporting then becomes insufficient, and the valuable information being integrated and analyzed becomes a requirement on an intra-day basis. An organization can then go full circle to doing whatever it can operationally.

With post-operational warehousing systems being bound by the extract, transform and load cycles that need to be performed there, real-time BI brings data analysis directly into operational systems. Business intelligence previously was viewed as “whatever we do to data in the data warehouse,” but now BI is being expanded into the discipline of data usage and exploitation, wherever that data may reside.

For more and more companies, another aspect of BI maturity involves coping with new data types, which usually means harnessing the “big three” forms of unstructured or semi-structured big data: sensor, social media and Web activity data. It can also mean an organization availing itself of the syndicated data marketplace and bringing external information into its BI systems.

Before making big investments, assess your existing information management architecture to determine if it will meet the new BI demands of the enterprise. Suggested questions to ask as part of that assessment include the following:
  • Is all company data, including unstructured and semi-structured data, under management?

  • Is there a solid data quality program in place to ensure the distribution of high-quality information?

  • Are all BI queries processed in post-operational batch systems now? Could they be done, and decisions made, in operational systems instead?

  • Do new query requests automatically require using a particular system or type of BI technology, such as multidimensional cubes?

  • Is the current usage of BI systems going to continue to scale? Is it time to consider alternatives?

  • Can data marts, data warehouses and cubes with overlapping functinoality be consolidated? And have the potential savings of doing so been calculated?

BI Challenges – And Ways Around Them

Maintaining sufficient performance levels is one of the top challenges in BI today, especially as data volumes grow. Alternative technologies for managing structured data – such as columnar databases, in-memory databases and data warehouse appliances (or a combination of them) – can provide additional performance on specific workloads beyond what conventional relational databases are capable of delivering.

Data integration also remains a major challenge. For many organizations, the answer may be the “perpetual short-term” solution of data virtualization, which can bring together data from separate, technically distinct data stores without requiring the information to be consolidated in a data warehouse.

Interest in the cloud is high as a place to store the data behind BI systems and manage the integration and data access components. Security fears and integration with on-premises data remain big concerns for many prospective users. Nonetheless, cloud and virtual deployments undoubtedly will house upwards of one-third of corporate databases within the next five years.

Centralizing management of master data is potentially valuable to BI initiatives as well because it replaces individual efforts and creates a governed process for master data management (MDM) along with the infrastructure required to support distribution of the master data to different systems. But there are architectural considerations to address when evaluating an MDM program. For example, does each system need to build its own master data? Is one system’s master data worthy of being used more broadly? Are the owners of master data ready for the responsibilities of providing it beyond their individual systems?

While few companies have a budget for “innovation,” the good news is that advanced business intelligence is not really about innovation. It’s about business, and when done well it provides a high return on investment even in the short periods that are demanded of IT investments these days. But there’s the rub – it must be done well.

By asking the following questions before investing in information management technologies to support BI activities, organizations should be well prepared to management their information assets effectively:
  • Can business decisions that data warehouse and business intelligence (BI) systems support be made earlier than when the data comes together in them?

  • For “column-selective” queries that need to access only a small number of the columns in database tables, can columnar databases be supported?

  • Are NoSQL and “big data” technologies being implemented or considered within the organization? If so, is there a plan for accommodating and managing them?

  • Are cloud BI and data management options on the table?

  • Has the syndicated data marketplace been surveyed for information that aligns with company needs and could be pulled into internal systems?

  • Has data virtualization been adopted for integration needs between different systems? Is it being used in cases where physically moving data would provide better performance?
Coupled with a focus on enabling better business outcomes as a result of efforts to increase a company’s level of BI maturity, the resulting information management architecture will be able to support the BI data demands of the foreseeable future.

Editor's Note:
To read the other articles featured with this article in our e-zine, BI Trends + Strategies, click here.

  • William McKnightWilliam McKnight
    William is the president of McKnight Consulting Group, a firm focused on delivering business value and solving business challenges utilizing proven, streamlined approaches in data warehousing, master data management and business intelligence, all with a focus on data quality and scalable architectures. William functions as strategist, information architect and program manager for complex, high-volume, full life-cycle implementations worldwide. William is a Southwest Entrepreneur of the Year finalist, a frequent best-practices judge, has authored hundreds of articles and white papers, and given hundreds of international keynotes and public seminars. His team's implementations from both IT and consultant positions have won Best Practices awards. He is a former IT Vice President of a Fortune company, a former software engineer, and holds an MBA. William is author of the book 90 Days to Success in Consulting. Contact William at wmcknight@mcknightcg.com.

    Editor's Note: More articles and resources are available in William's BeyeNETWORK Expert Channel. Be sure to visit today!

Recent articles by William McKnight



Want to post a comment? Login or become a member today!

Be the first to comment!