We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


AIDE – A Business Intelligence Consulting Methodology

Originally published April 21, 2010

Enterprises do not find it difficult to get funding approval for business intelligence (BI) projects because businesses realize the need for accurate, timely and intelligent data to stay competitive in the market. A Gartner survey reveals that business intelligence continues to be one of the top 10 priorities of the CIO community. Lately, there is a considerable spike in the number of commercial off-the-shelf packaged analytic application offerings from BI vendors. However, due to custom requirements, enterprises continue to build analytic applications. Regardless of a build or buy approach, BI consultants continue to play an important role in solution development. From an enterprise perspective, one of the true challenges lies in the availability of BI consultants with the right skills. This article analyzes how BI consulting is different from typical IT consulting and recommends a consulting methodology for BI consultants.

Who is a Business Intelligence Consultant?

A BI consultant is an individual with a computer science or relevant background who has the skills to advise the business on how to use the IT resources to arrive at a solution for the given BI problem. BI consultants typically acquire the required skills for an advisory capacity from multiple BI project implementations. A BI consultant can be an employee of an IT consulting services firm or can work independently as a freelancer.

What is the Skill Set of a BI Consultant?

Consulting skills are a combination of technological skills and personality traits as depicted in Figure 1. Project implementation experience, analysis, problem solving, appreciation of BI market trends, industry standard certifications and thought leadership are some of the key constituents that make up technical skills.


(mouse over image to enlarge)


Personality traits, on the other hand, are a package of interpersonal skills such as interviewing, decision making, negotiation, motivation, etiquette and engaging with others. Both sets of skills are essential for a BI consultant to be successful.

What Kinds of Tasks are Usually Performed by a BI Consultant?

Business goals drive BI projects across the organization. A consultant can play an active role during any phase of the BI project life cycle. A consultant’s help is typically sought whenever there is lack of specific BI skills within an enterprise IT team and/or IT services vendor. A consultant can also be pressed into service if an ongoing BI project faces difficulties and requires specific expertise to move the project in the right direction.


(mouse over image to enlarge)


As depicted in Figure 2, some of the tasks that are performed by a BI consultant during the course of execution of a BI project include, but are not limited to:
  • Gather requirements for an enterprise data warehouse (DW)
  • Define solution implementation road map
  • Define or assess BI solution architecture
  • Sizing or capacity planning for enterprise DW
  • Define or evaluate extraction, transformation and loading (ETL) architecture
  • Define or evaluate BI reporting architecture
  • Establish coding/development standards
  • Guide the BI competency center setup
  • Mentor the setup of data governance council
  • Advise the BI product/vendor selection process
  • Define and/or validate prototypes/proofs of concept
  • Define data warehouse testing strategy

How is BI Consulting Different from Typical IT Consulting?

In a classical IT consulting service, the focus is largely on development of an application that helps automate business processes, creation and maintenance of data, whereas the focus of a BI consulting service shifts toward usage and analysis of legacy data. They share a set of commonalities in the sense that both build an application and follow a development methodology. However, there are certain aspects where BI consulting differs from a traditional IT consulting service. Table 1 highlights some of those aspects:


 
The following subsections analyze each of the aspects that differentiate BI consulting from a traditional IT consulting service.

Enterprise-Wide Impact

In a traditional IT consulting service scenario, applications are developed to cater to the needs of a specific business division/unit. BI initiatives are, in contrast, a result of strategic business objectives. The BI end-user community cuts across all divisions or units of business and at all levels, whereas the end-user community of a typical application development project is restricted to a business division.

Cross Commitment

Typical IT consulting services require the commitment of people within the business unit. By the virtue of a BI project having an enterprise-wide impact, it requires commitment of people across various business units. A BI consultant not only works with client personnel like data architects, database administrators, business analysts, business managers and executives, but also with other IT service vendors of the client, software product vendors, independent contractors and members of various other groups from his/her own employer. It is critical for a BI consultant to keep the stakeholders apprised of the breadth and depth of cross commitment required for a successful and timely implementation of a BI project.

Project Life Cycle

Another aspect in which BI consulting differs from a typical IT consulting service is the duration of the project execution. In a traditional IT consulting engagement, applications can be developed in a relatively short span of time and deployed in one go, whereas it is not feasible to follow the same approach for a BI project implementation mainly because of the enterprise-wide scope of work.

Regulatory Requirements

In a classical IT consulting service, in general, there are little or no statutory compliance requirements. By and large, data warehousing and business intelligence are used for enterprise reporting, and often, an enterprise’s data warehouse data is also used for providing input to a company’s annual and/or quarterly reports. Therefore, it is necessary for a BI consultant to be aware of legal compliance requirements such as SOX and Basel II. This is generally not the case for a consultant in a typical IT consulting service.

Common Technological Challenges Faced by a BI Consultant

Every consulting assignment is unique in terms of challenges involved in the project implementation. These challenges can be broadly classified into non-technical and technical challenges. Non-technical challenges, as the name implies, refer to non-technological aspects such as lack of business domain knowledge, people and politics, critical time lines, budget and resource constraints.

 


Technical challenges, on the other hand, as depicted in Figure 3, refer to technological aspects of the challenges. The following subsections describe these in detail to help BI consultants better equip themselves for the challenges lying ahead.

Point of Departure (PoD) Architecture

In large companies, the vast majority of the day-to-day business data is processed and stored in legacy systems. These legacy systems have been running for several decades. For a DW/BI project, it is important for a BI consultant to arrive at the PoD architecture to help explain how the transformation will happen from the as-is state to the to-be state. In a legacy system environment, arriving at PoD architecture is always a daunting task due to the fact that knowledge behind the legacy systems and the jobs running on those systems are lost. This could be because the person who created those mainframe jobs has quit/retired and no documentation exists about the legacy systems. In such cases, a BI consultant should have the ability to interview various people to collect all possible information and use that information to arrive at PoD architecture.

Data Sourcing

Typically in large enterprises, there exists a mixture of legacy systems, homegrown applications and commercial off-the-shelf packaged applications. Determining the data source for every data entity such as customer, product and vendor data would require a mammoth effort. This is mainly because not only a data entity, for instance customer data, is available in two or more systems, but also due to the fact that the data is inconsistent across these systems. When such challenges arise, a BI consultant should be prepared to bring concerned stakeholders to the table, conduct a brainstorming session and be a catalyst in determining the system of reference for sourcing the data.

Volume of Data

In a conventional IT consulting service, the emphasis is on data creation and/or maintenance of data. However, the emphasis in a BI consulting service is on historical data. Data warehouses are built to cater to the enterprise-wide data requirements for various purposes including reporting, analytics, performance management and mining. Thus, BI project implementation processes data in massive volumes. Such enormous data volumes adversely affect the performance of the data warehouse. BI consultants should be well versed in techniques for managing huge data volumes such as performance tuning, delta capture, near-line storage, and data archiving to combat challenges posed by huge data volumes.

Quality of Data

One of the common challenges faced in a BI consulting assignment is the poor quality of data. The core objective of a BI initiative is to provide an integrated enterprise-wide view of data. This data integration is challenged by inconsistent and incomplete data in legacy systems. The BI consultant must be aware of techniques and methodologies involved in assessing the quality of legacy data, planning and implementation of data quality measures such as data profiling, data standardization, data consolidation and data augmentation.

Data Privacy and Security

A single version of the truth and a one-stop shop for all the data requirements are portrayed as some of the benefits of BI initiatives. In order for an enterprise to realize these benefits, integration of data across various business divisions into a repository (usually staging, warehouse and marts) is inevitable. However, some of the business divisions (finance, for instance) may not be prepared/willing to share their data for enterprise data warehousing and business intelligence purposes.

Often, the reason put forth for denial of data sharing is data privacy or security. BI consultants should anticipate such issues ahead of time and have a solid data security model in place to alleviate any fears of data misuse. Remember, sensitive business divisions are not worried about authentication (“who” can access), but they are bothered about authorization (“who” can access “what”).

Business Rules

Business rules are nothing but business logic. In a DW/BI project implementation, business rules are applied predominantly in the process of data integration and report generation. In the vast majority of cases, enterprises lack rules engines or centralized and well maintained rules repositories. This leads to a situation where a BI project team gets conflicting versions of metrics or key performance indicator (KPI) definitions.

In such scenarios, a BI consultant should identify concerned stakeholders and conduct an arbitration session to iron out the differences. BI consultants should educate their team about the changing nature of business rules and advise the team to code the business rules in a way that it is modular and configurable. Should an enterprise already have a rules engine in place, a BI consultant should advise the team to leverage it instead of coding it again.

Many of us have read stories that in UK, for example, as much as 87% of the BI projects end up in failure. So, what is a failure? Is inability to reach the implementation stage a failure? Is not meeting the client’s requirement a failure? Is lack of ability to fetch the return on investment a failure? The answer is a simple yes to all! Is it the technical complexities that cause these failures? Well, the chances are slim. Technical intricacies are highly unlikely to cause failure, but people and political intricacies can. It could be the process complexities as well. The sources of the failure vary from one project implementation to the next. However, lack of skills is invariably one of the major sources of failure. With the right skills and approach, complexities of any sort can be dealt with comfortably in BI project implementation.

AIDE – A BI Technology Consulting Model

AIDE is a BI technology consulting model, not a business consulting model. It comprises of four phases and outlines a set of tasks to be executed during each phase. Figure 4 depicts AIDE phases.

\

The distinction of this methodology is that it focuses more on “understanding” the landscape of the BI project prior to even beginning analysis of the problem. The following sections of this article describe each phase of this methodology.

Ascertain Problem Statement

BI consultants should always approach their assignments with a sound understanding of the project and its goals. It is extremely important for BI consultants to familiarize themselves with the business benefits of a BI project. It is suggested that the following set of tasks be executed during the “Ascertain” phase:
  • Understand the business case of the BI project
  • Comprehend the business drivers of the project
  • Identify the BI project objectives
  • Determine the scope, inclusions and exclusions of the BI project
  • Understand the problem statement
  • Understand and negotiate the expected role to be played
  • Deliver a consulting plan with a set of tasks, time lines, dependencies and deliverables
Once BI consultants gain a fair level of understanding about the BI project, its scope and the problem statement, they should negotiate their role in the assignment. During the negotiation, they should always be candid in terms of what they can offer toward the resolution of the problem. It is an important behavior for BI consultants to be open; it helps set the right expectations. Once they reach an agreement on the expected role to play, they should deliver a consulting plan (not to be confused with the BI project plan) on how they will achieve the expected outcome. A consulting plan could be a simple set of tasks defined with time lines and dependencies. Consultants should also include the deliverables from their end. For instance, if a BI consultant is expected to evaluate the BI architecture, the deliverables could be BI architecture gap analysis and recommendations.

Interpret Social Elements

BI projects are the result of corporate BI initiatives. These BI initiatives are an outcome of business strategies. Those strategies are derived from corporate goals – which are driven by corporate vision. It is therefore important for a BI consultant to understand the larger picture of the BI project; and that’s exactly what the emphasis of this phase is. It is advised that the following set of tasks be executed during the “Interpret” phase:
  • Understand corporate vision, business goals and strategies
  • Find out the various BI initiatives taken by the enterprise
  • Understand the enterprise IT landscape – hardware, software and service vendors
  • Familiarize corporate hierarchy, end user community and project team structure
  • Understand previously failed BI initiatives in the enterprise, if any
  • Socialize and get an insight into organizational politics
  • Recognize the upstream and downstream systems
  • Understand the pain points of the user community
  • Understand how it impacts the business
  • Deliver an appreciation document summarizing your understanding
It is extremely important to know if there are any failed BI initiatives in the past. Not only does this help BI consultants to learn from the past, it also helps them to realize what could be potential show-stoppers. It is critical for BI consultants to understand the people and politics in the organization. There is no formal approach to learn the political issues; a BI consultant must socialize with the members of the team from the top to the bottom of the pyramid to get a foothold on organizational politics.

Determine Root Cause

During the first two phases, a BI consultant would have gained significant understanding about the organization, its vision, goals and initiatives. By now, a BI consultant would have also obtained a good understanding about the BI project, its objectives and scope, team structure, various relevant business applications and the business pain points. Now, it’s time for a BI consultant to get into the project specifics and begin the actual investigation to find out the source(s) of each of the pain points expressed by the business. In this regard, the following set of tasks should be performed during the “Determine” phase:
  • Understand the technical and functional requirements of the BI project
  • Understand the existing solution architecture and design
  • Classify the pain points expressed by users into technical and functional issues
  • For every issue, analyze the root cause:
    • People & Politics?
    • Process?
    • Technology?
  • For all the technological issues, locate where the bottleneck is:
    • Architecture? Design? Security?
    • Data Model? Sourcing Strategy? Data Format?
    • Data Availability? Extraction? Transformation? Loading?
    • Information Delivery? Data Visualization? Data Exploration?
    • Hardware – Processor? Disk? Memory? Operating System? Network?
    • Tools – Version? Patches? Configuration?
    • Business Continuity Processes? Failover Processes?
  • Analyze logs and other relevant metadata and find out the root cause of the issue
  • Document and deliver the analysis and key findings
For issues of a political nature, a BI consultant should bring the concerned parties to a table and facilitate a discussion among them to iron out the differences. For issues of a technical nature, at times, metadata features such as logging could have been disabled to boost the system performance. BI consultants should act proactively and turn on such features in a development and/or test environment from the beginning.

Elucidate Resolution

During this final phase, BI consultants should arrive at a solution option(s) that addresses every pain point expressed by the business. The following set of tasks should be performed during the “Elucidate” phase:
  • For every issue, prepare a list of solution options
  • For every solution option, perform an impact analysis
  • For every solution recommendation, find out the list of dependencies
  • Summarize and document all the solution options
  • Present summary to the target audience
  • Articulate the pros and cons of each solution option
  • Capture any suggestions from the audience (suggestions may include another solution option!)
  • Seek feedback
BI consultants should realize the importance of proactively doing an impact analysis for every solution option that they think of. For instance, a small change in the column name of a source database may affect several stored procedures, multiple triggers, a huge chunk of ETL mappings, hundreds of users and possibly thousands of reports. Such an impact analysis study would not only impress the client but would also help them select the best one among the list of solution options available for them.

BI consultants should also uncover the list of dependencies for every solution option put forward to the client. This scores a high level of significance because it helps the client identify the right timing for the solution implementation. For instance, prior to performing a small change in a database column name, a database must first be backed up, database services need to be brought down, a database script needs to be created and run for incorporating the desired changes, database services need to be brought back up and finally test queries need to be executed for validation purposes.

Lower impact and lesser dependency options are not always the best solution options. That is why it is important for BI consultants to articulate the advantages and disadvantages of each solution option. Remember, a BI consultant’s job is just to help the client identify the right option.

Conclusion

BI consulting continued to gain traction amid turbulent times like the economy slowdown and recession; however, the supply of BI consultants with the right skills does not match the demand. BI consulting differs from typical IT consulting in an array of parameters; and it comes with plenty of challenges. To overcome those challenges and be successful, the AIDE BI consulting methodology suggests consultants spend three-quarters of their time understanding the project, its context, scope, requirements, existing IT landscape, people, politics, processes, technologies, problem statements and the analysis of the problem. It advocates not more than a quarter of the consulting time be spent on problem resolution. Thus, the AIDE model ensures that the outcome of a BI consultant’s work is always qualitative and productive. 
  • Arunachalam ThirunavukkarasuArunachalam Thirunavukkarasu
    Arunachalam is a Technology Architect, with Information Management Consulting Group, SETLabs, Infosys Technologies Ltd. He has several years of experience in business intelligence and data warehousing and has successfully implemented large data warehouses. He is an Inmon certified CIF Architect. He is also a TDWI Certified Business Intelligence Professional and DAMA Certified Data Management Professional. He is responsible for developing solutions and frameworks in the areas of data warehousing and business intelligence including data integration, information delivery, BI search, metadata, and migration of BI reporting data. He may be reached at Arunachalam_T@Infosys.com.
 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!