Competitive Intelligence: The Natural Extension to Business Intelligence

Originally published July 11, 2011

For many companies, competitive intelligence (CI) is vital in order to improve their visibility in the market and to increase their market share. Competitive intelligence is “the systematic, ongoing, legal collection and analysis of Information about competitors, similar products, market trends, branches, new patents and technologies and new customer expectations.“1

Business intelligence (BI) collects and analyzes information about a company to support its managers in identifying the need for action in the market. Competitive intelligence gathers and discloses important information for the company’s decision makers, but this information is found outside the company and delivers hints about the market segment in which the company is active. To clarify the differences: Business intelligence is intelligence about one’s own company and uses information from inside the company. Competitive intelligence is additional intelligence about the economical ecosystem in which the company is acting, and can be found outside the company.

It is wise for the company to use both perspectives to judge its activities in relation to the market, its customers and its products. Both business intelligence and competitive intelligence deliver facts, but also soft information that is useful for the correction of its strategy and tactics. The market conditions and parameters are dictated by customers, competitors, politics and trends, and can be collected and prepared by competitive intelligence in a form that allows decision makers to act fast in areas that need adjustment due to changes in market conditions. Business intelligence and competitive intelligence create significant information and build a solid base for wise decisions needed to achieve the company’s overall goals.

How to Work with Competitive Intelligence

A company can build a solid base of information for competitive intelligence by:

  • Collecting data

  • Extracting information

  • Gaining context for the information

Collecting data can be done in several ways. One method is using Web Crawlers. They systematically search Internet pages and extract content in raw form.

Other information sources are:

  • Public websites. These can be Web presences of customers, competitors or regulators

  • News services. There are specialized services available for subscription. Usually they describe market parameters (Reuters, Bloomberg, etc.).

  • Social media. Blogs, Twitter, Facebook, etc. deliver a lot of information about different players on the market.

  • Information services. For example, Edgar Online delivers finance information about companies such as balance sheet, cash flow, income statement, and number of employees, which might be of interest to their competitors.

  • Market analysts. Their studies give insight into markets and changes. They also identify trends and make predictions.

Information sources are diverse and while it is expected that information is presented in different formats, most of it is also unstructured.

In the next step, useful information needs to be extracted from the various data gathered from different information sources. Information collection and aggregation can be done by using different methods of text mining. The intervention of people is even more important in competitive intelligence than in business intelligence when it comes to evaluating information and putting different pieces of information into a meaningful and relevant relationship.

The last step in competitive intelligence is putting information into context so that it reflects reality as accurately as possible. This step requires a lot of time. Using the well-known 80/20 rule, you should use “only” 20% of the time for collecting and extracting the data, and 80% of the time to bring the information into context. This is discussed later in the article.

Competitive Intelligence Means Context

The market, competitors, customers and regulators represent the context in which a company is acting.

Competitors are an ongoing pressure; demanding customers, sinking customer loyalty and increasing price sensitivity can be addressed by having a wide and sound knowledge about a company’s economical ecosystem, which is essential for decision makers. Collecting information that feeds the knowledge base is the task of business intelligence, customer relationship management and competitive intelligence. Competitive intelligence directs the focus of the company not only to the customers, as customer relationship management does, but also to other market actors. Competitors can compete for the same customers or influence, and regulators can control the market.

The collected information in a CI environment has to fulfill the following criteria:

  • Relevance of information source (or data quality). What quality does the information source have? How clear is the information content? Does this source bring you a step further in gaining insight?

  • Timeliness. How up to date is the information? Is it one year old or only a few days? In general, up-to-date information has a deeper impact on the enterprise than older information.

  • Connection. Does the extracted information have a relevant connection to the company goal? What key people are connected to an important customer?

  • Message Strength. If a piece of information is used often in documents or has a huge impact in research activities, it has great strength.

If we try to apply the same criteria to a BI application, we discover that these questions are just as important for competitive intelligence. However, small differences must be observed when we apply the four criteria to business intelligence and competitive intelligence:

  • Relevance of information source (or data quality). This is easier to achieve in a BI environment. Under the condition that all data sources are connected to the business intelligence/data warehouse system, high quality of information is possible if clear data quality policies are in place.

  • Timeliness. This can be achieved in a BI application with a lot of time and money. Real-time solutions or “in-memory“ applications help achieve high quality in the actuality of the information.

  • Connection. Good modeling connects the right key figures to a certain question using a query or a report.

  • Message Strength. This is easier to achieve in business intelligence if we model smart key figures and dimensions that have a strong impact on the significance of the information.

The results of a CI analysis can be done in various ways: from tables using different structuring criteria to graphics and diagrams to visualize the most important concepts. For example, in the blogosphere, you can use tag clouds to visualize the strength of certain tags. Everything is accepted as long as the representation of information is intuitive, understandable and ergonomic.

BI vs. CI or BI and CI?

Paul Gray, Professor at Claremont Graduate University in California summarizes the differences between business intelligence and competitive intelligence: “Most business intelligence focuses on fact-based decision making that is based principally on understanding and using internal factors."2

The goal of both business intelligence and competitive intelligence is to put raw data into context and gain useful information. On the next level, you should gain new insights from the collected information. This can be achieved by using comparison, filtering, change of the perspective, or associative connection of different pieces of information. In other words, gaining intelligent information using competitive intelligence is very similar to human reasoning.

A first difference is to be found along the linkage Data → Information → Knowledge →Wisdom (this is how Larry P. English, an authority on information quality has defined it in his book Improving Data Warehouse and Business Information Quality: Methods for Reducing Costs and Increasing Profits) in the way business intelligence and competitive intelligence enrich their data. Business intelligence has achieved a high degree of automation in data collection and enrichment. Competitive intelligence relies more on manual steps or human intervention.

The process of gaining results in competitive intelligence is shown in Figure 1. We again encounter the Data → Information → Knowledge → Wisdom linkage, because the enrichment procedure and putting the data into a larger context is universal. The “Intelligence” step assumes that the decision maker uses the knowledge to make a decision and then observes the implications of this decision in the enterprise, prepared to make a change if necessary. Permanent maintenance of a CI environment can be of additional value for the company when it helps drive the business in an optimal way.

Figure 1: Description of the competitive intelligence process

If you look at the data sources for business intelligence and competitive intelligence, you can see that business intelligence uses primarily data from inside the company. Most BI data is structured data. Competitive intelligence uses information from outside the company and is never sure that all relevant data sources are being captured. Even in the ideal situation where almost all data sources are used, you can never be sure because the competition or other market agents may not be willing to tell the truth or to look for symmetry of information. And sometimes wrong or false information (FUD = fear, uncertainty and doubt) is spread deliberately in the market. Most information within the CI environment is unstructured, which emphasizes the complementarities of business intelligence and competitive intelligence.
 
Both business intelligence and competitive intelligence face the same big challenge: There is too much information inside and outside the enterprise that needs to be processed.

Knowledge is both explicit and implicit, but only explicit information can be processed automatically. Competitive intelligence starts off by not knowing at the very beginning what it is actually searching for. As time goes by, some insights are gained and information is sharpened. It is a deductive procedure, and the work is similar to that of a detective. Competitive intelligence wants to gain implicit knowledge, as sometimes explicit information is of low quality. Until recently, business intelligence processed only explicit information. It uses automated procedures, data warehouse data management, complex ETL processes, and calculation of key figures to feed the knowledge workers with relevant information. Recently, predictive analytics has become more important in the BI environment, supporting the efforts to extract implicit knowledge.

Instead of comparing business intelligence and competitive intelligence, consider how both may be integrated into the enterprise.

Methods in Data Analysis

Competitive intelligence follows a process that consists of two phases:

  • Phase 1 is known as secondary research. It collects and analyzes 80% of the data volume in 20% of the time allocated to the whole process.

  • Phase 2 is called primary research and evaluates 20% of the data volume in 80% of the time.

The following diagram explains the 80/20 method.

Figure 2: 80/20 rule in CI environment

In the first phase, you collect 80% of the data and use 20% of the time budget. It is called secondary research due to the fact that the research and the analysis are less important than in the second phase.

The results of the first phase will be processed in the second phase – the primary research.  The second phase uses the remaining 80% of the time allocated to the process.

The first phase can result in data overflow. Therefore, in the second phase you should mine the intelligence from the data that was collected in the first phase. In this second phase, the human factor is very important. Valuable information is gained about the business environment of the company with reflection and interviews with specialists and experts.

The secondary research seems to be easier because it is a quantitative method that can be easily automated using certain software technologies. The primary research is conducted by people and is difficult to automate. Curiosity, neutrality and patience are the skills a good CI employee needs.

Can we apply the 80/20 rule from the CI environment to a BI environment? The quick answer is: YES! BI users should use 80% of their time to get the last 20% of the valuable information that is not embedded explicitly in reports and graphs. Again, the human factor is important. A BI user has to find out why certain key figures have certain values, establish relationships between disparate numbers or  connect facts with information from outside the BI systems. Unfortunately, most BI systems store data from the past. New solutions offer to analyze detailed, real-time data. In predictive analytics, which makes predictions, people are not replaceable because predictive analytics is an iterative process that is guided by the user’s experience and knowledge. BI users are also mining after intelligence that is contained in the data of the enterprise.

Conclusion

Complementary Properties

What are the complementary properties of business intelligence and competitive intelligence?

In summary, business intelligence and competitive intelligence are complementary methods to gain information that helps a company act in its market environment.

  • In a BI environment, the context of information is immediately available to the company and easy to access. In a CI environment, you must deduce the correct context by using iterative and diverse methods. The context and information sources for competitive intelligence are hard to access and the quality of information should be routinely questioned.

  • BI data sources are mostly internal while competitive intelligence collects and evaluates data that exists outside of the company.

  • Business intelligence uses mostly structured data and is focused mainly on key figures. Competitive intelligence is primarily based on unstructured data and tries to build some key figures (e.g., revenues over time of the concurrence), but it takes text more seriously than business intelligence.

Similarities

Which are the common properties and similarities of business intelligence and competitive intelligence?

  • The 80/20 method of competitive intelligence can be used one-to-one in a BI environment.

  • The gathering of information is strongly automated in business intelligence. Competitive intelligence can achieve a certain level of automation, but needs a lot of human intervention. This difference might lead us to incorrectly assume that in business intelligence the human factor is not equally indispensable. In a BI environment, an analyst must put key figures in intelligent relationships, ask questions, and search for reasons and explanations.

  • Predictive analytics is becoming more important in the BI environment. Competitive intelligence is related to a deductive process and sometimes uses predictive analytics (e.g., text mining).

  • Both business intelligence and competitive intelligence can reach a level of excellence if BI and CI analysts take a look beyond their own horizons to gather information from outside.

Competitive intelligence is recommended as an extension to business intelligence. Together they build a kind of intrinsic intelligence of the company. Firms that use business intelligence and competitive intelligence in their strategies have a better market position, are more realistic in estimating their positioning compared to their competitors, and are able to make better decisions in the proper window of opportunity.

Using competitive intelligence, the human factor is again placed in the front. The recommendation of Hans-Georg Kemper and Henning Baars that technology should not get the upper hand must be internalized: “In the reality is often the case that in BI-/CI- projects exist a pronounced and dominant orientation to technology which in many cases get a counterproductive faith – a recurrent phenomenon which is known since decades."3

References

  1. Wikipedia

  2. BI Journal, Vol. 15, No. 4, (TDWI), "Competitive Intelligence", Paul Gray, Professor Emeritus of Information Science, Claremont Graduate University, California

  3. HMD 247, 43. Volume, February 2006, "Business Intelligence und Competitive Intelligence", Prof. Dr. Hans-Georg Kemper, Henning Baars

  4. BI Journal, Vol. 15, No. 4, (TDWI), “Learning Competitive Intelligence from a Bunch of Screwballs“, Troy Hiltbrand

  5. Internet,"Complexity, Competitive Intelligence and the First Mover Advantage", Philip Vos Fellman, School of Business, Southern New Hampshire University, Jonathan Vos Post, Computer Futures Altadena, California  

  6. White Paper “Self-Acting Data Mining: Das neue Paradigma der Datenanalyse“, P. Neckel, www.mayato.de
  • Alexandru DraghiciAlexandru Draghici
    Alexandru is a business intelligence consultant and works as freelancer. Since 1994, he has been active in OLAP, data warehouse and business intelligence. His strengths are in conceptual work and architecture of DW and BI solutions. He possesses wide knowledge and manifold experience in SAP BI technology and non-SAP BI technology such as Oracle, Hyperion, SAS Institute, and Cognos. His special interest is in "unstructured data.” He has several articles published in the German edition of the BeyeNetwork.
 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!