We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Blog: Barry Devlin Subscribe to this blog's RSS feed!

Barry Devlin

As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein.

About the author >

Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

January 2014 Archives

Mart.jpgBusiness unIntelligence emerged from my questioning of the fundamental assumptions underpinning BI in all its forms, from the enterprise data warehouse to big data analytics. The belief I'm questioning in this post is that the target audience of BI is everyone in the business. This springs from the very reasonable premise that BI should be used more widely throughout the organization than it currently is. But, somewhere along the way, I don't know when, the idea emerged that BI must be used by everybody in the enterprise, if we are to gain full business benefit. BI vendors thus lament low-penetration of their tools, agonizing over how to make them simpler, more appealing. Data marts, echoing the bright and breezy WalMarts and Kmarts of retail, were introduced in the mid-1990s as an alternative to the dark and dismal data warehouse. Today, self-service BI, self-service analytics, self-service everything will solve world BI hunger.

I'm sorry, for me, that dog don't hunt.

This post was triggered by Paxata's mission statement, which aims to "Empower EVERY PERSON in the enterprise to find and prepare analytical information..." In truth, Paxata is building a very impressive and powerful adaptive data preparation tool. But, their mission statement almost derailed my interest. Really, each and every person in the organization should be finding and preparing analytical information? From the janitor to the CEO? I'll return to Paxata later in this post, but first, let me check if anybody else feels the same level of discomfort with this concept as I do.

Let's start with self-service on a personal level. I'm pretty comfortable with self-service when I visit a computer retailer. Put me in a perfume store, and I need an assistant--quickly. My wife has the opposite experience. I conclude from this (and many other examples) that self-service works only if the self-server has (i) sufficient understanding of what she's trying to do and (ii) considerable knowledge of what is available, its characteristics and where it is in the store. My experience in BI is that business users typically satisfy the former condition but fail regularly on the latter. In my original 1988 data warehousing paper (or contact me if you want a copy), I distinguished between dependent and independent users. It was only the second group who satisfied both conditions above. At the simplest level, we may divide business users into two such groups, as I did back then. In reality, of course, it's more subtle. Some business users understand statistics, many don't. Some are more attuned to using information; others rely (often correctly) on their intuitions. In chapter 9 of Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data I discuss many of the ways in which decisions are influenced by many things other than information. So, it's horses for courses at a personal level: self-service BI only works for some of the people some of the time.

Organizationally, the idea that everyone is involved in decision making that demands analytical support is contrary to all concepts of division of labor and responsibility. The above-mentioned janitor has no incentive to analyze his cleaning performance. There still exist a myriad of tasks in every organization that are menial, performed by rote. Managers of such areas have, since the time of Fredrick Winslow Taylor in the early 1900s, been analyzing such work and finding ways to streamline it. While we recognize today that production line workers do have knowledge to contribute to process improvement, such knowledge is tacit, and unlikely, in my view, to be quantified by the workers themselves. Rather, that elucidation depends on another type of skill, that of independent users, power users or, as they like to be called today, data scientists. These are people whose skills and interests bridge the business/IT divide.

Which brings me back to Paxata and the concept of the biz-tech ecosystem, the symbiotic relationship between business and IT demanded by the speed and span of modern business. Finding and preparing data has long been the principal bottleneck of all BI. It is a type work that really requires a balanced mix of business acumen and IT skill. Its exploratory and ad hoc nature defies the old development approach of business requirements statements thrown over the fence to IT.

What's required is an exploratory environment for diverse data types that seamlessly blends business acumen and IT skills. This is precisely what Paxata does. Based more on the data content than on the metadata (field or column names and types), business analysts explore the actual contents of data sources and their inter-relationships in a highly visual manner that uses color and other cues to direct attention to aspect of interest identified by heuristics within the tool itself. Data can be simply cleansed and transformed, split and joined. The interface is deliberately spreadsheet-like, another comfort zone for the business analyst. But, unlike a spreadsheet, all action are recorded and tracked; they can be rolled back and they can be repeated elsewhere, vital aspects of the level of data governance needed to make this solution capable of being put into production. It's hard to describe in words; you really need to try it or see the demo. See further descriptions from Joseph A. di Paolantonio and Jon Reed.

It's tools like this that make the biz-tech ecosystem real, that blend business and IT data skills and knowledge for easier application to real business needs. They enable people from the business side of the enterprise to enhance their IT abilities, and vice versa, removing the barriers to data exploration and preparation that stand between the information and full business value. They make it easier for more people to become independent or power users, data scientists, or whatever they choose to call themselves, making their jobs easier, faster and more productive. That is the vital and visionary work that Paxata (and other vendors like them) are doing in this era of exploding data varieties and information volumes.

Business unIntelligence Cover.jpgBut I still believe that this role and these tools will never be for everyone. What do you think?

News flash: Business unIntelligence is now available as an ebook on Kindle and on Safari.


Posted January 23, 2014 2:29 AM
Permalink | No Comments |
Mart.jpgBusiness unIntelligence emerged from my questioning of the fundamental assumptions underpinning BI in all its forms, from the enterprise data warehouse to big data analytics. The belief I'm questioning in this post is that the target audience of BI is everyone in the business. This springs from the very reasonable premise that BI should be used more widely throughout the organization than it currently is. But, somewhere along the way, I don't know when, the idea emerged that BI must be used by everybody in the enterprise, if we are to gain full business benefit. BI vendors thus lament low-penetration of their tools, agonizing over how to make them simpler, more appealing. Data marts, echoing the bright and breezy WalMarts and Kmarts of retail, were introduced in the mid-1990s as an alternative to the dark and dismal data warehouse. Today, self-service BI, self-service analytics, self-service everything will solve world BI hunger.

I'm sorry, for me, that dog don't hunt.

This post was triggered by Paxata's mission statement, which aims to "Empower EVERY PERSON in the enterprise to find and prepare analytical information..." In truth, Paxata is building a very impressive and powerful adaptive data preparation tool. But, their mission statement almost derailed my interest. Really, each and every person in the organization should be finding and preparing analytical information? From the janitor to the CEO? I'll return to Paxata later in this post, but first, let me check if anybody else feels the same level of discomfort with this concept as I do.

Let's start with self-service on a personal level. I'm pretty comfortable with self-service when I visit a computer retailer. Put me in a perfume store, and I need an assistant--quickly. My wife has the opposite experience. I conclude from this (and many other examples) that self-service works only if the self-server has (i) sufficient understanding of what she's trying to do and (ii) considerable knowledge of what is available, its characteristics and where it is in the store. My experience in BI is that business users typically satisfy the former condition but fail regularly on the latter. In my original 1988 data warehousing paper (or contact me if you want a copy), I distinguished between dependent and independent users. It was only the second group who satisfied both conditions above. At the simplest level, we may divide business users into two such groups, as I did back then. In reality, of course, it's more subtle. Some business users understand statistics, many don't. Some are more attuned to using information; others rely (often correctly) on their intuitions. In chapter 9 of Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data I discuss many of the ways in which decisions are influenced by many things other than information. So, it's horses for courses at a personal level: self-service BI only works for some of the people some of the time.

Organizationally, the idea that everyone is involved in decision making that demands analytical support is contrary to all concepts of division of labor and responsibility. The above-mentioned janitor has no incentive to analyze his cleaning performance. There still exist a myriad of tasks in every organization that are menial, performed by rote. Managers of such areas have, since the time of Fredrick Winslow Taylor in the early 1900s, been analyzing such work and finding ways to streamline it. While we recognize today that production line workers do have knowledge to contribute to process improvement, such knowledge is tacit, and unlikely, in my view, to be quantified by the workers themselves. Rather, that elucidation depends on another type of skill, that of independent users, power users or, as they like to be called today, data scientists. These are people whose skills and interests bridge the business/IT divide.

Which brings me back to Paxata and the concept of the biz-tech ecosystem, the symbiotic relationship between business and IT demanded by the speed and span of modern business. Finding and preparing data has long been the principal bottleneck of all BI. It is a type work that really requires a balanced mix of business acumen and IT skill. Its exploratory and ad hoc nature defies the old development approach of business requirements statements thrown over the fence to IT.

What's required is an exploratory environment for diverse data types that seamlessly blends business acumen and IT skills. This is precisely what Paxata does. Based more on the data content than on the metadata (field or column names and types), business analysts explore the actual contents of data sources and their inter-relationships in a highly visual manner that uses color and other cues to direct attention to aspect of interest identified by heuristics within the tool itself. Data can be simply cleansed and transformed, split and joined. The interface is deliberately spreadsheet-like, another comfort zone for the business analyst. But, unlike a spreadsheet, all action are recorded and tracked; they can be rolled back and they can be repeated elsewhere, vital aspects of the level of data governance needed to make this solution capable of being put into production. It's hard to describe in words; you really need to try it or see the demo. See further descriptions from Joseph A. di Paolantonio and Jon Reed.

It's tools like this that make the biz-tech ecosystem real, that blend business and IT data skills and knowledge for easier application to real business needs. They enable people from the business side of the enterprise to enhance their IT abilities, and vice versa, removing the barriers to data exploration and preparation that stand between the information and full business value. They make it easier for more people to become independent or power users, data scientists, or whatever they choose to call themselves, making their jobs easier, faster and more productive. That is the vital and visionary work that Paxata (and other vendors like them) are doing in this era of exploding data varieties and information volumes.

Business unIntelligence Cover.jpgBut I still believe that this role and these tools will never be for everyone. What do you think?

News flash: Business unIntelligence is now available as an ebook on Kindle and on Safari.



Posted January 23, 2014 2:29 AM
Permalink | No Comments |
astrologer.jpgMore years ago than I care to remember, I took a strong interest in predicting behavior. Besotted by a woman who blew hot and cold, I turned to astrology seeking to know the future of the relationship. The prognosis was far from positive... and ultimately correct.

Modern thinking is highly skeptical of such esoteric arts. Correctly so in the case of many practitioners. But it has far less reason, in my opinion, to dismiss much of the underlying rationale and value of these approaches. That, however, is a topic for another forum. My point here is that I did not believe the prediction; I didn't want to. And that's part of a rather gory truth of any predictive tool.

As we flip the calendar to 2014, it is clear that the audience for predicting people's future behavior has grown far beyond the ragged ranks of lonely lovers. Marketers and advertisers, once content with psychological and sociological generalizations, have become increasingly besotted by big data and the promise it offers of knowing the customer so intimately that her individual and specific future behaviors can be predicted. And, probably of more value, influenced. This, of course, is the power and the glory of predictive analytics. But, as already noted, this blog's title actually says "gory". And here, in detail, is why.

Thumbnail image for Business unIntelligence Cover.jpgWhile writing in Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data about some of the uses of big data way back in February, 2012, I came across Charles Duhigg's New York Times piece on Target's ability to predict not only the pregnancy of female (obviously) customers but also their likely delivery dates. Over the New Year break, I finally found time to read Duhigg's book, The Power of Habit, in which that story first appeared. My point in my own book, and in a related blog Death by a thousand analytics, was that using big data in this manner is both a clear invasion of privacy and likely to creep people out. But Duhigg's book deals with a much broader topic--human habitual behavior and unconscious thought patterns--that has far deeper implications for predictive analytics and decision support in general.

The message of The Power of Habit is fairly easily summarized. A very significant proportion of our thinking, decision-making and action at personal, organizational and societal levels is entirely habitual. Given a particular cue (which can be anything from a thought to an action), a routine set of behaviors is immediately and automatically engaged, which provides an anticipated reward. Simply put: cue => routine => reward. This mechanism operates, both in its set up and execution, in one of the most primitive areas of the brain, the basal ganglia, a structure that derives in evolutionary terms from the earliest chordates. It is, in fact, a basic survival mechanism that enables us undertake a wide range of necessary life-preserving activities, from preparing food to getting home, with minimal expenditure of energy and attention. This, of course, initially freed the brain to deal with novel, life-threatening situations and more recently for higher-brain functions such as empathy and reasoning. However, as anyone who has ever tried to give up biting their nails or mid-afternoon snacking can testify, changing habitual patterns can be very difficult indeed, even those which are counterproductive or even downright dangerous.

This very primitive mental operation of cue-routine-reward has two (at least) important consequences for predictive analytics.

For marketing, the drive will be to move increasingly to an understanding of specific, individual habits and their cues, in order to create opportunities to influence behavior. This is more difficult than it sounds, as such cues are often difficult even for the individuals themselves to identify. Similarly, rewards are often far more subtle and diverse than might be imagined. The story of how Pepsodent toothpaste was sold to an American public famed for poor oral hygiene in the early 1900s has become a classic marketing exemplar of how to create a habit and a sales success. But, according to Duhigg, the actual cue and reward involved were long mistaken. Furthermore, although most Western people are by now inured to mass manipulation by advertising, the much more personal and private knowledge used for such individual influencing may raise resistance to such approaches. From the point of view of privacy, I am of the opinion that it most certainly should.

In the case of personal and organizational decision making support, habitual and other unconscious patterns are a serious impediment to efforts to become "data-driven". My unfortunate lovelorn experience of disbelieving a predicted but undesired outcome is a basic behavior pattern well-recognized in psychology but seldom mentioned in business intelligence. Habitual beliefs and behaviors at an organizational level are so deeply embedded and invisible to participants that they stymie efforts even to recognize problems, never mind take effective action.

The bottom line is that no matter how much data you gather or how elaborate the models you generate, Business unIntelligence operates finally, in full glory or gory fullness, in the minds of your customers and business users.


Posted January 6, 2014 3:40 AM
Permalink | No Comments |
astrologer.jpgMore years ago than I care to remember, I took a strong interest in predicting behavior. Besotted by a woman who blew hot and cold, I turned to astrology seeking to know the future of the relationship. The prognosis was far from positive... and ultimately correct.

Modern thinking is highly skeptical of such esoteric arts. Correctly so in the case of many practitioners. But it has far less reason, in my opinion, to dismiss much of the underlying rationale and value of these approaches. That, however, is a topic for another forum. My point here is that I did not believe the prediction; I didn't want to. And that's part of a rather gory truth of any predictive tool.

As we flip the calendar to 2014, it is clear that the audience for predicting people's future behavior has grown far beyond the ragged ranks of lonely lovers. Marketers and advertisers, once content with psychological and sociological generalizations, have become increasingly besotted by big data and the promise it offers of knowing the customer so intimately that her individual and specific future behaviors can be predicted. And, probably of more value, influenced. This, of course, is the power and the glory of predictive analytics. But, as already noted, this blog's title actually says "gory". And here, in detail, is why.

Thumbnail image for Business unIntelligence Cover.jpgWhile writing in Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data about some of the uses of big data way back in February, 2012, I came across Charles Duhigg's New York Times piece on Target's ability to predict not only the pregnancy of female (obviously) customers but also their likely delivery dates. Over the New Year break, I finally found time to read Duhigg's book, The Power of Habit, in which that story first appeared. My point in my own book, and in a related blog Death by a thousand analytics, was that using big data in this manner is both a clear invasion of privacy and likely to creep people out. But Duhigg's book deals with a much broader topic--human habitual behavior and unconscious thought patterns--that has far deeper implications for predictive analytics and decision support in general.

The message of The Power of Habit is fairly easily summarized. A very significant proportion of our thinking, decision-making and action at personal, organizational and societal levels is entirely habitual. Given a particular cue (which can be anything from a thought to an action), a routine set of behaviors is immediately and automatically engaged, which provides an anticipated reward. Simply put: cue => routine => reward. This mechanism operates, both in its set up and execution, in one of the most primitive areas of the brain, the basal ganglia, a structure that derives in evolutionary terms from the earliest chordates. It is, in fact, a basic survival mechanism that enables us undertake a wide range of necessary life-preserving activities, from preparing food to getting home, with minimal expenditure of energy and attention. This, of course, initially freed the brain to deal with novel, life-threatening situations and more recently for higher-brain functions such as empathy and reasoning. However, as anyone who has ever tried to give up biting their nails or mid-afternoon snacking can testify, changing habitual patterns can be very difficult indeed, even those which are counterproductive or even downright dangerous.

This very primitive mental operation of cue-routine-reward has two (at least) important consequences for predictive analytics.

For marketing, the drive will be to move increasingly to an understanding of specific, individual habits and their cues, in order to create opportunities to influence behavior. This is more difficult than it sounds, as such cues are often difficult even for the individuals themselves to identify. Similarly, rewards are often far more subtle and diverse than might be imagined. The story of how Pepsodent toothpaste was sold to an American public famed for poor oral hygiene in the early 1900s has become a classic marketing exemplar of how to create a habit and a sales success. But, according to Duhigg, the actual cue and reward involved were long mistaken. Furthermore, although most Western people are by now inured to mass manipulation by advertising, the much more personal and private knowledge used for such individual influencing may raise resistance to such approaches. From the point of view of privacy, I am of the opinion that it most certainly should.

In the case of personal and organizational decision making support, habitual and other unconscious patterns are a serious impediment to efforts to become "data-driven". My unfortunate lovelorn experience of disbelieving a predicted but undesired outcome is a basic behavior pattern well-recognized in psychology but seldom mentioned in business intelligence. Habitual beliefs and behaviors at an organizational level are so deeply embedded and invisible to participants that they stymie efforts even to recognize problems, never mind take effective action.

The bottom line is that no matter how much data you gather or how elaborate the models you generate, Business unIntelligence operates finally, in full glory or gory fullness, in the minds of your customers and business users.


Posted January 6, 2014 3:40 AM
Permalink | No Comments |


   VISIT MY EXPERT CHANNEL

Search this blog
Categories ›
Archives ›
Recent Entries ›