We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Blog: Barry Devlin Subscribe to this blog's RSS feed!

Barry Devlin

As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein.

About the author >

Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

December 2013 Archives

Sex_Lies_and_Videotape.jpgIn this last blog of 2013, I want to make some serious points that should be front and center (he said coyly) for all analytic, BI and big data professionals next year. In fact, they should be already. And if not, I urge you to use some reflection time over the coming holiday to make them so.

Let's start with sex... toys. Consider the value of deep analytic big data monitoring to enforce the following two verified laws in the USA: (i) you may not have more than two dildos in the same house in Arizona and (ii) it's illegal to own more than six dildos in Texas. (I refuse to speculate on why Texans may be less susceptible to moral degeneracy than citizens of Arizona.) I suspect most of us are mildly amused by such legislation. Many have probably long taken comfort in the belief that it is unenforceable; after all, who can imagine the local cops breaking into your home on suspicion that you are hoarding sex toys? But wait. Haven't the authorities in the US and other countries been rummaging through your digital drawers for years now, and without any just cause for suspicion? Hasn't at least one supermarket chain collated and analyzed data to determine if its (female) customers have recently become pregnant? Haven't two technology companies applied for patents to spy on (sorry, gather behavioral data to enable better targeted advertising to) you via your living room TV? How far will government and business go in mining our personal lives, our sexuality, our bodies, our social circles to (allegedly) protect us, cure us or sell us more (unnecessary) stuff.

ThanksforSharingPOSTER.jpgThe sad truth is that we have lost most of our privacy already, having entered into a Faustian pact to share, both knowingly and unwittingly, the details of our daily lives. That knowing part--the Facebook likes and Google +1s--may be said to represent a conscious tradeoff by the person sharing between a loss of privacy and a perceived increase in social capital or useful contextual information. Even the acceptance that our smartphones report our location minute by minute is driven by a consensual belief that we may be offered a coupon for a nearby coffee shop at any moment. The payoff for ultimate traceability. Apple iBeacon allows newer iPhones or Android phones with Bluetooth Low Energy devices to track their--and your--position in space with centimeter precision. Which aisle in the supermarket are you in? What about some very specific retail therapy recommendations? These, and other soon to emerge toys, have the addictive quality of sex to many of the current generation of CMOs and proponents of big analytics.

However, the smartphone is but the pioneer species of the internet of things, the ultimate in small toys for big data boys. As sensors become ever smaller, cheaper and ever more powerful, the utopian vision is of systems that respond instantly to our needs, that anticipate our very expectations. We are promised houses that know we're home and adjust lighting and heading accordingly. Wrist bands that sense when we awake in the morning, so that the coffee can be brewed, or know that our elderly aunt has not moved for the past hour and may have slipped and broken her hip. Software to the rescue, hardware to alleviate yet another chore. According to Computerworld, Gartner predicts that half of all BI implementations will incorporate machine data from the internet of things by 2017. Research conducted by EMA and 9sight during the summer of 2013, suggested that the future is already here; machine-generated data overtook human-sourced information as big data sources among our respondents. And the more details of our mundane activities that become available in the internet of things for analysis and correlation, the more specific and identifiable our individual patterns of behavior become and the more difficult it is to retain any degree of anonymity. IBM's 5 for 5 this year includes "a digital guardian that will protect you online" within 5 years. But it's mostly about security rather privacy, and already years too late.

Thumbnail image for Business unIntelligence Cover.jpgRealists may like to consider how useful such tools would have been to the Stasi in German Democratic Republic (East Germany) or the Soviet Union's KGB in the second half of the last century. Would the world be as we know it if they had? The more dystopian among us conjure up visions of Franz Kafka's The Trial or George Orwell's 1984 only 30 years late in arrival. But, it is not my intention to propose neo-Luddism. The big data jinni is already well out of the bottle. Despite the silly-bugger games we are currently playing with it, big data does hold the possibility to understand and address many of the most intractable environmental, climatic and social issues we face today. Although, as I've mentioned repeatedly in "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data", our success in acting upon, never mind solving, them will depend far more on our very human intention towards what we desire than on all the data, software and hardware we apply.

So, as we look to 2014, I urge all of us to keep three resolutions in our minds, a subtle matrix underpinning every business need we evaluate and every design decision we make:

  1. Understand and account for the relationship between big data and the traditional core business information long created and processed in existing operational and informational systems; each complements and contextualizes the other

  2. In gathering and analyzing big data, consider how its use can impact personal privacy, especially the ways in which such data can be combined with other big data sets and compromise anonymity

  3. Perhaps most importantly, consider the universal/global impact of the project: how does it contribute to or mitigate the real-world issues of environmental degradation, over-production and consumption, economic instability, and more. In short, how does it support the best in humanity and the world?

320px-Trinity_Test_Fireball_16ms.jpg
These, especially the last, may sound like utopian dreams. But, consider the almost unimaginable power unleashed by the unprecedented growth and interconnection of information currently in process. We are unleashing a potential for good or ill far greater than that created by a small team of physicists in Los Alamos during the Manhattan Project.



I wish you all a Happy Christmas and a Peaceful and Prosperous New Year, within whatever tradition you choose to celebrate this time of year.

Posted December 18, 2013 9:15 AM
Permalink | No Comments |
Sex_Lies_and_Videotape.jpgIn this last blog of 2013, I want to make some serious points that should be front and center (he said coyly) for all analytic, BI and big data professionals next year. In fact, they should be already. And if not, I urge you to use some reflection time over the coming holiday to make them so.

Let's start with sex... toys. Consider the value of deep analytic big data monitoring to enforce the following two verified laws in the USA: (i) you may not have more than two dildos in the same house in Arizona and (ii) it's illegal to own more than six dildos in Texas. (I refuse to speculate on why Texans may be less susceptible to moral degeneracy than citizens of Arizona.) I suspect most of us are mildly amused by such legislation. Many have probably long taken comfort in the belief that it is unenforceable; after all, who can imagine the local cops breaking into your home on suspicion that you are hoarding sex toys? But wait. Haven't the authorities in the US and other countries been rummaging through your digital drawers for years now, and without any just cause for suspicion? Hasn't at least one supermarket chain collated and analyzed data to determine if its (female) customers have recently become pregnant? Haven't two technology companies applied for patents to spy on (sorry, gather behavioral data to enable better targeted advertising to) you via your living room TV? How far will government and business go in mining our personal lives, our sexuality, our bodies, our social circles to (allegedly) protect us, cure us or sell us more (unnecessary) stuff.

ThanksforSharingPOSTER.jpgThe sad truth is that we have lost most of our privacy already, having entered into a Faustian pact to share, both knowingly and unwittingly, the details of our daily lives. That knowing part--the Facebook likes and Google +1s--may be said to represent a conscious tradeoff by the person sharing between a loss of privacy and a perceived increase in social capital or useful contextual information. Even the acceptance that our smartphones report our location minute by minute is driven by a consensual belief that we may be offered a coupon for a nearby coffee shop at any moment. The payoff for ultimate traceability. Apple iBeacon allows newer iPhones or Android phones with Bluetooth Low Energy devices to track their--and your--position in space with centimeter precision. Which aisle in the supermarket are you in? What about some very specific retail therapy recommendations? These, and other soon to emerge toys, have the addictive quality of sex to many of the current generation of CMOs and proponents of big analytics.

However, the smartphone is but the pioneer species of the internet of things, the ultimate in small toys for big data boys. As sensors become ever smaller, cheaper and ever more powerful, the utopian vision is of systems that respond instantly to our needs, that anticipate our very expectations. We are promised houses that know we're home and adjust lighting and heading accordingly. Wrist bands that sense when we awake in the morning, so that the coffee can be brewed, or know that our elderly aunt has not moved for the past hour and may have slipped and broken her hip. Software to the rescue, hardware to alleviate yet another chore. According to Computerworld, Gartner predicts that half of all BI implementations will incorporate machine data from the internet of things by 2017. Research conducted by EMA and 9sight during the summer of 2013, suggested that the future is already here; machine-generated data overtook human-sourced information as big data sources among our respondents. And the more details of our mundane activities that become available in the internet of things for analysis and correlation, the more specific and identifiable our individual patterns of behavior become and the more difficult it is to retain any degree of anonymity. IBM's 5 for 5 this year includes "a digital guardian that will protect you online" within 5 years. But it's mostly about security rather privacy, and already years too late.

Thumbnail image for Business unIntelligence Cover.jpgRealists may like to consider how useful such tools would have been to the Stasi in German Democratic Republic (East Germany) or the Soviet Union's KGB in the second half of the last century. Would the world be as we know it if they had? The more dystopian among us conjure up visions of Franz Kafka's The Trial or George Orwell's 1984 only 30 years late in arrival. But, it is not my intention to propose neo-Luddism. The big data jinni is already well out of the bottle. Despite the silly-bugger games we are currently playing with it, big data does hold the possibility to understand and address many of the most intractable environmental, climatic and social issues we face today. Although, as I've mentioned repeatedly in "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data", our success in acting upon, never mind solving, them will depend far more on our very human intention towards what we desire than on all the data, software and hardware we apply.

So, as we look to 2014, I urge all of us to keep three resolutions in our minds, a subtle matrix underpinning every business need we evaluate and every design decision we make:

  1. Understand and account for the relationship between big data and the traditional core business information long created and processed in existing operational and informational systems; each complements and contextualizes the other

  2. In gathering and analyzing big data, consider how its use can impact personal privacy, especially the ways in which such data can be combined with other big data sets and compromise anonymity

  3. Perhaps most importantly, consider the universal/global impact of the project: how does it contribute to or mitigate the real-world issues of environmental degradation, over-production and consumption, economic instability, and more. In short, how does it support the best in humanity and the world?

320px-Trinity_Test_Fireball_16ms.jpg
These, especially the last, may sound like utopian dreams. But, consider the almost unimaginable power unleashed by the unprecedented growth and interconnection of information currently in process. We are unleashing a potential for good or ill far greater than that created by a small team of physicists in Los Alamos during the Manhattan Project.



I wish you all a Happy Christmas and a Peaceful and Prosperous New Year, within whatever tradition you choose to celebrate this time of year.

Posted December 18, 2013 9:15 AM
Permalink | No Comments |
sounion_athens1.jpgTraditionally, BI has been a process-free zone. Decision makers are such free thinkers that suggesting their methods of working can be defined by some stogy process is generally met with sneers of derision. Or worse. BI vendors and developers have largely acquiesced; the only place you see process mentioned is in data integration, where activity flow diagrams abound to define the steps needed to populate the data warehouse and marts.

I, on the other hand, have long held - since the turn of the millennium, in fact - that all decision making follows a process, albeit a very flexible and adaptive one. The early proof emerges in operational BI (or decision management, as it's also called) where decision making steps are embedded in fairly traditional operational processes. As predictive and operational analytics has become increasingly popular, this intermingling of informational and operational is such that these once distinctly different business behaviors are becoming indistinguishable. A relatively easy thought experiment then leads to the conclusion that all decision making has an underlying process.

I was also fairly sure at an early stage that only a Service Oriented Architecture (SOA) approach could provide the flexible and adaptive activities and workflows required. I further saw that SOA could (and would need to) be a foundation for data integration as the demand for near real-time decision making grew. As a result, I have been discussing all this at seminars and conferences for many years now. But every time I'd mention SOA, the sound of discontent would rumble around the room. Too complex. Tried it and failed. And, more recently, isn't that all old hat now with cloud and mobile?

All of this is by way of introduction to a very interesting briefing I received this week from Pat Pruchnickyj, Director of Product Marketing at Talend, who restored my faith in SOA as an overall approach and in its practical application! Although perhaps best known for its open source ETL (extract, transform and load) and data integration tooling it first introduced in 2006, Talend today take a broader view and offers data focused solutions, such as ETL and data quality, as well as open source application integration solutions, such as enterprise service bus (ESB) and message queuing. These various approaches are united by common metadata, typically created and managed through a graphical, workflow-oriented tool, Talend Open Studio.

So, why is this important? If you follow the history of BI, you'll know that many well-established implementations are characterized by complex and often long-running batch processes that gather, consolidate and cleanse data from multiple internal operational sources into a data warehouse and then to marts. This is a model that scales poorly in an era where vast volumes of data are coming from external sources (a substantial part of big data) and analysis is increasingly demanding near real-time data. File-based data integration becomes a challenge in these circumstances. The simplest approach may be to move towards ever smaller files running in micro-batches. However, the ultimate requirement is to enable message-based communication between source and target applications/databases. This requires a fundamental change in thinking for most BI developers. So a starting point of ETL and an end point of messaging, both under a common ETL-like workflow, makes for easier growth. Developers can begin to see that a data transfer/cleansing service is conceptually similar to any business activity also offered as a service. And the possibility of creating workflows combining operational and informational processes emerges naturally to support operational BI.

Thumbnail image for Business unIntelligence Cover.jpgIs this to say that ETL tools are a dying species? Certainly not. For some types and sizes of data integration, a file-based approach will continue to offer higher performance or more extensive integration and cleansing function. The key is to ensure common, shared metadata (or as I prefer to call it, context-setting information, CSI) between all the different flavors of data and application integration.

Process, including both business and IT aspects, is the subject of Chapter 7 of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data".

Sunset Over Architecture (SOA) image: http://vorris.blogspot.com/2012/07/mr-cameron-you-are-darn-right-start.html

Posted December 12, 2013 3:46 AM
Permalink | No Comments |
google-glass-utopia-dystopia.jpgThe ninth in a series of posts introducing the concepts and messages of my new book, "Business unIntelligence--Insight and Innovation Beyond Analytics and Big Data", now available!

I took a few days off last week and read the first science fiction book in maybe twenty years...and the first computer science fiction I've ever read. The book was briefly recommended by an attendee at my presentation in DW2013 in Zurich a few weeks ago, a thought inspired (I imagine) by some of my comments about various possibilities that social collaboration could offer to innovation in business. The book in question is "The Circle" by Dave Eggers, which explores the logical and apparently inevitable conclusion to current developments in social networking and search in the world at large. Set in a post-Facebook, post-Google world, the Circle is the all-encompassing company that delivers and controls all aspects of social interaction on the Web. And it is moving towards a vision encapsulated by the slogan "Secrets are Lies. Sharing is Caring. Privacy is Theft" crafted by the somewhat clueless heroine, Mae, who goes Transparent (recording everything she sees, hears and says on a live Web feed) but seems totally unaware of the delicious irony of using "bathroom breaks" to conceal some activities strictly unrelated to toilet.

While Mae's unquestioning embrace of all things social may seem a tad naive to many of us old-timers, there is little doubt that many people and organizations believe implicitly in one or more aspects of what the Circle proposes. One example is the explosion of video surveillance, both overt and covert, that is purported to improve citizens' behavior on the basis that increasing the risk of getting caught decreases our propensity to misbehave. Dan Ariely in "The (Honest) Truth about Dishonesty" begs to differ. According to his research, the Simple Model of Rational Crime, devised by University of Chicago economist and Nobel laureate, Gary Becker, which proposes that people commit crimes based on a rational analysis of each situation, is too simplistic. Rather, he believes that our behavior is driven by two conflicting motivations, one is to see ourselves as honest, honorable people and the second is that we want to benefit from cheating and get as much money as possible. However, our cognitive flexibility allows us to rationalize that if cheat by only a little bit, we can benefit from cheating and still view ourselves as honest. The extent of that "little bit" differs from person to person and determines how we actually behave. Personally, I suspect that this model of human motivation is also too simplistic, but, not being a professor at either Duke or MIT, I don't have the resources to test that hypothesis... My fundamental concern is that the erosion of privacy inherent in ongoing surveillance is being ignored in the misguided belief that society is benefiting from improved behavior.

In my last post, I discussed similar privacy issues that seem to go unnoticed when advances in technology allow monitoring our behavior while watching TV, in the interest of targeting advertising or improving monetization of pay-per-view programming.

Of course, exactly the same ethical issues arise as we consider the use of informal information within the enterprise to improve insight on what motivates decision makers and increases their ability to innovate. In my book, I define informal information as information generated as part of every process, in both formal activities (project kick-off and plan review meetings, shareholder and board meetings, court proceedings, etc.) but also during informal activities (chats at the water cooler, ad hoc meetings, phone calls, conferences attended, and so on) that is increasingly captured and stored digitally. A brief look at the two lists of activities above will show just how many are or could easily be recorded. My belief is that such records can be analyzed to understand how innovative thinking emerges and is often suppressed in teams working together on any creative project. In video-conferenced meetings, for example, recording and analysis of facial micro-expressions could show when, how and by whom particular ideas are approved or otherwise by team members and how this affects their proponents. Combine it with spoken and written comments, project success or failure and other information and we can build over time an understanding of how to construct innovative teams by blending skills and attitudes in the optimal proportions to encourage invention and balance it with the constructive criticism and enthusiastic support needed to convert it to innovation.

So, you can probably see the benefits of that as a proponent of decision support software if you really want to look beyond traditional BI. The above thinking is the logical conclusion of analytics, using yet further sets of big data. We get to see not just how participants behave but also gain insight into their subconscious reactions and motivations. And if you think this is science fiction, a recent New York Times article, "When Algorithms Grow Accustomed to Your Face", will convince you otherwise, but be careful how your expression may reveal your reaction, especially if a Webcam is pointing at you!

In terms of ethics and privacy, we are now treading on very dangerous ground. Increasingly, the technology is allowing us to record and analyze human motivation and intention. It can be used for good or ill. We must apply some of the innovative thinking that created this technology to figuring out how to control and manage its use.


Thumbnail image for Thumbnail image for Business unIntelligence Cover.jpg
I will be further exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and "Beyond BI is... Business unIntelligence" recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Montage inspired by ExtremeTech article on Google Glass. 

Posted December 4, 2013 2:37 AM
Permalink | No Comments |


   VISIT MY EXPERT CHANNEL

Search this blog
Categories ›
Archives ›
Recent Entries ›