We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Blog: Barry Devlin Subscribe to this blog's RSS feed!

Barry Devlin

As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein.

About the author >

Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

November 2013 Archives

gravity.jpgThe eighth in a series of posts introducing the concepts and messages of my new book, "Business unIntelligence--Insight and Innovation Beyond Analytics and Big Data", now available!

Decision making seldom, if ever, occurs in a vacuum. Even in Gravity, all of Sandra Bullock's life-or-death decisions happened in tiny and delicate bubbles of air. All business decisions are likewise made in bubbles. One type of bubble is of the information variety, where the information available limits the view of what's possible in terms of action or outcome. The insidious effect of such bubbles, with particular emphasis on our personal lives is well described by Eli Pariser in The Filter Bubble: What the Internet Is Hiding from You, Penguin Press (2011). However, my interest here in in another bubble--an intention bubble that hides the broader consequences of decisions from their makers.

As I point out in Business unIntelligence, much of what drives our decisions--whether in business or in personal matters--is far from the fully rational, data-driven approach long espoused and gaining even more traction in this big data era. Modern psychology and neurobiology show us a growing set of internal, physical and mental processes that contribute significantly to all decision making. These include emotional state, energy level, psychological integration, intuition and more. In order of application, perhaps the first of these "non-rational" factors is the intentionality of the decision maker. A recent news story illustrates the danger of the intention bubble, particularly in the hyper-connected, big data world of modern marketing.

A search for LG Smart TV at the moment shows a story of snooping filling most of the first page. As first described by developer, tweaker and Linux enthusiast, DoctorBeet on his eponymous Blog, he was a little curious about the choice of advertisements his new TV was serving up on its landing screen. A packet analysis of data sent to LG unsurprisingly showed lots of information about channel choices. More concerning was that the names of personal files viewed from a USB device were also transmitted, all unencrypted. A link to an LG site, which has been "currently under maintenance" every time I've checked since, revealed LG's intention: "LG Smart Ad analyses users favourite programs, online behaviour, search keywords and other information to offer relevant ads to target audiences. For example, LG Smart Ad can feature sharp suits to men, or alluring cosmetics and fragrances to women. Furthermore, LG Smart Ad offers useful and various advertising performance reports. That live broadcasting ads cannot. To accurately identify actual advertising effectiveness," according to DoctorBeet. Following the online uproar, I imagine that LG may fall back on "coding error" or some such reason for the scope of data being collected and other excesses. However, few in the advertising industry or among proponents of big data will quibble with the intention. But, perhaps we should...

Some links in TechDirt's coverage of the story led to even creepier behavior-monitoring ideas, this time in patents applied for by Microsoft and Verizon. Microsoft's "consumer detector" could enable "the system [to] charge for the television show or movie based on the number of viewers in the room. Or, if the number of viewers exceeds the limits laid out by a particular content license, the system would halt playback unless additional viewing rights were purchased." Verizon was contemplating analyzing "ambient action" captured by the front camera now on most smart TV devices. According to the patent, "ambient action may include the user eating, exercising, laughing, reading, sleeping, talking, singing, humming, cleaning, playing a musical instrument, performing any other suitable action, and/or engaging in any other physical activity during the presentation of the media content." Should we assume that excluded from such "suitable actions" are those of a sexual nature that immediately spring to mind as often occurring in front of a TV?

At issue here is about how carefully we must attend to the intention of decision makers about what big data to collect, how it could be used, what could happen if it was combined with other available data, and so on. Further, we need to consider the intention of anybody who has legitimate access to such data. Let's be clear, the temptation for misuse extends far beyond NSA and other spy agency employees. Is anybody considering that there is a line that could or should be drawn here on how far businesses can go in behavioral analysis?

It was my intention to address innovation in this article. I decided the above was more topical, and at least equally important. I'll come back to innovative decision making in a later post.


Thumbnail image for Thumbnail image for Business unIntelligence Cover.jpg
I will be further exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and "Beyond BI is... Business unIntelligence" recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Posted November 25, 2013 7:37 AM
Permalink | No Comments |
gravity.jpgThe eighth in a series of posts introducing the concepts and messages of my new book, "Business unIntelligence--Insight and Innovation Beyond Analytics and Big Data", now available!

Decision making seldom, if ever, occurs in a vacuum. Even in Gravity, all of Sandra Bullock's life-or-death decisions happened in tiny and delicate bubbles of air. All business decisions are likewise made in bubbles. One type of bubble is of the information variety, where the information available limits the view of what's possible in terms of action or outcome. The insidious effect of such bubbles, with particular emphasis on our personal lives is well described by Eli Pariser in The Filter Bubble: What the Internet Is Hiding from You, Penguin Press (2011). However, my interest here in in another bubble--an intention bubble that hides the broader consequences of decisions from their makers.

As I point out in Business unIntelligence, much of what drives our decisions--whether in business or in personal matters--is far from the fully rational, data-driven approach long espoused and gaining even more traction in this big data era. Modern psychology and neurobiology show us a growing set of internal, physical and mental processes that contribute significantly to all decision making. These include emotional state, energy level, psychological integration, intuition and more. In order of application, perhaps the first of these "non-rational" factors is the intentionality of the decision maker. A recent news story illustrates the danger of the intention bubble, particularly in the hyper-connected, big data world of modern marketing.

A search for LG Smart TV at the moment shows a story of snooping filling most of the first page. As first described by developer, tweaker and Linux enthusiast, DoctorBeet on his eponymous Blog, he was a little curious about the choice of advertisements his new TV was serving up on its landing screen. A packet analysis of data sent to LG unsurprisingly showed lots of information about channel choices. More concerning was that the names of personal files viewed from a USB device were also transmitted, all unencrypted. A link to an LG site, which has been "currently under maintenance" every time I've checked since, revealed LG's intention: "LG Smart Ad analyses users favourite programs, online behaviour, search keywords and other information to offer relevant ads to target audiences. For example, LG Smart Ad can feature sharp suits to men, or alluring cosmetics and fragrances to women. Furthermore, LG Smart Ad offers useful and various advertising performance reports. That live broadcasting ads cannot. To accurately identify actual advertising effectiveness," according to DoctorBeet. Following the online uproar, I imagine that LG may fall back on "coding error" or some such reason for the scope of data being collected and other excesses. However, few in the advertising industry or among proponents of big data will quibble with the intention. But, perhaps we should...

Some links in TechDirt's coverage of the story led to even creepier behavior-monitoring ideas, this time in patents applied for by Microsoft and Verizon. Microsoft's "consumer detector" could enable "the system [to] charge for the television show or movie based on the number of viewers in the room. Or, if the number of viewers exceeds the limits laid out by a particular content license, the system would halt playback unless additional viewing rights were purchased." Verizon was contemplating analyzing "ambient action" captured by the front camera now on most smart TV devices. According to the patent, "ambient action may include the user eating, exercising, laughing, reading, sleeping, talking, singing, humming, cleaning, playing a musical instrument, performing any other suitable action, and/or engaging in any other physical activity during the presentation of the media content." Should we assume that excluded from such "suitable actions" are those of a sexual nature that immediately spring to mind as often occurring in front of a TV?

At issue here is about how carefully we must attend to the intention of decision makers about what big data to collect, how it could be used, what could happen if it was combined with other available data, and so on. Further, we need to consider the intention of anybody who has legitimate access to such data. Let's be clear, the temptation for misuse extends far beyond NSA and other spy agency employees. Is anybody considering that there is a line that could or should be drawn here on how far businesses can go in behavioral analysis?

It was my intention to address innovation in this article. I decided the above was more topical, and at least equally important. I'll come back to innovative decision making in a later post.


Thumbnail image for Thumbnail image for Business unIntelligence Cover.jpg
I will be further exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and "Beyond BI is... Business unIntelligence" recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Posted November 25, 2013 7:37 AM
Permalink | No Comments |
susurration.jpgIt seems to me that much of the drive behind NoSQL (whether No SQL or Not Only SQL) arose from a rather narrow view of the relational model and technology by web-oriented developers whose experience was constrained by the strengths and limitations of MySQL. Many of their criticisms of relational databases had actually been overcome by commercial products like DB2, Oracle and Teradata to varying extents and under certain circumstances. Although, of course, open source and commodity hardware pricing also continue to drive uptake.

A similar pattern can be seen with NewSQL in its original definition by Matt Aslett of the 451 group, back in April 2011. So, when it comes to products clamoring for inclusion in either category, I tend to be somewhat jaundiced. A class defined by what it is not (NoSQL) presents some logical difficulties. And one classed "new", when today's new is tomorrow's obsolete is not much better. I prefer to look at products in a more holistic sense. With that in mind, let's get to NuoDB, which announced version 2 in mid-October. With my travel schedule I didn't find time to blog then, but now that I'm back on terra firma in Cape Town, the time has come!

Back in October 2012, I encountered NuoDB prior to their initial launch, and their then positioning as part of the NewSQL wave. I also had a bit of a rant then about the NoSQL/NewSQL nomenclature (although no one listened then either), and commented on the technical innovation in the product, which quite impressed me, saying "NuoDB takes a highly innovative, object-oriented, transaction/messaging-system approach to the underlying database processing, eliminating the concept of a single control process responsible for all aspects of database integrity and organization. [T]he approach is described as elastically scalable - cashing in on the cloud and big data.  It also touts emergent behavior, a concept central to the theory of complex systems. Together with an in-memory model for data storage, NuoDB appears very well positioned to take advantage of the two key technological advances of recent years... extensive memory and multi-core processors."

The concept of emergent behavior (the idea that the database could be anything anybody wanted it to be, with SQL simply as first model) was interesting technically but challenging in positioning the product. Version 2 is more focused, with a tagline of distributed database and an emphasis on scale-out and geo-distribution within the relational paradigm. This makes more sense in marketing terms and the use case in a global VoIP support environment shows how the product can be used to reduce latency and improve data consistency. No need to harp on about "NewSQL" then...

Sales aside, the underlying novel technical architecture continues to interest me. A reading on the NuoDB Technical Whitepaper (registration required) revealed some additional gems. One, in particular, resonates with my thinking on the ongoing breakdown of one of the longest-standing postulates of decision support: the belief that operational and informational processes demand separate databases to support them, as discussed in Chapter 5 of my book. While there continue to be valid business reasons to build and maintain a separate store of core, historical information, real-time decision needs also demand the ability to support both operational and informational needs on the primary data store. NuoDB's Transaction Engine architecture and use of Multi-Version Concurrency Control together enable good performance of both read/write and longer-running read-only operations seen in operational BI applications.

Business unIntelligence Cover.jpgI will return to exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and Beyond BI is... Business unIntelligence, recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Susurration image: http://bigjoebuck.blogspot.com/2010_12_27_archive.html

Posted November 20, 2013 1:55 AM
Permalink | No Comments |
susurration.jpgIt seems to me that much of the drive behind NoSQL (whether No SQL or Not Only SQL) arose from a rather narrow view of the relational model and technology by web-oriented developers whose experience was constrained by the strengths and limitations of MySQL. Many of their criticisms of relational databases had actually been overcome by commercial products like DB2, Oracle and Teradata to varying extents and under certain circumstances. Although, of course, open source and commodity hardware pricing also continue to drive uptake.

A similar pattern can be seen with NewSQL in its original definition by Matt Aslett of the 451 group, back in April 2011. So, when it comes to products clamoring for inclusion in either category, I tend to be somewhat jaundiced. A class defined by what it is not (NoSQL) presents some logical difficulties. And one classed "new", when today's new is tomorrow's obsolete is not much better. I prefer to look at products in a more holistic sense. With that in mind, let's get to NuoDB, which announced version 2 in mid-October. With my travel schedule I didn't find time to blog then, but now that I'm back on terra firma in Cape Town, the time has come!

Back in October 2012, I encountered NuoDB prior to their initial launch, and their then positioning as part of the NewSQL wave. I also had a bit of a rant then about the NoSQL/NewSQL nomenclature (although no one listened then either), and commented on the technical innovation in the product, which quite impressed me, saying "NuoDB takes a highly innovative, object-oriented, transaction/messaging-system approach to the underlying database processing, eliminating the concept of a single control process responsible for all aspects of database integrity and organization. [T]he approach is described as elastically scalable - cashing in on the cloud and big data.  It also touts emergent behavior, a concept central to the theory of complex systems. Together with an in-memory model for data storage, NuoDB appears very well positioned to take advantage of the two key technological advances of recent years... extensive memory and multi-core processors."

The concept of emergent behavior (the idea that the database could be anything anybody wanted it to be, with SQL simply as first model) was interesting technically but challenging in positioning the product. Version 2 is more focused, with a tagline of distributed database and an emphasis on scale-out and geo-distribution within the relational paradigm. This makes more sense in marketing terms and the use case in a global VoIP support environment shows how the product can be used to reduce latency and improve data consistency. No need to harp on about "NewSQL" then...

Sales aside, the underlying novel technical architecture continues to interest me. A reading on the NuoDB Technical Whitepaper (registration required) revealed some additional gems. One, in particular, resonates with my thinking on the ongoing breakdown of one of the longest-standing postulates of decision support: the belief that operational and informational processes demand separate databases to support them, as discussed in Chapter 5 of my book. While there continue to be valid business reasons to build and maintain a separate store of core, historical information, real-time decision needs also demand the ability to support both operational and informational needs on the primary data store. NuoDB's Transaction Engine architecture and use of Multi-Version Concurrency Control together enable good performance of both read/write and longer-running read-only operations seen in operational BI applications.

Business unIntelligence Cover.jpgI will return to exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and Beyond BI is... Business unIntelligence, recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Susurration image: http://bigjoebuck.blogspot.com/2010_12_27_archive.html

Posted November 20, 2013 1:55 AM
Permalink | No Comments |