We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Blog: Barry Devlin Subscribe to this blog's RSS feed!

Barry Devlin

As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein.

About the author >

Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

google-glass-utopia-dystopia.jpgThe ninth in a series of posts introducing the concepts and messages of my new book, "Business unIntelligence--Insight and Innovation Beyond Analytics and Big Data", now available!

I took a few days off last week and read the first science fiction book in maybe twenty years...and the first computer science fiction I've ever read. The book was briefly recommended by an attendee at my presentation in DW2013 in Zurich a few weeks ago, a thought inspired (I imagine) by some of my comments about various possibilities that social collaboration could offer to innovation in business. The book in question is "The Circle" by Dave Eggers, which explores the logical and apparently inevitable conclusion to current developments in social networking and search in the world at large. Set in a post-Facebook, post-Google world, the Circle is the all-encompassing company that delivers and controls all aspects of social interaction on the Web. And it is moving towards a vision encapsulated by the slogan "Secrets are Lies. Sharing is Caring. Privacy is Theft" crafted by the somewhat clueless heroine, Mae, who goes Transparent (recording everything she sees, hears and says on a live Web feed) but seems totally unaware of the delicious irony of using "bathroom breaks" to conceal some activities strictly unrelated to toilet.

While Mae's unquestioning embrace of all things social may seem a tad naive to many of us old-timers, there is little doubt that many people and organizations believe implicitly in one or more aspects of what the Circle proposes. One example is the explosion of video surveillance, both overt and covert, that is purported to improve citizens' behavior on the basis that increasing the risk of getting caught decreases our propensity to misbehave. Dan Ariely in "The (Honest) Truth about Dishonesty" begs to differ. According to his research, the Simple Model of Rational Crime, devised by University of Chicago economist and Nobel laureate, Gary Becker, which proposes that people commit crimes based on a rational analysis of each situation, is too simplistic. Rather, he believes that our behavior is driven by two conflicting motivations, one is to see ourselves as honest, honorable people and the second is that we want to benefit from cheating and get as much money as possible. However, our cognitive flexibility allows us to rationalize that if cheat by only a little bit, we can benefit from cheating and still view ourselves as honest. The extent of that "little bit" differs from person to person and determines how we actually behave. Personally, I suspect that this model of human motivation is also too simplistic, but, not being a professor at either Duke or MIT, I don't have the resources to test that hypothesis... My fundamental concern is that the erosion of privacy inherent in ongoing surveillance is being ignored in the misguided belief that society is benefiting from improved behavior.

In my last post, I discussed similar privacy issues that seem to go unnoticed when advances in technology allow monitoring our behavior while watching TV, in the interest of targeting advertising or improving monetization of pay-per-view programming.

Of course, exactly the same ethical issues arise as we consider the use of informal information within the enterprise to improve insight on what motivates decision makers and increases their ability to innovate. In my book, I define informal information as information generated as part of every process, in both formal activities (project kick-off and plan review meetings, shareholder and board meetings, court proceedings, etc.) but also during informal activities (chats at the water cooler, ad hoc meetings, phone calls, conferences attended, and so on) that is increasingly captured and stored digitally. A brief look at the two lists of activities above will show just how many are or could easily be recorded. My belief is that such records can be analyzed to understand how innovative thinking emerges and is often suppressed in teams working together on any creative project. In video-conferenced meetings, for example, recording and analysis of facial micro-expressions could show when, how and by whom particular ideas are approved or otherwise by team members and how this affects their proponents. Combine it with spoken and written comments, project success or failure and other information and we can build over time an understanding of how to construct innovative teams by blending skills and attitudes in the optimal proportions to encourage invention and balance it with the constructive criticism and enthusiastic support needed to convert it to innovation.

So, you can probably see the benefits of that as a proponent of decision support software if you really want to look beyond traditional BI. The above thinking is the logical conclusion of analytics, using yet further sets of big data. We get to see not just how participants behave but also gain insight into their subconscious reactions and motivations. And if you think this is science fiction, a recent New York Times article, "When Algorithms Grow Accustomed to Your Face", will convince you otherwise, but be careful how your expression may reveal your reaction, especially if a Webcam is pointing at you!

In terms of ethics and privacy, we are now treading on very dangerous ground. Increasingly, the technology is allowing us to record and analyze human motivation and intention. It can be used for good or ill. We must apply some of the innovative thinking that created this technology to figuring out how to control and manage its use.


Thumbnail image for Thumbnail image for Business unIntelligence Cover.jpg
I will be further exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and "Beyond BI is... Business unIntelligence" recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Montage inspired by ExtremeTech article on Google Glass. 

Posted December 4, 2013 2:37 AM
Permalink | No Comments |
gravity.jpgThe eighth in a series of posts introducing the concepts and messages of my new book, "Business unIntelligence--Insight and Innovation Beyond Analytics and Big Data", now available!

Decision making seldom, if ever, occurs in a vacuum. Even in Gravity, all of Sandra Bullock's life-or-death decisions happened in tiny and delicate bubbles of air. All business decisions are likewise made in bubbles. One type of bubble is of the information variety, where the information available limits the view of what's possible in terms of action or outcome. The insidious effect of such bubbles, with particular emphasis on our personal lives is well described by Eli Pariser in The Filter Bubble: What the Internet Is Hiding from You, Penguin Press (2011). However, my interest here in in another bubble--an intention bubble that hides the broader consequences of decisions from their makers.

As I point out in Business unIntelligence, much of what drives our decisions--whether in business or in personal matters--is far from the fully rational, data-driven approach long espoused and gaining even more traction in this big data era. Modern psychology and neurobiology show us a growing set of internal, physical and mental processes that contribute significantly to all decision making. These include emotional state, energy level, psychological integration, intuition and more. In order of application, perhaps the first of these "non-rational" factors is the intentionality of the decision maker. A recent news story illustrates the danger of the intention bubble, particularly in the hyper-connected, big data world of modern marketing.

A search for LG Smart TV at the moment shows a story of snooping filling most of the first page. As first described by developer, tweaker and Linux enthusiast, DoctorBeet on his eponymous Blog, he was a little curious about the choice of advertisements his new TV was serving up on its landing screen. A packet analysis of data sent to LG unsurprisingly showed lots of information about channel choices. More concerning was that the names of personal files viewed from a USB device were also transmitted, all unencrypted. A link to an LG site, which has been "currently under maintenance" every time I've checked since, revealed LG's intention: "LG Smart Ad analyses users favourite programs, online behaviour, search keywords and other information to offer relevant ads to target audiences. For example, LG Smart Ad can feature sharp suits to men, or alluring cosmetics and fragrances to women. Furthermore, LG Smart Ad offers useful and various advertising performance reports. That live broadcasting ads cannot. To accurately identify actual advertising effectiveness," according to DoctorBeet. Following the online uproar, I imagine that LG may fall back on "coding error" or some such reason for the scope of data being collected and other excesses. However, few in the advertising industry or among proponents of big data will quibble with the intention. But, perhaps we should...

Some links in TechDirt's coverage of the story led to even creepier behavior-monitoring ideas, this time in patents applied for by Microsoft and Verizon. Microsoft's "consumer detector" could enable "the system [to] charge for the television show or movie based on the number of viewers in the room. Or, if the number of viewers exceeds the limits laid out by a particular content license, the system would halt playback unless additional viewing rights were purchased." Verizon was contemplating analyzing "ambient action" captured by the front camera now on most smart TV devices. According to the patent, "ambient action may include the user eating, exercising, laughing, reading, sleeping, talking, singing, humming, cleaning, playing a musical instrument, performing any other suitable action, and/or engaging in any other physical activity during the presentation of the media content." Should we assume that excluded from such "suitable actions" are those of a sexual nature that immediately spring to mind as often occurring in front of a TV?

At issue here is about how carefully we must attend to the intention of decision makers about what big data to collect, how it could be used, what could happen if it was combined with other available data, and so on. Further, we need to consider the intention of anybody who has legitimate access to such data. Let's be clear, the temptation for misuse extends far beyond NSA and other spy agency employees. Is anybody considering that there is a line that could or should be drawn here on how far businesses can go in behavioral analysis?

It was my intention to address innovation in this article. I decided the above was more topical, and at least equally important. I'll come back to innovative decision making in a later post.


Thumbnail image for Thumbnail image for Business unIntelligence Cover.jpg
I will be further exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and "Beyond BI is... Business unIntelligence" recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Posted November 25, 2013 7:37 AM
Permalink | No Comments |
susurration.jpgIt seems to me that much of the drive behind NoSQL (whether No SQL or Not Only SQL) arose from a rather narrow view of the relational model and technology by web-oriented developers whose experience was constrained by the strengths and limitations of MySQL. Many of their criticisms of relational databases had actually been overcome by commercial products like DB2, Oracle and Teradata to varying extents and under certain circumstances. Although, of course, open source and commodity hardware pricing also continue to drive uptake.

A similar pattern can be seen with NewSQL in its original definition by Matt Aslett of the 451 group, back in April 2011. So, when it comes to products clamoring for inclusion in either category, I tend to be somewhat jaundiced. A class defined by what it is not (NoSQL) presents some logical difficulties. And one classed "new", when today's new is tomorrow's obsolete is not much better. I prefer to look at products in a more holistic sense. With that in mind, let's get to NuoDB, which announced version 2 in mid-October. With my travel schedule I didn't find time to blog then, but now that I'm back on terra firma in Cape Town, the time has come!

Back in October 2012, I encountered NuoDB prior to their initial launch, and their then positioning as part of the NewSQL wave. I also had a bit of a rant then about the NoSQL/NewSQL nomenclature (although no one listened then either), and commented on the technical innovation in the product, which quite impressed me, saying "NuoDB takes a highly innovative, object-oriented, transaction/messaging-system approach to the underlying database processing, eliminating the concept of a single control process responsible for all aspects of database integrity and organization. [T]he approach is described as elastically scalable - cashing in on the cloud and big data.  It also touts emergent behavior, a concept central to the theory of complex systems. Together with an in-memory model for data storage, NuoDB appears very well positioned to take advantage of the two key technological advances of recent years... extensive memory and multi-core processors."

The concept of emergent behavior (the idea that the database could be anything anybody wanted it to be, with SQL simply as first model) was interesting technically but challenging in positioning the product. Version 2 is more focused, with a tagline of distributed database and an emphasis on scale-out and geo-distribution within the relational paradigm. This makes more sense in marketing terms and the use case in a global VoIP support environment shows how the product can be used to reduce latency and improve data consistency. No need to harp on about "NewSQL" then...

Sales aside, the underlying novel technical architecture continues to interest me. A reading on the NuoDB Technical Whitepaper (registration required) revealed some additional gems. One, in particular, resonates with my thinking on the ongoing breakdown of one of the longest-standing postulates of decision support: the belief that operational and informational processes demand separate databases to support them, as discussed in Chapter 5 of my book. While there continue to be valid business reasons to build and maintain a separate store of core, historical information, real-time decision needs also demand the ability to support both operational and informational needs on the primary data store. NuoDB's Transaction Engine architecture and use of Multi-Version Concurrency Control together enable good performance of both read/write and longer-running read-only operations seen in operational BI applications.

Business unIntelligence Cover.jpgI will return to exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and Beyond BI is... Business unIntelligence, recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Susurration image: http://bigjoebuck.blogspot.com/2010_12_27_archive.html

Posted November 20, 2013 1:55 AM
Permalink | No Comments |
The seventh in a series of posts introducing the concepts and messages of my new book, "Business unIntelligence--Insight and Innovation Beyond Analytics and Big Data", now available!

Fig 9-4 - Rodin The Thinker.jpgIt has long been assumed in the BI community that more information is "a good thing" when it comes to making better decisions.  Except when there is too much information...when we encounter information overload. Some qualify the thinking by requiring information to be relevant, although that raises questions of how to determine relevance, especially in entirely novel or poorly understood situations. Despite any such reservations, the BI industry generally focuses entirely on information-related issues, from preparation to presentation, and leaves any thinking about the process of how humans make decisions to some unidentified other party.

So, how does insight arise in the human process of decision making? Of course, information does play an important role, but the information we focus on in BI is but a very thin sliver of the actual information that the mind takes into account. Today, neurobiology and psychology tells us that our minds are absorbing information from the earliest days of life, and that such pre-verbal information has significant and unconscious impact on all of the decisions we make during our lives. Such early information conditions our social behavior and expectations of the world. In a similar, although probably in a somewhat more conscious manner, later life experiences create an informational background that colors our thinking in each and every decision we take.

BI purists may argue that such information is unquantifiable and therefore should be discounted.  However, its impact can be large and may be inferred from the ongoing behavior of people involved in the decision making process. When we work in teams, we automatically notice this information and adjust accordingly. Joe always brings up such-and-such a problem. Maria regularly took issue with the previous CFO, but supports the current one, even though the policies and practices are unchanged. This informal information could be gathered electronically and mined for patterns even today to create an entirely new view of how irrational most decision really are.

This leads us to look beyond information as the sole or, even, majority basis for decision making. Rational choice theory has long held sway as the foundation of thinking about business decision making. In recent years, the roles of intuition, gut-feeling, emotional state and intention are slowly coming to the fore as possible contributors.  The BI community has yet to catch up on such thinking. And, indeed, its integration into decision support (to recall that old phrase) requires software and methods that are far from those found in traditional and current BI.

Collaborative and social tools come closest to the foundation of what is required, and I'll address this as we look at innovation in the next article.

Thumbnail image for Thumbnail image for Business unIntelligence Cover.jpg
I will be further exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and "Beyond BI is... Business unIntelligence" recorded Sept. 26. Read my interview with Lindy Ryan in Rediscovering BI.

Upcoming conference appearances where I can share more are in Los Angeles at SPARK! on Oct. 15-16 and in Rome at a dedicated two-day seminar on October 30-31.

Posted October 11, 2013 11:36 AM
Permalink | No Comments |
The sixth in a series of posts introducing the concepts and messages of my new book, "Business unIntelligence--Insight and Innovation Beyond Analytics and Big Data", now available!

Pillars.jpgIn my last post, I introduced the three pillars of the REAL logical architecture of Business unIntelligence. To review briefly, the central, process-oriented data pillar contains traditional operational and core informational data, fed from legally binding transactions (both OLTP and contractual). It is centrally placed because it contains core business information, including traditional operational data and informational EDW and data marts in a single logical component. Machine-generated data and human-sourced information are placed as pillars on either side. The leftmost pillar focuses on real-time and well-structured data, while the one on the right emphasizes the less-structured and, at times, less timely information.

The concept of the pillars emerged from my struggle to clarify the concept of big data and how it relates to the data that businesses have been collecting since we began automating the processes of first running and later managing the business via computers some 50 or more years ago. The pillars are, to a first approximation, delineated my processing and storage concerns. However, there is a more fundamental but less obvious distinction that relates to the sources of the data and information stored and used. These sources are represented by the four clipped boxes at the bottom of the picture.

Let's start in the middle with transactions. Although we often think of transactions in a technical context, more fundamentally they stand for the legally binding agreements (or valid steps towards such agreements) on which all business is based. When we recognize this, we see that traditional operational and informational systems are designed to collect, manage and monitor the contractual and formal information of the business. Before the emergence of big data, we seldom considered in any detail what happened before such transactions occurred.

This thinking leads to the three fundamental information/data sources at the bottom of the picture. Measures and events come from the physical world of machines and show ongoing conditions (e.g. temperature, velocity, location) and changes in conditions (e.g. an acceleration, a button pressed, a call ended).  Messages are human-sourced communications in text, voice, image or video that represent something that one person wants to share with another.  All of these can and do generate transactions when processed in traditional system. Big data simply records them and analyzes them as "ambient data", as Dale Roberts calls it in Decision Sourcing. Because it precedes transactions and, in many cases, does not lead to any transactions, such information leads to a very different view of the world than our traditional systems provide. Where it precedes transactions, we can do predictive analytics about how the formal business will be affected. When it is completely unrelated to transactions, we can gain new insights into reality that can drive true innovation.

I'll talk more about insight and innovation in my next post...

Thumbnail image for Thumbnail image for Business unIntelligence Cover.jpgI will be further exploring the themes and messages of "Business unIntelligence: Insight and Innovation Beyond Analytics and Big Data" over the coming weeks.  You can order the book at the link above; it is now available. Also, check out my presentation at the BrightTALK Business Intelligence and Big Data Analytics Summit, recorded Sept. 11 and "Beyond BI is... Business unIntelligence" recorded Sept. 26.

Upcoming conference appearances where I can share more are in Amsterdam at the Business Analytics Congress on Oct. 9 and in Los Angeles at SPARK! on Oct. 15-16.

Posted October 4, 2013 3:03 AM
Permalink | No Comments |

1 2 3 4 NEXT

   VISIT MY EXPERT CHANNEL

Search this blog
Categories ›
Archives ›
Recent Entries ›