Blog: Barry Devlin http://www.b-eye-network.com/blogs/devlin/ As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein. Copyright 2014 Tue, 19 Aug 2014 03:28:03 -0700 http://www.movabletype.org/?v=4.261 http://blogs.law.harvard.edu/tech/rss Datameer offers Hadoop-based data mart Datameer could be seen as the Business Objects of the Hadoop world. And it's that thought that leads me to data marts.

As one of the oldest and most divisive debates in business intelligence, it's clear that the time-to-value discussions of data warehouse vs. data mart also apply to Hadoop. Hadoop is increasingly being used to integrate data from a wide variety of sources for analysis, begging the question: do it in advance for data quality or do it as part of the analysis to reduce time to value? Datameer is clearly a data mart.

And in the big data world, it's certainly not the only data mart type of offering. What's different about Datameer is that it has been around for nearly 5 years and has an impressive customer base.

At an architectural level, we should consider how the quality vs. timeliness, mart vs. warehouse trade-off applies in the world of big data. Read more on this at my new blog location.]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/08/datameers_hadoo.php http://www.b-eye-network.com/blogs/devlin/archives/2014/08/datameers_hadoo.php Tue, 19 Aug 2014 03:28:03 -0700
Datameer offers Hadoop-based data mart Datameer could be seen as the Business Objects of the Hadoop world. And it's that thought that leads me to data marts.

As one of the oldest and most divisive debates in business intelligence, it's clear that the time-to-value discussions of data warehouse vs. data mart also apply to Hadoop. Hadoop is increasingly being used to integrate data from a wide variety of sources for analysis, begging the question: do it in advance for data quality or do it as part of the analysis to reduce time to value? Datameer is clearly a data mart.

And in the big data world, it's certainly not the only data mart type of offering. What's different about Datameer is that it has been around for nearly 5 years and has an impressive customer base.

At an architectural level, we should consider how the quality vs. timeliness, mart vs. warehouse trade-off applies in the world of big data. Read more on this at my new blog location.]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/08/datameer_offers.php http://www.b-eye-network.com/blogs/devlin/archives/2014/08/datameer_offers.php Tue, 19 Aug 2014 03:28:03 -0700
Eating the elephant called Hadoop eat elephant.jpgHadoop vendors Hortonworks, Cloudera and, most recently, MapR have all amassed substantial cash stashes. This has triggered much speculation about both who will win the lion's share of the the big data market and how the elephant will rampage through the data warehousing landscape. Missing from such debate is an understanding of the central role of information management and its automation in the evolution and eventual success of data warehousing.

Although showing rapid evolution, the Hadoop software environment is still focused on fundamental database, data manipulation and similar technologies. In data warehousing, the focus long ago shifted to ensuring data quality and consistency, from modeling business requirements all the way through to production delivery and ongoing maintenance. We see this in tools such as Wherescape and Kalido, built by teams who had to develop and support real, ongoing and changing business intelligence needs.

Read the full story at my new blog location: Now... Business unIntelligence.]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/07/eating_the_elep.php http://www.b-eye-network.com/blogs/devlin/archives/2014/07/eating_the_elep.php Fri, 11 Jul 2014 00:43:40 -0700
Eating the elephant called Hadoop eat elephant.jpgHadoop vendors Hortonworks, Cloudera and, most recently, MapR have all amassed substantial cash stashes. This has triggered much speculation about both who will win the lion's share of the the big data market and how the elephant will rampage through the data warehousing landscape. Missing from such debate is an understanding of the central role of information management and its automation in the evolution and eventual success of data warehousing.

Although showing rapid evolution, the Hadoop software environment is still focused on fundamental database, data manipulation and similar technologies. In data warehousing, the focus long ago shifted to ensuring data quality and consistency, from modeling business requirements all the way through to production delivery and ongoing maintenance. We see this in tools such as Wherescape and Kalido, built by teams who had to develop and support real, ongoing and changing business intelligence needs.

Read the full story at my new blog location: Now... Business unIntelligence.]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/07/eating_the_elep_1.php http://www.b-eye-network.com/blogs/devlin/archives/2014/07/eating_the_elep_1.php Fri, 11 Jul 2014 00:43:40 -0700
Link - Hadoop: the Third Wave Breaks Although the yellow elephant continues to trample all over the world of Information Management, it is becoming increasingly difficult to say where more traditional technologies end and Hadoop begins.

Flying Elephant londonjunglebook8.jpg

Actian's (@ActianCorp) presentation at the #BBBT on 24 June emphasized again that the boundaries of the Hadoop world are becoming very ill-defined indeed, as more traditional engines are adapted to run on or in the Hadoop cluster.

The Actian Analytics Platform - Hadoop SQL Edition embeds their existing X100 / Vectorwise SQL engine directly in the nodes of the Hadoop environment. The approach offers the full range of SQL support previously available in Vectorwise on Hadoop. Architecturally as interesting, is the creation and use of column-based, binary, compressed vector files by the X100 engine for improved performance and the subsequent replication of these files by the Hadoop system. These latter files support co-location of data for joins for a further performance boost.

This is, of course, the type of integration one would expect from seasoned database developers when they migrate to a new platform. Pivotal's HAWQ has Greenplum technology embedded. It would be surprising if IBM's on-Hadoop Big SQL offering is not based on DB2 knowledge at the very least.

The real point is that the mix and match of functionality and data seen here emphasizes the conundrum I posed at the top of the blog. Where does Hadoop end? And where does "NoHadoop" (well, if we can have NoSQL...) begin? What does this all mean for the evolution of Information Management technology over the coming few years?

Read full post.

]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/06/hadoop_the_thir.php http://www.b-eye-network.com/blogs/devlin/archives/2014/06/hadoop_the_thir.php Thu, 26 Jun 2014 08:44:11 -0700
Link - Hadoop: the Third Wave Breaks Although the yellow elephant continues to trample all over the world of Information Management, it is becoming increasingly difficult to say where more traditional technologies end and Hadoop begins.

Flying Elephant londonjunglebook8.jpg

Actian's (@ActianCorp) presentation at the #BBBT on 24 June emphasized again that the boundaries of the Hadoop world are becoming very ill-defined indeed, as more traditional engines are adapted to run on or in the Hadoop cluster.

The Actian Analytics Platform - Hadoop SQL Edition embeds their existing X100 / Vectorwise SQL engine directly in the nodes of the Hadoop environment. The approach offers the full range of SQL support previously available in Vectorwise on Hadoop. Architecturally as interesting, is the creation and use of column-based, binary, compressed vector files by the X100 engine for improved performance and the subsequent replication of these files by the Hadoop system. These latter files support co-location of data for joins for a further performance boost.

This is, of course, the type of integration one would expect from seasoned database developers when they migrate to a new platform. Pivotal's HAWQ has Greenplum technology embedded. It would be surprising if IBM's on-Hadoop Big SQL offering is not based on DB2 knowledge at the very least.

The real point is that the mix and match of functionality and data seen here emphasizes the conundrum I posed at the top of the blog. Where does Hadoop end? And where does "NoHadoop" (well, if we can have NoSQL...) begin? What does this all mean for the evolution of Information Management technology over the coming few years?

Read full post.

]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/06/link_-_hadoop_t.php http://www.b-eye-network.com/blogs/devlin/archives/2014/06/link_-_hadoop_t.php Thu, 26 Jun 2014 08:44:11 -0700
Analytics, Big Data and Protecting Privacy Privacy Padlock.pngIn the year since Edward Snowden spoke out on governmental spying, much has been written about privacy but little enough done to protect personal information, either from governments or from big business.

It's now a year since the material gathered by Edward Snowden at the NSA was first published by the Guardian and Washington Post newspapers. In one of a number of anniversary-related items, Vodafone revealed that secret wires are mandated in "about six" of the 29 countries in which it operates. It also noted that, in addition, Albania, Egypt, Hungary, India, Malta, Qatar, Romania, South Africa and Turkey deem it unlawful to disclose any information related to wiretapping or content interception. Vodafone's move is to be welcomed. Hopefully, it will encourage further transparency from other telecommunications providers on governmental demands for information.

However, governmental big data collection and analysis is only one aspect of this issue. Personal data is also of keen interest to a range of commercial enterprises, from telcos themselves to retailers and financial institutions, not to mention the Internet giants, such as Google and Facebook, which are the most voracious consumers of such information. Many people are rightly concerned about how governments--from allegedly democratic to manifestly totalitarian--may use our personal data. To be frank, the dangers are obvious. However, commercial uses of personal data are more insidious, and potentially more dangerous and destructive to humanity. Governments at least purport to represent the people to a greater or lesser extent; commercial enterprises don't even wear that minimal fig leaf.

Take, as one example among many, indoor proximity detection systems based on Bluetooth Low Energy devices such as Apple's iBeacon and Google's rumored upcoming Nearby. The inexorable progress of communications technology--smaller, faster, cheaper, lower power--enables more and more ways of determining the location of your smartphone or tablet and, by extension, you. The operating system or app on your phone requires an opt-in to enable it to transmit your location. However, it is becoming increasingly difficult to avoid opting-in as many apps require it to work at all. More worrying are the systems that record and track without asking permission the MAC addresses of smartphones and tablets that poll public Wi-Fi network routers, which all such devices automatically do. (See, for example, this article, subscription required.) The only way to avoid such tracking is to turn off the device's Wi-Fi receiver. On the desktop, the situation is little better, with Facebook last week joining Google and Yahoo! in ignoring browser "do not track" settings.

It would be simple to blame the businesses involved--both the technology companies that develop the systems and the businesses that buy or use the data. They certainly must take their fair share of responsibility, together with the data scientists and other IT staff involved in building the systems. But the reality is that it is we, the general public, who hand over our personal data without a second thought about its possible uses, who must step up to demanding real change in the collection and use of such data. This demands significant rethinking in at least two areas.

First is the oft-repeated marketing story that "people want more targeted advertising", reiterated again last week by Facebook's Brian Boland. A more nuanced view is provided by Sara M. Watson, a Fellow at the Berkman Center for Internet and Society at Harvard University, in a recent Atlantic article Data Doppelgängers and the Uncanny Valley of Personalization: "Data tracking and personalized advertising is often described as 'creepy.' Personalized ads and experiences are supposed to reflect individuals, so when these systems miss their mark, they can interfere with a person's sense of self. It's hard to tell whether the algorithm doesn't know us at all, or if it actually knows us better than we know ourselves. And it's disconcerting to think that there might be a glimmer of truth in what otherwise seems unfamiliar. This goes beyond creepy, and even beyond the sense of being watched."

I would suggest that given the choice between less irrelevant advertising or, simply, less advertising on the Web, many people would opt for the latter, particularly given the increasing invasiveness of the data collection needed to drive allegedly more accurate targeting. Clearly, this latter choice would not be in the interest of the advertising industry, a position that crystalizes in the widespread resistance to limits on data gathering, especially in the United States. An obvious first step in addressing this issue is a people-driven, legally mandated move from opt-out data gathering to a formal opt-in approach. To be really useful, of course, this would need to be preceded by a widespread mass deletion of previously gathered data.

This leads directly to the second area in need of substantial rethinking--the funding model for Internet business. Most of us accept that "there's no such thing as a free lunch". But a free email service, Cloud store or search engine, well apparently that's eminently reasonable. Of course, it isn't. All these services cost money to build and run, costs that are covered (with significant profits in many cases) by advertising. More of it and supposedly better targeted via big data and analytics.

There is little doubt that the majority of people using the Internet gain real, daily value from it. Today, that value is paid for through personal data. The loss of privacy seems barely noticed. People I ask are largely disinterested in any possible consequences. However, privacy is the foundation for many aspects of society, including democracy--as can be clearly seen in totalitarian states, where widespread surveillance and destruction of privacy are among the first orders of business. We, the users of the Web, must do the unthinkable: we must demand the right to pay real money for mobile access, search, email and so on in exchange for an end to tracking personal data.

These are but two arguably simplistic suggestions to address issues that have been made more obvious by Snowden's revelations. A more complete theoretical and legal foundation for a new approach is urgently needed. One possible starting point is The Dangers of Surveillance by Neil Richards, Professor of Law at Washington University Law, published in the Harvard Law Review a few short months before Snowden spilled at least some of the beans.

Image courtesy Marc Kjerland
]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/06/analytics_big_d.php http://www.b-eye-network.com/blogs/devlin/archives/2014/06/analytics_big_d.php Business unIntelligence Thu, 19 Jun 2014 00:53:23 -0700
Analytics, Big Data and Protecting Privacy Privacy Padlock.pngIn the year since Edward Snowden spoke out on governmental spying, much has been written about privacy but little enough done to protect personal information, either from governments or from big business.

It's now a year since the material gathered by Edward Snowden at the NSA was first published by the Guardian and Washington Post newspapers. In one of a number of anniversary-related items, Vodafone revealed that secret wires are mandated in "about six" of the 29 countries in which it operates. It also noted that, in addition, Albania, Egypt, Hungary, India, Malta, Qatar, Romania, South Africa and Turkey deem it unlawful to disclose any information related to wiretapping or content interception. Vodafone's move is to be welcomed. Hopefully, it will encourage further transparency from other telecommunications providers on governmental demands for information.

However, governmental big data collection and analysis is only one aspect of this issue. Personal data is also of keen interest to a range of commercial enterprises, from telcos themselves to retailers and financial institutions, not to mention the Internet giants, such as Google and Facebook, which are the most voracious consumers of such information. Many people are rightly concerned about how governments--from allegedly democratic to manifestly totalitarian--may use our personal data. To be frank, the dangers are obvious. However, commercial uses of personal data are more insidious, and potentially more dangerous and destructive to humanity. Governments at least purport to represent the people to a greater or lesser extent; commercial enterprises don't even wear that minimal fig leaf.

Take, as one example among many, indoor proximity detection systems based on Bluetooth Low Energy devices such as Apple's iBeacon and Google's rumored upcoming Nearby. The inexorable progress of communications technology--smaller, faster, cheaper, lower power--enables more and more ways of determining the location of your smartphone or tablet and, by extension, you. The operating system or app on your phone requires an opt-in to enable it to transmit your location. However, it is becoming increasingly difficult to avoid opting-in as many apps require it to work at all. More worrying are the systems that record and track without asking permission the MAC addresses of smartphones and tablets that poll public Wi-Fi network routers, which all such devices automatically do. (See, for example, this article, subscription required.) The only way to avoid such tracking is to turn off the device's Wi-Fi receiver. On the desktop, the situation is little better, with Facebook last week joining Google and Yahoo! in ignoring browser "do not track" settings.

It would be simple to blame the businesses involved--both the technology companies that develop the systems and the businesses that buy or use the data. They certainly must take their fair share of responsibility, together with the data scientists and other IT staff involved in building the systems. But the reality is that it is we, the general public, who hand over our personal data without a second thought about its possible uses, who must step up to demanding real change in the collection and use of such data. This demands significant rethinking in at least two areas.

First is the oft-repeated marketing story that "people want more targeted advertising", reiterated again last week by Facebook's Brian Boland. A more nuanced view is provided by Sara M. Watson, a Fellow at the Berkman Center for Internet and Society at Harvard University, in a recent Atlantic article Data Doppelgí¤ngers and the Uncanny Valley of Personalization: "Data tracking and personalized advertising is often described as 'creepy.' Personalized ads and experiences are supposed to reflect individuals, so when these systems miss their mark, they can interfere with a person's sense of self. It's hard to tell whether the algorithm doesn't know us at all, or if it actually knows us better than we know ourselves. And it's disconcerting to think that there might be a glimmer of truth in what otherwise seems unfamiliar. This goes beyond creepy, and even beyond the sense of being watched."

I would suggest that given the choice between less irrelevant advertising or, simply, less advertising on the Web, many people would opt for the latter, particularly given the increasing invasiveness of the data collection needed to drive allegedly more accurate targeting. Clearly, this latter choice would not be in the interest of the advertising industry, a position that crystalizes in the widespread resistance to limits on data gathering, especially in the United States. An obvious first step in addressing this issue is a people-driven, legally mandated move from opt-out data gathering to a formal opt-in approach. To be really useful, of course, this would need to be preceded by a widespread mass deletion of previously gathered data.

This leads directly to the second area in need of substantial rethinking--the funding model for Internet business. Most of us accept that "there's no such thing as a free lunch". But a free email service, Cloud store or search engine, well apparently that's eminently reasonable. Of course, it isn't. All these services cost money to build and run, costs that are covered (with significant profits in many cases) by advertising. More of it and supposedly better targeted via big data and analytics.

There is little doubt that the majority of people using the Internet gain real, daily value from it. Today, that value is paid for through personal data. The loss of privacy seems barely noticed. People I ask are largely disinterested in any possible consequences. However, privacy is the foundation for many aspects of society, including democracy--as can be clearly seen in totalitarian states, where widespread surveillance and destruction of privacy are among the first orders of business. We, the users of the Web, must do the unthinkable: we must demand the right to pay real money for mobile access, search, email and so on in exchange for an end to tracking personal data.

These are but two arguably simplistic suggestions to address issues that have been made more obvious by Snowden's revelations. A more complete theoretical and legal foundation for a new approach is urgently needed. One possible starting point is The Dangers of Surveillance by Neil Richards, Professor of Law at Washington University Law, published in the Harvard Law Review a few short months before Snowden spilled at least some of the beans.

Image courtesy Marc Kjerland
]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/06/analytics_big_d_1.php http://www.b-eye-network.com/blogs/devlin/archives/2014/06/analytics_big_d_1.php Thu, 19 Jun 2014 00:53:23 -0700
Reining in the Internet of Things Thoughts on the societal impact of the Internet of Things inspired by a unique dashboard product.

VisualCue tile.pngNewcomer to the BBBT, on 2nd May, Kerry Gilger, Founder of VisualCue took the members by storm with an elegant, visually intuitive and, to me at least, novel approach to delivering dashboards. VisualCue is based on the concept of a tile that represents a set of metrics as icons colored according to their state relative to defined threshold values. The main icon in the tile shown here represents the overall performance of a call center agent, with the secondary icons showing other KPIs, such as total calls answered, average handling time, sales per hour worked, customer satisfaction, etc. Tiles are assembled into mosaics, which function rather like visual bar charts that can be sorted according to the different metrics, drilled down to related items and displayed in other formats, including tabular numbers.

Visual Cue Mosaic.jpgThe product seems particularly useful in operational BI applications, with Kerry showing examples from call centers, logistics and educational settings. The response of the BBBT members was overwhelmingly positive. @rick_vanderlans described it as "revolutionary technology", while @gildardorojas asked "why we didn't have before something as neat and logical?" @marcusborba opined "@VisualCue's capability is amazing, and the data visualization is gorgeous!"

So, am I being a Luddite, or even a curmudgeon, to have made the only negative comments of the call? My concern was not about the product at all, but rather around the power it unleashes simply by being so good at what it does. Combine this level of ease-of-use in analytics with big data and, especially, data from the Internet of Things, and we take a quantum leap from measurement to invasiveness, from management to Big-Brother-like control.

Each of the three example use cases described by Gilger provided wonderful examples of real and significant business benefit; but, taken together, they also opened up appalling possibilities of abuse of privacy, misappropriation of personal information and disempowerment of the people involved. I'll briefly explore the three examples, realizing that in the absence of the full story, I'm undoubtedly imagining some aspects. Nor is this about VisualCue (who tweeted that "Privacy is certainly a critical issue! We focus on presenting data that an organization already has--maybe we make it obvious") or the companies using it; it's meant to be a warning that we who know some of the possibilities--positive and negative--offered by big data analytics must consider in advance the unintended consequences.

Detailed monitoring of call center agents' performance is nothing new. Indeed, it is widely seen as best practice and key to improving both individual and overall call center results. VisualCue, according to Gilger, has provided outstanding performance gains, including one center where agents in competition with peers have personally sought out training to improve their own metrics, something that is apparently unheard of in the industry. Based on past best practices and detailed knowledge of where the agent is weak, VisualCue can provide individually customized advice. In a sense, this example illustrates the pinnacle of such use of monitoring data and analytics to drive personnel performance. But, within it lies the seeds of its own destruction. As the agent's job is more and more broken down into repeatable tasks, each measurable by a different metric, human innovation and empathy is removed and the job prepared for automation. In fact, a 2013 study puts at 99% the probability that certain call center jobs, particularly telemarketing, will be soon eliminated by technology.

The old adage "what you can't measure, you can't manage" is at the heart of traditional BI. In an era when data was scarce and often incoherent, this focus makes sense. However, applying it to all aspects of life today is, to me, ethically problematical. The example of monitoring the entire scope of an educational institution in a single dashboard--from financials through administration to student performance--is a case where our ability to analyze so many data points leads to the illusion that we can manage the entire process mechanically. The Latin root of "educate" means "to draw forth" from the student, the success of which simply cannot be gauged through basic numerical measures, and is certainly not correlated with the business measures of the institution.

vehtrack.jpgThe final example of tracking the operational performance of a waste management company's routes, trucks and drivers emphasizes our growing ability to measure and monitor the details of real life minute by minute. By continuously tracking the location and engine management signals from its trucks, the dashboard created by this company enabled it to make significant financial savings and improvements to its operational performance. However, it also enables supervisors to drill into the ongoing behavior of the company's drivers: deviations from planned routes, long stops with the engine running, extreme braking, exceeding the speed limit, etc. While presumably covered by their employment contract, such micromanagement of employees is at best disempowering and at worst open to abuse by increasingly all-seeing supervisors. Of much greater concern is the fact that these sensors are increasingly embedded in private automobiles and that such tracking capability is already being applied without owners' consent to smartphones. As far as a year back, Euclid Analytics had already tracked about 50 million devices in 4,000 locations according to a New York Times blog.

1984-big-brother-is-watching-you.jpgI'm grateful to Kerry Gilger for sharing the use cases that inspired my speculations above. Of course, my point is beyond the individual companies involved and products used. At issue is the range of social and ethical dilemmas raised by the rapid advances in sensor technology, data gathered and the power of analytic software. Our every action online is already monitored by the likes of Google and Facebook for profit and by organizations like the NSA allegedly for security and crime prevention. The level of monitoring of our physical lives is now rapidly increasing. Anonymity is rapidly disappearing, if not already extinct. Our personal privacy rights are being usurped by the data gathering and analysis programs of these commercial and governmental organizations, as eloquently described by Shoshana Zuboff of Harvard Business and Law schools in a recent article in Frankfurter Allgemeine Zeitung.

It is imperative that those of us who have grown up with and nurtured business intelligence over the past three decades--from hardware and software vendors, to consultants and analysts, to BI managers and implementers in businesses everywhere--begin to deeply consider the ethical, legal and societal issues now being raised and take action to guide the industry and society appropriately through the development of new codes of ethical behavior and use of information, and input to national and international legislation.

]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/05/reining_in_the_1.php http://www.b-eye-network.com/blogs/devlin/archives/2014/05/reining_in_the_1.php Sun, 04 May 2014 06:26:47 -0700
Reining in the Internet of Things Thoughts on the societal impact of the Internet of Things inspired by a unique dashboard product.

VisualCue tile.pngNewcomer to the BBBT, on 2nd May, Kerry Gilger, Founder of VisualCue took the members by storm with an elegant, visually intuitive and, to me at least, novel approach to delivering dashboards. VisualCue is based on the concept of a tile that represents a set of metrics as icons colored according to their state relative to defined threshold values. The main icon in the tile shown here represents the overall performance of a call center agent, with the secondary icons showing other KPIs, such as total calls answered, average handling time, sales per hour worked, customer satisfaction, etc. Tiles are assembled into mosaics, which function rather like visual bar charts that can be sorted according to the different metrics, drilled down to related items and displayed in other formats, including tabular numbers.

Visual Cue Mosaic.jpgThe product seems particularly useful in operational BI applications, with Kerry showing examples from call centers, logistics and educational settings. The response of the BBBT members was overwhelmingly positive. @rick_vanderlans described it as "revolutionary technology", while @gildardorojas asked "why we didn't have before something as neat and logical?" @marcusborba opined "@VisualCue's capability is amazing, and the data visualization is gorgeous!"

So, am I being a Luddite, or even a curmudgeon, to have made the only negative comments of the call? My concern was not about the product at all, but rather around the power it unleashes simply by being so good at what it does. Combine this level of ease-of-use in analytics with big data and, especially, data from the Internet of Things, and we take a quantum leap from measurement to invasiveness, from management to Big-Brother-like control.

Each of the three example use cases described by Gilger provided wonderful examples of real and significant business benefit; but, taken together, they also opened up appalling possibilities of abuse of privacy, misappropriation of personal information and disempowerment of the people involved. I'll briefly explore the three examples, realizing that in the absence of the full story, I'm undoubtedly imagining some aspects. Nor is this about VisualCue (who tweeted that "Privacy is certainly a critical issue! We focus on presenting data that an organization already has--maybe we make it obvious") or the companies using it; it's meant to be a warning that we who know some of the possibilities--positive and negative--offered by big data analytics must consider in advance the unintended consequences.

Detailed monitoring of call center agents' performance is nothing new. Indeed, it is widely seen as best practice and key to improving both individual and overall call center results. VisualCue, according to Gilger, has provided outstanding performance gains, including one center where agents in competition with peers have personally sought out training to improve their own metrics, something that is apparently unheard of in the industry. Based on past best practices and detailed knowledge of where the agent is weak, VisualCue can provide individually customized advice. In a sense, this example illustrates the pinnacle of such use of monitoring data and analytics to drive personnel performance. But, within it lies the seeds of its own destruction. As the agent's job is more and more broken down into repeatable tasks, each measurable by a different metric, human innovation and empathy is removed and the job prepared for automation. In fact, a 2013 study puts at 99% the probability that certain call center jobs, particularly telemarketing, will be soon eliminated by technology.

The old adage "what you can't measure, you can't manage" is at the heart of traditional BI. In an era when data was scarce and often incoherent, this focus makes sense. However, applying it to all aspects of life today is, to me, ethically problematical. The example of monitoring the entire scope of an educational institution in a single dashboard--from financials through administration to student performance--is a case where our ability to analyze so many data points leads to the illusion that we can manage the entire process mechanically. The Latin root of "educate" means "to draw forth" from the student, the success of which simply cannot be gauged through basic numerical measures, and is certainly not correlated with the business measures of the institution.

vehtrack.jpgThe final example of tracking the operational performance of a waste management company's routes, trucks and drivers emphasizes our growing ability to measure and monitor the details of real life minute by minute. By continuously tracking the location and engine management signals from its trucks, the dashboard created by this company enabled it to make significant financial savings and improvements to its operational performance. However, it also enables supervisors to drill into the ongoing behavior of the company's drivers: deviations from planned routes, long stops with the engine running, extreme braking, exceeding the speed limit, etc. While presumably covered by their employment contract, such micromanagement of employees is at best disempowering and at worst open to abuse by increasingly all-seeing supervisors. Of much greater concern is the fact that these sensors are increasingly embedded in private automobiles and that such tracking capability is already being applied without owners' consent to smartphones. As far as a year back, Euclid Analytics had already tracked about 50 million devices in 4,000 locations according to a New York Times blog.

1984-big-brother-is-watching-you.jpgI'm grateful to Kerry Gilger for sharing the use cases that inspired my speculations above. Of course, my point is beyond the individual companies involved and products used. At issue is the range of social and ethical dilemmas raised by the rapid advances in sensor technology, data gathered and the power of analytic software. Our every action online is already monitored by the likes of Google and Facebook for profit and by organizations like the NSA allegedly for security and crime prevention. The level of monitoring of our physical lives is now rapidly increasing. Anonymity is rapidly disappearing, if not already extinct. Our personal privacy rights are being usurped by the data gathering and analysis programs of these commercial and governmental organizations, as eloquently described by Shoshana Zuboff of Harvard Business and Law schools in a recent article in Frankfurter Allgemeine Zeitung.

It is imperative that those of us who have grown up with and nurtured business intelligence over the past three decades--from hardware and software vendors, to consultants and analysts, to BI managers and implementers in businesses everywhere--begin to deeply consider the ethical, legal and societal issues now being raised and take action to guide the industry and society appropriately through the development of new codes of ethical behavior and use of information, and input to national and international legislation.

]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/05/reining_in_the.php http://www.b-eye-network.com/blogs/devlin/archives/2014/05/reining_in_the.php Internet of Things Sun, 04 May 2014 06:26:47 -0700
Automating the Data Warehouse (and beyond) eco-skyscraper-by-vikas-pawar-2a.jpgIn an era of "big data this" and "Internet of Things that", it's refreshing to step back to some of the basic principles of defining, building and maintaining data stores that support the process of decision making... or data warehousing, as we old-fashioned folks call it. Kalido did an excellent job last Friday of reminding the BBBT just what is needed to automate the process of data warehouse management. But, before the denizens of the data lake swim away with a bored flick of their tails, let me point out that this matters for big data too--maybe even more so. I'll return to this towards the end of this post.

In the first flush of considering a BI or analytics opportunity in the business and conceiving a solution that delivers exactly the right data needed to address that pesky problem, it's easy to forget the often rocky road of design and development ahead. More often forgotten, or sometimes ignored, is the ongoing drama of maintenance. Kalido, with their origins as an internal IT team solving a real problem for the real business of Royal Dutch Shell in the late '90s, have kept these challenges front and center.

All IT projects begin with business requirements, but data warehouses have a second, equally important, staring point: existing data sources. These twin origins typically lead to two largely disconnected processes. First, there is the requirements activity often called data modeling, but more correctly seen as the elucidation of a business model, consisting of function required by the business and data needed to support it. Second, there is the ETL-centric process of finding and understanding the existing sources of this data, figuring out how to prepare and condition it, and designing the physical database elements needed to support the function required.

Most data warehouse practitioners recognize that the disconnect between these two development processes is the origin of much of the cost and time expended in delivering a data warehouse. And they figure out a way through it. Unfortunately, they often fail to recognize that each time a new set of data must be added or an existing set updated, they have to work around the problem yet again. So, not only is initial development impacted, but future maintenance remains an expensive and time-consuming task. An ideal approach is to create an integrated environment that automates the entire set of tasks from business requirements documentation, through the definition and execution of data preparation, all the way to database design and tuning. Kalido is one of a small number of vendors who have taken this all-inclusive approach. They report build effort reductions of 60-85% in data warehouse development.

Conceptually, we move from focusing on the detailed steps (ETL) of preparing data to managing the metadata that relates the business model to the physical database design. The repetitive and error-prone donkey-work of ETL, job management and administration is automated. The skills required in IT change from programming-like to modeling-like. This has none of the sexiness of predictive analytics or self-service BI. Rather, it's about real IT productivity. Arguably, good IT shops always create some or all of this process- and metadata-management infrastructure themselves around their chosen modeling, ETL and database tools. Kalido is "just" a rather complete administrative environment for these processes.

Which brings me finally back to the shores of the data lake. As described, the data lake consists of a Hadoop-based store of all the data a business could ever need, in its original structure and form, and into which any business user can dip a bucket and retrieve the data required without IT blocking the way. However, whether IT is involved or not, the process of understanding the business need and getting the data from the lake into a form that is useful and usable for a decision-making requirement is exactly identical to that described in my third paragraph above. The same problems apply. Trust me, similar solutions will be required.

Image: http://inhabitat.com/vikas-pawar-skyscraper/


]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/03/automating_the_1.php http://www.b-eye-network.com/blogs/devlin/archives/2014/03/automating_the_1.php Mon, 17 Mar 2014 05:33:30 -0700
Automating the Data Warehouse (and beyond) eco-skyscraper-by-vikas-pawar-2a.jpgIn an era of "big data this" and "Internet of Things that", it's refreshing to step back to some of the basic principles of defining, building and maintaining data stores that support the process of decision making... or data warehousing, as we old-fashioned folks call it. Kalido did an excellent job last Friday of reminding the BBBT just what is needed to automate the process of data warehouse management. But, before the denizens of the data lake swim away with a bored flick of their tails, let me point out that this matters for big data too--maybe even more so. I'll return to this towards the end of this post.

In the first flush of considering a BI or analytics opportunity in the business and conceiving a solution that delivers exactly the right data needed to address that pesky problem, it's easy to forget the often rocky road of design and development ahead. More often forgotten, or sometimes ignored, is the ongoing drama of maintenance. Kalido, with their origins as an internal IT team solving a real problem for the real business of Royal Dutch Shell in the late '90s, have kept these challenges front and center.

All IT projects begin with business requirements, but data warehouses have a second, equally important, staring point: existing data sources. These twin origins typically lead to two largely disconnected processes. First, there is the requirements activity often called data modeling, but more correctly seen as the elucidation of a business model, consisting of function required by the business and data needed to support it. Second, there is the ETL-centric process of finding and understanding the existing sources of this data, figuring out how to prepare and condition it, and designing the physical database elements needed to support the function required.

Most data warehouse practitioners recognize that the disconnect between these two development processes is the origin of much of the cost and time expended in delivering a data warehouse. And they figure out a way through it. Unfortunately, they often fail to recognize that each time a new set of data must be added or an existing set updated, they have to work around the problem yet again. So, not only is initial development impacted, but future maintenance remains an expensive and time-consuming task. An ideal approach is to create an integrated environment that automates the entire set of tasks from business requirements documentation, through the definition and execution of data preparation, all the way to database design and tuning. Kalido is one of a small number of vendors who have taken this all-inclusive approach. They report build effort reductions of 60-85% in data warehouse development.

Conceptually, we move from focusing on the detailed steps (ETL) of preparing data to managing the metadata that relates the business model to the physical database design. The repetitive and error-prone donkey-work of ETL, job management and administration is automated. The skills required in IT change from programming-like to modeling-like. This has none of the sexiness of predictive analytics or self-service BI. Rather, it's about real IT productivity. Arguably, good IT shops always create some or all of this process- and metadata-management infrastructure themselves around their chosen modeling, ETL and database tools. Kalido is "just" a rather complete administrative environment for these processes.

Which brings me finally back to the shores of the data lake. As described, the data lake consists of a Hadoop-based store of all the data a business could ever need, in its original structure and form, and into which any business user can dip a bucket and retrieve the data required without IT blocking the way. However, whether IT is involved or not, the process of understanding the business need and getting the data from the lake into a form that is useful and usable for a decision-making requirement is exactly identical to that described in my third paragraph above. The same problems apply. Trust me, similar solutions will be required.

Image: http://inhabitat.com/vikas-pawar-skyscraper/


]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/03/automating_the.php http://www.b-eye-network.com/blogs/devlin/archives/2014/03/automating_the.php Data warehouse Mon, 17 Mar 2014 05:33:30 -0700
Big Data, the Internet of Things and the Death of Capitalism? Part 5 Parts 1, 2, 3, 4 and 4A of this series explored the problem as I see it. Now, finally, I consider what we might do if my titular question actually makes sense.

Mammoth kill.jpgTo start, let's review my basic thesis. Mass production and competition, facilitated by ever improving technology, have been delivering better and cheaper products and improving many people's lives (at least in the developed world) for nearly two centuries. Capital, in the form of technology, and people--labor--work together in today's system to produce goods that people purchase using earnings from their labor. As technology grows exponentially better, an ever greater range of jobs are open to displacement. When technology displaces some yet to be determined percentage of labor, this system swings out of balance; there are simply not enough people with sufficient money to buy the products made, no matter how cheaply. We have not yet reached this tipping point because, throughout most of this period, the new jobs created by technology have largely offset the losses. However, employment trends in the past 10-15 years in the Western world suggest that this effect is no longer operating to the extent that it was, if at all.

In brief, the problem is that although technology produces greater wealth (as all economists agree), without its transfer to the masses through wages paid for labor, the number of consumers becomes insufficient to justify further production. The owners of the capital assets accumulate more wealth--and we see this happening in the increasing inequality in society--but they cannot match the consumption of the masses. Capitalism, or perhaps more precisely, the free market then collapses.

Let's first look at the production side of the above equation. What can be done to prevent job losses outpacing job creation as a result of technological advances? Can we prevent or put a damper on the great hollowing out of middle-income jobs that is creating a dumbbell-shaped distribution of a few highly-paid experts at one end and a shrinking swathe of lower-paid, less-skilled workers at the other? Can (or should) we move to address the growing imbalance of power and wealth between capital (especially technologically based) and labor? Let's be clear at the start, however, turning off automation is not an option I consider.

My suggestions, emerging mainly from the thinking discussed earlier, are mainly economic and social in nature. An obvious approach is to use the levers of taxation--as is done in many other areas--to drive a desired social outcome. We could, for example, reform taxation and social charges on labor to reduce the cost difference between using people and automating a process. In a similar vein, shifting taxation from labor to capital could also be tried. I can already hear the Tea Party screaming to protect the free market from the damn socialist. But, if my analysis is correct, the free market is about to undergo, at best, a radical change, if employment drops below some critical level. Pulling these levers soon and fairly dramatically is probably necessary; this is an approach that can only delay the inevitable. Another approach is for industry itself to take steps to protect employment. Mark Bonchek , writing in a recent Harvard Business Review blog, describes a few "job entrepreneurs" who maximize jobs instead of profits (but still make profits as well), including one in the Detroit area aimed at creating jobs for unemployed auto workers.

Moving from the producer's side to the consumer's view, profit aside, why did we set off down the road of the Industrial Revolution? To improve people's daily lives, to lessen the load of hard labor, to alleviate drudgery. The early path was not clear. Seven-day labor on the farm was replaced by seven-day labor in the factory. But, by the middle of the last century, working hours were being reduced in the workplace and in the home, food was cheaper and more plentiful; money and time were available for leisure. In theory, the result should have been an improvement in the human condition. In practice, the improvement was subverted by the mass producers. They needed to sell ever more of the goods they could produce so cheaply that profit came mainly through volume sales. Economist Victor Lebow's 1955 proclamation of "The Real Meaning of Consumer Demand" sums it up: "Our enormously productive economy demands that we make consumption our way of life... that we seek our spiritual satisfaction and our ego satisfaction in consumption... We need things consumed, burned up, worn out, replaced and discarded at an ever-increasing rate". Of course, some part of this is human nature, but it has been driven inexorably by advertising. We've ended up in the classic race to the bottom, even to the extent of products being produced with ever shorter lifespans to drive earlier replacement. Such consumption is becoming increasingly unsustainable as the world population grows, finite resources run out and the energy consumed in both production and use drives increasing climate change. As the president of Uruguay, Jose Mujica, asked of the Rio+20 Summit in 2012, "Does this planet have enough resources so seven or eight billion can have the same level of consumption and waste that today is seen in rich societies?"

My counter-intuitive suggestion here, and one I have not seen raised by economists (surprisingly?), is to ramp down consumerism, mainly through a reinvention of the purposes and practices of advertising. Reducing over-competition and over-consumption would probably drive interesting changes in the production side of the equation, including reduced demand for further automation, lower energy consumption, product quality being favored over quantity, higher savings rates by (non-)consumers, and more. Turning down the engine of consumption could also enable changes for the better in the financial markets, reducing the focus on quarterly results in favor of strategically sounder investment. Input from economists would be much appreciated.

But, let's wrap up. The title of this series asked: will automation through big data and the Internet of Things drive the death of capitalism? Although some readers may have assumed that this was my preferred outcome, I am more of the opinion that capitalism and the free market need to evolve rather quickly if they are to survive and, preferably, thrive. But, this would mean some radical changes. For example, a French think-tank, LH Forum, suggests the development of a positive economy that: "reorients capitalism towards long-term challenges. Altruism toward future generations is a much more powerful incentive than [the] selfishness which is supposed to steer the market economy". Other fundamental rethinking comes from British/Scottish historian, Niall Ferguson, who takes a wider view of "The Great Degeneration" of Western civilization. In a word, this is a topic that requires broad, deep and urgent thought.

For my more IT-oriented readers, I suspect this blog series has taken you far from basic ground. For this, I do not apologize. As described in Chapter 2 of "Business unIntelligence", I believe that the future of business and IT is to be joined at the hip. The biz-tech ecosystem declares that technology is at the heart of all business development. Business must understand IT. IT must be involved in the business. I suggest that understanding the impact of automation on business and society is a task for IT strategists and architects as much, if not more, as it is for economists and business planners.

Image from: www.moonbattery.com/archives/2010/07/prehistoric-cli.html. All elephant photos in the earlier posts are my own work!

]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/03/big_data_the_in_6.php http://www.b-eye-network.com/blogs/devlin/archives/2014/03/big_data_the_in_6.php Tue, 11 Mar 2014 06:47:55 -0700
Big Data, the Internet of Things and the Death of Capitalism? Part 5 Parts 1, 2, 3, 4 and 4A of this series explored the problem as I see it. Now, finally, I consider what we might do if my titular question actually makes sense.

Mammoth kill.jpgTo start, let's review my basic thesis. Mass production and competition, facilitated by ever improving technology, have been delivering better and cheaper products and improving many people's lives (at least in the developed world) for nearly two centuries. Capital, in the form of technology, and people--labor--work together in today's system to produce goods that people purchase using earnings from their labor. As technology grows exponentially better, an ever greater range of jobs are open to displacement. When technology displaces some yet to be determined percentage of labor, this system swings out of balance; there are simply not enough people with sufficient money to buy the products made, no matter how cheaply. We have not yet reached this tipping point because, throughout most of this period, the new jobs created by technology have largely offset the losses. However, employment trends in the past 10-15 years in the Western world suggest that this effect is no longer operating to the extent that it was, if at all.

In brief, the problem is that although technology produces greater wealth (as all economists agree), without its transfer to the masses through wages paid for labor, the number of consumers becomes insufficient to justify further production. The owners of the capital assets accumulate more wealth--and we see this happening in the increasing inequality in society--but they cannot match the consumption of the masses. Capitalism, or perhaps more precisely, the free market then collapses.

Let's first look at the production side of the above equation. What can be done to prevent job losses outpacing job creation as a result of technological advances? Can we prevent or put a damper on the great hollowing out of middle-income jobs that is creating a dumbbell-shaped distribution of a few highly-paid experts at one end and a shrinking swathe of lower-paid, less-skilled workers at the other? Can (or should) we move to address the growing imbalance of power and wealth between capital (especially technologically based) and labor? Let's be clear at the start, however, turning off automation is not an option I consider.

My suggestions, emerging mainly from the thinking discussed earlier, are mainly economic and social in nature. An obvious approach is to use the levers of taxation--as is done in many other areas--to drive a desired social outcome. We could, for example, reform taxation and social charges on labor to reduce the cost difference between using people and automating a process. In a similar vein, shifting taxation from labor to capital could also be tried. I can already hear the Tea Party screaming to protect the free market from the damn socialist. But, if my analysis is correct, the free market is about to undergo, at best, a radical change, if employment drops below some critical level. Pulling these levers soon and fairly dramatically is probably necessary; this is an approach that can only delay the inevitable. Another approach is for industry itself to take steps to protect employment. Mark Bonchek , writing in a recent Harvard Business Review blog, describes a few "job entrepreneurs" who maximize jobs instead of profits (but still make profits as well), including one in the Detroit area aimed at creating jobs for unemployed auto workers.

Moving from the producer's side to the consumer's view, profit aside, why did we set off down the road of the Industrial Revolution? To improve people's daily lives, to lessen the load of hard labor, to alleviate drudgery. The early path was not clear. Seven-day labor on the farm was replaced by seven-day labor in the factory. But, by the middle of the last century, working hours were being reduced in the workplace and in the home, food was cheaper and more plentiful; money and time were available for leisure. In theory, the result should have been an improvement in the human condition. In practice, the improvement was subverted by the mass producers. They needed to sell ever more of the goods they could produce so cheaply that profit came mainly through volume sales. Economist Victor Lebow's 1955 proclamation of "The Real Meaning of Consumer Demand" sums it up: "Our enormously productive economy demands that we make consumption our way of life... that we seek our spiritual satisfaction and our ego satisfaction in consumption... We need things consumed, burned up, worn out, replaced and discarded at an ever-increasing rate". Of course, some part of this is human nature, but it has been driven inexorably by advertising. We've ended up in the classic race to the bottom, even to the extent of products being produced with ever shorter lifespans to drive earlier replacement. Such consumption is becoming increasingly unsustainable as the world population grows, finite resources run out and the energy consumed in both production and use drives increasing climate change. As the president of Uruguay, Jose Mujica, asked of the Rio+20 Summit in 2012, "Does this planet have enough resources so seven or eight billion can have the same level of consumption and waste that today is seen in rich societies?"

My counter-intuitive suggestion here, and one I have not seen raised by economists (surprisingly?), is to ramp down consumerism, mainly through a reinvention of the purposes and practices of advertising. Reducing over-competition and over-consumption would probably drive interesting changes in the production side of the equation, including reduced demand for further automation, lower energy consumption, product quality being favored over quantity, higher savings rates by (non-)consumers, and more. Turning down the engine of consumption could also enable changes for the better in the financial markets, reducing the focus on quarterly results in favor of strategically sounder investment. Input from economists would be much appreciated.

But, let's wrap up. The title of this series asked: will automation through big data and the Internet of Things drive the death of capitalism? Although some readers may have assumed that this was my preferred outcome, I am more of the opinion that capitalism and the free market need to evolve rather quickly if they are to survive and, preferably, thrive. But, this would mean some radical changes. For example, a French think-tank, LH Forum, suggests the development of a positive economy that: "reorients capitalism towards long-term challenges. Altruism toward future generations is a much more powerful incentive than [the] selfishness which is supposed to steer the market economy". Other fundamental rethinking comes from British/Scottish historian, Niall Ferguson, who takes a wider view of "The Great Degeneration" of Western civilization. In a word, this is a topic that requires broad, deep and urgent thought.

For my more IT-oriented readers, I suspect this blog series has taken you far from basic ground. For this, I do not apologize. As described in Chapter 2 of "Business unIntelligence", I believe that the future of business and IT is to be joined at the hip. The biz-tech ecosystem declares that technology is at the heart of all business development. Business must understand IT. IT must be involved in the business. I suggest that understanding the impact of automation on business and society is a task for IT strategists and architects as much, if not more, as it is for economists and business planners.

Image from: www.moonbattery.com/archives/2010/07/prehistoric-cli.html. All elephant photos in the earlier posts are my own work!

]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/03/big_data_the_in_5.php http://www.b-eye-network.com/blogs/devlin/archives/2014/03/big_data_the_in_5.php Internet of Things Tue, 11 Mar 2014 06:47:55 -0700
Big Data, the Internet of Things and the Death of Capitalism? Part 4A Parts 1, 2, 3, and 4 of this series explored the problem as I see it and examined the views of some economists and other authors. This was supposed to be the final part, where I answered the question of the title. But, instead I have a bonus blog... inspired by Brynjolfsson and McAfee's Second Machine Age.

Addo Elephant B.JPG"The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies" by Erik Brynjolfsson and Andrew McAfee, published as recently as last January, has been lying half-read on my Kindle desk for some weeks, mainly because its early chapters overlap with much of the technological material I'd referenced already. But a couple of tweets in response to Part 4 (thanks to Michael @mjcavaretta and Patrick @itworks sent me back to reexamine the content. There I found enough interesting thinking to fill a new post. I have to say that I was also hoping that the authors might come to my rescue as I searched for answers to my question above.

The central discussion of the book about the economic effect of advances in technology is couched in terms of bounty and spread. The former is the overall benefit accruing to the economy and the people whose lives are affected. The authors believe this bounty is enormous and growing, although they do admit that traditional economic measures, such as GDP, are no longer adequate. Nonetheless, my gut feel is that technology has, by many measures, improved the lot of humanity, or has the potential to do so. Spread is perhaps more easily quantified: it is the gap in wealth, income, mobility and more between people at the top and bottom of the ladder. And it has been demonstrably growing wider in recent years--which is socially and economically bad news. The authors' question then is whether the growth in bounty can counteract that in spread. Will the rising tide raise all boats, even though the super-yachts may be raised considerably further than the simple rowboats or even the rafts made of society's junk?

The news is bad. The authors report that "between 1983 and 2009, Americans became vastly wealthier overall as the total value of their assets increased. However... the bottom 80 percent of the income distribution actually saw a net decrease in their wealth." Combining this with a second observation that income distribution is moving from a normal (bell) curve towards a power law curve with a long tail of lower incomes, my conclusion is that the negative effect of spread is outpacing the positive lift of bounty. Note finally, that the above discussion is focused on the developed economies. A brief visit to one of the emerging economies should suffice to convince that the inequality there is far greater. The old saw that the rich get richer and the poor get poorer appears increasingly appropriate.

And Brynjolfsson and McAfee do end up agreeing with me in Chapter 11 as they sort through various arguments on the relative importance of bounty and spread. "Instead of being confident that the bounty from technology will more than compensate for the spread it generates, we are instead concerned about something close to the reverse: that the spread could actually reduce the bounty in years to come."

Turning their attention to the main underlying cause, the advance of technology and the possibility of "technological unemployment", they come desperately close to my argument: "there is a floor on how low wages for human labor can go. In turn, that floor can lead to unemployment: people who want to work, but are unable to find jobs. If neither the worker nor any entrepreneur can think of a profitable task that requires that worker's skills and capabilities, then that worker will go unemployed indefinitely. Over history, this has happened to many other inputs to production that were once valuable, from whale oil to horse labor. They are no longer needed in today's economy even at zero price. In other words, just as technology can create inequality, it can also create unemployment. And in theory, this can affect a large number of people, even a majority of the population, and even if the overall economic pie is growing."

But, they instantly--and, in my view, unjustifiably--veer back to the new economic orthodoxy that machines are more likely to complement humans than displace them and then focus on that scenario, suggesting that we must focus on equipping humans for this role through better and different education. While accepting that there are certainly instances where this is true, I see no evidence in recent years that this is the primary scenario we need to address. Only in Chapter 14, do they finally make some "Long-Term Recommendations" that suggest how changes in taxation might be required to rebalance the inequality in income and unemployment that they see eventually emerging. In this they echo both Martin Ford and Tyler Cohen, as we've seen earlier.

But I cannot help but feel this is far too little and far too late. So, what would I suggest? I believe we need some rather counter-intuitive thinking. And that is the long-promised topic of the real Part 5.

]]>
http://www.b-eye-network.com/blogs/devlin/archives/2014/03/big_data_the_in_7.php http://www.b-eye-network.com/blogs/devlin/archives/2014/03/big_data_the_in_7.php Tue, 04 Mar 2014 05:46:35 -0700