http://www.nytimes.com/2009/01/07/technology/business-computing/07program.html?em

The article suggests R is a threat to SAS. Any perspectives on this or the use of R for data analysis?

The business intelligence (BI) marketplace has often been immune to industry downturns. This is because companies often turn to BI in difficult times to help them identify areas where revenues can be increased and costs can be reduced. This is especially the case in front office sales, marketing, and support organizations. Given the potential size of the coming downturn, however, can even BI be immune? I doubt it.

I believe, however, that there are ways BI can ride out the coming storm and be of benefit to the business. I think the main task that organizations should focus on is using new BI technologies to reduce costs (rather than increasing revenues). This can be achieved by reducing the cost of delivering new BI business solutions and by increasing business user productivity.

The BI solutions that will have the most impact in 2009 will be those that provide IT and business users quick and low-cost approaches for discovering, accessing, integrating, analyzing, delivering and sharing information in a way that that helps business users become more productive and more self-sufficient.

This means that there will be increased interest in open source software, BI software-as-a-service, low-cost application appliances, search, the integration of BI with collaborative and social computing software, rich internet applications, web syndication, and data and presentation mashups. Many of these solutions will come from small innovative BI companies, rather than large software companies who are still struggling to integrate the morass of BI software they acquired in 2008.

The technologies mentioned support low cost and fast BI application deployment. Many of them will be used by line-of-business IT rather than the enterprise IT organization. This could result in a turf war where enterprise IT tries to control and govern the use of these new technologies by the business. This would be a huge mistake. Instead enterprise IT should look for best practices in the use of these technologies by business groups, replicate them in other parts of the organization, and look for ways of incorporating the cream of the crop into the existing IT environment.

The purists will cry that this will lead to anarchy and islands of data and software. If that is the case then so be it. In the coming 12 months we need to do what ever it takes to be productive and reduce short-term costs. This is not the time for fancy architectures, purist approaches, academic debates, or large projects.

Have a great 2009!

MapReduce, a software framework introduced by Google for supporting parallel processing over large petabyte files has garnered significant attention of late. IBM is experimenting with this in conjunction with Google, and GreenPlum recently announced support.

The significant interest in MapReduce, and related technologies such as Hadoop and HDFS, has led to a backlash from the relational camp. David DeWitt and Michael Stonebraker have been especially outspoken (see www.databasecolumn.com/2008/01/mapreduce-a-major-step-back.html and www.databasecolumn.com/2008/01/mapreduce-continued.html).

Here is a small quote from their thoughts on the topic:

"As both educators and researchers, we are amazed at the hype that the MapReduce proponents have spread about how it represents a paradigm shift in the development of scalable, data-intensive applications. MapReduce may be a good idea for writing certain types of general-purpose computations, but to the database community, it is:

1. A giant step backward in the programming paradigm for large-scale data intensive applications

2. A sub-optimal implementation, in that it uses brute force instead of indexing

3. Not novel at all -- it represents a specific implementation of well known techniques developed nearly 25 years ago

4. Missing most of the features that are routinely included in current DBMS

5. Incompatible with all of the tools DBMS users have come to depend on"

Does this mean the database wars are starting up again?

My opinion is that MapReduce is not intended for general purpose commercial database processing and is therefore not a major threat to relational systems. However, it does have its uses (as Google has demonstrated) for certain types of high volume processing. It also demonstrates that as data volumes get bigger, and the complexity of data and data structures increases, other types of database technology may start to gain traction in certain niche marketplaces. The use by IBM of the SPADE language, instead of StreamSQL, in its InfoSphere Streams product (System S) also demonstrates the changes going on in the database market.

What do you think?

At the recent SAP TechEd, I had the opportunity to address this topic with John Schwartz, CEO of Business Objects. I made the point that mobile computing was enjoying more success in Europe than the US, and asked him if he thought the Apple iPhone would change this. He agreed that mobile computing use was higher in Europe, but felt the majority of this was still for personal purposes. He noted that mobile BI was seeing growth in both the US and Europe. However, he said the device of choice for mobile BI was still the Blackberry, because its architecture was more suited to this type of processing. So far, he said, Business Objects has seen little demand for iPhone support.

Donald Cambell, CTO of Cognos, took the same position when I asked him this question at a recent IBM Cognos analyst meeting. He said IT is still primarily supplying Backberry’s for business use, but in some cases IT will support personally purchased iPhones. He said the iPhone still doesn’t have the promised capability to run processes in the background, which limits its use for BI. He noted that the new Google G1 has an excellent development platform, and if the device is successful it could have a major impact on mobile business applications. He also made the interesting point that Windows Mobile was the preferred mobile platform for packaged embedded solutions used in locations such as hospitals, for example.

I would be interested to hear from other people about how their organization is deploying mobile BI solutions.

The acquisition of Datallegro created a significant amount of interest, and over the last two weeks I have discussed the acquisition with a wide range of people. Although opinions vary, some general consensus has emerged from those discussions.

The first question is why did Microsoft acquire Datallegro? The answer is that Microsoft wants to market SQL Server as a solution for large-scale data warehousing, and to do this they need to compete with the main DBMS vendors who have massively parallel database products. The Microsoft research group in San Francisco was spearheading key efforts to expand SQL Server in this area, but with the tragic death of Jim Gray it would appear the group lost momentum. To regain this momentum, Microsoft felt it needed to purchase an MPP database appliance.

Why Datallegro? There are a wide range of database appliances on the market, but some of them use proprietary hardware and software techniques, and many of them are tightly integrated into the underlying system. What Microsoft needed was a conventional relational DBMS appliance where they could quickly replace Linux by Windows, and the open source relational DBMS by SQL Server. A smaller number of appliances fit this requirement. Datallegro was one of them, and as we saw the ultimate winner.

What has Microsoft bought? This is a more controversial question. The first thing to note is the purchase price. The rumor mill reports the number to be $275 million. From my perspective this is a staggering sum for a company with a limited track record in data warehousing and few customers. Many people I spoke to were appalled by the sum paid by Microsoft. The feeling is that Microsoft purchased a marketing position rather than any real technology. One shouldn't of course underestimate the power of this marketing position. Whereas I don't think IBM or Teradata will feel threatened by the acquisition, it does put Oracle in a difficult position.

Oracle has spent years trying to sell its shared everything solution as a competitor to shared nothing MPP approaches for data warehousing. The result is that it is losing market share to competitors such as IBM, Teradata and appliance vendors in the high-end data warehousing sector. From a marketing perspective, Microsoft is now adding to this competitive pressure. Oracle now has the choice of eating humble pie and acquiring or building an MPP solution, or seeing the continuing erosion of its large-scale data warehousing market share as customers see the benefits of alternative solutions. My money is on Oracle burying its head in the sand and doing nothing.

The remaining question is whether Microsoft will succeed in using the Datallegro acquisition to penetrate high-end data warehousing. Given that SQL Server 2008 has just shipped, it means that the next release of SQL Server will not be for about another three years. A lot can happen in that time. Stuart Frost, the CEO of Datallegro, disputes this three year figure in blogs he has written, but Microsoft has a more rigid development, test, and release cycle than startups like Datallegro do. Even if the SQL Server MPP capability can be put into an interim release, it is unlikely we will see anything for two years.

Another issue concerns how Microsoft is managing its data warehousing development. It would appear that the BI team in Redmond has ceded Microsoft's data warehousing strategy to the SQL Server Group. In turn, the SQL Group has set up a data warehousing center of excellence at the Datallegro HQ in Southern California. Datallegro has limited data warehousing expertise, and for Redmond to have this remote group driving its data warehousing strategy appears very risky.

This brings me to the departure of Bill Baker to become CTO of Visible Technologies. This is a major loss for Microsoft. Bill is a visionary, an incredible leader, and was the driving force behind putting Microsoft on the BI and data warehousing roadmap. Microsoft has now lost two key BI and data warehousing people, Jim Gray and Bill Baker. They are irreplaceable, and setting up a data warehousing center of excellence in Southern California doesn't come close to making up for this loss.

This year was the seventh year the event has been held and it surpassed even the excellence of previous summits. The four analysts and consultants (Jill Dyche, Claudia Imhoff, William McKnight, and myself) were joined by representatives from Composite Software, Dataflux, Eyeris, HP, IBM Cognos, Infocentricity, Microsoft, Paraccel, PivotLink, SAP Business Objects, Teradata, Xactly Corporation, and of course the BI Network.

The informal discussions covered a wide range of topics from BI to politics! The acquisition of Datallegro by Microsoft had just happened and this was a big topic of discussion. Although views varied, several people expressed the opinion that Microsoft was really buying an enterprise marketing position (especially against Oracle), rather than any real technology. By this time the industry blogging machine was working overtime and several blogs had already reported the purchase price to be $275 million, which staggered everyone.

Towards the end of the summit, news was leaking out that Bill Baker was leaving Microsoft and everyone agreed this was a tragic loss for the company. Microsoft is certainly going through some dramatic changes in the BI area.

The formal discussions focused on Software as a Service BI (led by Claudia Imhoff), CRM (led by Jill Dyche), Operational BI (led by myself) and IT Leadership (led by William McKnight). The volume of information and discussion is too lengthy to report here, but the BI Network will be releasing a number of podcasts on some of the discussions in the near future. Podcasts with each of the vendors are already available.

Some key points I got from these four sessions were:

1. There is considerable interest in SaaS BI by both vendors and customers. BI is being used not only by SMBs, but also groups within large organizations. SaaS BI is often used to get a project started and many companies would like to bring the project in house once it matures. Many people felt that the pay as your go model will gradually become the norm for both SaaS and on-premises solutions (as pointed out at the summit "on-premises" is correct English usage, but "on-premise" is not). Lastly, like in-house application packages in the past, SaaS companies and solutions will merge and be acquired to provide a set of application solutions, rather than remain as stand-alone silos.

2. CRM is going through a reemergence with companies focusing on micromarketing, social computing as a new CRM information source, and increased interest in master data management.

3. There was universal agreement that operational BI is a big growth area, but that the range of solutions and vendors both inside and outside BI is large and confusing. One point of discussion was the convergence of operational BI with business process management and complex event processing. Other discussions focused on the impact of operational BI being process driven, rather than data driven, and on whether BI is the best term to use to describe analytical and decision making solutions moving forward.

4. The discussion on IT leadership generated many different viewpoints. There was universal agreement that companies need to focus less on reducing IT costs, and more on recognizing IT as a essential business component of the organization, in the same way, for example, that HR is. There was also a lot of discussion about the need for IT to modernize its thinking, and create a more flexible governance environment to handle emerging technologies such as social computing.

The summit offered several opportunities to enjoy the many tourist activities of the Rogue River. The highlight for me was a visit to the Wildlife Images animal rehabilitation and education center. Everyone fell in love with a badger called Nubs with the result that the group donated $1,000 to support Nubs and other animals at the center.

I can't wait until next year!

Stuart Frost, CEO of DATAllegro, had some interesting things to say about the acquisition on his blog immediately after the announcement. Here are some excerpts:

"... just as the VC community started to recover from the Internet ‘bubble’ in 2003, I came up with the vision for DATAllegro. Since that time, we’ve raised just under $65m in venture capital and created a hugely successful exit for my investors, my great team and last, but not least, me!"

"As soon as the acquisition closes, we'll start the work of moving our technology from Ingres & Linux to SQL Server and Windows. Our feasibility studies over the last few months indicate that SQL Server is a significant improvement in terms of performance - especially in key areas such as star joins, I/O throughput and in-memory operations. The engineering team here at DATAllegro is VERY excited about the next version of the product."

"Over the last few years, it's been incredibly frustrating to have prospects tell us that we have the best technology, vision and people, but that they can't buy from a startup - I think that will change radically under the Microsoft brand! As a result, I'm starting to think that it could be a long term home for me. It will certainly make a nice change from having to raise VC money every few months!"

"It will be interesting to see the impact this acquisition has on the rest of the market. My guess is that the other incumbents will scramble to respond to Microsoft's pre-emptive strike and that this could lead to a few of the other startups being acquired. The ones left out will find life very hard over the next few years."

There are a few interesting points to note in these quotes. The first is that DATAllegro will move from using an open source DBMS to SQL Server. Microsoft wants to move SQL Server up market to compete with the high-end DBMSs for data warehousing such as IBM DB2, Oracle and Teradata. It is interesting to note that at a Teradata analyst meeting this week, Teradata talked about how it is moving in the opposite direction of providing lower-cost and smaller appliances for data marts. Teradata hopes that this strategy will introduce Teradata to companies outside of the Fortune 2000 who will then hopefully move gradually to an enterprise data warehouse environment based on Teradata.

Before the acquisition, DATAllegro was clearly targeting Teradata customers by suggesting that they could offload some of their Teradata workloads to a lower-cost DATAllegro solution, i.e., DATAllegro was selling coexistence, rather than rip and replace. Obviously this strategy will change now that Microsoft is driving the marketing program.

The second point to note is that DATAllegro had burned though nearly $65 million in VC funding. Like many of startups they were achieving visibility, but not sales. In fact, it was difficult to find good DATAllegro case studies. To be fair, this is true for several of the other appliance vendors as well. Netezza, who has been in this market longer, is an exception here. I wonder if Microsoft perhaps took advantage of the VCs getting nervous about the viability of DATAllegro. We may get some insight into this when the purchase price is leaked.

I think Stuart's comments about other startups being acquired is correct. There are simply too many appliance vendors out in the market. The question is who is likely to acquire them? Leading DW DBMS vendors such as IBM, Oracle, and Teradata are unlikely candidates. One possibility is HP who are struggling to get NeoView off the ground. Another is Sun who have relationships with several appliance vendors. There is no question, however, that several of the appliance vendors are likely to go out of business. History shows that selling price/performance is not a good longterm strategy. Success may come for those appliance vendors that move toward selling more application-focused solutions.

Regardless, the DBMS and DW markets continue to be exciting!

The report proposes a new definition for Enterprise 2.0: "A system of Web-based technologies that provide rapid and agile collaboration, information sharing, emergence, and integration capabilities in the extended enterprise." One of the reasons given for this new definition is the poor understanding of Web 2.0, which is often closely associated with Enterprise 2.0. In fact many people describe Enterprise 2.0 as the use of Web 2.0 in the enterprise.

Some 44% of the 441 survey respondents said that Enterprise 2.0 (as a technology and a practice) is imperative or signifiant to achieving the organization's overall business goals and success. However, 41% of respondents said there was no clear understanding of what Enterprise 2.0 is in their organization. These latter two results seem to contradict each other - people think the concept is critical to success, but are unsure what it is!

The reason for these odd results is that most people are familiar with several of the technologies usually associated with Enterprise 2.0, but are still unsure how they fit together to provide a complete solution.

Improved collaboration was often cited in the report as the main objective of Enterprise 2.0, but the term "collaboration" here has a broader meaning than its traditional definition. In this case, collaboration includes information and knowledge sharing, and also social networking. This is an important point because I think that collaboration today has a broader meaning than it has in the past.

The main inhibitor to adoption of Enterprise 2.0 appears to be lack of control and increased security concerns (42% of respondents). There is nothing new here, but it does show that traditional governance approaches have to be modernized in order for Enterprise 2.0 to be successful.

The results of this report are clearly inline with the direction of the BI industry, which is beginning to focus on the impact of modern collaborative and social networking techniques and Web content on BI processing.

During interviews, business users always seem comfortable with the term business analytics, but often view BI as a vague and imprecise technical term. Some vendors and IT folks also seem to prefer business analytics to BI. In the area of operational BI, for example, new solutions are appearing on the market that employ embedded or stream analytics. These solutions often use data from sources other than a data warehouse. The vendor concern here is that BI and data warehousing are often seen as being tightly linked and one cannot occur without the other. The term operational analytics is sometimes preferred because it is viewed as being more dynamic than operational BI.

At last week’s SAS conference, CEO Jim Goodnight commented that business intelligence has been watered down by competitors such as Business Objects and Cognos, whose query and reporting tools couldn’t approach the sophistication of the advanced analytics produced by SAS products.

What do you think? Has the term business intelligence become so abused that it is now confusing and meaningless?

During 2007 both vendors and industry pundits said that operational BI and getting BI out to the masses were key directions. There was certainly a significant amount of discussion on these two related topics during the year, but did customers actually succeed in making progress here?

To my way of thinking moving BI out to the masses involves making it easier to use. From this perspective I think 2007 was a failure. Yes, products did make significant progress in supporting Microsoft Office, but did this really solve the usability problem for less experienced users? Are these users really major users of Microsoft Excel, for example? I think not.

There are two models in the market. One is the IBM and Microsoft model, where the emphasis on product functionality and thus complexity. The other is the Google and Apple model where the focus is on usability and agility. The IBM and Microsoft model is important and is likely to be the cornerstone of IT systems and infrastructure for many years to come, but we need to find more user-friendly solutions.

For less experienced users the way to go is the Google and Apple model. This may provide be less functionality and less stability from an IT perspective, but end user acceptance and growth is likely to benefit most from this model.

The issue for vendors and IT is how to marry the two models. Most BI vendors are still committed to the IBM and Microsoft model, and this opens the door to innovative new BI vendors and I believe in some cases open source products. Many software-as-a-service (really applications-as-a-service) analytical solutions are also starting to gain traction because they offer user-friendly options at a reasonable cost.

I think some of the mainstream BI vendors (e.g., Actuate) are beginning to realize that piling more function into older architectures is not the way forward. Hopefully other vendors will soon realize this as well. However, the size of many BI vendors and the number of products they have is working against fast and easy.

My focus for 2008 then is on BI usability. I think the products that can provide this will be the winners.

The Oracle acquisition of BEA is no surprise. The battle for the infrastructure market is now clearly between Oracle and IBM, with Microsoft, SAP and Sun watching from the sidelines. One interesting aspect of the BEA acquisition is that Oracle now has four portal products. I am glad I'm not an Oracle salesperson!

Sun's proposed acquisition of open source database vendor MySQL was more of a surprise. To date, Sun has not been a database player. MySQL is the most popular open source database products on the market, and it is used by several major Web players including Google and Facebook. MySQL claims that 100 million copies of the product has been downloaded and that an additional 50,000 copies are downloaded daily.

The risk for any open source user is that the software can be acquired by a commercial company. The open source license usually ensures that the product source code remains freely available up to the development level at the time of acquisition. Sun says they will continue to develop the product on multiple platforms including Linux, Mac OS X, Microsoft Windows, and OpenSolaris. Given Sun's commitment to open source (e.g., OpenSolaris, Java System Portal Server/OpenPortal, Glassfish application server) there is no reason to disbelieve them.

One interesting aspect of the MySQL acquisition is that Oracle owns InnoDB, which is a storage engine for MySQL. InnoDB is not a standalone product: it is distributed with MySQL. InnoDB has a contractual relationship with MySQL. It will be interesting to see what happens here.

This point is brought home by the recent announcement by Vertica of a strategic relationship with Talend, an open-source data integration vendor. Vertica markets the column-oriented Vertica Database, and has a relationship with HP and Red Hat to offer a bundled hardware and software analytical database solution. Given that Talend also has relationship with open source BI tool vendor Jaspersoft, it means the combination of Vertica, Talend, JasperSoft and Red Hat software on top of an HP hardware platform provides a cost-effective and open source data warehouse and BI environment. If this package was offered as a single solution, and supported by a single vendor, it would represent the ideal data warehouse and BI appliance solution.

In my opinion, the appliance vendors must move toward offering these types of packaged software environments if they are to survive. Simply offering better price/performance is not a viable long-term strategy. The database machine vendors discovered this. In fact, I believe the vendors need to go one step further and provide application appliances that provide a complete business solution. Although it had a proprietary architecture, the IBM AS/400 was a tremendous success because it offered a complete application solution to business users.

The diving line between master data and reference data is not always clear cut. One solution is to break master data into two types: master reference data and master business entity data. Master reference data has well defined and simple data structures, has simple keys and governance rules, is often standardized (US state codes, for example), involves only a few applications, and is reasonably stable. Master business entity data, such as customer, on the other hand, is usually ill-defined, has complex data structures and relationships, requires compound and intelligent keys and complex governance rules, is not usually standardized, involves many business processes, and changes frequently.

Does this distinction really matter? When developing data quality management and master data management systems it can do. Cleaning and managing master reference status is a reasonable easy job. The opposite is true for master business entity data.

Any comments?