I've posted already about TDWI's San Diego event, but I still haven't exhausted the thoughts I wanted to share. That's a measure of just how important and successful I think the show was. Three things jumped out at me:

• The audience is back, and it's ready to spend. The event was buzzing; I was told by organizers that the numbers significantly exceeded expectations. That was easy to see; speeches, booths, and hallways were packed. Vendors told me booth traffic was great, and that visitors (although typically not budget holders) were in or preparing for projects and product acquisitions.
• The hunger for content continues. In my session and in others, I saw show-of-hands responses to questions like "how many of you have been here before?" "How many of you have built this kind of system?" "How many of you have been trained on [pick a DW-related topic]?"  The responses made it clear that like other TDWI events I've been to, this one was packed with people who were new or intermediate users with training in mind. TDWI's basic training mission has never been healthier.
• Agile matters. A lot. My first post on the event was put up rather quickly and as the event progressed, I heard the theme flesh out well, with real stories from users who applied the techniques to their projects. My initial impression that we might be looking at another buzzword poorly applied was wrong. Agile's real, and TDWI's coverage and guidance is rich and well worth investigating. The vendors? Well, they're doing what they always do. Caveat emptor. I repeat: it's not an adjective.  Learn what it means and apply it. You can't buy it.

The last point above drives a few more thoughts about Wayne Eckerson's keynote and my comments on it. Wayne had little time to work with, and left the nuance and details of Agile to the agenda speakers who followed. As chairperson, he made the right decision, an unselfish one. Having chaired conferences of my own in my Giga and Forrester days, I applaud his willingness to cut his own time to literally less than a half hour to let his speakers shine.  I was hasty in my comments about his choices - he was clear on the topics he would have covered if he had more time, and subsequent diligence on my part (and his gentle prodding and pointers to prior work TDWI had done on the topic) reveals more detailed examination and training of Agile than I knew was in place.

As I said earlier, the speech itself was a good one, well delivered. And I withdraw my content-based "unsatisfied" comment, Wayne built a conference to tell the Agile story, and didn't attempt to cram it into too little time in his own speech. Instead he delivered some tips to a crowd that hopefully understood how surprisingly radical some of them were. As I've said elsewhere, we who live in the future sometimes forget what's going on in the present - TDWI's strong connection to current user data keeps it grounded, and Wayne's tips captured what leading organizations are doing today - some of which are different in surprising ways from past practice.

TDWI gave Agile credit, and covered it well. BI developers should learn what it can do for them and use if as a bridge to their colleagues in other programming groups - it has the potential to be a shared set of assumptions, processes and practices that bridge what often are separate organizations.

On my second day at TDWI, I was in meetings all day - events like this are a great opportunity for analysts to catch up with many of the companies they follow at one time, and this particular one was packed with sponsors. Congrats to the folks who sell sponsorships - they had a packed exhibit hall, and a lot of very interested attendees. I got a chance to chat at a few booths (all buzzing), ask a few attendees some real-world questions (and was asked some surprising ones myself), and get a sense of the workload in the trenches (heavy and growing.)

more...

My friend Curt Monash has taken Oracle to task for the way it labels its web pages that contain download links for analyst reports, and I took some collateral damage in the process. It was embarrassing to me, but an important discussion, and I thought I ought to share some ideas about the whole issue. For example, I found that other vendor sites don't always label white papers as sponsored either.

Some of my pieces are published by vendors who simply buy the rights to make available things I've posted here or elsewhere. Those are not "sponsored"; no discussion about what I will or will not say has taken place in advance, and there is no promise by me to write, or to pay by them. Other pieces are specifically commissioned from me, under editorial agreements I've described elsewhere. In brief, though - vendors get to check facts, but not dictate what I say. And they don't buy comparisons, favorable or otherwise, to competitors - I don't accept that kind of work for publication, at any price.

read more...

In a recent post I discussed Oracle's market share in BI, based on a press-published chart taken from IDC data - showing Oracle coming in second. As often happens in such discussions, I got quite a few direct emails and twitter messages - some in no uncertain terms - about why the particular metric I chose was not sufficiently nuanced or representative of the true picture. I freely admit: that's true. In general, market observers know Oracle is not typically placed second overall - but the picture is more complex than a single ranking. My point was, and is, that it's too easy to slip into a "who's on top" mentality that obscures true market dynamics. In this post, I'll dig a bit deeper, and describe what different approaches or categorizations show us - and what they don't. Finally I'll talk about how much this matters - and to whom.

More..

Last time I mentioned GoodData, it was in passing, as I discussed YouCalc and other SaaS BI players. In the ensuing year, many other toes have been dipped into the water. I sat down with GoodData CEO and founder Roman Stanek and Marketing VP Sam Boonin this week to catch up on how it's all going, and from where they sit, the news seems to look pretty good. With 40 employees, 25 customers since last November, and a funding round from the likes of Marc Andreesen and Tim O'Reilly, GoodData seems to be off to a GoodStart. And now it has a new initiative: free analytics for other SaaS players to expand its presence.

more

One of the more philosophical questions analysts like to ask is "What is Big Data?" It's relative - it begs the question, "what's big?" And that is a constantly moving number, and always assessed by comparison to the ridiculous amounts some companies work with. But Big Data as a concept in IT parlance today tends to mean something fairly specific, not just about size but also about composition and the nature of the processing. So I considered a serious attempt at a fairly rigorous discussion about the nature of the workload, structure of the data and the kinds of analytics that comprise what people think of as Big Data....and then I thought of Steve Martin, who would have considered this carefully and then looked into the camera and said "Naaaahh." So I determined to emulate him and have a bit of fun instead, by crowdsourcing some help completing the sentence "You know you have Big Data when..." Here's what some Twitter folks said. Some are funny, some more serious ...

You know you have Big Data when....

... you get a call from the utility company asking you not to run 'that brownout query' again. (@aristippus303 at Datawatch)

... your IT spends more time purchasing storage capacity than making sure the business has the data they need - @judyiko (Informatica)

.,. EMC name a new product after you (@aristippus303 at Datawatch)

...  it piles up so high that it disappears into the clouds (@evertlammerts - I assume pun was intended?)

...  the SAN undergoes gravitational collapse and you get cited by OSHA for an unlicensed singularity. (@datamartist)

...  a query is long enough to require a couple of DBA generations to see it returning first data. (@Stray_Cat)

...  your datacenter manager divides time between installing a new NAS in the kitchen and googling for vacant aircraft hangars. (@alanjharrison)

And a few of mine:

...  you conduct an audit, including external files, and add more in to the databases than you take out.

...  you think Flomax is a new ETL product.

...  the first item on your bucket list is "finish data model."

...  you've never gotten to the "Reduce" part.

...  your Dad won't let you have the keys to the table you want to join to because he's still doing the schema update he started on your birthday. No, your BIRTH day.

OK - that's way more than enough. Don't you have a schema to update? Get back to work. If you get bored, send me some more.

Oracle's newest BI release is massive, spans multiple product categories, and raises the bar for competitors in dramatic fashion. In my prior post I focused on its rollout and competitive posture. The market has waited a long time as the reconciliation of many moving parts was accomplished - most notably the convergence of the Hyperion Enterprise Performance Management (EPM) offering and Oracle Business Intelligence Enterprise Edition (OBIEE). Hyperion integration with its Essbase acquisition was not complete. In 2007, OBI's newest release (10.1.3) was most notable in many eyes for its new Microsoft Office support. PeopleSoft and Siebel had been acquired some two years before that, and Master Data Management was already a topic of discussion then (2005). There was a long way to go. And analysts? Well, think of us as the kids in the back: "Are we there yet?"

More - warning - it's quite long.

Tibco, fresh from a Q2 with license revenue up 23% over last year's, continuing a two year run of beating consensus earnings estimates, has stepped up and out ahead to pursue the long-coveted mid-market customers who don't use BI but find that spreadsheets don't do enough.  Tibco believes, like Microsoft, that many are social technology users: they have blogs and use other channels available to them, and they will build and share reports given the chance. So, says Tibco, here it is: building on the Silver cloud platform it's had in beta for about a year, Tibco is introducing Silver Spotfire, with an offer tuned to the cloud user - a no-cost, no-obligation, no-risk 1-year trial of a Spotfire play in the cloud requiring no IT involvement. "All you need is a browser," is the pitch, and this is not from a new company you don't know, but an established  player with a sizable roster of enterprise BI customers.
more

Oracle is not first in BI, and wants to change that - that was the clear message of a well executed, multi-site "real plus virtual" event with top executives showing off the result of a multi-year effort to rationalize and integrate a set of leading but overlapping components into a seamless suite. Oracle Business Intelligence Enterprise Edition 11g (OBIEE) deserves the accolades it has already received from analysts who welcomed its announcement - it makes bold and serious bets on effective centralized metadata administration, data integration/ unification and optimized analytic architecture, collaboration, globalization, mobile device support, and a powerful link to action that will be most effective (unsurprisingly) with its own business applications. While it misses some pieces - fully integrated in-memory processing, SaaS and cloud support among them - these will be forthcoming, and Oracle is clearly committed to a quicker release cycle now that the thorny internal politics around legacy products seem to be resolved. But its competitive focus may be misdirected; while SAP is still ahead in market share, IBM is the bigger threat in the marketplace.

more

EMC's acquisition of Greenplum, announced today as a cash transaction, reaffirms the obvious: the Big Data tsunami upends conventional wisdom. It has already reshaped the market, spawning the most ferment in the RDBMS (and non-R DBMS via the noSQL players) space in years. When I first posted on Greenplum over a year ago, I said that

"Open source + capital has created an intriguing new model of rapid innovation in "mature" markets, and the database space - like BI - is not a done deal. It is indeed possible to escape the gravity well, if you execute. Greenplum is getting it done, and is among the new stars to watch."

Why the open source reference? Greenplum uses a parallelization layer atop PostgreSQL (like Aster, another of the new breed of ADBMS.)

Now EMC has written the next chapter in that story. In the process, it adds a new piece (after literally dozens of others in the past few years) to its own portfolio, which already includes unstructured data (via Documentum) and virtualization (via VMWare), layered in among the industry-leading storage and information management pieces. Disruptive? You bet. Is EMC finished? I doubt it. Candidates? BI tools, ETL, MDM, data integration come to mind. Losers? At least one big one. Read on.

more