What happens when an irresistible force meets an immovable object? We are about to find out. The irresistible force of BI, eDiscovery, compliance, fraud detection, governance, risk management, and other analytic and regulatory mandates is heading straight toward the immovable rock of year-to-year 10% reductions in information technology budgets.

The convergence of the markets for structured and unstructured data has been heralded many times, but maybe the time has come. We think that the new generation of solutions with increasing overlap of structured and unstructured data and multi-functionality will emerge and that BMMsoft EDMT® Server is the pioneer in that space. Looking into the crystal ball, what will happen is that an increasing overlap already underway will disrupt incumbents across these diverse markets.

The world wide BI Market as defined by Gartner is sized at $8.8B.[1] Realistically that includes a lot of Business Objects (SAP), SAS applications, and IBM solutions so the database part of that is probably closer to $6 billion.[2] The document management software market is estimated at nearly $3 billion.[3] While email archiving is relatively new and growing rapidly due to near federal regulations, it has now reached the $1 billion "take off" point. In short, at nearly $10 billion total, a product that addressed requirements across all three of these markets with a reasonable prospect of response from even one third of the enterprises, would have an outside boundary of over $3 billion. This is a substantial market under any interpretation.

In the meantime, the exiting markets for these three classes of products is fragmented into silos of the traditional data warehousing vendors, email archiving, and document management, the latter sometimes including compliance and governance software. The first are well known in the market - extending from such stalwarts as HP, IBM, Oracle, Microsoft, SAP, to data warehousing appliances and column-oriented databases - and will not be repeated here (though one new developments will be noted below). Document management systems include IBM FileNet Business Process Manager (www.ibm.com), EMC Documentum (www.emc.com), OpenText LiveLink ECM (www.opentext.com), Autonomy Cardiff Liquid Office (www.cardiff.com). Strictly speaking, risk management is considered a separate market from document management. Risk management and compliance offerings include Aventis (www.aventis.com), BWise (www.bWise.com), Cura (www.curasoftware.com), Protiviti (www.protiviti.com), Compliance 360 (www.compliance360.com) and IBM, which has at least two offerings one based on Lotus Notes and one based on FileNet. This list is partial and could easily be expanded with many best of breed offerings. The result? Fragmentation. Diversity, though not in a positive sense. Many offerings instead of a comprehensive approach to unified access and unified analysis.

Five years from now data will be as heterogeneous as ever and the uses of data even more so, but individual products - single instance products, not solutions - will characterize a transformed market for database management that traverses the boundaries between email archiving, document management, and data warehousing with agility that is only dreamt about in today's world. Video clips are now common on social networking sites such as Facebook and YouTube. Corporate sponsorship of such opportunities for viral marketing is becoming more common. The requirement to track and manage product brands and images will necessitate the archiving of such material, so multi-media (image/video/audio) are being added to the mix.

This future is being driven and realized by the imperative for business transparency, risk management and compliance, and growing regulatory requirements layered on top of existing business intelligence and document management requirements. Still, document management is distinct from workflow. If an enterprise needs workflow, then it will continue to require a special purpose document management system. Workflow was invented by FileNet in 1985, acquired by IBM in 2006, and continues to lead the pack where detailed step-by-step process engineering is required. Elaborate rules-engines for enterprise decision management are different than compliance. If an enterprise requires a rule-engine for compliance and governance, then it will need a special purpose compliance, risk management, and governance system. Such solutions would be over-kill for those enterprises that require email archiving for eDiscovery, document management for first order compliance, and cross references to transactional data in the data warehouse. While the future is uncertain, one of the vendors to watch is BMMsoft.

Innovation Happens

BMMsoft has put together a product delivering functionality across these three previously unrelated silos - data warehousing, eDiscovery (e-mail), and document management - and able to be purchased as a EDMT^®Server - a single part number from BMMsoft (EDMT stands for "E-Mail, Documents, Media, Transactions"). The database "under the hood" is Sybase IQ, a column-oriented data store with a proven track record and several large objectively audited benchmarks. The latest of these weighs in at 1000 terabytes - a petabyte - and was audited by Francois Raab, the same professional who audits the TPC.org benchmarks.[4]  The business need is real and based on customer acceptance. So is the product.

The three keys to connect and make intelligible the data from the three different sources are:

1) extreme scalability to handle the data volumes - this is where a column-oriented database would come in handy since the storage compaction is intrinsic and prior to the additional compression that could be applied;

2) parallel, real-time high performance ETL functionality to load all the data; and finally

3) search capabilities that enable high performance inquiries against the data.

Such unified access to diverse data types, intelligently connected by metadata, is also sometimes described as a "data mashup."

A part of the challenge that a start up - and up start - such as BMMsoft will face is building credibility, which BMMsoft has already solved with numerous client installations in production and success stories. In the case of BMMsoft EDMT® Server there is another consideration:  metadata is an underestimated and underdeveloped opportunity. Innovations in metadata that make possible many applications that require cross referencing emails, documents, and the transactional data. For example, fraud detection, threat identification, enhanced customer relations - all require navigating across the different data types. Metadata makes that possible. That is not an easy problem to solve; and BMMsoft has demonstrated significant progress with it. Second, the column-oriented database is intrinsically skinny in terms of data storage in comparison with the standard relational database, which continues to be challenged by database obesity. As data warehouses scale up, the cost of storage technology becomes a disproportionably large part of the price of the entire system. Note that for the column-oriented approach proportional cost savings come into view and are realized. Third, this also has significant performance implications, since if there is less data - in terms of volume points - to manage, then it is faster to do so. So when all the reasons are considered, the claims are quite modest, or at least in line with common sense. The wonder is that no one thought of it sooner.

When you think about it for a minute, there is every reason that an underlying database should be capable of storing a variety of different data types and doing so intelligently. The latter intelligence is the "secret sauce" that differentiates BMMsoft. The relationships between the different types of data are built as the data is being loaded by BMMsoft using multiple software technology patents.  The column-orientation of the underlying data store - Sybase IQ - intrinsically condenses the amount of space required to persist the information, yielding up to an order of magnitude - more typically a factor of two or three - in storage savings, even prior to the application of formal compression algorithms. This fights database obesity across all segments - email, document, media, transactional (structured) data warehousing information. This means that the application that lives off of the underlying data is able to take advantage of performance improvements since less data is being stored and more being fetched with every data retrieval. For those enterprises with a commitment to installed Oracle or MySQL infrastructure, BMMsoft provides investment protection. The EDMT^® Server runs also on Oracle, Netezza and MySQL and can be easily ported to any other relational Database. 

Thus, BMMsoft is a triple threat and is able to function as a standalone product addressing data warehousing, email archiving, and document management requirements as separate silos. But just as importantly, for those enterprises that need or want to compete with advanced applications in fraud detection, security threat assessment, customer data mining beyond structured data, BMMsoft offers the infrastructure and application to do so. For example, the ability to perform cross-analysis between securities traded on the stock market and those companies named in email and voice mail (remember multimedia handling) will immediately provide a short list for follow up detection on on-going insider trading or other fraudulent scheme. While hindsight is 20-20, a similar method of identifying emerging patterns through cross-analysis would have been be useful in surfacing the 8 billion dollar Societe General fraud, Madoff's nonexistent options plays at the basis of the pyramid, the Georgia Tech shooter, and relevant chatter that shows up prior to terrorist attacks. Going forward, this technology is distinct in that it can be deployed on a small, medium, or large scale to highlight emerging hot spots that require attention.

One may object - but won't the competition be able to reverse engineer the functionality and provide something similar using different methods? Of course, eventually every innovation will be competitively attacked by some more-or-less effective "work around." Read the prospectus - new start ups and existing software laboratories at HP, IBM, etc. will eventually produce innovations that challenge the contender. However, that could require three to five years. Then there is the matter of bringing it to market. IBM provides an example, based on publicly available news reports. IBM went out and purchased FileNet for about $5 billion dollars. FileNet is a great company, which virtually invented workflow, and if one requires advanced workflow capabilities, it is hands down a good choice. However, it does not do data warehousing or email archiving. As a subsidiary of IBM which delivers substantial revenue to the "mother ship," the executives in charge will set a high bar on any IBM innovations which combine email archiving and structured data warehousing with document management. In short, IBM is faced with the classic innovator's dilemma.[5] The price points that interest it - both internally and externally - are further up on the curve than the deals that BMMsoft will be able to complete. Given that BMMsoft has established presence in the market, it has a good chance of marching up market, displacing the installed, legacy solutions as it goes. This happened before in the client server revolution when IBM mainframe deals at the several million dollar price point were undercut by a copy of PowerBuilder and a copy of Sybase, albeit a different version of the database. Given that BMMsoft has a head-start, it is exploiting first mover advantages and building an installed base that will be challenged only with great difficulty. The relevance of such technology in the context of healthcare information technology (HIT) will be explored in a pending post. Please stand by for update - and keep in touch!