Blog: Dan E. Linstedt

« ELT and ETL - candid view of pros and cons. | Main | Nanomorphing feedback loops, terminator Eyeballs. »

Look into the future: Appliance Data Warehouses

The market is shifting, vendors are packing more and more features and functionality into their devices, they are also making their devices smaller and smaller. What does the future Data Warehouse look like? Can it be an appliance like device? What kind of partnerships or acquisitions can we expect? Why would we choose an appliance DW over our own component selections?

In this blog I look into the future, just to see if we can answer these questions. I believe there are changes coming, long overdue changes.

In the land of yesterday we would have to go in search of "best-of-breed" software, and then pair that up with best-of-breed hardware. Size it appropriately, install it all, and integrate it ourselves (within IT). I believe all that is changing. If it hasn't already, it certainly will shortly.

New vendors on the market are offering coupled hardware with built-in RDBMS's. This is just the start and as good of a start it is, it still has a little ways to go. Let's talk for a minute. What if you could walk out and buy an ADW appliance (active data warehouse) - self-configured to perform optimally on the machine, embedded within the BIOS, encapsulated storage, and a black-box interface... Would you do it? Especially at a cheaper cost than buying RDBMS vendor 1, and Hardware vendor 2.

So what does the future device look like?
It should contain not only the RDBMS, but also ELT software. This software should be embedded onto the machine for fastest performance, along with optimized disk routines, and mechanized load balancing. The ELT software should have two types of inputs: the flat file loading process, and the real-time network plug which reads JMS queues after configuration. The ETLT should be fully self-contained on it's own processor slot so that it doesn't interfere with the RDBMS operating in parallel, at high speeds on the disk.

There should also be a BI (reporting tool) card built in. It should have it's own IP connections, and reside on it's own processor slot as well. The tool and the box configuration should all be browser based, all administration could be fat client I suppose, but why? Why not make it all web/app server? It's separated from the RDBMS and ETLT engine slots, again so that it can run in parallel. Although the BI tool and the ETLT tool should be based on a common metadata framework.

Now, depending on the number of nodes purchased - hooking them together through a third pre-configured IP allows them to load-balance across a high-speed backbone. Again, nothing to do with each other but distribute the work-load.

What kind of partnerships or acquisitions can we expect?
I think in the future, you'll see storage vendors partner more heavily with RDBMS vendors, who are already working on "blade servers" - some vendors have the pre-packaged solution there. Other vendors are coming up to speed. I also think you'll see a larger effort to integrate the BI and ETLT software onto hardware platforms. It's getting cheaper to architect and build hardware - and most of the time we need the extra performance boost, even if the BI application card is running on a dual CPU at 450 Mhz, it's the RDBMS that needs the power.

That's all fine and dandy, but where's the value proposition?
The value comes as follows: automatic updates to the software and firmware over the web, little to no configuration needed (all comes pre-installed, and factory tuned), no fancy load-balancing, or parallelism software needed to gain performance. No messy dual environments for upgrades, no multi-cost purchases, plug & play scalability, speed and performance built into the hardware/firmware.

I think you may see compliance vendors entering this game too, they already are partnering with storage vendors for appliance based storage.

What makes this work and why?
The RDBMS engine on the appliance must be extremely fast, and extremely scalable. It must focus on bulk-applying data sets or images of data that is time-stamped. It must have high-quality, high-end compression, data quality, delta-processing capabilities available. As we consolidate our resources, it becomes easier to manage, upgrade, replace. In this environment you're paying for the engineering to be "done", out of the box, into the rack - load your data and away you go. All data is denormalized inside the box, so compression ratios are very very high, storage needs are low, performance is super fast - we don't have to worry about data modeling or indexing any more. (the end goal)

There are a number of companies to watch out there who are moving in these directions. It won't be long before they can meet all these needs with one appliance.

Of course it wouldn't hurt for these companies to consider a metadata appliance either, or possibly incorporate that directly into the warehouse appliance.

Just a few random thoughts, See you next time.

Share: