Blog: Jill Dyché Subscribe to this blog's RSS feed!

Jill Dyché

There you are! What took you so long? This is my blog and it's about YOU.

Yes, you. Or at least it's about your company. Or people you work with in your company. Or people at other companies that are a lot like you. Or people at other companies that you'd rather not resemble at all. Or it's about your competitors and what they're doing, and whether you're doing it better. You get the idea. There's a swarm of swamis, shrinks, and gurus out there already, but I'm just a consultant who works with lots of clients, and the dirty little secret - shhh! - is my clients share a lot of the same challenges around data management, data governance, and data integration. Many of their stories are universal, and that's where you come in.

I'm hoping you'll pour a cup of tea (if this were another Web site, it would be a tumbler of single-malt, but never mind), open the blog, read a little bit and go, "Jeez, that sounds just like me." Or not. Either way, welcome on in. It really is all about you.

About the author >

Jill is a partner co-founder of Baseline Consulting, a technology and management consulting firm specializing in data integration and business analytics. Jill is the author of three acclaimed business books, the latest of which is Customer Data Integration: Reaching a Single Version of the Truth, co-authored with Evan Levy. Her blog, Inside the Biz, focuses on the business value of IT.

Editor's Note: More articles and resources are available in Jill's BeyeNETWORK Expert Channel. Be sure to visit today!

By Stephen Putman, Senior Consultant

Netflix_PseudoGil
I just finished reading a post on the Netflix blog - 5 Lessons We've Learned Using Amazon Web Services (AWS). Even though this article is specific to a high-traffic cloud-based technology platform, I think that it holds a great lesson for the optimization of any computer system, and especially a system that relies on outside sources such as a business intelligence system.  

Netflix develops their systems with the attitude that anything can fail at any point in the technology stack, and their systems should respond in as graceful a way as possible. This is a wonderful attitude to have for any system, and their lessons can be applied to a BI system just as easily:

1. You must unlearn what you have learned. Many people who develop and maintain BI systems come from the transactional application world, and apply their experience to a BI system, which is fundamentally different in several ways - for example, the optimization goal of a transactional system is the individual transaction, while the optimization point of a BI system is the retrieval and manipulation of often huge data sets. Managers and developers that do not realize these differences are doomed to failure with their systems, while people who  successfully  make the transition meet organizational goals much more easily.

2. Co-tenancy is hard. The BI system must manage many different types of loads and requests on a daily basis while simultaneously appearing to be as responsive to the user as all other software used. The system administrator must balance data loads, operational reporting requests, and the construction and manipulation of analysis data sets, often at the same time. This is the same sort of paradigm shift as in lesson 1 - people who do not realize the complications of this environment are doomed to failure since the success of a BI system is directly proportional to the frequency of use, and an inefficient system quickly becomes unused.

3. The best way to avoid failure is to fail constantly. This lesson seems counter-intuitive, but I've seen a lot of failed systems that always assumed that things would work perfectly - source feeds would always have valid data, in the same place, at the same time, always - that this philosophy gains more credence daily. Systems should always be tested for outages at any step of the process, and coded so that the response is graceful and as invisible to end-users as possible. If you don't rehearse this in development, you will fail in production - take that to the bank.

4. Learn with real scale, not toy models. It would seem that proper performance testing on systems equivalent to production hardware and networking with full data sets would be self-evident, but many development shops look at this as an unnecessary expense that adds little to the finished product. But, as in lesson 3 above, if you do not rehearse the operation of your system on the same size of system as your production environment, you have no way of knowing how the system will respond in real-world situations, and are effectively gambling with your career. The smart manager avoids this sort of gamble.

5. Commit yourself. This message surfaces in many different discussions, but it should be re-emphasized frequently - a system as important as your enterprise business intelligence system should have strong and unwavering commitment from all levels of your organization to survive the inevitable struggles that occur in the implementation of such a large computer system.

It is sometimes surprising to realize that even though technology continues to become more complex and distributed, the same simple lessons can be learned from every system and applied to new systems. These lessons should be reviewed frequently in your quest to implement successful data processing systems.

photo by PseudoGil via Flickr (Creative Commons License)


StevePutman_bw_100Stephen Putman has over 20 years experience supporting client/server and internet-based operations from small offices to major corporations.   He has extensive experience in a variety of front-end development tools, as well as relational database design and administration, and is extremely effective in project management and leadership roles. He is the co-author of The Data Governance eBook, available at information-management.com.



Posted January 18, 2011 6:00 AM
Permalink | No Comments |