We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Blog: Shawn Rogers Subscribe to this blog's RSS feed!

Not Pictured

Thank you for visiting my blog. I am a Cofounder & Technical Advisor here at the BeyeNETWORK. Having covered the business intelligence and data warehousing industry for more than 15 years, I'm looking forward to a more interactive form of communication with all of you. Please share your comments and thoughts!



I am getting ready to start a cloud based business intelligence project here at the BeyeNETWORK and thought it might be interesting to share details along the way. I will try to post as we achieve milestones for the project.

So first off let me share a little about the scope and our goals. As I am sure you are aware we capture lots of transactional data on what our readers interact with across the sites that make up the Network in January we made changes to our global passport system that allowed us to normalize the base data we collect enabling a cleaner view of the content and how it related to readers. Our goal is to aggregate the information and to identify trends in the industry, popular topics and authors and to deliver more finely focused dynamic content to readers.

I'm sure you won't be surprised to hear that data grew quicker than we expected and initially we used straight SQL queries to gain some insight into what was happening with our data. We then created a simple dashboard to deliver views of the queries, gave access to others on the team and quickly brought our systems to a grinding halt. It was a good example of how not to implement a data warehouse and business intelligence project. The data was stored on our web servers and every time I asked a serious question of the data it affected the Networks performance. Not to mentioned we were storing the information in a simple series of tables not optimized or organized for analytics.

So, this brings us to the challenge. I need a system that serves my business needs, integrated our data to the cloud in a model optimized for analytics so I can deliver business intelligence insights into what our customers are most interested in. We plan to do the following with help from friends and partners and I will share about that in my next post. In the meanwhile we are beginning the data modeling stage and implementing a methodology for handling our data that will provide a flexible platform as we move forward. We have chosen Amazon as our cloud host and will virtualize a copy of our existing platform there to support the business intelligence platform we require and to keep the environment consistent for our engineering team.

I see the cloud as a perfect platform for this project and as more vendors have embraced the cloud we found solutions that fit our needs and offered the flexibility we require. The data is small now (8M rows, growing 1M per month) but as it scales the platform will as well. The costs are very manageable as well.

That's all for now I will bring you up to date on our progress again soon as well as providing insight into the process and the solutions we are using.

Posted July 20, 2009 9:56 AM
Permalink | No Comments |

Leave a comment

Search this blog
Categories ›
Archives ›
Recent Entries ›
Twitter Updates