Recently in BI Cloud Project Category

Its not always about the technology....
As we continue to work with Steve Dine and his group at Datasource Consulting I have identified a trait that is critical to a successful project. If you had asked me prior to beginning this I would have provided a standard answer like the technology or the platform or the data quality and while all of these play a key role in the success of the project none of them is as important as the ability to listen. Steve and his group have this skill and their ability to understand our data and its traits is very compelling. As they have utilized Talend to populate our cloud based architecture the Q and A between his team and mine is growing at a fever pitch and the questions illustrate just how seriously they take our data. Today's call included a quick conversation about 290 records that didn't have the required attributes we determined that a process issue in our session variable for the sites was partially to blame and if Steve didn't have a firm grasp of the data this may have gone completely unnoticed. The by product of good listening has been the leadership that comes from it. It insures we can move forward at the highest possible speed towards what I believe is going to be a very strong solution. Bringing together cloud technology, JasperSoft and Talend is a highly technical undertaking but as mentioned above its clear that listening can over shadow the technology at least in the early execution stages of the project.

For all of you out there looking to start a business intelligence project be sure you have a serious comfort level when it comes to explaining your business and the data that supports it. If you even think for a moment that the consultant doesn't understand or care deeply about your business or data run the other way!

Congratulations to Steve and his team for executing on this critical skill.

The project continues to move forward at a very fast pace and I have to say much of that success is due to the team we have assembled. As I mentioned in my first post I am working with some excellent service and solution providers on this project so I thought it was about time that I let you know who they are.

Heading up the project is Steve Dine founder of Datasource Consulting here in Denver. His firm has embraced the challenge and brought a set of best practices and methodologies to the project that provide a solid scalable infrastructure but also provide the flexibility to help us evolve and grow. Datasource Consulting's expertise around lean BI, integration and cloud technologies made them the perfect project lead.

We couldn't get this job completed without the right solutions and as many of you know I am a huge supporter of open source technology. The entire platform for the Network is built on a LAMP architecture so when we sought out solutions we went to the open source community first.

Our initial challenge around this project was to identify a way to move our data off of our production environment that supports the  Network web servers to a cloud based platform. More importantly we required this ETL functionality to work easily on the Amazon Cloud and it had to be powerful enough to feed the data model and processes designed by the team at Datasource Consulting. The logical choice for us was Talend Integration Suite an open source enterprise data integration solution, designed for scalability, process complexity and multi-user development. I'd like to thank Yves de Montcheuil, Vice President of Marketing at Talend for working with us on the project. As we progress, Talend will also supply expert level assistance as we bring the data integration online.

At the front end of our project we will be using JasperSoft Business Intelligence Suite where we will be able to leverage the advanced reporting, production reporting, ad-hoc, dash board and data exploration and analysis functionalities that are delivered in the solution. JasperSoft works well with cloud based data and Nick Halsey Vice President of Marketing and Product Management has also worked with us to supply professional services to leverage the platform when it goes into production.

An additional advantage to this line up is that Datasource is a Talend certified partner and Talend and JasperSoft are also partners so the synergy between the companies is exceptional.

That's it for now, I have to prepare for my reporting scope meeting with Matt Caton of Datasource I am looking forward to exploring my options.

I am getting ready to start a cloud based business intelligence project here at the BeyeNETWORK and thought it might be interesting to share details along the way. I will try to post as we achieve milestones for the project.

So first off let me share a little about the scope and our goals. As I am sure you are aware we capture lots of transactional data on what our readers interact with across the sites that make up the Network in January we made changes to our global passport system that allowed us to normalize the base data we collect enabling a cleaner view of the content and how it related to readers. Our goal is to aggregate the information and to identify trends in the industry, popular topics and authors and to deliver more finely focused dynamic content to readers.

I'm sure you won't be surprised to hear that data grew quicker than we expected and initially we used straight SQL queries to gain some insight into what was happening with our data. We then created a simple dashboard to deliver views of the queries, gave access to others on the team and quickly brought our systems to a grinding halt. It was a good example of how not to implement a data warehouse and business intelligence project. The data was stored on our web servers and every time I asked a serious question of the data it affected the Networks performance. Not to mentioned we were storing the information in a simple series of tables not optimized or organized for analytics.

So, this brings us to the challenge. I need a system that serves my business needs, integrated our data to the cloud in a model optimized for analytics so I can deliver business intelligence insights into what our customers are most interested in. We plan to do the following with help from friends and partners and I will share about that in my next post. In the meanwhile we are beginning the data modeling stage and implementing a methodology for handling our data that will provide a flexible platform as we move forward. We have chosen Amazon as our cloud host and will virtualize a copy of our existing platform there to support the business intelligence platform we require and to keep the environment consistent for our engineering team.

I see the cloud as a perfect platform for this project and as more vendors have embraced the cloud we found solutions that fit our needs and offered the flexibility we require. The data is small now (8M rows, growing 1M per month) but as it scales the platform will as well. The costs are very manageable as well.

That's all for now I will bring you up to date on our progress again soon as well as providing insight into the process and the solutions we are using.