http://www.b-eye-network.com/blogs/hackathorn/ Welcome to my blog that focuses on business intelligence in the global context. We live in a increasingly small world. Our economy is globalized to a large degree in every industry, whether we like it or not! Please join me by commenting on these entries over the coming months. Let's explore the globalization of business intelligence together. en Copyright 2008 Fri, 04 Jul 2008 14:45:04 -0700 http://www.sixapart.com/movabletype/ http://blogs.law.harvard.edu/tech/rss blogged about Wordle.net and gave a word cloud of his blogs. This is what I got when I submitted my collection of blogs. So, if you want to know what the Independent Analysts Platform did this last week, just ponder the following this illustration. ]]>]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_the_word_cl.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_the_word_cl.php IAP08 Fri, 04 Jul 2008 14:45:04 -0700 Composite Software with their Discovery Appliance. David Besemer gave an insighful demo. + Progress Software with their latest SOA offerings. Hub Vandervoort and his pipes and side-pipes. + Kalido with their metadata mgt. Bill Hewitt and Robert Dickson did well with a tough concept to motivate and demonstrate. + HP with Neoview. The one-liners from Greg Battas cut to the chase. + Dataupia with Satori Server. John O'Brien sliced and diced thru the architecture. Rick van der Lans did a super job herding us cats and bringing all the logistics together. For a glimpse at the cast of characters, view our homemade video that congratulated Rick on the event. The blogs about IAP are collected here. There was definitely synergism flowing among this group of 'independent' analysts. If you are an analyst with similar interests, please contact us. The analysts attending are: Peter Aiken, VCU & Data Blueprint Barry Devlin, 9sight Consulting Jos van Dongen, Tholis Consulting Clive Finkelstein, Information Engineering Services Mike Ferguson, Intelligent Business Strategies Beth Gold-Bernstein, ebizQ Richard Hackathorn, Bolder Technology Jan Henderychx, Brainware Claudia Imhoff, Intelligent Solutions Bill Inmon, Inmon Data Systems Krish Krishnan, Sixth Sense John Ladley, IMCue Solutions Rick van der Lans, R20 Consultancy Stan Locke, Zachman Framework Associates David Loshin , Knowledge Integrity Mark Madsen, Third Nature David McGovern, Alternative Technologies William McKnight, Conversion Services International Shawn Rogers, Powell Media Alec Sharp, Clariteq Systems Consulting Gwen Thomas, Data Governance Institute Nancy Williams, DecisionPath Consulting Lyndsay Wise, WiseAnalytics John Zachman, Zachman International]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_reflections.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_reflections.php IAP08 Thu, 03 Jul 2008 16:14:01 -0700 Mike Boyarski, Director of Product Mgt of Ingres, talked on Appliance-Based Computing for BI. He has working at Oracle for eleven years and then to On-Demand. Mike mentioned the long history of Ingres. However, the current goal is to be the enterprise-class open source database. A number of companies are contributing to the open-source base, such as DATAllegro contributing hash key encoding, sort performance and partitioned indexing. The academic institutions are doing significant database research on the code base. He remarked that the market fits better with the mid-market companies. When compared with MySQL, Mike inserts that Ingres is better in performance. Companies that are basing products on the Ingres codebase are: Pentaho/JasperSoft, Business Objects, DATAllegro, GoldenGate. Ingres tends to be a good technology alternative for the SaaS market. The rapid innovation of open-source software has become a problem for most data centers that require more stability. Mike defined Appliance-Based Computing as a “complete pre-integrated and standardized BI software stack”. He asserts that hardware is not part of the appliance, despite the sensitivity of a BI system to processor/network balance. The Ingress Icebreaker BI Appliance consists of the following stack: Jasper, Apache/Tomcat, Java, Ingres 2006, Linux. Mike characterizes that this system results in a capability similar to Business Objects two years ago. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_ingres_thin.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_ingres_thin.php IAP08 Wed, 02 Jul 2008 18:34:46 -0700 John O’Brien, CTO and co-founder of Dataupia, and Samantha Stone, VP of Marketing, remarked that their tag line is to 'free your data' to provide open access to data for everyone. The goal is unlocking the power of data for business users. Dataupia has just experienced the one-year birthday from their first product shipment. Their product is the Satori Server, which John asserts is a true DW appliance, since it is shipped as a hardware/software integrated product. He said that their product is non-disruptive because their product brings the power of a MMP database to Oracle, Microsoft SQL Server, and IBM DB2 (called OMI as a shorthand for the three database engines). It works with the customer’s existing applications and existing databases. Pushing toward core IT processing, John said that their target market is the data warehouse mainstream that consists of: + long term data archiving + routine BI using massive data sets + ODS John gave the example of the Dataupia architecture as being the backend to OMI to provide the parallelism to existing applications without changes. Their storage is custom designed based on records, rather than blocks. It is record-oriented data store. John asserts that their record-orient database has the same performance levels as column-oriented databases. They are not using compression at the data item level, but do utilize it for aggregations, which often result in 80% compression. John outlined the requirements of several current customers, such as Subex, Sendio, Tektronix (network traffic ) and Focus (teleco). The main advantage of Dataupia seems to be their transparency at the application level by being a backend to OMI database engines. They have extended their SQL functionality to correspond to the uniqueness of the front-end.]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_dataupia_fr.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_dataupia_fr.php IAP08 Wed, 02 Jul 2008 17:41:09 -0700 Dave Schrader, director of Strategy and Marketing at Teradata, returned for a second hour and give an overview of what is new at Teradata. He quickly focused on what was new with Teradata, which he was divided into the following seven areas: 1) Data mart consolidation: emphasized the value of cross function data, so combining data marts has great value potential than separate ones. 2) New Platforms: quick overview of the product family: 550, 2500, and 5550 (A side remark was that Teradata now has three customers that have over a petabyte of data.) 3) Database Improvements: moving more functionality into the parallel database, 4) Master Data Management: i2 capability was folded into the MDM product. 5) Application Extensions: + Teradata Relationship Manager added AssetLink to support Market Resource Mgt (where are the dollars going), extended campaign mgt with an arbitrary number of steps, and support of mobile marketing/sales. + Financial and Performance Management partnered with Hyperion for more metrics, integrated with SAP, 6) Partnerships: The big 3 for Teradata are SAS, Microsoft, and SAP. SAS partnership focused on consolidation of their data sources and embedded SAS analytical processing into the Teradata database. SAP partnership focused on keeping the SAP interface but using Teradata for the larger cube creation and other analytics. Microsoft partnership focused on using Microsoft analytical services on top of Teradata, to provide high end solutions. 7) Education and Services: reaching out to various groups more. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_teradata_co.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_teradata_co.php IAP08 Wed, 02 Jul 2008 15:21:36 -0700 Mike Hoskins, CTO and General Manager of the Data Integration for Pervasive Software, described his company as an established mid-level software firm. Pervasive started in 1982 as SoftCraft selling the Btrieve database, which is still being used in many places in the world. Another early data integration vendor, Data Junction, was acquired in 2003. Over the years, Pervasive has become a leader within the data integration market characterized by their 'agile embeddable integration'. Pervasive products are: DataIntegrator and DataProfiler. There is flexibility for embedding within a SOA environment. In the future, the Pervasive data services platform allows modular configuration through a data mediation services. A Metadata Studio is being developed to collect metadata from a variety of products and stored it into a common repository, to be managed from the studio. DataRush is being developed to solve the larger data integration problems. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_pervasive_m_1.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_pervasive_m_1.php IAP08 Wed, 02 Jul 2008 14:06:54 -0700 Greg Battas, Chief Technologist for the BI Group, works for Ben Barnes in HP Software Group. He started with a basic definition of Neoview, which is integrated H/S warehouse platform by NP, built out of low cost commodity parts. It supports EDW, ODS and 'challenging' data marts, by having shared services, mixed workload, 3 to 300TB of raw data, complex query processing, and high SLA. Neoview is winning in situation of mart/DW consolidations, Teradata migration (replacement or dual vendor strategy, offload for high SLA), and outgrown Oracle implementations. Greg summarized their positioning as being the 'extreme sports' guys of BI. BI is evolving! It is no longer my mother’s data warehouse, which has the simple left-to-right flow from input data to output reports. Now, the DW has many operational systems feed and feed off of the DW. Doing things quickly is a critical requirement. Greg used the analogy that HP has been known as an 'arms dealer' that sells stuff to all the key BI vendors, without having a BI solution of their own. This has changed with the development of Neoview. Greg gave a concise history of the evolution of Neoview from a rewrite of Tandem for high-end performance requirements. The first test bed for Neoview was HP’s internal IT with Randy Mott’s team. Neoview is designed for operational BI for hundreds of terabytes with 24x7x265 always-on, fault tolerant, and black-box administration. Greg mentioned that, to get a rough estimate, he went to the HP.com site and configured a server with 100+ processors and many TBs, all of standard parts. Give a heart attack to the web analytics folks! Workload flexibility pipelined data flow to handle a wide spectrum from big complex queries to short, high-SLA queries to long-running queries, as shown next. As an example of extreme DW requirements, a teleco firm required the ingest of 3B rows into 72 B rows of detail data, while also incremental extract of 3B rows for fraud detection, concurrent aggregate build, a massive affinity analysis, a large adhoc report, and 24 K queries from customer service reps who need 3-second responses. There are no windows to introduce new features or fix bugs. The world is not moving rapidly toward batch. The system must be up all the time. Neoview has several partner integration projects, such as embedding SAS algorithms embedded into the Neoview engine. Both ETL and ELT are bad approaches. What is really needed is to take the data transformations and other data integration pieces and place this DI processing close to the file/storage system. The normal query execution has dual access to the data. With the acquisition of Knightsbridge and recently EDS, HP has the ability to provide enterprise-level professional services to augment their product offering of Neoview. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/hp_neoview_desi_1.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/hp_neoview_desi_1.php IAP08 Wed, 02 Jul 2008 13:03:54 -0700 Bill Hewitt, president and CEO of Kalido, showed how companies can get their hands around the changes in their markets. Does IT help companies get better? Bill summarized their focus as "Information should be delivered as a service, where producers and consumers will create/consume/distribute information in near real-time." By the time that data gets to the proper people, the information is too late, too old. Managers need the context and the relevance for their business. The Kalido Information Engine consists of: + Business Information Modeler + Dynamic Information Warehouse + Universal Information Director + Master Data Management Robert Dickson, principal resales consultant, continued with a detail demonstration of augmenting a Cognos tool with enhance metadata. In particular, the field of Reseller Type was added as an attribute of Reseller so that users can query on this new field. It is interesting that Kalido does not supply a database engine (since it uses SQL Server, DB2 and the like) or query tools (since it uses Cognos, Business Objects, and the like). It automates the metadata management layer above database/query products so that the data warehouse is more responsive to changing business requirements. P.S. During the break David Loshin and I discussed the Kalido presentation and felt that it was the best so far. The reason is that the CEO came and give the value proposition, all in 20 minutes. Then, a knowledgeable technical person gave a detailed demo with real product, all in 30 minutes. And, the demo actually worked! ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_kalido_prov_1.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_kalido_prov_1.php Events in the BI Sector IAP08 Wed, 02 Jul 2008 11:33:36 -0700 Roger Hodskins, VP of Marketing and Alliances, described Lumigent as providing solutions that secure their business application data from unauthorized access, eliminate vulnerabilities, and assure privacy. The market niche is labeled GRC - Governance, Risk, and Compliance. Now that is a mouth-full. Does it really mean anything? Roger remarked that audit gets a bad rap. The purpose of an audit is to ensure that proper controls are in place within your company. Next, he mentioned compliance, which is usually imposed by government agencies. The levels of compliance are: regulations, governance framework, control framework, and finally the actual controls. Consequences of weak compliance ranges from going to jail to the loss of customer confidence. The steps in compliance are: + discover: uncover location/paths to be watched + assess: create baseline to measure changes + monitor: implement controls on key biz processes + alert: report on unusual activity + secure: identify vulnerabilities and close gaps Roger used the analogy of being the surveillance camera, rather than a guard. A real story, first reports from their product indicated that the DBA was doing changes during production hours, which was a violation of company policy. A call from the VP to the offender stopped that practice by all DBAs within a week. The market for compliance solution is estimated by Forrester to be 1% to 5% of annual IT application expenses. Using this percentage along with other measures, the market for compliance solutions is in the billions! The magic occurs in three stages of: policy engine, auditing engine, reporting/alerting. Data is collected from three sources: + native auditing of the application (such as Oracle Financials) + transaction log reading + network monitoring A unique feature of Lumigent solutions is to determine that a database operation was performed by a certain user, along with before/after values. This is possible because of close partnerships with application vendors (Deltek for government project mgt, Motorola for criminal justice, Trizetto for health care, and the like) Some customers are generating over a terabyte of compliance data per week! A large stock exchange has hundreds of millions of transactions per day. Pricing errors can create huge liabilities very quickly, which the Lumigent solution can detect quickly.]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_lumigent_te_1.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_lumigent_te_1.php IAP08 Wed, 02 Jul 2008 10:14:34 -0700 Mark Troester, director of product marketing, talked about their role as a supplier of data access technologies via their DataDirect Connect, XQuery, and Shadow (from NEON Systems). DataDirect is an independent operating unit of Progress Software. Mark requested that his discussion be considered confidential, so this blog ends. Sorry...]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_progress_as_2.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_progress_as_2.php IAP08 Tue, 01 Jul 2008 17:55:36 -0700 Edwin van der Sanden, CTO of Corizon, talked about composite applications (read 'mashups') within the enterprise. The architecture spans a User Interface, composition, reusable services, and connectivity into legacy systems and web services, as shown in the following figure. The tool set consists of: Corizon Studio and Composer, the UI Service Provider, US Service Builder, and UI Service Extractor. Corizon offers three deployment options: stand-alone, embedded in packaged app, and as a portlet. A concise product overview is available here.]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_corizon_doe.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_corizon_doe.php IAP08 Tue, 01 Jul 2008 17:15:38 -0700 Mark Theissen, COO of DATAllegro, remarked that he is responsible for everything that touches a customer. The company received $19.6M of additional funding in May for a total of $65M. They have experienced 300% growth annually and have 100 employees. Mark distinguished their product as a second-generation of data warehouse appliance (DWA) that uses industry standard hardware and software, such as EMC storage, CISCO networking, Dell server, Ingres database, along with tool partners like SAS, Business Objects, MicroStrategy, Cognos, and Informatica. Mark asserts that they have "ultra shared nothing" architecture, as shown in the following diagram. Dual control nodes handles incoming queries, parse/optimize queries, and distributes the query across the compute nodes. The compute nodes have separate disks that are separate from the main data on the EMI storage. Dual fibre channel is used to connect two compute node to a storage node, while dual infiniBand does the interconnect of the compute nodes. Dual control node Mark said that a key competitive advantage for the company is grid technology that can move data at 500 GB per minute. They offer three different configurations: hub-and-spoke, multi-temperature, and in the future disaster recovery. Teradata integration is important because DATAllegro has perked the interest of many Teradata customers. The use cases are: batch accelerator, BI offload, and online archive. They have four Teradata customers currently. The key is to get the data out of Teradata quickly. When compared with the new Teradata appliance offerings, Mark felt that DATAllegro offers same capability at a lower price. Further, he felt that the Teradata offering are underpowered with reduced functionality. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_datallegro_1.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_datallegro_1.php IAP08 Tue, 01 Jul 2008 16:26:38 -0700 Dave Schrader, director of Strategy and Marketing at Teradata, presented the framework of Active Enterprise Intelligence (AEI). The role of the Enterprise Data Warehouse (EDW) is to integrate cross-functional data. AEI is the next step beyond EDW, involving the alignment of strategic intelligence with operational intelligence and the opposite. This results in value from cross analytics, along with value from AEI to support accelerated decisions. The AEI architecture consists of six key features: active access, active loading, active events, active workloads, and active availability. Dave motivated the technical aspects with a variety of real world examples. He stressed the need for intelligent decisions at all levels of the company and noted that being intelligent is not always obvious. For example, telecomm firm monitor the balances of their subscribers to detect unusually large amounts and then proactively contacting subscribers to assist them in paying their balances. This results in lower debt and greater customer satisfaction. Technology is not the limiting issue. Of the triad of People-Process-Data, the people and process are the issues that need attention in today’s business environment. Teradata differentiates on their AEI framework and asserts that their customers are doing this. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_teradata_go.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_teradata_go.php IAP08 Tue, 01 Jul 2008 14:09:38 -0700 Kristina Kerr, Senior Product Manager at Microsoft, presented the topic of Business Intelligence. Kristina started with a classic Dilbert cartoon (dated 5-14-07) about the futility of using dashboards at least by the pointed-hair manager. With a background with Crystal Reports and Business Objects, she became interested in Microsoft’s efforts with dash boards. Customers are thinking bigger than BI and even bigger than ERP. The vision of Microsoft is to providing business insights to ALL employees leading to better, faster, and more relevant decisions. Typical day for information workers involves email and the like, but never do they explicitly deal with BI. The best BI is that which is hidden under Excel, email and such. She showed a table with two dimensions of Scope as personal to shared and Development as organic to intentional. The result in many current situation is confused or frustrating. At the personal-organic, it should be self-service. At the shared-intentional, then it should be performance management in terms of KPIs. Then, the result is empowered for personal-organic or aligned for shared-intentional. The implication is three levels of context for BI: personal BI, team BI, organizational BI. Kristina walked through an extensive scenario, starting with SharePoint scorecard for AdventureWorks sales. By clicking on various elements, relevant data would appear in other windows. SharePoint maintains a live data connection along with the proper permissions for the user. SharePoint also does a search across all metadata returning a list of objects, such as reports, people and a variety of things. Excel is extended with data mining functions, like analyze key influencers, detect categories, fill from example, forecast, highlight exceptions, and what-if scenarios. The point is that a lot of BI is being hidden within SharePoint, controlling security. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_microsoft_h_1.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_microsoft_h_1.php IAP08 Tue, 01 Jul 2008 13:00:20 -0700 Hub Vandervoort, CTO of Progress Software, presented. The company gets little recognition in the industry because 65% of revenue comes from OEM partners. He talked on the topic of Faster than a Speeding Bullet, which illustrated complex event processing within SOA as velocity within a pipeline. He illustrated the concepts and problems with a series of examples from manufacturing to equity trading. The obvious issue is how fast or fat is the pipeline. However, most overlook some key side-pipelines. A pipeline requires information provisioning that interacts with complex data with the packets flowing on the pipeline. Another side-pipe is the business visibility that reports on the status of the pipeline’s operation. One must alignment (balance) the velocity of the main pipeline with the velocity of these two side-pipes. Poor alignment causes higher error rates in the business operation. As an example, equity trading was 3-day settling period, which has been reduced to two hours. However, the update cycle on the data warehouse stayed with 24 hours. And, the reporting on those trades also stayed with daily cycles. The margin of safety must be as least 3x, implying that a two-hour latency in a pipeline requires at least a 40-minute latency in both the information provisioning and in the business visibility. Hub asserted that the margin should be 10x to 100x if a company is concerned about compliance. Why SOA? First, it is the basis for the event-driven pipeline. Second, it supports asynchronous, distributed and federated requirements. Third, it permits the non-invasive insertion of real-time side-pipes with less disruption. Progress Software is acquiring Iona Technologies to expand its service-oriented architecture product offerings. With its expertise in CORBA object request broker with pub-sub features, Iona has sold Progress products for years to bring smart endpoints together with smart networks. ]]> http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_progress_as_1.php http://www.b-eye-network.com/blogs/hackathorn/archives/2008/07/iap_progress_as_1.php IAP08 Tue, 01 Jul 2008 11:42:41 -0700