Blog: Colin White Subscribe to this blog's RSS feed!

Colin White

I like the various blogs associated with my many hobbies and even those to do with work. I find them very useful and I was excited when the Business Intelligence Network invited me to write my very own blog. At last I now have somewhere to park all the various tidbits that I know are useful, but I am not sure what to do with. I am interested in a wide range of information technologies and so you might find my thoughts will bounce around a bit. I hope these thoughts will provoke some interesting discussions.

About the author >

Colin White is the founder of BI Research and president of DataBase Associates Inc. As an analyst, educator and writer, he is well known for his in-depth knowledge of data management, information integration, and business intelligence technologies and how they can be used for building the smart and agile business. With many years of IT experience, he has consulted for dozens of companies throughout the world and is a frequent speaker at leading IT events. Colin has written numerous articles and papers on deploying new and evolving information technologies for business benefit and is a regular contributor to several leading print- and web-based industry journals. For ten years he was the conference chair of the Shared Insights Portals, Content Management, and Collaboration conference. He was also the conference director of the DB/EXPO trade show and conference.

Editor's Note: More articles and resourcesare available in Colin's BeyeNETWORK Expert Channel. Be sure to visit today!

Last week I presented at the Big Data Summit and attended Hadoop World in New York. Both events focused on the use of Hadoop and MapReduce for the processing and analyzing of very large amounts of data.

The Big Data Summit was organized by Aster Data and sponsored by Informatica and Microstrategy. Given that the summit was in the same hotel as that used for Hadoop World the following day, it would be reasonable to expect that most of the attendees would be attending both events. This was not entirely the case. Many of the summit attendees came from enterprise IT backgrounds and these folks were clearly interested in the role of Hadoop in enterprise systems. Whereas many of them were knowledgeable about Hadoop, an equal number were not.

The message coming out of the event was that Hadoop is a powerful tool for the batch processing of huge quantities of data, but coexistence with existing enterprise systems is fundamental to success. This is why Aster Data decided to use the event to launch their Hadoop Data Connector, which uses Aster's SQL-MapReduce (SQL-MR) capabilities to support the bi-directional exchange of data between Aster's analytical database system and the Hadoop Distributed File System (HDFS). One important use of Hadoop is to preprocess, filter, and transform vast quantities of semi-structured and unstructured data for loading into a data warehouse. This can be thought of as Hadoop ETL. Good load performance in this environment is critical.

Case studies from Comscore and LinkedIn demonstrated the power MapReduce in processing pedabytes of data. In the case of Comscore they are aiming to manage and analyze 3 months of detailed records (160 billion records) using Aster SQL/MR. LinkedIn, on the other hand is using a combination of Hadoop and Aster's MapReduce capabilities and moving data between the two environments. Performance and parallel processing is important for efficiently managing this exchange of data. This latter message was repeated by several other case studies at both events.

Hadoop World had a much more open source and developer feel to it. It was organized by Cloudera and had about 500 attendees. About half the audience was using Amazon Web Services and clearly experienced in Hadoop. Sponsors included Amazon Web Services, IBM, Facebook and Yahoo, all of whom gave keynotes. These keynotes were great for big numbers. Yahoo, for example, has 25,000 nodes running Hadoop (the biggest cluster has 4,000 nodes). Floor space and power consumption become major issues when deploying this level of commodity hardware. Yahoo processes 490 terabytes of data to construct its web index. This index takes 73 hours to build and has experienced a 50% growth in a year. This highlights the issues facing many web-based companies today, and potentially other organizations in the future.  

Although the event was clearly designed to evangelize the benefits of Hadoop, all of the keynotes emphasized interoperability with, rather than replacement of, existing systems. Two relational DBMS connectors were presented at the event including Sqoop from Cloudera and support for the Cloudera DBInputFormat interface from Vertica. Cloudera also took the opportunity of announcing it was evolving from a Hadoop services company to being a developer of Hadoop software.

The track sessions were grassroots Hadoop-related presentations. There was a strong focus on improving the usability of Hadoop and adding database and SQL query features to the system. I felt on several occasions many people were trying to reinvent the wheel and trying to solve problems that had already been solved by both open source and commercial database products. There is a clear danger of trying to expand Hadoop and MapReduce from being an excellent system for the batch processing of vast quantities of information to being a more generalized DBMS.  

The only real attack on existing database systems came surprisingly from the J. P. Morgan financial services company. The presentation started off by denigrating current systems and presenting Hadoop as an open source solution that solved everyone's problems at a much lower cost. When it came to use cases, however, the speakers positioned Hadoop as suitable for processing large amounts of unstructured data with high data latency. They also listed a number of "must have" features for the use of Hadoop in traditional enterprise situations: improved SQL interfaces, enhanced security, support for a relational container, reduced data latency, better management and monitoring tools, and an easier to use developer programming model. Sounds like a relational DBMS to me. Somehow the rhetoric at the beginning of the session didn't match the more practical perspectives of the latter part of the presentation.

In summary, it is clear that Hadoop and MapReduce have an important role to play in data warehousing and analytical processing. They will not replace existing environments, but will interoperate with them when traditional systems are incapable of processing big data and when certain sectors of an organization use Hadoop to mine and explore the vast data mountain that exists both inside and outside of organizations. This makes the current trend toward hybrid RDBMS SQL and MR solutions from companies such as Aster Data, Greenplum and Vertica an interesting proposition. It is important to point out, however, that each of these vendors takes a different approach to providing this hybrid support and it is essential that potential users match the hybrid solution to application requirements and developer skills. It is also important to note that Hadoop is more than simply MapReduce.   

If you want to get up to speed on all things Hadoop, read some case studies, and gain an understanding of its pros and cons versus existing systems then get Tom White's (I am not related!) excellent new book "Hadoop: The Definitive Guide" published by O'Reilly.

Posted October 6, 2009 1:37 PM
Permalink | No Comments |

I commented in a my previous blog entry that the controversy over ParAccel's TPC-H benchmark has become quite heated. This is especially true on Curt Monash's blog where at one point he made some personal comments about Kim Stanick, ParAccel's VP of Marketing. See this link for details.

This is the second blog this month that I have read where an analyst makes an attack, not only on the vendor, but also one of its employees. The other blog (and an associated article) was by Stephen Few entitled "Business is Personal - Let's Stop Pretending It Isn't." See this link for details.

The good thing about social computing is that it provides a fast way of sharing and collaborating about industry developments. However, these technologies have the same problems as e-mail and instant messaging, they enable people to react immediately to something that upsets or annoys them. With blogging, unlike email and instant messaging, everyone gets to see the results!

As analysts our job is to write balanced reviews of industry developments that provide useful information to the reader. My concern is that some analysts are behaving as though they are on cable television or writing for the tabloids. I believe we can critique a product without attacking a company, its products or its employees. Personal attacks by analysts are unprofessional, even if the company fights back against a review they take exception to. What do you think?

Posted June 25, 2009 1:52 PM
Permalink | 3 Comments |

ParAccel, one of the new analytic DBMS vendors, recently announced some impressive TPC-H benchmark results. A good review of these results can be found on Merv Adrian's blog at this link.

Not everyone agreed with Merv's balanced review. Curt Monash commented that "The TPC-H benchmark is a blight upon the industry." See his blog entry at this link.

This blog entry resulted in some 41 (somewhat heated) responses. At one point Curt made some negative comments about ParAccel's VP of Marketing, Kim Stanick, which in turn led to accusations that his blog entry was influenced by personal feelings.

I have two comments to make about this controversy. The first concerns the TPC-H benchmark and the second is about an increasing lack of social networking etiquette by analysts.

TPC benchmarks have always been controversial. People often argue that that do not represent real life workloads. What this really means is that you mileage may vary. These benchmarks are expensive to run and vendors throw every piece of technology at the benchmark in order to get good results. Some vendors are rumored to have even added special features to their products to improve the results. The upside of the benchmarks is that they are audited and reasonably well documented.

The use of TPC benchmarks has slowed over recent years. This is not only because they are expensive to run, but also because they have less marketing impact than in the past. In general, they have been of more use to hardware vendors because they demonstrate hardware scalability and provide hardware price/performance numbers. Oracle was perhaps an exception here because they liked to run full-page advertisements saying they were the fastest database system in existence.

TPC benchmarks do have some value to both the vendor and the customer. The benefits to the vendor are are increased visibility and credibility. Merv Adrian described this as a "rite of passage." It helps the vendor get on the short list. For the customer these benchmarks show the solution to be credible and scalable. All products work well in PowerPoint, but the TPC benchmarks demonstrate that the solution is more than just vaporware.

I think most customers are knowledgeable enough to realize that the benchmark may not match their own workloads or scale as well in their own environments. This is where the proof of concept (POC) benchmark comes in. The POC enables the customer to evaluate the product using their own workloads.

TPC benchmarks are not perfect, but they do provide some helpful information in the decision making process.

I will address the issue of blog etiquette in a separate blog entry.  

Posted June 25, 2009 1:43 PM
Permalink | 2 Comments |

My post a couple of days ago about data warehousing in the cloud led to requests for more information about this topic and related SaaS BI solutions.  

Claudia Imhoff and I recently published a research report on the BeyeNETWORK entitled "Pay as You Go: Software-as-a-Service Business Intelligence and Data Management." The report was sponsored by Blinklogic, Host Analytics, PivotLink and SAP BusinessObjects who offer SaaS BI solutions. It was also sponsored by Kognitio who (like Aster, GreenPlum and Vertica mentioned in my previous blog) have a data warehousing in the cloud offering. The report discusses SaaS BI and data warehousing and reviews the pros and cons of using this type of deployment model.

The report can be found on

Posted June 11, 2009 4:58 PM
Permalink | No Comments |

The use of cloud computing for data warehousing is getting a lot of attention from vendors. Following hot on the heels of Vertica's Analytic Database v3.0 for the Cloud announcement on June 1 was yesterday's Greenplum announcement of its Enterprise Data Cloud™ platform and today's announcement by Aster of .NET MapReduce support for its nCluster Cloud Edition.

I have interviewed all three vendors over the past week and while there are some common characteristics in the approaches being taken by the three vendors to cloud computing, there are also some differences.

Common characteristics include:

  • Software only analytic DBMS solutions running on commodity hardware
  • Massively parallel processing
  • Focus on elastic scaling, high availability through software, and easy administration
  • Acceptance of alternative database models such as MapReduce
  • Very large databases supporting near-real-time user-facing applications, scientific applications, and new types of business solution
The emphasis of Greenplum is on a platform that enables organizations to create and manage data warehouses and data marts using a common pool of physical, virtual or public cloud infrastructure resources. The concept here is that multiple data warehouses and data marts are a fact life and the best approach is to put these multiple data stores onto a common and flexible analytical processing platform that provides easy administration and fast deployment using good enough data. Greenplum sees this approach being used initially on private clouds, but the use of public clouds growing over time.

Aster's emphasis is on extending analytical processing to the large audience of Java, C++ and C# programmers who don't know SQL. They see these developers creating custom analytical MapReduce functions for use by BI developers and analysts who can use these functions in SQL statements without any programming involved.

Although MapReduce has typically been used by Java programmers, there is also a large audience of Microsoft .NET developers who potentially could use MapReduce. A recent report by Forrester, for example, shows 64% of organizations use Java and 43% use C#. The objective of Aster is to extend the use of MapReduce from web-centric organizations into large enterprises by improving its programming, availability and administration capabilities over and above open source MapReduce solutions such as HADOOP.

Vertica see its data warehouse cloud computing environment being used for proof of concept projects, spill over capacity for enterprise projects and for software-as-service (SaaS) applications. Like Greenplum it supports virtualization. Its Analytic Database v3.0 for the Cloud adds support for more cloud platforms including Amazon Machine Images and early support for the Sun Compute Cloud. It also adds several cloud-friendly administration features based on open source solutions such as Webmin and Ganglia.

It is important for organizations to understand where cloud computing and new approaches such as MapReduce fit into the enterprise data warehousing environment. Over the course of the next few months my monthly newsletter on the BeyeNETWORK will look at these topics in more detail and review the pros and cons of these new approaches.

Posted June 9, 2009 12:00 AM
Permalink | No Comments |
Aster Data Systems has announced the Aster nCluster Cloud Edition. The Aster nCluster is an analytical database system for supporting very large relational databases. Aster joins companies such as Vertica and Kognitio in providing data warehousing capabilities on a cloud computing platform. Aster supports both the Amazon and AppNextus cloud computing environments.

This announcement demonstrates the growing demand for analytical processing from companies that have a significant presence in the web marketplace. One of the users of the CloudEdition, for example, is Didit, a web ad marketing company. These types of companies need to analyze huge amounts of web-related data.

One interesting aspect of Aster is its support for MapReduce, which is a growing trend by database companies Greenplum has MapReduce support and IBM is working on supporting it (System S research project). MapReduce provides a framework for massive parallel processing and is used by a number of web-centric companies such as Google. A key distinguishing feature for Aster is that it supports custom SQL functions that exploit MapReduce. MapReduce capabilities could become a key differentiator in high volume cloud computing data warehousing environments.  

You can find out more about the use of MapReduce in database systems in an article I published on the BeyeNetwork entitled "Are Relational Database Systems Keeping Up with the Information Processing Needs of Companies?"

Given my database background, it's exciting to see that database products are moving away from being commodity items are now offering some important distinguishing features.

Posted February 10, 2009 8:45 AM
Permalink | No Comments |
Microsoft today announced major changes to its PerformancePoint BI product. The current version of the product, PerformancePoint 2007, provides three BI related capabilities: business performance management (BPM), BI analytics, and business planning.

The BPM component evolved from the Microsoft Business Scorecard Manager, whereas the analytics component is based on a subset of the functionality acquired from ProClarity. The planning component was a new component developed by the Microsoft BI group aimed specifically at financial planning and budgeting.

Today's announcement breaks out the BPM and analytics components from PerformancePoint and merges them into the Enterprise Edition of Microsoft SharePoint Server. Existing SharePoint Enterprise Edition users (with Software Assurance) will now get these components as a part of their licensing agreement. These customers will be able to download the PerformancePoint components starting April 1.

In the summer, Microsoft will release Service Pack 3 of PerformancePoint 2007. This will be the final release of the product, which will be supported for ten years.

Microsoft's strategy is to move the responsibility for financial planning and budgeting to the Microsoft Dynamics FRX and Forecaster products. However, horizontal planning capabilities will continue be added to Microsoft SQL Server, Excel, and SharePoint over future releases.

This new direction makes sense for Microsoft. Although Microsoft was emphasizing the BPM and planning capabilities of PerformancePoint, it was achieving limited success in these areas. Instead, the majority of customers were buying the product for its analytics capabilities. This was especially true for ProClarity users.

Another reason why this makes sense is that Microsoft SharePoint is a very successful product, and this is leading to companies purchasing related Microsoft solutions. Over 80 percent of PerformancePoint customers, for example, are also SharePoint Server users. The penetration of SharePoint in the market is also a key factor in the success of SQL Server and its BI components.

Given that Microsoft Office is also increasingly being integrated with Microsoft SharePoint, it means that customers will now be motivated to purchase Microsoft Office, SQL Server, and SharePoint Server as a product set in order to deploy business intelligence and related collaborative tools to a mass business user audience.

Posted January 23, 2009 9:00 AM
Permalink | 2 Comments |

I came across this article in the New York Times about the R programming language. It was interesting to note that it was the number 2 most read article in the technology section.

The article suggests R is a threat to SAS. Any perspectives on this or the use of R for data analysis?

Posted January 7, 2009 8:06 AM
Permalink | No Comments |

Well it’s the last day of 2008, and it’s tradition at this time of year to make predictions for the coming year. If the financial chaos of the last few months continues into 2009, which most financial pundits say it will, then the IT industry is heading for a tough time over the coming year. This makes predicting industry directions really difficult because IT organizations are less inclined to purchase new products and technologies when budgets are tight.

The business intelligence (BI) marketplace has often been immune to industry downturns. This is because companies often turn to BI in difficult times to help them identify areas where revenues can be increased and costs can be reduced. This is especially the case in front office sales, marketing, and support organizations. Given the potential size of the coming downturn, however, can even BI be immune? I doubt it.

I believe, however, that there are ways BI can ride out the coming storm and be of benefit to the business. I think the main task that organizations should focus on is using new BI technologies to reduce costs (rather than increasing revenues). This can be achieved by reducing the cost of delivering new BI business solutions and by increasing business user productivity.

The BI solutions that will have the most impact in 2009 will be those that provide IT and business users quick and low-cost approaches for discovering, accessing, integrating, analyzing, delivering and sharing information in a way that that helps business users become more productive and more self-sufficient.

This means that there will be increased interest in open source software, BI software-as-a-service, low-cost application appliances, search, the integration of BI with collaborative and social computing software, rich internet applications, web syndication, and data and presentation mashups. Many of these solutions will come from small innovative BI companies, rather than large software companies who are still struggling to integrate the morass of BI software they acquired in 2008.

The technologies mentioned support low cost and fast BI application deployment. Many of them will be used by line-of-business IT rather than the enterprise IT organization. This could result in a turf war where enterprise IT tries to control and govern the use of these new technologies by the business. This would be a huge mistake. Instead enterprise IT should look for best practices in the use of these technologies by business groups, replicate them in other parts of the organization, and look for ways of incorporating the cream of the crop into the existing IT environment.

The purists will cry that this will lead to anarchy and islands of data and software. If that is the case then so be it. In the coming 12 months we need to do what ever it takes to be productive and reduce short-term costs. This is not the time for fancy architectures, purist approaches, academic debates, or large projects.

Have a great 2009!

Posted December 31, 2008 4:14 PM
Permalink | No Comments |

Relational database systems, such as IBM DB2 and Oracle Database, have undergone over a quarter century of development. During that time they have managed to successfully fight off competing database technologies for supporting mainstream database management. Do you remember the object/relational wars of the eighties?

MapReduce, a software framework introduced by Google for supporting parallel processing over large petabyte files has garnered significant attention of late. IBM is experimenting with this in conjunction with Google, and GreenPlum recently announced support.

The significant interest in MapReduce, and related technologies such as Hadoop and HDFS, has led to a backlash from the relational camp. David DeWitt and Michael Stonebraker have been especially outspoken (see and

Here is a small quote from their thoughts on the topic:

"As both educators and researchers, we are amazed at the hype that the MapReduce proponents have spread about how it represents a paradigm shift in the development of scalable, data-intensive applications. MapReduce may be a good idea for writing certain types of general-purpose computations, but to the database community, it is:

1. A giant step backward in the programming paradigm for large-scale data intensive applications

2. A sub-optimal implementation, in that it uses brute force instead of indexing

3. Not novel at all -- it represents a specific implementation of well known techniques developed nearly 25 years ago

4. Missing most of the features that are routinely included in current DBMS

5. Incompatible with all of the tools DBMS users have come to depend on"

Does this mean the database wars are starting up again?

My opinion is that MapReduce is not intended for general purpose commercial database processing and is therefore not a major threat to relational systems. However, it does have its uses (as Google has demonstrated) for certain types of high volume processing. It also demonstrates that as data volumes get bigger, and the complexity of data and data structures increases, other types of database technology may start to gain traction in certain niche marketplaces. The use by IBM of the SPADE language, instead of StreamSQL, in its InfoSphere Streams product (System S) also demonstrates the changes going on in the database market.

What do you think?

Posted November 25, 2008 4:20 PM
Permalink | No Comments |