BeyeNETWORK Netherlands Blogs BeyeNETWORK Netherlands Blogs. Copyright BeyeNETWORK 2005 - 2019 150 31 BeyeNETWORK Netherlands Blogs Netezza: Pioneer Appliance for Large Data Management Discovery Days kicked off last week in Indianapolis and part of the focus was on Netezza. I gave a talk on the origins of appliances, based in part on the linear progression from uniprocessing to SMP to Clusters to MPP and made a point that I see appliances in that lineage. However, it was with the caveat that it's no longer linear and each appliance is putting different nuance to MPP. Appliances do represent something different than just bundled MPP systems.

Alan Edwards of IBM filled in some of the details of the Netezza story and value proposition. Netezza is very much a strong part of the IBM information management story today. Below are some points from Alan's talk. Netezza-aware professionals will already understand most, but keep in mind the lack of familiarity of the audience. Everyone can at least be reminded of these high points for Netezza.

Nearly 70% of data warehouses experience performance-constrained issues

Traditional systems are just too complex, too long to get answers

Netezza means results in Urdu.

Twinfin 3rd gen is a line of surfboards (and the latest line of Netezza appliances)

Netezza is purpose built for analytics

There a no "hints", etc. (none necessary)

They have an analytics package that runs in hardware (I believe this is a reference to the FPGA)

Appliances start at 1 tb and go up to 1 pb+


]]> Sun, 8 May 2011 18:02:27 MST
Dallas-based Open Source Columnar vendor Calpont latest release

Open Source. Check. Columnar. Check. Dallas' answer to a couple of others checking these boxes out there is InfiniDB from Calpont. I had a chance to catch up with them this week as they announced their latest release, 2.1, about 15 months after their commercial launch.

InfiniDB does not have indexes, does late materialization (or "just in time" as they call it) and multi-table hash joins.

Some of the items they were stressing were scalability, SQL extension, compression and performance.

The modules that perform user functions and performance functions can be scaled individually to accommodate need in either area. This is part of their plan to provide linear scalability.

Partitioning can be vertical or horizontal and comes by default with CREATE TABLE.

There was a lot of stressing of the predictable, linear performance.


Here are some SQL extensions that were added in last few releases that were stressed:


Limit keyword

User Defined Functions

STDDEV and related functions


Auto incrementing

Partition drop - to take individual partitions offline

Insert into table select from... where the from can be to/from InfiniDB and MySQL

Vertical and horizontal partitioning

Most of these are available in the enterprise edition. If you want their syntax guide, drop me an email.

Compression is also new and improved. Using the data set, they achieved 3x to 9x compression. Compression can be set on individual columns.

Performance with compression was a key takeaway:


Several recommendations for effectively using InfiniDB were given:

]]> Fri, 6 May 2011 07:35:57 MST
Love and marriage Go together like horse and carriage. This is a well known song made popular by Al Bundy back in the eighties when Business Intelligence (BI) was still known as the monthly sales report. Just like horse and carriage there is another relation. One that strongly influences the quality of your BI solutions. It is the relationship between business and IT. As in most relations there is one partner doing the demand and another taking care of the supply. So it is easy to imagine that business and IT are in some kind of relation or marriage if you will. However there are dark clouds appearing at the horizon.

IT is starting to fall back in their supply. The business, always wanting more, is suddenly confronted with less. In some cases the quality of the information is getting worse or the information is received too slow or even late. Business is getting frustrated because they want it now and they are not getting anything. The reason for this behavior by IT is explained best by the economic circumstances forcing the IT budgets down. The most easy and often made action, not the most smart one in my opinion, is to kill or at least minimize demand. So less BI projects are started and change request are not or slowly executed. The business is getting so frustrated by this lack of supply by their IT partner that they start to look for information elsewhere (google) or do it themselves. Enter the famous self service BI solutions. In the end it is fair to concluded that the relationship between IT and business is troubled. Time for relationship counseling.

In relationship counseling there are some models based on a letter of the alphabet that try to explain the nature of the relation. The capital letter V for example stand for a relationship where you once had the same expectations of your relation but you both headed in a different direction. This is the model that leads to explosion. Dishes flying everywhere and accusations like: ”you don’t understand me”. Not a good model. There is also the X-model. Like ships passing in the night you briefly share something good but all too soon you both head in different directions. Also not a perfect model. The model that brings most benefits is the H-model. Both partners stay the way they are. There is always some give and take but overall both keep on doing what they like or what they are good at. However you need to make sure that both partners are still heading in the same direction. That is where the horizontal bar comes in. This line makes sure that both partners are aligned. They communicate together and work on their expectations of the relationship. Most of the time IT and Business has a different relationship or model. The BICC or COIE is based on a support model. There is some central organization, often within IT, that support the BI strategy, support the data, support the end user and so on. In this model both partners are trying to meet half way or even doing the other partners work. For example IT building business reports instead of focusing on getting the data available on time and of good quality. It is only when you organize your BI competences and services according to the H-model that the task and responsibilities of all parties involved are getting clear. If IT does what is does best and the same applies for business, while at the same time they organize their shared expectations in a BI solution center, their relation starts to sparkle again. To quote Barnett Brickner: ”Success in marriage does not come merely through finding the right mate, but through being the right mate”.

]]> Fri, 22 Apr 2011 11:29:58 MST
Indianapolis, St. Louis, Phoenix, Columbus, New York: Come out to Discovery Days I'm about to keynote a fun series of talks with IBM called Discovery Days. We'll be covering topics around optimizing systems for speed, efficiency and analytics. If you're in any of the below named cities I hope to see you there! And if you're in St. Louis, Phoenix, Columbus or New York City and you want me to consult or speak to you or your team (end user organizations, software companies) while I'm in town, let me know as I can plan some time before or after the conference.

The known days so far are:

May 5

Indianapolis, IN

JW Marriott Indianapolis

May 10

St. Louis, MO

Renaissance St. Louis Grand Hotel

May 12

Phoenix, AZ

Ritz Carlton Phoenix

June 9

Columbus, OH

Westin Columbus

October 4

New York, NY


There will be a few more. I will post as they become known.

Here is the link to the first event in Indianapolis.

]]> Tue, 19 Apr 2011 12:54:45 MST
Trends from the Expo Hall I made it to the Oracle Collaborate show this week in Orlando. I presented a case study presentation on a MDM initiative. I spent several hours in the expo hall talking with people in some interesting concepts. A couple of themes stood out as being somewhat new to being on an expo hall floor.

1. Lifecycle data support

There continues to be evidence that new approaches are necessary to keep data alive and do it at a lower cost than keeping "all data online everywhere".

Sure, we've got major compression and columnar orientation to compress complete rows of data everywhere it resides. However, we usually don't usually want to carry production data loads in the development and test environments, yet we need data that approximates production data there. One feature of IBM Infosphere Optim, among others, is the ability to get a representative sampling of production data into those environments according to the direction you set.

Also, there were the archival features of TierData, which smartly determines which production data, even columns, you can safely archive to improve the overall performance of the application.

2. Part Time Help

The commoditization of certain functions that were always just there in the shop and taken for granted continues. For better or worse, services like SmartHelp from Circular Edge and DBA on Demand from SmartDog give their clients flexible work arrangements for a variety of technical enterprise functions.

]]> Sat, 16 Apr 2011 14:35:40 MST
BI Consultant van het jaar nominatie De verkiezing van BI Consultant van het jaar is weer aanstaande. Afgelopen jaren waren John Minkjan en Kasper de Jonge de ”BI consultant van het jaar”. Dit jaar is de uitreiking op het BI Event van Array, 17 mei a.s.

Vanaf vandaag begint de nominatieronde, deze duurt 2 weken (t/m 19 april). Daarna volgt via de website van Array de definitieve verkiezing.

Het belangrijkste criterium voor een kandidaat is de mate van kennisdeling in de BI community. Wil je iemand nomineren, omschrijf deze nominatie dan met voorbeelden van wat deze consultant voor actieve kennisdeling in de community doet.

Nominaties kunnen als antwoord op dit bericht of via de LinkedIn groep.

]]> Wed, 6 Apr 2011 12:04:36 MST
Setting up Shop with Master Data Management, Part 4 Finally, the MDM team needs to consider how many environments it maintains. Often, 2 development environments are required. One is a replica of production, which is used to work on production hot fixes and quick development activities as a result of production findings. The other is the environment that MDM is doing its ongoing development on. From the development environments, MDM maintains the path-to-production which incorporates all other pre-production environments.

It could come up as to which MDM environment application development environments will use and who will maintain those environments. If applications are going to use one of the MDM environments, there may be conflict with MDM's development. If a copy is used, space must be procured and that copy must be managed.

There are also MDM workflows, where we find a ton of confusion in regards to who will update them, since they are very focused on the business. If the project is providing data to MDM, the data will need to be sourced. If an ESB is present in the shop, its utilization must be considered.

Pre-production testing will likely need to involve multiple teams. Certainly, this is nothing new as many projects have had this need before. However, it increases the need for coordination of personnel, environments and testing resources. It is likely that each team will have its own set of tests, as well as the overall integrated test.

And finally, let's not forget good old production support. MDM normally needs to be on a 24 x 7 x 365 support schedule. Shops vary in terms of activity that might occur with MDM data overnight. Regardless, production support carries with it all the usual concerns of utilizing and educating existing first level support personnel and creating the escalation plan.

]]> Tue, 8 Mar 2011 12:46:06 MST
ParAccel CEO on the Teradata acquisition of Aster Data I spoke with Chuck Berger, CEO of ParAccel, today about the Teradata acquisition of Aster Data. "ParAccel is the lone remaining stand-alone analytic appliance" said Chuck.

Chuck sees this as validation that companies have more data than they can process and that traditional architectures are not optimized for processing huge data in short amounts of time. "Companies are feeling the frustration about the value they are unable to realize from their data."

"HP, IBM, EMC and Teradata wouldn't be making high value acquisitions if they weren't seeing demand" said Chuck. He also talked about the "natural enemies" that these acquisions were making and how ParAccel was staying away from making natural enemies. As an example, Chuck said "All the BI guys are leary of IBM because of Cognos." Sales teams will bring in their overlay representatives to deals and this will keep doors open to ParAccel.

ParAccel's strategy is not changing. It is to "excel with the best product and maintain independence. Create no natural enemies. Many still need partners in the analytic space."

Finally, ParAccel will be free to continue to pursue "innovation and not integration" as a result of the merger activity, according to Chuck.

]]> Thu, 3 Mar 2011 15:36:57 MST
Teradata to acquire Aster Data We should have seen it coming, but Teradata announced its investment (11% acquired in September, 2010) in Aster Data is going to the level of acquisition. Aster Data has an MPP implementation of a columnar data store and integration with MapReduce. Teradata has been quiet on the columnar and MapReduce fronts and now we know why. Columnar is undeniably important going forward into the analysis of massive data volumes. Co-existence with row-oriented databases is already happening.

We find MapReduce in those fringier cases where companies are going beyond needing to store human-generated structured data and into computer-generated data and unstructured data. Started by Google and implemented by many who are Google-like in their need to manage data, MapReduce represents a new market. It's one that is different from an EDW profile market where known queries against static data occur. It's also one where many companies will find themselves in during the next few years. Teradata had already formed a relationship with Cloudera to acquire data from Cloudera's CDH platform.

A Teradata EDW and an Aster Data store could easily co-exist at a client and handle different workloads. While both would have query workloads, Aster Data could use its advanced real-time data capture capabilities to feed summaries to the EDW. Before I go too far with this, Teradata did say the code bases would be separate for the known future. This is not dissimilar to Microsoft with their PDW (formerly the DataAllegro product).

You also need to recognize EMC (Greenplum), HP (Vertica), SAP (Sybase) and IBM (Netezza) mostly have multiple data stores, but all have analytic data stores on their hands! Teradata already had other appliances, having spent the bulk of the last 2 years rolling out its appliance line. And Aster Data was not just for Hadoop-level data need. It's managing several single terabyte and below implementations. The possibilities just with Teradata, the company, just exploded.

After previous market activity like this, many a company's sales and marketing teams can get unfocused and even in serious competitive mode. Teradata is very well run, but let's see if they can manage this challenge.

As Hadoop and its ecosystem matures, the game is changing from an era of not-enough to an era of plenty and a need to sort through the offerings.

Let us finally say Teradata is full stack and sees its immediate future as a full stack specialized player, alongside the better-known big players, offering an unparalled focus on information management.

]]> Thu, 3 Mar 2011 15:34:56 MST
Setting up Shop with Master Data Management, Part 3

As applications discover the need to work with MDM for their projects to be successful, and the data and the parties are identified, the MDM team needs to be able to engage the requirement. This is what I call setting up shop.

You will need to balance structure and agility.

Early in the project's planning/research phase, a general conversation about the project and its data needs should be conducted with MDM Leadership. As enough information is made available to complete the project plan tasks, the MDM team could provide the project team with the appropriate tasks for inclusion into their project plan. A description of a superset of those tasks follows.

All projects engaging MDM data need to share these documents with the MDM team or put MDM Leadership on outright sign-off for the following documents during the Requirements phase:

  • Business Requirements
  • Non-functional Requirements

The Business Requirements should contain diagrams and commentary on the interface(s) that the project will have to MDM. Depending on your working model, either the MDM team or the application team provides the technical aspects of the application integration with MDM. Figure out which in the manifesto or be prepared for MDM to be labeled "hard to work with."

]]> Sat, 19 Feb 2011 09:39:48 MST
Mobile Business Intelligence: When Mobility Matters, a BeyeNetwork Custom Research Report Prepared for MicroStrategy

I have completed this paper, where I make the case for one of the largest trends in BI, mobility.

"No matter what business you are in, you are in the business of information. And it's business intelligence that has long been the discipline to deliver the needed information. Demand for business intelligence as a means to get maximum value from information has never been higher as businesses increasingly compete in real time and require information that is integrated from across the enterprise. The old saw about business intelligence is that it gets "the right information to the right people at the right time." It's really time to add "right medium" to that mix.

Automating business decisions and action is one path to business intelligence maturity. Determining what actions to trigger automatically based on changes in corporate data can come from a solid understanding of how decisions are made today. However, many decisions are multifaceted, and a knowledge worker's analysis will continue to be a part of effective business intelligence.

Effective analysis is getting more complicated for knowledge workers. The more complicated aspects include sensing what is happening and combining that with summarized historical data to build a set of possible actions. These "analytics" are the basis of competitive advantage for organizations today. Once calculated, they must be put to effective use, again utilizing the best medium available for real-time delivery."

Please see here for the full paper. The contents include:

Business Intelligence Deployment Option History

Business Mobility

Mobile Business Intelligence Deployed

GUESS? Store Managers Don't Have to Second Guess Data

PriceLYNX: Going Mobile to Curb Supply Spend

What These Stories Tell Us Tips

Approaches to Mobile Business Intelligence

MicroStrategy Mobile

]]> Tue, 1 Feb 2011 08:33:49 MST
Setting up Shop with Master Data Management, Part 2 MDM programs are generally designed to provide the data needed by a cross-section of applications or for data that can utilize its workflow capabilities for its origination and updates. It's an approach usually not taken for data needed by a single application, although it may be done as a set-up for future applications.

Part of the MDM manifesto must include how teams will source its data. Over 75% of the post-implementation requests of MDM will be around this question.

In order to acquire the data, it must be mapped to the data structures of the target application. Who does this? The MDM team, the application team, a separate integration team, or a separate architecture team?

Regardless, all new projects should meet with MDM Leadership in a very early phase of their project to determine:

  1. Data available in MDM that should be used in the project
  2. Data not available in MDM that should be sourced by the MDM team for the project (and other projects)
  3. Data that the project is generating that the MDM team should source into MDM
  4. Time and resource estimate for the MDM team contribution

Data not in MDM that needs to be may be done in 1 of 2 ways:

  1. MDM (the MDM team usually) can source the data from its origination point or a third party system
  2. MDM can update or add to its workflow environment, which incorporates manual entry of the information at the right point

]]> Sat, 29 Jan 2011 09:45:10 MST
Setting up Shop with Master Data Management, Part 1 Several MDM programs out there are in development and about to go to production. Several others are struggling in production as they try to move the program into a second subject area or to engage more publishing or subscribing systems to the information. Others need to extend the data governance beyond a single business group.

Few have made the leap to successfully setting up their MDM program as a fully functioning member of the 'major system' ecosystem of the company. The guidelines in these blog entries will help those shops make that transition and address the questions that the REST of the company may have about MDM. It is absolutely essential that MDM be properly positioned to these important evaluators of the program success.

Enterprise MDM cannot be successful "in a vacuum" - built to meet the need of a single application/subject area that is well-known. Building MDM with this hyper-focus to the exclusion of all concerns for scalability results in just what I am seeing now: MDM re-dos and multiple MDMs where there could be one, enterprise MDM.

These questions include (again, from the perspective of those not in the MDM team):

What is MDM?

What data is available there?

Do I have to use MDM's data?

Do I really have to use MDM's data? Who will care if I don't?

What if the MDM data is not suitable for my application needs?

How long does it take to incorporate my data?

Whose role is it to add data to MDM?

Is it push or pull?

I'm just going to bring this third-party data into my application, not MDM, OK?

Is my need an extension of a subject area or a new one?

Who do I talk to about MDM?

Do I have to contribute my data?

How do I modify the existing MDM workflows?

Does the MDM team carry a separate project plan for my need?

Who builds the plan and manages those tasks?

How do I unit test, do quality assurance testing, etc. with MDM data?

Just knowing these questions could trigger the necessary action, but in case it doesn't, I'll keep posting here (and you can as well) some tips to setting up shop with MDM.

]]> Sun, 23 Jan 2011 21:29:17 MST
Super Fancy Sexy (again) Data Visualization HierarchyThose of you that know me or read my blog know that I am a strong advocate of visualization in Business Intelligence. I honestly belief that using images, charts, or other visual stimuli greatly helps users of a BI environment to quickly understand the data presented and work with it. I also think that great looking reports, dashboards or other BI products will be much more quickly accepted by business users if they just look nice. I often call this (within our Capgemini practice): It has to look super fancy sexy.Stephen Few is somebody who understand this and has written some great books about this as well (see Another believer is Hans Rosling with his Gapminder explorations ( More recently I came across this website: This is a nice website where you will soon discover that data-journalist Davic McCandless take data visualization serious. His Tetris like animation comparing the financial crisis with let’s say the fight against hunger in Africa is brilliant.Definitely something to check out if you have a couple of minutes. I strongly urge all BI professionals to work on a visual intelligence strategy and framework. 

]]> Fri, 7 Jan 2011 08:11:36 MST
Business en IT samen in de wolken (Artikel, Dutch) Je kunt momenteel geen vakblad openslaan, internetdiscussie volgen of seminar bezoeken zonder dat er vragen worden gesteld over Cloud. Veel vragen komen voort uit het simpele feit dat mensen moeite lijken te hebben met Cloud als concept. Waarom kies ik voor Cloud, wanneer gaan we in de Cloud, waar liggen de voordelen en hoe zit het met de veiligheid op de Cloud? Al met al kun je wel zeggen dat er sprake is van een stevige hype. We willen het hebben, ook al weten we niet eens wat het is. Iedereen denkt er het zijne van en het gevolg is een overvloed van definities waarbij met name de verschillen in interpretatie tussen business en IT opvallen. En gemeenschappelijk element komt wel terug in al die definities, het gaat over ‘delen’ op n of ander niveau.Zie bijgaand artikel in de cloud special van DBMBusiness en IT samen in de wolken

]]> Wed, 8 Dec 2010 07:59:54 MST