after going on hiatus for quite a long time I’m back with my favorite issue: The correct aggregation along diamond-shaped hierarchies.

So far I was thinking that MS Analysis would calculate the right values by using the function “distinct sum”.

As far as I understand this function it sums up all distinct values along a hierarchy. IMO this is not the right approach to sum up all values of all distinct members along a hierarchy.

Let’s modify our example data a bit:

On the right you can see a part of the sales force of a financial services company.
C1-C4 is a subset of the clients, R1 and R2 are two sales reps and M1 is a regional manager.
For each client, the measure “a” is given, which means the yearly amount of money available to the client for investment. In order to calculate the investment potential for each node in the hierarchy, “a” has to be summed up.

It is typical for financial services companies, that each rep is expected to fully exploit a client’s potential. That means, that it is no option to use weighting factors to share the client’s potential between reps. As you can see on the right, C2 is assigned to two reps, whereas C2’s full investment potential has to be assigned to R1 and to R2 respectively.

For M1 each client’s potential must be taken into account only once.

To have something to work with, I give you the investment potentials for each client:

• C1: 7.000
• C2: 7.000
• C3: 8.000
• C4: 4.000

Here are the expected results of the aggregation:

• R1: 14.000
• R2: 19.000
• M1: 26.000

IMO, “distinct sum” would deliver the following values, which are NOT correct:

• R1: 7.000
• R2: 19.000
• M1: 19.000

I would highly appreciate to receive further comments by MSAS gurus . Maybe I’m all wrong and the “distinct sum” function would do the job perfectly.

Before you start to wonder at the cryptic names of the tables and columns, I’d like to stress the importance of naming conventions.

Here, all the tables are assigned to a schema called “cmdm”, where “mdm” means “multi dimensional model” and “c” is the prefix for “Cubica”, the name of the data warehouse framework I have developed.

The ER-diagram contains three tables, which will be the mininum number of required tables for a dimension. Please note, that this model is very basic and does not contain any tables or columns for versioning or time-variance. As we go along and cover further aspects of data warehousing the model will successively be enhanced.

So why are there at least three tables instead of just one dimension table like in a “normal” star-schema? Here’s why:

1. there can be raw data on any hierarchy level
2. one raw data member can be a part of any number of consolidation paths
3. there can be non-leaf members with data and three different ways to aggregate them
4. the hierarchy can be irregular and include diamond shapes
5. the data model should be as generic as possible

Let’s examine the tables in detail:

1. Root Dimension Table (Prefix “dim_r_”)
   A word about terminology first: I consider the lower, fine-grained levels of a hierarchy as root levels. In the Microsoft world they are called leaf levels (see also the term “non-leaf members with data”). The name “Root Dimension Table” comes from the fact, that most raw data can be found at the lower (root or leaf) levels of a hierarchy. If you like you can also say “Raw Dimension Tables” (as long as you keep the “R” as the first letter ).The Root Dimension Table contains all hierarchy members, which potentially can be assigned to raw data. E.g. in a sales force, clients can be visited by regular sales reps as well as by regional managers, who are placed on a higher level in the regional hierarchy. That means, that sales raw data can occur on different hierarchy levels. Other members of the hierarchy might never be assigned to raw data and their data is exclusively calculated be aggregating the raw data of their descendants. Those members are not included in the Root Dimension Table.As you can see, the Root Dimension Table is absolutely flat. There is no hierarchy or some other kind of relationship between the data rows. It is just a plain collection of all members, which can potentially be assigned to raw data. Here is a description of the three fields of a Root Dimension Table:

   • r_skey
     Root Surrogate Key: The primary key of the Root Dimension Table. It is an artificial (surrogate) key and wherever the name “r_skey” can be found in the data model it always has the same meaning: the primary surrogate key of a Root Dimension Table.
   • node_type
     This is an integer field to determine the type of entity (or hierarchy node) of the hierarchy member. Normally the hierarchy level can be used, but there can be very weird hierarchies that do not have classical levels. Thus, the term “node type” is used for the sake of generity.
   • e_key
     Entity Key: Each node type refers to a specific entity type. In a geographical dimension there can be stores, districts, cities, regions, states, etc., which are all entities of different types. To provide a connection to the “outside” world, each dimension member must have a business key. Because of the different node types the number and the data types of the key components are not the same for each member. To handle that, the components of the business keys are concatenated (using an appropriate seperator between the components) into one varchar field (of arbitrary size). Alternatively it is also possible to have a collection of all business keys of all entity (or node) types.
2. Dimension Table (prefix: “dim_”)
   This table is very similar to a dimension table in a normal star-schema. It contains all the members of the dimension including the members with raw data.The Dimension table contains the following attributes:
   • skey
     Surrogate Key: The primary key of the Dimension Table. It is an artificial (surrogate) key and wherever the name “skey” can be found in the data model it always has the same meaning: the primary surrogate key of a Dimension Table.
   • node_type
     Same as above
   • p_skey
     The skey of the parent of the current member. That means, that the Dimension Table is generally designed as a parent-child hierarchy with a self-referencing foreign key.
   • r_skey
     By now, you already know what an r_skey is: the primary surrogate key of a Root Dimension Table. It’s a foreign key referencing the Root Dimension Table. This key is not null for each row in the Dimension Table that refers to a member that can potentially have raw data. For all the other members r_skey is set to null.
   • e_key_1..n
     The business keys of all relevant entitiy (or node) types. Because the Dimension Table can be accessed from outside the data warehouse core, it is appropriate to a have seperate fields for each relevant entity business key. The field types must not necessarily be varchar (as in the above example).

   You might ask yourself why we have two member keys here: skey and r_skey. This is because a Dimension Table does not contain dimension members!

   “WHAT?” - you might ask. In the common data warehouse literature a dimension table is always defined as a collection of dimension members. This is with no doubt true in many cases, but not in a generic multi-dimensional data model.

   I will discuss that in my next post, and I’m also going to describe the third table in the ER-diagram above. Until then, stay tuned.
   

1. Raw data from external data sources
2. Staging area
3. Consolidated and enriched raw data
4. Multidimensional data (dimension tables, fact tables, etc.)
5. Data marts

When we take a look at the data flow in the warehouse process we can find the most client-specific or application-specific requirements at the transitions 1->2, 2->3, and 4->5. In most cases, the transition 2->3 is particularly specific, complex, and elaborate.

On the other hand, the transition 3->4 is a good candidate for a generic approach. The complete process can be defined by a set of meta data and a set of procedures, which, based on the meta data, can dynamically build and execute the required SQL statements. The transition 3->4 can still be very complex and elaborate, but it is by far not as application-specific as the other transitions.

Sections 3 and 4 form what I call the Warehouse Core. This is the place where you can normally find some sort of star- or snowflake-schemas. The warehouse process of the core is primarily made up of the following steps:

• Aggregations
• Management of historical changes
• Management of structural changes

One might say, that these operations can be very application-specific. This is absolutely right, but, compared to the other sections, they can easily be customized and configured by enhancing the star-schema and the meta data a little bit.

This post shall just set the stage for the upcoming detailed description of the enhanced star-schema and the associated meta data , which are both part of the Data Warehouse Framework I have developed over the past years. With the help of this framework I have succesfully implemented a number of Data Warehouse solutions, many of them containing irregular and ragged hierarchies and non-additive measures.

Now, let’s put it up another notch by adding another real-world requirement: Non-leaf members with data.

When there are non-leaf members with data, the aggregation can be done in one of the following three ways:

1. Distinct sum over the descendants of the non-leaf member
2. Distinct sum over the descendants of the non-leaf member and the non-leaf member itself
3. Data value of the non-leaf member itself

The newest crucial question: Which BI-Tool is able to aggregate distinct sums over many-to-many dimensions with non-leaf members with data?

Again, we’re looking at a dimension with diamond shapes:

On the right you can see a part of the sales force of a financial services company.
C1-C4 is a subset of the clients, R1 and R2 are two sales reps and M1 is a regional manager.

For each client, the measure “a” is given, which means the yearly amount of money available to the client for investment. In order to calculate the investment potential for each node in the hierarchy, “a” has to be summed up.

It is typical for financial services companies, that each rep is expected to fully exploit a client’s potential. That means, that it is no option to use weighting factors to share the client’s potential between reps. As you can see on the right, C2 is assigned to two reps, whereas C2’s full investment potential has to be assigned to R1 and to R2 respectively.

For M1 each client’s potential must be taken into account only once.

To have something to work with, I give you the investment potentials for each client:

• C1: 2.000
• C2: 7.000
• C3: 8.000
• C4: 4.000

Here are the expected results of the aggregation:

• R1: 9.000
• R2: 19.000
• M1: 21.000

Please do not hesitate to leave your comments here. I’m still convinced, that there is no BI-tool available, which would be able to solve to above mentioned problem. These kinds of aggregations and hierarchies are everywhere and it would be a quantum leap to finally have an out-of-the-box solution that could cope with them.

If this is done, requirements engineering, specification, documentation, and project management is nothing more than collecting meta data and assessing the completeness of the meta data set. Through priorities and processing sequences it is possible to completely define a procedural model for the DWH project.

When I speak of meta data, I do not only mean the more or less technical data, which describes dimensions and facts, but also data, which describes the warehouse process (ETL), and, most important, “political” data like target groups, stakeholders, team members, and other important people.

To get the most out of the meta data and to alleviate the collecting and administration of the meta data set, I frequently use a relational database. That allows me to generate a GUI for entering data and a number of different reports. Plus, this database can be used as a central repository for each member of the project team. For the project manager it can be of great help, if it contains typical project information like target date, status, estimated effort, remaining effort, responsibilities, etc. for the relevant entities.

A big advantage, which is based on the completeness of the meta data set, is, that certain pitfalls and showstoppers can be identified at a very early stage of the project.

Here is an example from one of my projects: I’m always especially paranoid with historical variability like slowly moving dimensions (which often turn out to be rapidly changing dimensions). Hence there are a number of attributes in my meta data model, which describe SMDs. In the (meta data based) process of specification and requirements engineering I asked the client about the historical variability of the product hierarchy. The people I asked were very amazed and apparently, nobody in the company had ever though about it. The question was: What happens with historical data when the product hierarchy changes? Has the change to be applied to the historical data (especially aggregated data)? Through the procedural model implied by the meta data we were able to address the implications of the historical variability at a very early stage in the project and we could force the client’s management to make a reliable decision. Very often, these kinds of aspects finally occur when the BI system is already in production, jeopardizing the success of the entire project.

In one of my next posts, I’m going to describe the meta data model in more detail by identifying the different sections of the model and describing the attributes, which make up the different meta data entities .

The picture on the left depicts a typical hierarchy, which can often be found as the structure of a sales force.

C1-C4 are clients, who are assigned to the sales reps R1 and R2. The sales reps are both managed by regional manager M1.

A quite important measure for sales reps, managers, sales unit, and, of course, the company as a whole is the number of associated clients.

How would a typical BI tool be set up to calculate the number of clients based on the hierarchy on the left?

• The client level with members C1-C4 is defined as the raw data level. Each member has a client-id as a primary key.
  .
• The measures for upper levels for the sales reps and the managers are aggregated by the system. These aggregations are either pre-calculated or take place on-the-fly.
  The aggregation rule is “count(distinct client-id)”.
  .
• First, the measures for the sales reps are calculated with the following results: R1: 2, R2: 3
  .
• Based on the results for the reps, the measures for the managers are calculated. The result for M1 would be 2+3=5, which is obviously wrong!

OK, most Data Warehouse and BI people know that problem, which is often referred to as “diamond shapes in hierarchies”. This term is derived from the typical diamond shape, which arises when two knots have a child knot and a parent knot in common. You can (vaguely ) see the diamond formed by C2, R1, R2, and M1 in the picture above.

The diamond shape problem is sometimes solved with weighting factors for each parent, which share a common child. To achieve correct results on the higher levels, the sum of the weighting factors has to be 1. In the hierarchy above, R1 und R2 would both have a factor of 0.5.
The weighted measure would be:

• M1: 1+1+0.5+0.5=4, which is absolutely right!
• R1: 1+0.5=1.5; R2:1+1+0.5=2.5, which is, at least in my opinion, not only wrong, but also very cruel! Human beings are cut in half!

Another fine example for diamond shapes: In my first post I mentioned the “legacy” analytical software we wanted to replace with a new solution. One of the eye-opening flaws of that software occured when we were aggregating sales data for Bayer. As you might know, Bayer is the producer of Aspirine. Now, Aspirine can be used for different indications, e.g. as a pain reliever (analgetic) or as a blood thinner.A typical example for a hierarchy knot with more than one parent. I don’t have to tell you, that all the calculated measures on the upper levels were wrong.

The quintessence of all this is, that, in order to achieve correct aggregations, you have to calculate them yourself in the Data Warehouse before you render the results in a BI application or report. You cannot expect out-of-the-box systems to be able to deliver the correct results.

How to do it right and how to model your relational Data Warehouse by using an enhanced star schema will be described and discussed in the upcoming posts.

I will also address the problem of having raw data on different hierarchy levels.

Back around 1995 I was working for a software company, which had been the market leader for Sales Force Automation (SFA) and Electronic Territory Managment Systems (ETMS) software for the pharmaceutical industry in Germany. As the leader of the server development team I was in charge of the programs and the underlying Oracle database, which guaranteed a multi-directional data flow between the different sales reps and the head office.Reps were reporting sales calls and additional data about their activities as well as certain characteristics of the doctors, pharmacies, and hospitals they visited. Additionally, pharmaceutical companies were buying turnover and sales volumes data from external providers based on geographical segments, time, and different levels of a self-defined “product hierarchy”.

All this valuable data was residing in the database on the server, which obviously triggered the desire for analytical applications.

Long before I had been responsible for the server, an analytical application had been developed by a another team. This application did no longer satisfy the growing demands of the very heterogeneous user base and thus, we decided to abandon it and to develop a new suite of analytical applications, based on a real data warehouse, from scratch.

Then, someone had the very smart idea to check the market for out-of-the-box solutions! Plus, some of our customers were already using BI systems like Cognos. After contacting Cognos we agreed to hold a three day in-house workshop as a proof of concept and soon, three Cognos staffers showed up at our office in Heidelberg: The unavoidable sales guy, a senior pre-sales consultant, and a quite attractive and very nice young woman, who seemed to be new to the company and apparently doing some training on the job. I can’t recall her name today, so let’s call her Lucy.

It seemed as if the Cognos people thought: “Yet another prospect with some data to load, to aggregate, and to render in a couple of fancy reports. That’s just right for Lucy to get some more hands-on experience. We have already taught her how to deal with multi-dimensional data and how to customize the system according to the client’s requirements.”

To make a long story short, here is what happened: At the end of the third day we were all heading to our conference room, where Lucy was expected to present what she had been working on for the last three days. As we entered the room we found Lucy in a confused state. She was crying, unable to say a word. We all looked at each other and everybody was really shocked. It was a quite embarrassing situation.

Lucy explained to us, that she couldn’t even manage to model the product hierarchy, let alone to aggregate any data. She had asked almost every expert at her company and even though we also tried to supply as many information and help as possible, she couldn’t figure out a solution. The truth was, that there was no solution. The Cognos system simply was not ready for this kind of product hierarchy, which in turn is common for almost every pharmaceutical company in the world. The same holds true for the organizational and geographical dimensions of a pharmaceutical sales force.

I’m going to describe the above mentioned dimensions and hierarchies in great detail in the upcoming posts, but, in a nutshell, their main characteristics are:

• There are no fixed hierarchy levels, the hierarchies are purely parent-child
• Each knot can be either the parent or the child of a knot of any type
• Each knot can have any number of parents
• Raw data can occur at any knot in the hierarchy
• the data value of a knot can be the result of an aggregation of the data values of its children
• the data value of a knot can be raw data from an external data source, deviating from the aggregate of the children
• Very often, the data values (facts) are non-additive

After it had been clear to me, that there was not a single out-of-the-box solution available which could cope with these hierarchies, I dedided to pick up the original plan to develop a data warehouse model from scratch.

In the course of the next couple of weeks or even months I’m going to describe the statical and dynamical data warehouse model, which now has matured over the last ten years. The hierarchies described above are only one of several features, that can severely risk the success of a data warehouse project. Many of those features will be covered in the upcoming posts (e.g. time variance, completeness of meta data, ETL tools, skills of the team members, hardware, etc.).

Finally, I can’t resist to tell you what I found out just a few weeks ago. I was doing a presentation of the prototype of an OLEDB for OLAP provider I have developed recenty (more on that in an upcoming post). It took place in the German head office of one of the world’s leading players in the phama market. They are using Cognos as their BI platform and obviously I asked them how they deal with the organizational sales force hierarchy. In a mixture of sarcasm and embarrassment they replied: “C’mon, you know it! We have to tell users not to look at any numbers above a certain level in the organization, because they are all wrong!”.

I don’t know what happened to Lucy after our workshop more than ten years ago. I hope she is doing as well as Cognos is doing today.