BB0005 – Example Project For Data Vault Strengths

0
77

Hello and welcome to our latest blog post here on BIDA Brains.

I am Mihai Neacsu, and I am the Business Development Manager here at BIDA.

I am pleased you have come back to see our latest blog post.

Thank you.

In our prior blog post I talked about the men we consider to be the most influential in our industry.

I also talked about how a suite of dimensional models were developed by the Data Warehouse Network.

They were called the Vertical Packaged Solutions, and they were able to implement archives as well as Operational Data Stores under certain conditions.

As a result Sybase was selling copies of the IWS Models like hot cakes.

When SAP bought Sybase there was no role for the IWS models and they were put in mothballs.

At some point in time SAP published these models.

They were still published as late as 2018.

You can go and read the blog post, that now has broken links, on the button on the blog post.

SAP IWS Models

By using the one modeling style for the end user query interface, the archive and the operational data store, the overall costs of implementing IWS were vastly lower than the third normal form models that Teradata were selling at the time.

At the same time, the benefits of a Sybase IWS data warehouse were approximately the same as using different modeling techniques.

Although most people will tell you that it’s not possible to properly archive data in a dimensional model, this was perfected twenty five years ago, and it has been implemented in to many companies.


 

So, you might ask the question, if dimensional models can archive data effectively why would you ever use a Data Vault or more complex archiving mechanisms?

Well, there are situations where this sort of dimensional modeling is not well suited for the archive.

So let us consider this example.

Just so you know, this is a real example.

Consider that most western countries have a government department that is responsible for the payment of welfare payments to citizens.

I will use Australia as an example.

Australia has a government department now called “Services Australia”.

This department is responsible for payments such as child care, child support, unemployment benefits, medical insurance and many others.

It makes payments in the order of billions of dollars per year using the taxes collected in the country.

These sorts of departments have some very complex historical data requirements.


 

Changing Systems

It is very common for government departments to be merged and split up.

Services Australia is the combination of what used to be many different government departments.

So the computer systems that these departments had would have evolved over long periods of time and been developed by different people.

They would be profoundly different for each department.

Then, as departments are merged data would be migrated from some departments to other departments computer systems, but the need for historical data is there for reasons I will mention next.

These sorts of large government departments have a far greater need for support of changing systems than your average large company.


 

Changing Legislation

Governments love to change legislation with respect to welfare payments.

This legislation can change quite dramatically with a change of government, from left of politics, to right of politics.

Australia is also a two party country for all intents and purposes.

There is a right leaning party and a left leaning party.

When the left leaning party comes in to power they spend money like a drunken sailor, and these alterations to the legislation governing payments have to be changed, and reflected, in the data warehouse.

When the right learning party comes in to power they try and reduce the over-spending, and this requires another large batch of legislation changes have to be applied to systems, including the data warehouse.

However, both parties need to version the legislation because during the election season they need to have the department staff run calculations, and forecasts, to present proposed legislation to the electorate in an effort to win votes.

So not only does the data warehouse need to support the changing of legislation.

It needs to support both past and future proposed legislation changes, along with being able to calculate the effect of these changes on groups as small as approximately 200 households.


 

Combining Complex Data With Other Complex Data

Not only does a data warehouse for such a department need to integrate all the data it has, along with versions both past, present and proposed future legislation, along with calculation scenarios for different proposed future legislation.

It also needs to integrate data from other departments and be able to integrate that data with all these similar complexities.

For example, Services Australia needs to be able to integrate data from the Australian Tax Office and their rules for taxation and their taxation records.

After all, there is not much point promising people more money in welfare payments with the left hand, if the government is only going to take that money back in increased taxes with the right hand.

Though, to be honest, that is pretty much what the Australian government does.


 

When you have such a complex environment that is so rapidly changing and so many versions of everything need to be kept?

That is a tough job for a dimensional model.

The way we archive data in a dimensional model is not well suited to this level of change and volatility.

This very complex, and relatively rare, set of requirements is far better suited to a modeling style that is meant to “archive data”.

The data also needs to be archived without much concern to how it will be queried or the costs of the overheads required to maintain different modeling styles.

In short, such a situation is better suited to Data Vault than it is suited to dimensional models.

Just like a telco or retailer or bank is better suited to using a dimensional model than a Data Vault model for the archive portion of the data warehouse.

Now sure.

Given this above example, some people will argue that a Data Vault Model offers some benefits over archiving data in a dimensional model.

While this is true, those benefits come at a cost.

That extra cost has to be justified.

In the areas of high transaction volume consumer businesses?

That extra cost is tough to justify.

In the area of complex government departments paying out billions of dollars in welfare payments?

The extra cost is relatively simple to justify.

Indeed, those people who build dimensional models won’t go near such government departments because they have been there before and they don’t want to go there again. LOL!


 

In Summary

In this blog post I just wanted to do a short post to point out that when it comes to data modeling we must somewhat observe that there are horses for courses.

There are a class of problems that are well suited to dimensional models including archiving in a dimensional model.

There are a class of problems that are well suited to archival models.

And there are a class of problems that are in the area overlapping these two modeling types.

It is the job of the Data Warehouse Architect to decide which modeling technique should be selected.

So, ladies and gentlemen, when you are proposing data warehousing modeling styles?

Please choose carefully.

Your choice will be reviewed down the track, and you will want to be able to defend your choice.

Blindly implementing just one or the other modeling style because that’s what you know?

Such an approach will not stand the test of time.

So consider your self warned, that your choices will be reviewed in the future.

This is inevitable.

Ok?

Thank you very much for your time and attention.

We really appreciate you dropping by to read our blogs.

Best Regards.

Mihai Neacsu.

Business Development Manager.

The BIDA Team.

Ask Us Anything

You are welcome to ask us any question you would like below on our contact form.

We will be emailing our subscribers useful and
valuable information regarding business intelligence.
Please subscribe here to be on our general emailing list.

LEAVE A REPLY

Please enter your comment!
Please enter your name here