Picture the scene: You arrive at work bright and early, switch on your PC and settle down to some important work. You’re confident that the data you need will be ready and waiting for you. I have a question for you: Do you know how that data gets there and where it comes from? The Solvency II Directive has now made the question of where and how very important to insurers and reinsurers across Europe.
Over the years I have come across numerous situations that run something along these lines:
Team A: Our data is loaded up by IT.
IT: No, we don’t touch that data. It’s a manual data load by Team B.
Team B: We just send the spreadsheet to Team A. We’re sure that they load the data.
Team A: No, we really don’t load up that data.
A friend and colleague of mine, Justin York, has coined the phrase “data fairies” for precisely such situations, and it seems to describe the situation accurately. For many people, this is the truth of the situation and whilst they may not believe in the data fairies, they do not know how they get their data or even sometimes where it comes from. Sadly, often, they do not care.
I was lucky enough to hear Peter Aiken speak last year, and he summed up the situation perfectly with this analogy.
“Most people consider data in the same way that they think of air, in that they don’t consider it all. [They] just assume that it will always be there and will always be good enough to use. It’s only when something goes wrong that we consider the quality of the data that we are using.”
And I would add to that— even then, do we think about where the data comes from?
All too often, I find that the answer to this question is no. Having been forced to consider the quality of the data, since it is not good enough to use, the usual reaction is to go for a tactical fix of the data in front of you. After all, if you don’t know where the data comes from or how it got there, how can you consider a more permanent, strategic solution? Unfortunately this attitude starts a cycle of tactical data cleanses or fixes that soon become part of the usual way of working.
I think that anyone who has the opportunity to step back and consider this would agree that this situation makes no sense at all, but if you don’t know where your data comes from, how do you go about finding the source of a problem and fixing it?
This is why data lineage (knowing where the data comes from and what happens to it on its journey to you) is so important. Data lineage allows you to easily trace the problem back to its origin so that a more strategic solution can be considered, if it is appropriate and cost effective to do so.
For insurers and reinsurers, knowing and documenting the path that your data travels has become vital. The Solvency II directive requires you to proactively assess the sufficiency and quality of data used in both your technical provisions and internal model calculations. Some companies may already collect and hold this information and may,simply have to change their current approach. Unfortunately, others will have to start from scratch. Download this free white paper, Data management and Solvency II, for more detailed information.
Documenting data lineage for an entire company – from scratch – is not a small task. So for those organizations that are not under the Solvency II gun, it may be more difficult to find support for the wholesale project of investigating and documenting data lineage. In those cases, I recommend an iterative approach. Slowly build up a repository of data lineage information. Use what you find out when investigating data quality issues and document for reuse in the future. Adding to such a repository over time will get your data lineage started. And, using it to more easily resolve issues will help you build a business case for documenting your company’s most critical data items in the future.
So upon reflection, and in keeping with the true traditions of a good fairy tale, it occurs to me that my story should also have a sub-title. Therefore, I do hope you have enjoyed reading “Do you believe in the Data Fairies (or The importance of Data Lineage).
Nicola Askham writes a regular blog as the Data Governance Coach. Subscribe to ensure you never miss a post.