Setting the Stage

Imagine that you are an average ordinary Qlik Developer. You start to create a new application for some healthcare big cheese so you find the virtual data warehouse of QVD files and all goes well. The data loads just fine and just for giggles and grins you utilize the Data Preview in the Load Script to see the values and panic hits you. What on earth is wrong with the Age field? That Qlik Dork guy is falling apart and could be 99 for all you know. But Dan and Levi? C’mon they can’t possibly be that old. Suddenly you are reminded of the importance of metadata. If you knew how that field was defined, it might help you understand those crazy values. Right?

Anomalies in data showcase the importance of metadata

Comments = Metadata

Since I first started coding I had a love/hate relationship with comments. You might know the feeling. You want to type your code as fast as you can while the algorithm is fresh in your head. It will be quick … you don’t need comments in your script.

After sleeping for a few weeks/months/years … you have to go back in and edit that comment-less code. You smack yourself in the head because without comments, your well thought out algorithm and flow now makes no sense. Why did you have an IF statement for a code value of 930XQ? After spending 5 days of trial and error, the spider webs begin to clear out of your head. You are able to update the code and this time you do what your first coding instructor told you that you should always do … you add comments to the code. Now you, and any other developer who looks at your fine work can fully understand it.

What does any of that have to do with the work you do in extracting and transforming data and storing those data tables into QVDs for other developers?

Everything. Without metadata (comments from you describing the data) those QVDs in a Qlik Virtual Data warehouse environment, the data is like a bunch of spaghetti code with no comments. When an anomaly arises, weeks of trying to clear the spider webs.

Creating Metadata with Comments

It was so important to the developers they literally called the function to add “metadata” about the data “Comment.” You are able to add the much needed secret decoder metadata with table comments and field comments. They even provided a way for you to add metadata tags to your fields.

Metadata is added to tables and fields simply through the use of the Comment or Tag functions

I know what you are thinking “Qlik Dork you would still have the same problem loading QVD files because you won’t see the code that extracts and transforms and stores the QVD files into the Qlik Virtual Data warehouse.”

Metadata Travels

The beauty is that all of the metadata comments or tags you add … actually become part of the data model and the QVD files themselves. Because Qlik absolutely understands the importance of metadata in helping everyone trust the data.

Data Model Viewer of the creating or utilizing application reveals the Metadata

While you are 100% correct … I will never see those comments in the application you built, me and any developer who has to maintain your application can quickly understand why the Age values are so high. You were using artificial intelligence to gain an understanding of the cognitive abilities. No doubt in any of our minds that cognitively Dan Wizard Pilla and Levi TheMan Turner are clearly working like someone with the wisdom gained from 999 years.

That few minutes of creating comments/metadata just saved both of us weeks of panic and work. I sure hope you added metadata for the Encounters table I loaded as well. Because understanding the Arrival and Discharge times are NOT the values when someone enters data into the Electronic Health Record will be huge, especially if the “know it all” down the hall presents screenshots of the electronic health record system showing your times don’t match. “Oh good you did. You must have had that issue in the past.”

Code revealing how to add metadata to the Encounters table

Why isn’t PatientID commented for the Encounters table?

I’m just asking for a friend. I applaud you being so vigilant in your work, but the PatientID field is in the Encounters table, but you didn’t add a comment for it. I will never see your code anyway, but they might and they might be curious. Or if you realize I’m teasing you and I wrote the above code and you are seeing it, you might be curious.

The answer is the same as for the question about “what makes Qlik so unique and so powerful?” The Associative Engine of course.

Fields don’t exist as part of “tables.” They just exist in their own symbol tables, and bit pointers map the associations across everything.

Luckily, there is an alternative method of adding all this wonderful metadata and that is through a variation of the “Comment” function that lets you create mapping tables and simply “map” all of the comments/tags at 1 time. Faster to type without all of the syntax characters like line by line, and I prefer this method because it won’t confuse anyone either. That’s kind of cool. Right?

Code that reveals that rather than adding metadata 1 table/field/tag at a time you can create mapping tables and add all the metadata from them.

Maybe this easier method will help you lean a bit more towards love, than hate, when it comes to adding the needed metadata.

Summary of The Importance of Metadata

The goal of this post was to help you realize how important it is for you, and to encourage you “the data modeler:”

• To really take to heart the importance of metadata for yourself and other developers
• To understand how that metadata makes data so much clearer for everyone involved, especially in a decentralized development world
• To take the few minutes needed to input key metadata with: the future you, anyone that later maintains the code, or tries to utilize the QVD files that you have contributed to the organizations Qlik Virtual Data warehouse library will thank you.

If you have already been doing this I’d love to see your comments about how valuable it is in your organization. 😎

If you didn’t know this could be done and are open to sharing I’d love to read your comments about the times you sure wish you had. 😉

Bonus Concept

While I won’t go into a deeper dive I did want to bring up a bonus concept. One that just hit me when looked at the image of the comment/tag maps. Purely by accident of course. Not pre-planned in anyway of course.  😉

“What if that metadata was read from a catalog system or database instead of being hand typed?” Interesting huh?

Metadata: The Secret Decoder Ring of Data

Quiz

Unlike other recent posts … you are off the hook and there is no quiz. You either understand the importance of metadata after reading this, or it is me that has failed. 😭