Skip to main content

What lies beneath – Exposing the data behind the data

Martijn Groot

Market and reference data provide contextual information to financial transactions: the terms and conditions of products, information on counterparty or client and execution venues. It also includes information to value positions and correctly process corporate actions such as dividends.

Keeping track of all this data is complex because of the number of terms products can have (ranging from a few to many thousands of terms), the variety of execution venues and tracking what products can be sold to whom. Regulators increasingly require more information on this data and put demands on the data sourcing and preparation processes.

However, the values of the specific data points are just the result of data collection, sourcing and verification processes. It is the visible and directly used part of the information. There is much additional information that documents how these values were arrived at and that govern their usage. This metadata is often hard to access but is critical in judging whether data is fit for purpose. It includes:

  • Provenance or lineage: what sources were used?
  • Audit information: who has changed it and when?
  • Permissions: are there content license restrictions, legal restrictions or other data sensitivity considerations?
  • Quality assurance: what checks have the data gone through? When was the information last updated? Is it accurate and complete?
  • Derivation logic: if an algorithm or financial model produced the data, which model was used?
  • Governance: who can change the source of the data or business rules operating on the data?
  • Usage: where is the data going? Are all users or applications requiring this data using this source? Can it be easily accessed?

Tracking the metadata is increasingly critical due to regulatory scrutiny on processes: firms need to explain their data points. Also, monitoring data usage is required to optimize data sourcing: identifying underused data sets or data acquired multiple times can improve operations.

Market and reference data represent a significant spend of financial services firms. Keeping good tracking of the metadata helps firms make the most of that spend. It is an essential part of overall data management: efficiently sourcing and mobilizing the data resources required by the business.

Next time you see a piece of data, ask what lies beneath and how easy it is to expose that. There is often more than meets the eye.