Data Virtualization and the Semantic Layer

Discussion for EA874 Topic 3 > Data/Information Architecture Layer

Post # 3

Ultimately, there are 2 key questions we ask as data architects in service to our business customers:

1. How can business gain access to the data to gain insights?
2. How do we provide consistency and quality -- while maintaining agility?

You can easily imagine the challenges of traversing traditional data warehouse solutions as depicted in the diagram below, in order to bring data to our internal enterprise customers.

source:
http://www.slideshare.net/Denodo/data-warehousesmdmdatavirtualizationdenodopackedlunchseriessession5
Implementing Data Virtualization for Data Warehouses and Master Data Management extensions
By Denodo Technologies, published on Jan 22, 2015

As Chris Daniels points out in his article, "inadequate reporting capabilities seem to be a common characteristic of both off-the-shelf and bespoke systems. It’s a real shame because it can prevent firms from exploiting the full potential of the information they hold." He points to several factors that result in computer systems that continue to be delivered without adequate reporting facilities, and that organisations wishing to make the most of their information will need to find ways to plug the gaps. Beyond the first-order solutions using Query-Reporting tools, he goes on to point out another solution which architects would also need to keep in mind: the provision of a Semantic Layer.

What is a Semantic Layer?
A semantic layer is a business representation of corporate data that helps end users access data using common business terms. The aim is to insulate users from the technical details of the data store and allow them to create queries in terms that are familiar and meaningful.

An increasing number of reporting tools allow users to make use of a “semantic layer.” Some vendors give the semantic layer a name that is specific to their particular product (for example, Business Objects calls it a “universe”), while others describe it as a business model or metadata layer, that should make it easier and intuitive of end-users to create their own reports by themselves.  However, it is typical that the semantic layer is still mainly used by developers -- and not business users.  Unfortunately, developers are SQL-savvy enough to perceive the semantic layer as a layer of nuisance i.e. another set of technical paradigm to deal with. As a result, it can be tempting for developers to bypass the semantic layer and revert to type and write queries by hand for each and every report.

As Chris notes in his conclusion, there are real benefits to be gained through the use of semantic layers. Semantic layers are increasingly becoming available within all types of reporting environments, and their use should be encouraged for the benefit of placing the power of harvesting information in the hands of the end-users themselves. Moreover, I think the technologies that create semantic layers prepares us to appreciate the more abstracted data access solution: Data virtualization.

What is Data virtualization?
Data virtualization is defined as an umbrella term used to describe any approach to data management that allows an application to retrieve and manipulate data without requiring technical details about the data, such as how it is formatted or where it is physically located i.e. it is another layering concept.

Data virtualization software may include functions for development, operation, and/or management.

Benefits include:

• Reduce risk of data errors
• Reduce systems workload through not moving data around
• Increase speed of access to data on a real-time basis
• Significantly reduce development and support time
• Increase governance and reduce risk through the use of policies
• Reduce data storage required 

Drawbacks include:

• May impact Operational systems response time, particularly if under-scaled to cope with unanticipated user queries or not tuned early on
• Does not impose a heterogeneous data model, meaning the user has to interpret the data, unless combined with Data Federation and business understanding of the data
• Requires a defined Governance approach to avoid budgeting issues with the shared services
• Not suitable for recording the historic snapshots of data - data warehouse is better for this
• Change management "is a huge overhead, as any changes need to be accepted by all applications and users sharing the same virtualization kit" 

I won't be able to expand on virtualization in this post, but I think it's sufficient to say that data architects would need to be familiar with the never-ending solutions coming out that help fulfill mission for the business, and works well with the enterprise information principle of accessibility.

References:
Chris Daniels. (2010 March).  What is a Semantic Layer and Why Would I Want One? http://www.b-eye-network.co.uk/view/12763

Data Virtualization. https://en.wikipedia.org/wiki/Data_virtualization

Semantic Layer. https://en.wikipedia.org/wiki/Semantic_layer
/