Data Virtualisation Simply Explained

Although many companies are aware of the value of their data, they are still wondering how they should achieve this value. The data itself is managed and used in very different systems, so that a uniform data view would be needed to connect this data with each other and gain the desired valuable insights.

‍

This requirement is not new — solutions have already been developed for several decades that integrate decentralized data and make it available to specialists at a single access point. However, the way in which these solutions are implemented has fundamentally changed with the requirements of modern data environments.

‍

In this blog article, we therefore provide an overview of the differences between physical and virtual data integration and explain how decentralized data architectures such as data mesh and data fabric relate to data virtualization.

‍

The data integration

Data integration is the process in which data from different systems within an organization is unified into one and the same data view. In principle, this can take place in two ways: either physically or virtually.

‍

In physical data integration, the data is first extracted from the individual data storage systems using ETL (Extract – Transform – Load), then transformed to be transferred to the uniform structure and finally loaded into a new storage location — usually a data warehouse. This data is either completely transferred from the original storage location to the data warehouse or copied, the latter being the case much more frequently. As a result, traditional data integration in most cases results in redundant data within an organization, which in itself has very different implications.

‍

Either way, physical data integration actually requires data to be moved. This is a problem, especially for data that is generally subject to a certain degree of dynamism, as no real-time view of the integrated data can be guaranteed. A new ETL process is needed again and again to obtain an up-to-date data view, which can lead to complex and expensive data processes.

‍

With virtual data integration, only one virtual layer is placed over all data sources to be integrated and the structured and unstructured data contained therein is made accessible through this virtual layer — in real time and completely independently of where the data is stored and without replicating the data itself. Only metadata and data governance rules are replicated at this level, because, as a link to the data sources, they enable data consumers to query later – regardless of whether they are applications, processes, data scientists or business users. This simplifies data integration immensely and offers a significant advantage over physical data integration, especially in dynamic contexts.

‍

Data virtualization is a technique of virtual data integration and as such supports decentralized data architectures such as Data Fabric or Data Mesh, as described in the next section.

‍

The connection between data virtualization and decentralized data architectures

In principle, a data architecture takes care of organizing and managing data and formats and regulates the flow of data between the individual systems. Decentralized data architectures democratize access to data and thus simplify, for example, the use of analytics tools for all users, regardless of their expertise. At the same time, decentralized data architectures represent more flexible and scalable solutions for organizations with more complex data structures and larger amounts of data.

‍

Data virtualization enables these decentralized data architectures by providing real-time access to data from any source. In addition, data virtualization provides the necessary framework for using these architectures in terms of data governance, interoperability of systems and processes, and security.

‍

Data Fabric

Data fabric is a concept that combines several components. It is based on the principle of data virtualization and uses various tools, systems and processes — including ETL, data warehouses and master data management (MDM) – to implement an organization-wide decentralized data architecture. Data Fabric provides data scientists or data analysts with central access to data, which significantly facilitates their use and further processing.

‍

Data Mesh

A data mesh is a decentralized data architecture that follows a domain-based principle. Individual business units are transferred ownership of their specific data — the idea is that their data is treated as their own products and distributed to the other organizational units. The divisions are therefore responsible for providing answers regarding “their” data and ensuring the quality of this data. This is intended to avoid the “bottleneck” of the data team responsible for everyone in other scenarios.

‍

Added value through data virtualization

Especially compared to physical data integration, data virtualization offers significant benefits for business and IT:

‍

• Higher efficiency and lower costs. Instead of transferring data from the data source to a data warehouse each time using ETL, data virtualization allows a real-time view of the relevant business data. This saves time and lowers the costs of transformation processes.
• Scalability. Large amounts of data pose problems neither for virtual data integration nor for ETL processes. However, ETL processes have performance difficulties with high data speeds, which can quickly become a problem, especially in modern (big) data scenarios.
• Agility. Virtual data integration also makes it easy to exchange different data sources. Existing integration processes remain unaffected by changes in data sources, formats or integration scenarios.
• Automation. Data virtualization offers enormous potential for automating a wide variety of data processes and helps companies, for example, to implement BI and analytics measures extremely economically.

‍

Any more questions?

Even though the potential of data virtualization for various business purposes is great, the actual added value for a company is very individual. Arrange a non-binding consultation with our data experts and find out what options your data organization offers you!

‍

‍