Rudra Pratap Deb Nath

Summary

SETL

SETL_CONSTRUCT

SETL_BI

SETL_AUTO

SETL_ONDEMAND

SETL_SPATIAL

SETL_OLAP (On going project)

Summary

In order to create better decisions for business analytics, organizations increasingly use external structured, semi-structured, and unstructured data in addition to the (mostly structured) internal data. Current Extract-Transform-Load (ETL) tools are not suitable for this “open world scenario” because they do not consider semantic issues in the integration processing. Current ETL tools neither support processing semantic data nor create a semantic Data Warehouse (DW), a repository of semantically integrated data. Thus, a DW with semantic data sources in addition to traditional data sources requires more powerful techniques to define, integrate, transform, update, and load data semantically, which are the research and technical solutions provided by SETL: A unified framework for semantic Extract-Transform-Load.

Our hypothesis is “In the context of highly heterogeneous data integration processes, considering semantics as first-class citizen facilitates the integration process by providing automation and lower entry barriers for non-technical users."

The mapping between the objectives of this research and provided solutions are illustrated in Figure 1:

Figure 1: Objectives to Solutions

Resources:

Aspects of Semantic ETL
Rudra Pratap Deb Nath
Aalborg University Press; page: 221; October 2020

Source Code: GitHubLink

Demo: YouTube Link

SETL

This section outlines the Semantic ETL Framework (SETL in short). To design an SDW, SETL follows a demand-driven approach. The two steps of the demand-driven approach are: 1) requirements engineering: identifying and analyzing the requirements of business users and decision makers, and 2) data integration: building the target TBox and the ETL based on the gathered requirements. The first step is beyond the scope of this thesis, and SETL focuses on the second step. In short, the integration steps SETL supports are: defining a target TBox for the SDW based on the given requirements, extracting data from multiple heterogeneous data sources, transforming source data into RDF triples following the semantics encoded in the target TBox, linking data internally and externally, and loading the data into a triple store, and/or publishing the data on the Web as Linked Data.

As shown in Figure 2, SETL is divided into three layers (separated by red-colored dotted lines): the Definition Layer, ETL Layer, and Data Warehouse Layer. In the Definition Layer, an ETL designer defines the target TBox, data sources, and the mappings among the source and target TBoxes. Using the SDW TBox Definition component, the designer defines the target TBox based on the requirements; the QB4OLAP vocabulary is used to annotate the target TBox with MD semantics. The Define Mapping component is used to map between a source and the target TBox. To create a semantic layer on top of a non-semantic data source, the TBox Extraction component is used to extract a TBox from a non-semantic data source. The R2RML component is used to generate RDB to RDF Mapping Language (R2RML) mappings for a nonsemantic source, which is later used to generate RDF triples for the source using an R2RML engine.

In the ETL Layer, the designer can design an ETL process to create the ABox of the SDW from the available sources. The Extraction component retrieves data from the sources, which are further cleansed and formatted by the Traditional Transformation component. Then, the Semantic Transformation component transforms the data into RDF triples according to the semantics encoded in the target TBox. As a sub-task, Semantic Transformation stores the IRI information of each resource/data in the Provenance graph to preserve the uniqueness of each resource. Internal data can be linked with other external knowledge bases using the External Linking component. The RDF triples can be dumped as a local file using the SaveToFile component. Finally, the Load component loads the RDF triples, directly from the Semantic Transformation component or from the dumped file, into a triplestore that can be queried by end-users using OLAP or SPARQL queries. The intermediate results from each phase are stored in a data staging area. As ETL is an iterative block and repeated for each flow, a curved arrow is used in Figure 2. The Data Warehouse Layer stores the RDF triples.

Figure 2: SETL Architecture

Resources:

SETL: A programmable semantic extract-transform-load framework for semantic data warehouses

RPD Nath, K Hose, TB Pedersen, O Romero

Information Systems 68, 17-43, 2017; 2017

Towards a programmable semantic extract-transform-load framework for semantic data warehouses (Best Paper Award DOLAP 2015)

RP Deb Nath, K Hose, TB Pedersen

Proceedings of the ACM Eighteenth International Workshop on Data Warehousing and OLAP 2015; 2015

SETL_CONSTRUCT

SETL_CONSTRUCT proposes a strong foundation for an RDF-based semantic integration process and a set of high-level ETL constructs that allows defining, mapping, processing, and integrating semantic data.

We structure the integration process into two layers: Definition Layer and Execution Layer. Different to SETL or other ETL tools, here, we propose a new paradigm: the ETL flow transformations are characterized once and for all at the Definition Layer instead of independently within each ETL operation (in the Execution Layer). This is done by generating a mapping file that gives an overall view of the integration process. This mapping file is our primary metadata source, and it will be fed to the ETL operators, orchestrated in the ETL flow (Execution Layer), to parametrize themselves automatically. Thus, we are unifying the creation of the required metadata to automate the ETL process in the Definition layer. We propose an OWL-based Source-To-target Mapping (S2TMAP) vocabulary to express the source to target mappings.

Overall the data integration process requires the following four steps in the detailed order.

Defining the target TBox with MD semantics using QB and QB4OLAP constructs. In addition, the TBox can be enriched with RDFS/OWL classes and properties. However, we do not validate the correctness of the added semantics beyond the MD model. This step is done at the Definition Layer.
Extracting source TBoxes from the given sources. This step is done at the Definition Layer.
Creating mappings among source and target constructs to characterize ETL flows. The created mappings are stored using the S2TMAP vocabulary proposed. This step is also done at the Definition Layer.
Populating the ABox of the SDW implementing ETL flows. This step is done at the Execution Layer.

Figure 3: The overall semantic integration process

Figure 3 illustrates the whole integration process and how the constructs of each layer communicate with each other. Here, we introduce two types of constructs: tasks and operations. On the one hand, a task requires developer interactions with the interface of the system to produce an output. Intuitively, one may consider the tasks output as the required metadata to automate operations. On the other hand, from the given metadata, an operation produces an output. The Definition Layer consists of two tasks (TargetTBoxDefinition and SourceToTargetMapping) and one operation (TBoxExtraction). These two tasks respectively address the first and third steps of the integration process mentioned above, while the TBoxExtraction operation addresses the second step. This is the only operation shared by both layers (see the input of SourceToTargetMapping in Figure 3). Therefore, the Definition Layer creates three types of metadata: target TBox (created by TargetTBoxDefinition), source TBoxes (create by TBoxExtraction), and source-to-target mappings (created by SourceToTargetMapping). The Execution Layer covers the fourth step of the integration process and includes a set of operations to create data flows from the sources to the target. Figure 3 shows constructs (i.e., the Mediatory Constructs) used by the ETL task/operations to communicate between them. These mediatory constructs store the required metadata created to automate the process. In the figure, Ext_op, Trans_op, and Load_op are the set of extraction, transformation, and load operations used to create the flows at the instance level. The ETL data flows created in the Execution Layer are automatically validated by checking the compatibility of the operations. Precisely, if an operation O1’s output is accepted by O2, then we say O2 is compatible with O1 and express it as O1 ! O2. Since traditional ETL tools (e.g., PDI) do not have ETL operations supporting the creation of an SDW, we propose a set of ETL operations for each phase of the ETL to process semantic data sources. The operations are categorized based on their functionality. Table B.1 summarizes each operation with its corresponding category name, compatible successors, and its objectives.

Resources:

High-Level ETL for Semantic Data Warehouses

RPD Nath, O Romero, TB Pedersen, K HoseIn: Semantic Web Journal , vol. 13, no. 1, pp. 85-132, 2022 [Pdf]

Source Code: GitHubLink

SETL_BI

SETL_BI extends SETL_CONSTRUCT by 1) adding operations to process non-semantic data sources in the ETL Layer, 2) adding an OLAP Layer that allows users to create OLAP queries using a GUI, without requiring SPARQL knowledge, and 3) providing an ontology, which shows the compatibility among the constructs of different layers

Figure 4 shows that SETLBI is organized into three layers: the Definition Layer, ETL Layer, and OLAP Layer. The orange-colored constructs are the new ETL constructs introduced over SETL_CONSTRUCT. The OLAP Layer enables the self-service analysis over the produced MD SDW. The connections among the ETL constructs (tasks/operations) of inter- and intra-layers are shown, in Figure 4 by using an ontology, where a class represents an ETL construct (a task/operation) of a layer and an arrow represents the relationship between two constructs of layers.

Figure 4: The Ontology of SETLBI

Prototype Specification: The system is developed in Java 8. All GUIs are implemented in Standard Widget Toolkit (SWT). Jena 3.4.0 is used to process, store, and query RDF and as a triple store, Jena TDB is used. A comprehensive video explaining the functionalities of SETLBI is available in the following link

Resources:

1. SETLBI: An Integrated Platform for Semantic Business Intelligence

RP Deb Nath, K Hose, TB Pedersen, O Romero, A Bhattacharjee

Companion Proceedings of the Web Conference 2020, 167-171, 2020; 2020

Source Code: GitHubLink

SETL_AUTO

See High-level ETL paper

SETL_ONDEMAND

As the application of Semantic Web technology continues to grow, conducting Online Analytical Processing (OLAP) over Semantic Data Warehouses (SDW) has become an essential task. The Extract-Transform-Load (ETL) process loads data from external sources into an SDW. Generally, OLAP operations function efficiently on static data with minimal changes over time. However, challenges arise when dealing with frequent data changes, such as prices, populations, or social media data, requiring constant updates to the SDW for query processing. Additionally, not all dimensions of the cube may be crucial for answering a query. In this paper, we propose an approach named

SETLonDEM AND, which extracts an OLAP query to determine the required data, fetches the necessary data from respective sources using the ETL pipeline, and then executes the query. This approach additionally enables data retrieval from external data endpoints and seamlessly integrates the outcomes from these external endpoints with local data sources, particularly in cases where the query is federated. We assess the performance, productivity, and quality of SETLonDEM AND compared to a traditional ETLQ approach.

Resources:

1. SETLonDEMAND : Towards an on Demand ETL Approach for Semantic Data Warehouses

RP Deb Nath, A Bhattacharjee

2024 6th International Conference on Electrical Engineering and Information & Communication Technology (ICEEICT) , 2024

Source Code: GitHubLink

SETL_SPATIAL

On going project

SETL_OLAP (On going project)

Initiatives such as Open Data, FAIR, and Linked Data principles encourage organizations to publish datasets in a standardized, semantically enriched, and non-proprietary format using Resource Description Framework (RDF). Since most of these data contain facts and figures, they must be capable of supporting Online Analytical Processing (OLAP)-like analysis. To achieve this, QB and QB4OLAP vocabularies allow data annotation with multi-dimensional (MD) semantics, facilitating the creation of semantic data warehouses (SDWs). However, an OLAP platform that allows interactive creation of OLAP queries over these SDWs is currently lacking. This is where $SETL_{OLAP}$ comes in, as it empowers users to generate and execute OLAP queries using user-friendly GUI components, thereby relieving users from the burden of learning SPARQL query language. Finally, $SETL_{OLAP}$ undergoes experimentation with a number of users and is evaluated using a real-world SDWs. The assessment successfully establishes the feasibility of our approach, showcasing that our tool not only expedites the query generation process but also ensures accurate results in practical scenarios.

Page updated

Report abuse