Meta-Modeling of Data Sources and Ingestion Big Data Layers

Abstract

A few years ago, the notion of big was introduced as a concept, and now it becomes a concrete reality in the field of information technology. The proliferation of data types from multiple sources such as social media, mobile devices, etc. creates a great deal of diversity beyond traditional transactional data. Hence, the data are no longer in net structures, easy to consume, but they are in different types of structures, namely structured, unstructured and semi-structured data. All of these data types lie at the Big Data architecture level in the data sources layer, which is the starting point for any further processing of Big Data. Indeed, this layer has a direct relationship with the Ingestion layer, which is in charge of validating, cleaning, transforming, reducing and integrating the data in order to use it later on by the Hadoop ecosystem. At this point, we deem it necessary to say that this work is a further development of the first two comparative studies we have undertaken before on the main five Hadoop Big Data distributions. Yet, in this article, we shall apply the techniques related to the Model Driven Engineering “MDE” to present a universal Meta-modeling for both data sources and Ingestion Big Data layers.

Keywords: Meta-modeling, Big Data, Data Sources layer, Ingestion layer, Model Driven Engineering, MDE