Social Media Big Data Integration: A New Approach Based on Calibration

Abstract

In recent years, the growing availability of huge amounts of information, generated in every sector at high speed and in a wide variety of forms and formats, is unprecedented. The ability to harness big data is an opportunity to obtain more accurate analyses and to improve decision-making in industry, government and many other organizations. However, handling big data may be challenging and proper data integration is a key dimension in achieving high information quality. In this paper, we propose a novel approach to data integration that calibrates online generated big data with customer survey data. A common issue of customer surveys is that responses are often overly positive, making it difficult to identify areas of weaknesses in organizations. On the other hand, online reviews are often overly negative, hampering an accurate evaluation of areas of excellence. The proposed methodology calibrates the levels of unbalanced responses in different data sources via resampling and performs data integration using Bayesian Networks to propagate the new re-balanced information. In this paper we will show how, with a real data example, the novel data integration approach allows businesses and organization to correctly appraise the level of satisfaction of their customers. The real application is based on the integration of online data of review blogs and customer satisfaction surveys from the San Fransisco airport. We will illustrate that this integration enhances the information quality of the data analytic work in four of its dimensions, namely, Data Structure, Data Integration, Temporal Relevance and Chronology of Data and Goal.

Keywords: Bayesian Networks, Big Data, CCalibration, Data Integration, Social Media, Information Quality, Resampling Techniques