Analysis of Big Data: Challenges and Fundamentals in the Computing System

Abstract

We are living in Digital universe with data prolife ring by Individuals, Institutions and Machines at an extremely high rate. This data is groups as "Big Data" due to its Volume, Velocity and Variety. Most of the data is not structured, quasi structured or semi structured and it is mixed in nature. The capacity and the heterogeneity of information with the speediness is generated, makes it challenging for the current computing structure to administer Big Data. The development of Big Data and its increasing ability to process and store a wide array of data has become more efficient with the introduction of High-Performance Computing (HPC) technologies. Its integration with Big Data workloads ensures reliability and safety of data for large corporate sectors. However, with a massive growth of wide spectrum of data that is generated through super computing, its efficient usage has become time-demanding and challenging for computational space to warrant successful analysis and processing of data. Though technologies such as Hadoop and Predictive Analysis necessitate the indispensability of Big Data, some software like Apache and YARN often disturb the analyzing process of HPC for Big data as a result of which performances of such systems gets disruptive. This paper will entail the resolution of such problems and a subsequent implementation of Big Data system in HPC technologies in a more efficient way.

Keywords: Big Data, Computing System, Hadoop, Predictive Analysis