A Reliable Workload Management Based on Predictive Analysis and Characterization of Workload Resources in HPC

Abstract

Execution of Big Data workloads upon High Performance Computing (HPC) infrastructures has become an attractive way to improve their performances. In resource management, a large volume of multi-structured log data of Cloud Data Center (CDC) is generated regarding job arrival patterns, CPU memory consumption, task duration and many others. The system also provides the mechanism for CACO Cauchy matrix method for automatic data recovery from disk failure; it can also remove the lengthy process like replica management. In this paper we proposed a system for dynamic load rebalancing and resource allocation technique using machine learning algorithm. Q-Learning based on ML algorithm has used for validating the system. Experimental analysis illustrates that how proposed system eliminates the present approaches drawbacks. Our key is to identify the various workload patterns generation in heterogeneous storage environments using machine learning algorithm.

Keywords: cost management, data center, dynamic job ordering, energy, load balancing and rebalancing

JEL Classification: Y60