Promenade: A Big Data Platform for Handling City Complex Networks with Dynamic Graphs

Abstract

Continuous data streams, generated by modern sensed cities, open many opportunities and perspectives in terms of developing new innovative services. To exploit this potential, flexible and scalable platforms are needed to ease the design, development, deployment, and operations of new city services. In recent years, several problem-specific platforms have been proposed in different application domains; however, to boost the evolution of smart cities, we claim the need for city-oriented platforms that can be easily customized to address different day-to-day life challenging problems. In this paper, we present the main architectural challenges and solutions proposed for the design of a novel open-source platform (named PROMENADE) characterized by: (i) a graph-based data-driven modeling support to ensure high generality for addressing disparate problems related to the networked nature of many city infrastructures and systems, (ii) the big dynamic nature of its data/graph entities which come in a real-time fashion from different sources (e.g., IoT/Edge networks, data providers, etc.), and (iii) its high efficiency, flexibility, and scalability to easily support new city services. The platform is designed around a core that provides a set of built-in functions for graph metrics computation exposed as a collection of containerized microservices. A specialization of the platform, called PROMENADE-v2.0, has been developed for road networks monitoring. It has been deployed in Openshift/Kubernetes and tested using realistic datasets collected from the city of Lyon, France. The analysis addresses an important problem of big data processing pipelines: the synchronization between data ingestion and processing in order to produce an accurate result in useful time. To this end, we study different approaches for synchronization and show how end-to-end latency is kept under control by leveraging the scalability of the platform.

Keywords: Smart Cities, Complex dynamic networks, Big Data, Microservices architecture, Container-based microservices, DevOps