Abstract

Big data processing is rapidly growing in recent years due to the immediate demanding of many applications. This growth compels industries to leverage scheduling in order to optimally allocate the resources to the big data streams which require data-driven big data analysis. Moreover, optimal scheduling of big data stream process should guarantee the QoS requirements of computing tasks. Execution time of tasks within the streams is specified as one of the most significant QoS factors.
In this paper, I will introduce the currently widely used stream processing framework Storm, a distributed real-time computation platform, and study the scheduling and execution strategies of big data stream processes within it. First, a queueing theory approach to the modeling of the streams as a collection of sequential and parallel tasks is proposed. It is assumed that heterogeneous threads are required to handle various big data tasks such as processing, storing and searching which may have quite general service time distributions. Then, with the proposed model, an optimization problem is defined to minimize the total number of resources required to serve the big data streams while guarantying the QoS requirements of their tasks. An algorithm is also proposed to mitigate the complexity order of the optimization problem. The objective of this research is to minimize the stream processing resources in terms of threads with constraints over the task waiting time of the application tasks. I apply the proposed scheduling algorithm to Apache Storm to optimize the cloud resource requirements. The experiment results validate the analysis.