Login | Register

The Analysis of Parallelism of Apache Storm

Title:

The Analysis of Parallelism of Apache Storm

Zhang, Xinyao (2016) The Analysis of Parallelism of Apache Storm. Masters thesis, Concordia University.

[img]
Preview
Text (application/pdf)
Zhang_MASc_F2016.pdf - Accepted Version
1MB

Abstract

Big data processing is rapidly growing in recent years due to the immediate demanding of many applications. This growth compels industries to leverage scheduling in order to optimally allocate the resources to the big data streams which require data-driven big data analysis. Moreover, optimal scheduling of big data stream process should guarantee the QoS requirements of computing tasks. Execution time of tasks within the streams is specified as one of the most significant QoS factors.
In this paper, I will introduce the currently widely used stream processing framework Storm, a distributed real-time computation platform, and study the scheduling and execution strategies of big data stream processes within it. First, a queueing theory approach to the modeling of the streams as a collection of sequential and parallel tasks is proposed. It is assumed that heterogeneous threads are required to handle various big data tasks such as processing, storing and searching which may have quite general service time distributions. Then, with the proposed model, an optimization problem is defined to minimize the total number of resources required to serve the big data streams while guarantying the QoS requirements of their tasks. An algorithm is also proposed to mitigate the complexity order of the optimization problem. The objective of this research is to minimize the stream processing resources in terms of threads with constraints over the task waiting time of the application tasks. I apply the proposed scheduling algorithm to Apache Storm to optimize the cloud resource requirements. The experiment results validate the analysis.

Divisions:Concordia University > Gina Cody School of Engineering and Computer Science > Electrical and Computer Engineering
Item Type:Thesis (Masters)
Authors:Zhang, Xinyao
Institution:Concordia University
Degree Name:M.A. Sc.
Program:Electrical and Computer Engineering
Date:30 August 2016
Thesis Supervisor(s):Qiu, Dongyu
ID Code:981565
Deposited By: XINYAO ZHANG
Deposited On:08 Nov 2016 15:15
Last Modified:18 Jan 2018 17:53
All items in Spectrum are protected by copyright, with all rights reserved. The use of items is governed by Spectrum's terms of access.

Repository Staff Only: item control page

Downloads per month over past year

Back to top Back to top