Abstract

Amazon EC2 offers inactive virtual machines (VM) as spot instances at up to 90% discount. In return, the least expensive option requires the customers' usage to be tolerated with a low availability level agreement. Thus, many studies proposed forecasting and prediction mechanisms to assess finding the best set of maximum prices.

In this research, we study the model's efficiency in predicting spot EC2 prices by assessing the performance of forecasting algorithms: RFR, XGBoost, k-NNR, and SVR. We evaluate the models using six metrics, including MAPE, RMSE, MAE, and MSE, commonly used in related work, as well as the Regression Error Characteristics (REC) curve and the Area under the curve (AUC-REC). Our experiments consider dataset time per year, training window (1-day, 1-week, and 1-month ahead), and instance location.

The REC curve and AUC-REC are superior performance measurements for evaluating models over different accuracy-loss thresholds. Our findings suggest that the cross-validation technique is unnecessary to improve the models' accuracy, except for the SVR model. Our study highlights the limitations of using threshold-based metrics, which can be misleading, and the importance of using representative metrics such as AUC-REC to evaluate machine learning models.

Our study has limitations, including the choice of algorithms, which may impact the results. Additionally, our experiments are limited to AWS cloud services, and our results may not be generalizable to other cloud providers. In future work, we plan to evaluate other forecasting methods, including deep learning and statistical methods, and investigate the results of other regions and training windows.