Abstract

Honeypots are traps designed to resemble easy-to-compromise computer systems in order to tempt attackers to invade them. When attackers target a honeypot, all their actions, tools and techniques are recorded and analyzed in order to help security professionals in their conflict against the attackers and the botmasters. However, botmasters might be able to detect honeypots. In particular, they can command compromised machines to perform illicit actions in which the targeted victims work as sensors that measure the machine's willingness to perform these actions. If honeypots were designed to completely ignore these commands, then they can be easily detected by botmasters. On the other hand, full participation by honeypots in such activities has its associated costs and may lead to legal liabilities. This raises the need for finding the optimal response strategy needed by honeypots in order to prolong their stay within botnets without exposing them to liability.
In this work, we show that current honeypot architectures and operation limitations may allow botmasters to uncover honeypots in their botnet. In particular, we show how botmasters can systematically collect, combine and analyze evidence about the true nature of the machines they compromise using Dempster-Shafer theory.
To determine the currently available optimal response for honeypots,
we provide a Bayesian game theoretic framework that models the interaction between honeypots and botmasters as a non-zero-sum noncooperative game with uncertainty.
However, the solution of the game shows that botmasters always have the upper hand in the conflict with honeypots since botmasters can update their belief about the true nature of the opponents and consequently act optimally based on the new belief value.
This motivated us to investigate a better strategy that enables honeypots to maximize their outcome by optimally responding to the probes of the botmasters. In particular, we provide a Markov Decision Processes model that helps security professionals to determine the optimal strategy that enables the honeypots to prolong their stay in the botnets while minimizing the cost of possible legal liability.
Throughout this thesis, we also provide different scenarios that illustrate and support our proposed analysis and solutions.