Security techniques for virtual machine based systems

Virtualising computing environments has generated significant research and commercial interest in the past decade and half. The main factors contributing to this resurgence include, but not limited to, greater isolation, ease of deployment and low total cost of ownership that virtualisation provides while improving availability and increasingeffciency of resources. Security has become an important part of the whole virtualisation landscape due to the increased use of virtualised environments that demand diverse security requirements and various levels of trust associated with different platforms, applications and users. This thesis is essentially focusing on security attacks and security architecture for virtual machine based systems. First, the thesis considers virtual machine introspection based techniques for detecting anomalies in virtual machine based applications and services. In particular, it addresses Domain Naming System (DNS) and Web Server based applications, and analyses the different types of security attacks that are possible against these services and the security measures and architecture that can be adopted to counteract these attacks. Then, the thesis proposes a graph-based reputation model for domain name system (DNS) to further strengthen the attacks detection capabilities of the proposed architecture. A detailed analysis of the attacks against the reputation system and technique sto counteract them are described. Finally, an integrated security model combining access control with proposed attack and intrusion detection capabilities together with trust management is proposed for virtual machine based systems. The integrated security architecture can detect intrusions and attacks against the monitored services and is able to dynamically update the access controls policies of the live system to isolate the problematic/compromised service from accessing all other resources while the service is still running. This enables administrators to address the security threats without affecting the rest of the infrastructure. Such a feature helps to enhance the resilience of the system, which is vitally important in securing virtual infrastructures. The thesis describes the design of the integrated security architecture and its implementation. Then the analysis of a number of attacks against DNS and Web servers using the architecture is discussed. Dynamic nature of detecting attacks, proactively protecting specific applications from previously unknown threats and isolating virtual machines without additional resources can stand out as unique features of the security techniques proposed in this thesis.