Resources matter: machine learning-based optimisation for distributed systems from cloud data centres to mobile devices
Cloud resources are the key to fueling different services and large-scale systems we can see everywhere, from electronic banking to social media platforms which have become an essential part of peoples’ everyday lives. Hence, resource management mechanisms play a crucial role in providing an acceptable level of performance and end users’ satisfaction. More importantly, it impacts the economic aspect, necessitating modern techniques and approaches to achieve cost efficiency while relying on available resources (e.g., on-premise) to minimize dependency while improving utilization and overall performance.
In addition to using cloud resources for supporting information and communications technology (ICT) services, such services and the software development’s world have also significantly changed due to the emergence of blockchain. Blockchain and its use have become the hottest trends in many other industries as a dominant paradigm to enforce transparency and security while reducing the risk of any technical transaction due to its distributed nature and scalability across different internet nodes. Moreover, the fast-paced trend of new technologies and hardware improvements for smartphones have provided a paradigm for various on-the-go services to people. It has made an ideal target to bring more computation-based jobs to smartphones, reducing the reliance on off-premises resources, particularly clouds. It can provide a more independent and customizable way to be adapted to different situations or needs, such as malware detection, to shift the dependence on central repositories (e.g., antivirus software) to smartphones. This will result in nearly real-time detection and more independent and self-aware protection solutions.
This thesis details research studies, including algorithms and frameworks, developed and presented for three important topics such as (1) cost-efficient scheduling in a multi-cloud environment, (2) blockchain-based application programming interface (API) for providing traceability and verification, and (3) the use of smartphones for independent but collaborative malware detection. This thesis targets five research areas through a series of scientific publications in 9 chapters (1) scheduling of resource-intensive scientific workflows that consist of many interdependent tasks dictated by data dependencies discussed in Chapter 2, (2) resource allocation of trace workloads in multi-clouds when the utility cost and the privacy are taken into account presented in Chapter 3, (3) the usage of machine learning techniques for assisting the scheduling in a multi-cloud environment by profiling jobs and resources explained in Chapters 4 and 5, (4) the use of blockchain for enforcing traceability and verification in an agricultural sector studied in Chapters 6 and 7, and (5) the use of deep neural networks on the resource-constrained smartphones for building robust software for improving the malware detection investigated in Chapters 8, 9, and 10.
The research outcomes, especially using machine learning techniques, have shown their significance in different research areas, such as scheduling for a coarse-grained resource allocation that leads to cost efficiency in multi-clouds. Scheduling decisions are intended for more than just workload surges that may happen in the private cloud due to limited resources. However, they consider workload, resource characteristics, and different cost-driven factors, such as varying electricity rates in private clouds and billing cycles in public clouds. The use of blockchain oracle can be fitted for different use cases and domains that are not only sensitive (i.e., agriculture and food) but also require the ability of traceability and verification of products. Finally, deep learning and different machine-learning techniques can facilitate the development of applications to protect and preserve privacy, leading toward more independent malware detection software.
