Deep learning-based worker quality control in crowdsourcing

Crowdsourcing is recognized as an effective way to gain knowledge or get problems solved in many areas. There are crowdsourcing platforms to engage workers for tasks in various scientific and business areas, such as Amazon Mechanical Turk, Figure Eight, Freelancer, and Upwork. The primary concern of using crowdsourcing is the output quality. Workers may provide incorrect answers as they could be non-experts, and few of them may even be spammers. Therefore, worker quality control has become a toppriority demand in crowdsourcing. To achieve effective quality control, four challenging problems, including group-aware worker quality evaluation, low-redundancy worker quality evaluation, participating behavior based worker performance prediction, and comprehensive worker performance prediction, have to be tackled.

In this dissertation, we discuss four main challenges in this domain and propose novel worker quality control approaches to address them. We first develop a groupaware worker quality evaluation model that considers the group feature and the worker feature. Then, we focus on the low-redundancy situation in crowdsourcing and propose a low-redundancy worker quality evaluation approach by learning the representations of workers. We also propose a participating behavior based worker performance prediction approach by exploiting the behavior of workers. Finally, we propose a comprehensive worker performance prediction approach by considering the features of workers, tasks, and requesters. The experimental results on real datasets demonstrate the effectiveness of our approaches compared to the existing worker quality control methods.