A Two-stage Iterative Approach to Improve Crowdsourcing-Based Relevance Assessment

作者：Wang, Yongzhen^*; Lin, Yan; Gao, Zheng; Chen, Yan

来源：ARABIAN JOURNAL FOR SCIENCE AND ENGINEERING, 2019, 44(4): 3155-3172.

DOI：10.1007/s13369-018-3456-z

• Crowdsourcing
• Relevance assessment
• Ensemble classifier
• Expectation maximization
• Expert guidance

Crowdsourcing has emerged as a viable platform to implement the relevance assessment of information retrieval. However, since crowdsourcing touches independent and anonymous workers, the quality control on assessment results has become a hotspot within academia and industry. For this problem, we propose a two-stage iterative approach by integrating the ensemble classifier and expert guidance. Specifically, in first stage an ensemble classifier is employed to select unreliable assessment objects for experts to validate. Then in second stage, the expectation maximization is utilized to update all assessment results in terms of the validation feedback. This loop continues until the cost limit is reached. Simulation experiment demonstrates that compared with existing solutions, our approach can eliminate more noise and thereby achieve a higher accuracy, while maintaining an acceptable running time and a low labor cost.

• 出版日期2019-4
• 单位大连海事大学