Assignment strategies for ground truths in the crowdsourcing of labeling tasks

作者：Kubota Takuya; Aritsugi Masayoshi^*

来源：Journal of Systems and Software, 2017, 126: 113-126.

DOI：10.1016/j.jss.2016.06.061

• Expectation maximization algorithm
• Human computation
• Condorcet jury theorem
• Confidence interval

It is expected that ground truths can result in many good labels in the crowdsourcing of labeling tasks. However, the use of ground truths has so far not been adequately addressed. In this paper, we develop algorithms that determine the number of ground truths that are necessary. We determine this number by iteratively calculating the expected quality of labels for tasks with various sets of ground truths, and then comparing the quality with the limit of the estimated label quality expected to be obtained by crowd sourcing. We assume that each worker has a different unknown labeling ability and performs a different number of tasks. Under this assumption, we develop assignment strategies for ground truths based on the estimated confidence intervals of the workers. Our algorithms can utilize different approaches based on the expectation maximization to estimate good-quality consensus labels. An experimental evaluation demonstrates that our algorithms work well in various situations.

• 出版日期2017-4