Notes & discussion document
9:00 – 10:00 Keynote 1: Drazen Prelec (MIT): Bayesian truth serum and the ‘wine tasting problem.’ (slides)
The term “Bayesian truth serum” refers to an information-theoretic scoring algorithm that rewards respondents for honest reporting of private information, using the reports of other people as the only input (individual honesty is non-verifiable). The algorithm can also function as an objective truth-detector, identifying which answer to a multiple-choice question is most likely to be true. I will start by reviewing some of the theoretical and experimental results obtained with this approach, and then turn to the problem of incentivizing respondents to create their own questions. This will be illustrated by the ‘wine tasting problem.’ An unstructured wine tasting may generate a great deal of tantalizing discussion, sometimes ending in consensus and sometimes not. It is hard to know if disagreements are caused by an ambiguous vocabulary or genuine differences in perceptual experience. I will discuss how scoring systems, related to the Bayesian truth serum, can train respondents to stabilize their phenomenological vocabulary.
10:00 – 10:30 Coffee break
10:30 – 12:00 Lightning Talks + Posters + Discussion I
- Using Online Games to Collect Data about Disagreements.
- Crowdsourced Measure of News Articles Bias: Assessing Contributors’ Reliability
- Metrics for Capturing Ambiguity in Crowdsourcing by Interlinking Workers, Annotations and Input Data
- Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation
- A Case for a Range of Acceptable Annotations
12:00 – 13:30 Lunch
13:30 – 14:30 Keynote 2: Brent Hecht (Northwestern University): Disagreement in Crowdsourcing due to Cultural Context. (slides)
The people who do crowdwork online exist in a large variety of cultural contexts. In this talk, focusing on our research looking at Wikipedia, I will show how these diverse cultural contexts often result in significant disagreements between members of the crowd. I will then discuss how our research into this phenomenon helped to establish what is now known as “algorithmic bias”. Finally, I will highlight our work showing that these disagreements can inform a new class of applications powered by “algorithmic diversity”.
14:30 – 15:30 Lightning Talks + Posters II
- Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis
- Crowd disagreement for medical images is informative
- CaptureBias: Using Ambiguity to Support Media Scientists in News Videos Bias Detection
- Bounding Ambiguity: Experiences with an Image Annotation System
15:30 – 16:00 Coffee break
16:00 – 16:30 Discussion II
16:30 – 17:00 Wrapup