With this workshop, we would like to bring together different ideas from a variety of perspectives on how to improve our understanding of subjectivity, ambiguity and disagreement in crowdsourcing.
Cheatham & Hitzler (2014) look at disagreement as a signal of inherent uncertainty in the domain knowledge found when assessing the Ontology Alignment Evaluation Initiative (OAEI) benchmark. Plank, Hovy & Søgaard (2014) find debatable cases in linguistic theory, rather than faulty annotation, from disagreement in their part-of-speech tagging task. Bayerl & Paul (2011) used disagreement to find ambiguity inherent in natural language as do Aroyo & Welty (2014), focusing on ambiguity at the sentence level and in the target semantics. Schaekermann et al. (2016) also saw a link between uncertainty and disagreement in crowdsourcing. Chang, Amershi & Kamar (2017), found that ambiguous cases cannot simply be resolved by better annotation guidelines or through worker quality control. Lin & Weld (2014) found that machine learning classifiers can often achieve a higher accuracy when trained with noisy crowdsourcing data that preserved the disagreement. Sharmanska et al. (2016) show how “ambiguity helps” in image classification with disagreement in crowdsourced annotations. Bhattacharya et al. (2017), collected data from a much larger number of raters (50 instead of the usual 3) and with questions posed differently to reveal increased ambiguity in semantic similarity data.
There is a substantial body of work on topics related to subjectivity, ambiguity and disagreement in fields beyond computer science such as political science, communications and linguistics. Workshop organizers have existing collaborations with researchers in these fields and are planning to invite them to join the multidisciplinary community for this workshop. Communications scientists have a long-standing tradition in research on public disputes and the role of disagreement in it. They explain disagreement in terms of diverging or opposing frames (Entman, 1993). Dardis (2008) demonstrates how framing affects disagreement by distinguishing between conflict-reinforcing frames and conflict-displacing frames. Political scientists often invoke the concepts of ideology and beliefs and the disagreement between people on them as a way to explain or predict attitudes towards particular topics. Linguists in the school of “Critical Discourse Analysis” (CDA) point to the dialectical relation between language and societal institutions: language use reflects and shapes relations of power and dominance, and it therefore plays a crucial role in reproducing disagreement. Ideology, according to this tradition, is defined as common sense, or more precisely as a “pattern of meaning or frame of interpretation […] felt to be commonsensical, and often functioning in a normative way” (Verschueren, 2012). Ideological disagreement therefore entails a contestation of these commonsensical norms and prescriptions held by specific segments of society. Structuralist linguists attempted to unearth language patterns that elicit or reduce disagreement, by scrutinizing how conflict is initiated (the linguistic or communicational devices used) and how it develops (Kakava, 2001). This boiled down to an analysis of the structure of arguments and the sequential organization of disagreement.