Ambiguity creates uncertainty in practically every facet of the web. Machine learning, recommender systems, web search, as well as news and systems that support human discourse, experience aspects of ambiguity and subjectivity that leads to inaccuracies, poor and undesirable performance. In crowdsourcing, human computation, and human-in-the-loop systems, topics central to TheWebConf, recent research has attempted to delve deeper into subjectivity and ambiguity, finding it to manifest in human-collected data primarily as disagreement [Cheatham and Hitzler, 2014; Plank, Hovy and Sogaard, 2014; Bayerl and Paul, 2011; Aroyo and Welty, 2015; Schaekermann, Law, Williams and Callaghan, 2016; Chang, Amershi and Kamar, 2017; Lin and Weld, 2014; etc.]. This includes the information presented to workers as part of a crowdsourcing task, the instructions for what to do with it, and the information they are asked to provide. These ambiguities become deeply tied into our machine learning models and metrics, as they are in the gold data. Similar ambiguity is found in interpreting and deriving utility from user generated data from large scale systems such as social media and search engines. Another aspect of disagreement surfaces in collaborative projects such as Wikipedia, online forums, and semantic markups. In language, ambiguity can result from missing details, contradictions and subjectivity. Subjectivity may stem from differences in cultural context, life experiences, or individual perception of hard-to-quantify properties. All of these can leave people with conflicting interpretations, leading to results that system builders would regard as “wrong”. These issues share common ground with many areas of interest to TheWebConf.

SAD2019 (Subjectivity, Ambiguity and Disagreement in Crowdsourcing) Workshop aims to bring together a latent community of researchers who treat disagreement (and subjectivity and ambiguity) as signal, rather than noise. Such researchers use theoretical and empirical methodology to characterize, utilize, mitigate and derive value from subjectivity, ambiguity and disagreement. The workshop will include invited talks, short technical talks and a discussion of medium- and long-term challenges to fuel future work.

We encourage interdisciplinary submissions from the broad spectrum of crowdsourcing and human computation research in fields and application areas, such as computer science, information sciences, law, medical data analysis, communication science and political science, as well as those primarily working on human computation and crowdsourcing. Solutions to these challenging problems will benefit from a diverse set of perspectives. Topics of interest (but not limited to) are:

  • Interaction/relation between disagreement, ambiguity and subjectivity
  • Costs and challenges introduced by ambiguity
  • Designing tasks with high subjectivity and low inter-rater reliability (e.g., semantic, linguistic, common sense, moral judgements.)
  • Better metrics for characterizing disagreement (over traditional inter-rater reliability)
  • Ambiguity in human computation task design, how to identify it and what to do about it
  • Theoretical ambiguity-aware frameworks for collecting data
  • Teasing apart different sources of ambiguity
  • Best practices for collecting subjective data
  • Benchmarks and datasets for studying ambiguity
  • Grand challenges that will further our understanding of subjectivity, ambiguity and disagreement
  • Disagreement, ambiguity, subjectivity and their representation and interaction in different content modalities, e.g. text, images, videos, audio
  • Interdisciplinary perspectives on disagreement, ambiguity and subjectivity, e.g., law, political science, humanities