2018 Submissions

We invite and encourage interdisciplinary submissions from the broad spectrum of crowdsourcing and human computation research in fields and application areas dealing with (but not limited to) the topics below:

  • Interaction/relation between disagreement, ambiguity and subjectivity
  • Costs and challenges introduced by ambiguity
  • Designing tasks with high subjectivity and low inter-rater reliability (e.g., semantic, linguistic, commonsense, moral judgments.)
  • Better metrics for characterizing disagreement (over traditional inter-rater reliability)
  • Ambiguity in human computation task design, how to identify it and what to do about it
  • Theoretical ambiguity-aware frameworks for collecting data
  • Teasing apart different sources of ambiguity
  • Best practices for collecting subjective data
  • Benchmarks and datasets for studying ambiguity
  • Grand challenges that will further our understanding of subjectivity, ambiguity and disagreement
  • Disagreement, ambiguity, subjectivity and their representation and interaction in different content modalities, e.g. text, images, videos, audio
  • Interdisciplinary perspectives on disagreement, ambiguity and subjectivity, e.g., law, political science, humanities

Submissions may present ideas, results from experimental studies, methodologies, work-in-progress and/or applications of systems that explore the topics of subjectivity, ambiguity and disagreement  in crowdsourcing and human computation in relation to computational systems, applications, or services. All submissions should follow the main conference formatting guidelines as follows:

  • Paper Length. Long papers of up to 8 pages, short papers & demos of up to 4 pages may be submitted. References can extend unlimitedly.
  • Formatting. Submissions must be formatted in AAAI two-column, camera-ready style. (See the AAAI 2018 Author Kit). Papers must be in trouble-free, high-resolution PDF format, formatted for US Letter (8.5″ x 11″) paper, using Type 1 or TrueType fonts.
  • Supplemental Materials. Authors are invited to provide supplemental materials such as data, code, executables, videos, etc.
  • Non-archival. Accepted full papers will be published on the SAD2018 workshop website. However, submissions are not intended to be considered archival, or to preclude submission of the reported work to archival journals.
  • Encore papers. We also invite submissions of “encore papers” – i.e. relevant work that has previously been published, although not at HCOMP 2018. These will be presented at the conference in the same manner, but they will not be posted on the workshop web site (we will instead link to the original source).
  • Presentation. Each accepted long paper will be presented in a 5-7 minutes presentation (excluding questions) at the workshop.
  • Discussion. All paper presentations in the same session will be followed by a 30 min poster session and then by a 30 min joint discussion and question & answer session.
  • Poster. Authors of all accepted papers will be asked to submit a poster summarizing their submission.
  • Submission. Papers should be submitted via EasyChair.
  • At least one author of each accepted paper must register for the conference to present the work or acceptance will be withdrawn.