Important Dates

What are participating organisations in the RCIR task expected to do?
Participating researchers in the task will develop and benchmark approaches to integrate multi-modal signals (e.g. eye tracking, screenshots, etc) into the retrieval process for two sub-tasks: a) a comprehension-evaluation task (CET) that aims to sort texts in terms of comprehension levels, and b) a comprehension-based retrieval task (CRT) that aims to rank texts (for a variety of topics) by integrating text comprehension-evidence into the IR process. Both of these sub-tasks are exploratory in nature, but are designed to facilitate initial experimentation on the topic by the community. There is also an Insights task (IT) that aims to explore alternative uses/analyses/approaches with the dataset.

How will my results be benchmarked against other participanting organisations?
Participating organisations will be evaluated in the Comprehension-evaluation Task (CET) task using MSE (Mean Squared Error), and in the Comprehension-based retrieval task (CRT) using NDCG (Normalized Discounted Cumulative Gain). There is no evaluation metric used for the Insights task.

Experiment Description
The RCIR task dataset consists of multi-modal sensor data captured from 10 experimental participants who each read 96 distinct pieces of text during an experimental session covering a variety of topics. Each experimental participant has had their eye movements recorded as they read each piece of text, and following reading each piece of text were required to answer 3 multiple choice questions in order to measure their comprehension of the text. Before reading each piece of text, experimental participants were given an instruction to induce different types of reading behaviour (i.e. reading conditions).

These are:

  • Sequential Reading: Read the text at your own pace in order to be able to answer questions on it
  • Skimming: “Quickly skim the text in order to be able to answer questions on it”
  • Scanning: “Scan the text looking for X” (X will reflect an information need)
  • Proofreading: “Proofread the text and count the number of grammatical, syntactical and spelling errors you can find”
A number of additional data sources have been captured while participants are reading text that include: facial expression measures via a webcam, EOG (Electrooculography), screenshots, mouse/keyboard interactions and more.

What data is available?
Each experimental participant read 96 pieces of text and there are 24 texts per reading condition type i.e. sequential, skiming, scanning and proofreading. Each text used belongs to one of 12 frequently-occorring topics present in the RACE dataset e.g. university/education, trains, nature and animals, music, art, energy and climate change, sleep, stress and mental health, etc. There are an equal topics for each experimental participant across each reading behaviour condition i.e. they are balanced. Each of the 960 texts is unique.

For each text for each experimental participant, the following data is available:
  • Eye-movement measurements captured using an eye tracker system while the participant was reading the texts. Both the raw signal data and pre-extracted features from this are available.
  • Identifying information for the texts and the associated questions used to measure comprehension.
  • The calculated comprehension score for all texts and the participant responses to each multiple choice question.
  • Pre-extracted NLP features.
  • And a variety of other useful and related features to support task participation.
50% of the dataset will be provided as a training set (480 texts with associated metadata and comprehension scores), and 50% of the dataset will be kept aside as a testing dataset.

Note: The training dataset will contain 6 distinct topics from the 6 topics in the testing dataset.

Are these raw signals or have suitable representative features already been extracted?

In order to ease participation, the data for each signal source for each activity has been processed to extract features e.g. co-registered eye-movements and fixation.

How do I make submissions?

Details are contained in the participation pack regarding how organisations should submit their relevance judgements