Important Dates

What is RCIR?


RCIR (Reading Comprehension in Information Retrieval)
is an NTCIR-16 workshop data challenge task. The NTCIR-16 RCIR pilot task aims to motivate the development of a first generation of personalised retrieval techniques that integrate reading comprehension measures from eye tracking as a source of evidence when ranking text content. Participating researchers in the task will develop and benchmark approaches to integrate multi-modal signals (e.g. eye tracking, screenshots, etc) into the retrieval process for two sub-tasks: a) a comprehension-evaluation task (CET) that aims to sort texts in terms of comprehension levels, and b) a comprehension-based retrieval task (CRT) that aims to rank texts (for a variety of topics) by integrating text comprehension-evidence into the IR process. Both of these sub-tasks are exploratory in nature, but are designed to facilitate initial experimentation on the topic by the community.

Although this is a collaborative evaluation where the performance of participating organisation's strategies will be ranked for the CET and CRT sub-tasks, it is expected that many good approaches that may perform sub optimally to others in terms of a particular metric (e.g. MSE, nDCG, etc) may offer other advantages in terms of speed, insight generation, interpretability and/or cross-subject applicability. Papers exploring such aspects of the data and tasks set are highly encouraged as part of the Insights task.

A new dataset will be generated by the organisers and will consist of textual data selected from the RACE data (the content texts), comprehension measures captured from experimental participants from reading the text, as well as a range of preprocessed signal data measuring the reader e.g. eye movement features. Prior to the NTCIR-16 June 2022 conference, participating organisations will be expected to submit a paper outlining their approach which they will then present at the RCIR workshop in Tokyo, Japan this in June 2022.

Important:

There will be three sub-tasks in the pilot RCIR task:

  • A) Comprehension-evaluation Task (CET): that aims to sort texts in terms of comprehension levels.
  • B) Comprehension-based retrieval task (CRT): that aims to rank texts (for a variety of topics) by integrating text comprehension-evidence into the IR process.
  • C) Insights task (IT): that aims to explore alternative uses/analyses/approaches with the dataset.
Further information on the dataset: DATA.


How to Participate?
In order to gain access to the dataset and participate in the RCIR task you should consult the NTCIR-16 page detailing registration information for task participation:
http://research.nii.ac.jp/ntcir/ntcir-16/howto.html
After registering for the task please contact graham.healy@dcu.ie to gain access to the dataset.


Who is organising this?
Graham Healy (* Lead Contact / Lead Organiser) - Dublin City University, Ireland
Cathal Gurrin - Dublin City University, Ireland
Tu-Khiem Le - Dublin City University, Ireland
Mai Boi Quach - Dublin City University, Ireland
Nguyễn Thanh Binh
- Ho Chi Minh University of Science, Vietnam