CAIR

Causality-driven Adhoc Information Retrieval
Forum for Information Retrieval Evaluation (FIRE) - 2020
December 16th-20th (Online Event)

Call for Participation

Why will you participate?

In contrast to a traditional search system, a Causal Search System seeks to retrieve documents that provide information on the likely causes leading to a query event. In this extended search system, in addition to the topically relevant ranked list of documents, the user will also be presented with a list of causally relevant documents. On submitted queries pertaining to an event (e.g. ‘drop of pound’ or ‘housing crisis’), the system then retrieves adequate information required to construct further analysis for the purpose of automated (or semi-automated with humans-in-loop) decision and policy making. Moreover, information extracted from causally related documents could also serve as the necessary explanations in order to support an automatically generated decision prescribing ways to eradicate a likely cause.

Read More

What to do?

Participants will be given a static test collection of documents and a list of queries related to events that are likely to be caused by a number of other past events. The participants are then required to develop ranking models that can effectively retrieve documents containing information on such past events that are likely candidates to lead to the query event. The officially submitted ranked lists of different participating systems will then be evaluated by comparing them against a set of manually judged relevant documents.

Repositories

Corpus

We provide you a static test collection of news articles constituting the official English ad hoc IR collection of FIRE. Access to the data will be provided on receipt of the below access form.

To have an access to the corpus, please email the complete and signed organisational-access form to both fire@isical.ac.in and cair.miners@gmail.com. Organisations may use the individual-access form to manage access rights internally and these individual access forms need not be sent to us.

Click here to download corpus

Train Topics

We will release a training set comprising 5 topics (with the relevance assessments) followed by 20 test topics to the participants. Each topic follows the standard TREC format, i.e., is comprised of a 'title' (usually a small number of keywords) and a 'narrative' (a paragraph describing the relevance criteria in details).

Download training topics

Relevance Judgements

For the 5 training topics, we provide you binary relevance judgements following TREC format. This will help to analyze the causal relevance which you need to address (instead of the topical relevance). This will also enable you to tune prototype systems and explore a number of early approaches and with the help of the evaluation (using the provided manual assessments) to see what works and what does not.

Related Research

If you are sans suitable reading resources, here is the preprint of our SIGIR'20 paper. Might be helpul, we believe. You can also find the paper at : SIGIR'20 Proceedings

Bulletins

Important Dates

Training Data Release 19th June, 2020
Test Data Release 31th July, 2020
Run Submission Deadline 27th August, 2020
Results Declaration 21th September, 2020
Working Note Submission 12th October, 2020 (AOE)
Review Notifications 23th October, 2020 (AOE)
Final Version of Working Note 2nd November, 2020 (AOE)

Latest Updates

Check results... find here
Results coming soon...
Run submission deadline extended...
Test data released... find here
Training data released... find here

Guidelines

What Do You Submit?

Your proposed system must generate a 6 column .tsv file following the standard TREC format. In order to encourage the investigation of different kind of features, three runs per participating group are allowed. You can send on runs over cair.miners@gmail.com.

How Do You Get Evaluated?

We will employ standard evaluation metrics, such as nDCG and MAP, to take into account both precision and recall (in the graded and binary cases respectively) of the submitted runs. Additionally, we will also rank systems based on precision alone using nDCG and P@5.

Know Where You are?

Run Name MAP P@5
UCSC-Run-query_narratives
(University of California, Santa Cruz)
0.4553 0.7000
UCSC-Run-query_title
(University of California, Santa Cruz)
0.4066 0.5400
UCSC-Run-post_event_terms_expansion
(University of California, Santa Cruz)
0.3885 0.5000
NIT-Run-1
(National Institute of Technology, Silchar)
0.0577 0.2600

You can also evaluate your models using the Relevance Judgements provided here.

Articulate Your Model

We are now accepting submissions. Working notes have to be submitted at cair.miners@gmail.com. Look at the guidelines below before you go -

Submission Guidelines

  • Submissions must describe substantial, original and unpublished work. Wherever appropriate, concrete evaluation and analysis must be included. If the paper being submitted is under review at any other venue, the same should be explicitly mentioned when making the submission. Such papers, if accepted, should be withdrawn from all other places.
  • All submissions should be in Single column CEUR format. Authors should use one of the CEUR Templates below:
  • There is no page limit for submissions, but should be more than 4 pages.
  • We will NOT be allowing multiple working notes from the same set of authors.
  • Each paper should have a copyright clause included in the paper (See the Author agreement variants)
  • Each author should also submit a copyright agreement signed by the authors. (Partially filled agreement will be shared shortly).

Basic Guidelines

  • Check basic sanity of the paper for grammatical errors and reported results.
  • Papers should have sufficient information for reproducing the mentioned results.
  • Check the papers for text reuse/Plagiarism. This includes self-plagiarism as well. Any paper found to have plagiarized content will be rejected without further consideration.
  • Author names should not have any salutations like Dr., Prof., etc.

Correspondence

May We Help You?

Debasis Ganguly, IBM Research Lab, Dublin
Charles Jochim, IBM Research Lab, Dublin
Francesca Bonin, IBM Research Lab, Dublin
Suchana Datta, University College Dublin
Dwaipayan Roy, GESIS, Cologne
Derek Greene, University College Dublin

Reach Out Here

Please reach out to the below in case you have any kind of queries related to the task.

cair.miners@gmail.com

Subscribe below for all latest updates.

cairminers@googlegroups.com