One of the objectives of the ReNeuIR workshop is to promote a holistic evaluation and a sustainable development of models in neural information retrieval (NIR), noting that efficacy matters but so do the computational and environmental costs incurred to achieve it. In particular, we draw attention to the lessons learnt from past information retrieval studies and encourage a multi-faceted evaluation of NIR models from quality to efficiency, and the design of reusable benchmarks and standardized metrics.
To that end, the workshop will hold a panel discussion to allow participants to brainstorm ideas and help set a new research agenda in this space. The goal of this discussion is to find relevant, formal, and measurable targets to include in our collective research agenda for the next year. To facilitate a meaningful conversation, we seek comments, ideas, and proposals from the community prior to the date of the workshop. We are interested in hearing your thoughts on the following and related questions:
Benchmarking: How do we set up a measurable and reproducible benchmarking platform? What tasks and datasets would be most appropriate? What hardware should we adopt as standard platforms? Can we borrow and extend to the NIR community ideas from similar benchmarking efforts such as the Big ANN Benchmarks (https://big-ann-benchmarks.com/)?
Evaluation: What metrics capture the various trade-offs pertinent to the objectives of the workshop? When can we say that a system is sufficiently accurate and efficient? Should we adopt a non-NIR system as a reference point for efficiency and/or effectiveness?
Environmental Impact: How do we approach the challenge of quantifying the environmental impact of the training and inference of NIR models? Should research publications in the community be encouraged to report an estimate of the environmental impact of proposed methods? How do we design metrics and tools to capture accurate estimates to allow researchers to measure and report concrete quantities within research publications?
Community Building: What action do you see that can strengthen the research community around this topic? Workshops? PhD summer schools? Publication of specific software?
Please send us a note at reneuir2022 [at] easychair [dot] org with comments, ideas, and questions. We will invite contributors to join us as co-authors of a SIGIR Forum submission later in the year.