SIGIR Keynote: Proof by Experimentation? Towards Better IR Research

  • Citation-Key:
    Fuhr:20a
  • Title:
    SIGIR Keynote: Proof by Experimentation? Towards Better IR Research
  • Author(s):
    Norbert Fuhr
  • Journal:
    SIGIR Forum
  • Volume:
    54
  • Number:
    2
  • Year:
    2020

Abstract:


Most IR experiments lack both internal and external validity. Top performance is mostly an illusion, given the lack of solid statistical evidence. Thus, reviewers should ignore performance numbers in their judgement, and pay more attention to proper methodology. Researchers need to focus more on performance analysis and prediction.