Proof by Experimentation? Towards Better IR Research

  • Citation-Key:
    Fuhr:20ta
  • Title:
    Proof by Experimentation? Towards Better IR Research
  • Author(s):
    Norbert Fuhr
  • How published:
    Keynote Talk at SIGIR 2020
  • Year:
    2020

Abstract:


In IR research, empirical foundation in the form of experiments plays an important role. However, the methods applied often are not at the level of scientific standards that hold in many other disciplines, as IR experiments are frequently flawed in several ways, like using invalid measures, inappropriate statistical methods or comparison to weak baselines. Moreover, neither re-used test collections nor leaderboards can yield statistically valid results - only evaluation campaigns or other reproduced results on new collections will lead to scientific findings. In order to make statements that do not only hold for the few tested collections, IR research should focus more on external validity, by aiming at understanding why certain methods work (or don't work) under certain circumstances. Ultimately, research should aim more at performance prediction than at performance measurement.

Fulltext as PDF (Presentation)