Experimental Information Retrieval
(The TREC Seminar):
11-744

Description:

This seminar studies the experimental evaluation of information retrieval systems in community-wide evaluation forums such as TREC, CLEF, NTCIR, INEX, TAC, and other annual research evaluations. The content changes from year to year, but the general format is an in-depth introduction to the evaluation forum; its tracks or tasks, test collections, evaluation methodologies, and metrics; and several of the most competitive or interesting systems in each track or task. Class discussions explore and develop new methods that might be expected to be competitive. The seminar includes a significant project component in which small teams develop systems intended to be competive with the best recent systems.

Students are not required to participate in actual TREC, CLEF, etc., evaluations, however some students may wish to do so. A specific goal of the seminar is to prepare students to compete effectively in such evaluations.

The course meets twice a week during the first half of the semester. This part of the course is a combination of seminar-style presentations and brainstorming sessions about how to build competitive systems. The course meets once a week during the second half of the semester, when students are doing their projects. This part of the class is essentially weekly progress reports about student projects.

The Spring 2013 offering will focus on TREC, and more specifically on the Contextual Suggestions, Knowledge Base Acceleration, Microblog, Session, and Web tracks, all of which will be repeated in the TREC 2013 evaluations (Summer 2013).

Class size:

8-12 students.

Units:

This is a 12-unit course. The course satisfies the IR lab requirement for LTI students.

Prerequisites:

11-741, Information Retrieval, 11-641, Search Engines and Web Mining, or consent of the instructor

Location & Time:

The class meets in WEH 5328 Tu/Th, 1:30-2:50 during the first half of the semester, and Th, 1:30-2:50 during the second half of the semester.

Instructor:

Jamie Callan

Instructional Materials:

TREC 2013 notebook papers, and past TREC conference papers, Project Resources web page.

Workload:

In the first half of the semester the workload consists of reading 3-5 (usually short) papers for each class, and actively participating in discussions. Each person is expected to lead or co-lead the discussion of one track (good practice for public speaking and managing meetings). In the second half of the semester your time is spent working on your project, and attending a weekly meeting to report on your progress. Half of the grade is for the project, so expect to spend significant time developing a good TREC system. (You are free to use open-source or other tools.) A TREC- or SIGIR-style paper describing the project is required at the end of the semester.

Grading:

Class participation: 30%
Discussion leadership: 20%
Project: 50%

Tentative schedule
  1. Jan 14, Course overview and introduction (pdf, pdf)
  2. Jan 16, Introduction to resources, organizational meeting, ClueWeb09, ClueWeb12
    Readings: Discussion Leadership Guidelines, TREC 2013 Overview
  3. Jan 21, Web Track
    Readings: 2013 Overview, 2012 Overview, 2 more TBA
  4. Jan 23, Web Track
  5. Jan 28, Session Track
    Readings: 2013 Overview, 2012 Overview, 2 more TBA
  6. Jan 30, Session Track
  7. Feb 4, Microblog Track
    Readings: 2013 Overview, 2012 Overview, 2 more TBA
  8. Feb 6, Microblog Track
  9. Feb 11, Contextual Suggestions Track
    Readings: 2013 Overview, 2012 Overview, 2 more TBA
  10. Feb 13, Contextual Suggestions Track
  11. Feb 18, Project meeting: Presentation of initial project plans
    Reading: Project meeting guidelines
  12. Feb 20, Project meeting: Presentation of initial project plans
  13. Feb 25, Federated Search Track
    Readings: 2013 Overview, 2 more TBA
  14. Feb 27, Federated Search Track
  15. Mar 6, Project meeting
  16. Mar 11, Spring break
  17. Mar 13, Spring break
  18. Mar 20, Project meeting
  19. Mar 27, Project meeting
  20. Apr 3, Project meeting
  21. Apr 10, Mid-semester break
  22. Apr 17, Project meeting
  23. Apr 24, Project meeting
  24. Apr 29, Project presentations
  25. May 1, Project presentations
    Optional: First draft of project reports due at 9:00am
  26. May 9, Final project reports due at 9:00am

Updated on December 20, 2013
Jamie Callan