Description:

This seminar studies the experimental evaluation of information retrieval systems in community-wide evaluation forums such as TREC, CLEF, NTCIR, INEX, TAC, and other annual research evaluations. The content will change from year to year, but the general format will be an in-depth introduction to the evaluation forum; its tracks or tasks, test collections, evaluation methodologies, and metrics; and several of the most competitive or interesting systems in each track or task. Class discussions will explore and develop new methods that might be expected to be competitive. The seminar includes a significant project component in which small teams develop systems intended to be competive with the best recent systems.

Students are not required to participate in actual TREC, CLEF, etc., evaluations, however some students may wish to do so. A specific goal of the seminar is to prepare students to compete effectively in such evaluations.

The course meets twice a week during the first half of the semester. This part of the course is a combination of seminar-style presentations and brainstorming sessions about how to build competitive systems. The course meets once a week during the second half of the semester, when students are doing their projects. This part of the class is essentially weekly progress reports about student projects.

The Spring 2010 offering will focus on TREC, and more specifically on the Blog, Chemical IR, Entity, Legal, Million Query, Relevance Feedback, and Web tracks, most of which will be repeated in the TREC 2011 evaluations (Summer 2011). This list will be updated after the TREC conference, which occurs Nov 18-20, 2009.

Class size:

6-12 students.

Units:

This is a 12-unit course. The course satisfies the IR lab requirement.

Prerequisites:

11-741, Information Retrieval, or consent of the instructor

Time & Location:

The class is tentatively scheduled for Tu/Th, 1:30-2:50 during the first half of the semester, and Th, 1:30-2:50 during the second half of the semester. The time may be adjusted after the first class.

Instructor:

Jamie Callan

Instructional Materials:

TREC conference papers.

Workload:

In the first half of the semester the workload consists of reading 2-3 papers for each class, and actively participating in discussions. Each person is expected to lead or co-lead the discussion of one track (good practice for public speaking and managing meetings). In the second half of the semester your time is spent working on your project, and attending a weekly meeting to report on your progress. Half of the grade is for the project, so expect to spend significant time developing a good TREC system. (You are free to use open-source or other tools.) A TREC- or SIGIR-style paper describing the project is required at the end of the semester.

Grading:

Class participation: 30%
Discussion leadership: 20%
Project: 50%

1. Jan 12, Course overview and introduction
2. Jan 14, Brief organizational meeting
3. Jan 19, Blog track
4. Jan 21, Blog track
5. Jan 26, Blog track planning
6. Jan 28, Web track
7. Feb 2, Web track
8. Feb 4, Web track planning
9. Feb 9, Relevance feedback
10. Feb 11, Relevance feedback
11. Feb 16, Relevance feedback planning
12. Feb 18, Entity
13. Feb 23, Legal
14. Feb 25, Chemical IR, Million Query
15. Mar 2, Project planning
16. Mar 4, Project planning
17. Mar 9, Spring break
18. Mar 11, Spring break
19. Mar 18, Project meeting
20. Mar 25, Project meeting
21. Apr 1, Project meeting
22. Apr 8, Project meeting
23. Apr 15, Project meeting
24. Apr 22, Final project reports
25. Apr 29, Final project reports