Description:

This seminar studies the experimental evaluation of information retrieval systems in community-wide evaluation forums such as TREC, CLEF, NTCIR, INEX, TAC, and other annual research evaluations. The content changes from year to year, but the general format is an in-depth introduction to the evaluation forum; its tracks or tasks, test collections, evaluation methodologies, and metrics; and several of the most competitive or interesting systems in each track or task. Class discussions explore and develop new methods that might be expected to be competitive. The seminar includes a significant project component in which small teams develop systems intended to be competive with the best recent systems.

Students are not required to participate in actual TREC, CLEF, etc., evaluations, however some students may wish to do so. A specific goal of the seminar is to prepare students to compete effectively in such evaluations.

The course meets twice a week during the first half of the semester. This part of the course is a combination of seminar-style presentations and brainstorming sessions about how to build competitive systems. The course meets once a week during the second half of the semester, when students are doing their projects. This part of the class is essentially weekly progress reports about student projects.

The Spring 2013 offering will focus on TREC, and more specifically on the Contextual Suggestions, Knowledge Base Acceleration, Microblog, Session, and Web tracks, all of which will be repeated in the TREC 2013 evaluations (Summer 2013).

Class size:

6-12 students.

Units:

This is a 12-unit course. The course satisfies the IR lab requirement for LTI students.

Prerequisites:

11-741, Information Retrieval, 11-641, Search Engines and Web Mining, or consent of the instructor

Location & Time:

The class meets in BH 235B Tu/Th, 1:30-2:50 during the first half of the semester, and Th, 1:30-2:50 during the second half of the semester.

Instructor:

Jamie Callan

Instructional Materials:

TREC conference papers and Project Resources web page.

Workload:

In the first half of the semester the workload consists of reading 3-5 (usually short) papers for each class, and actively participating in discussions. Each person is expected to lead or co-lead the discussion of one track (good practice for public speaking and managing meetings). In the second half of the semester your time is spent working on your project, and attending a weekly meeting to report on your progress. Half of the grade is for the project, so expect to spend significant time developing a good TREC system. (You are free to use open-source or other tools.) A TREC- or SIGIR-style paper describing the project is required at the end of the semester.

Grading:

Class participation: 30%
Discussion leadership: 20%
Project: 50%

1. Jan 15, Course overview and introduction (slides)
2. Jan 17, Introduction to resources, organizational meeting, ClueWeb09 (slides)
   Readings: Discussion Leadership Guidelines, TREC 2012 Overview
3. Jan 22, Web Track [Jamie] (slides)
   Readings: 2012 Web Track overview, 2011 Web Track overview, Glasgow, Twente, QUT
4. Jan 24, Web Track [Jamie] (slides)
   Readings: Ottawa, Srchvrs, CAS ICT
5. Jan 29, Session Track [Shifa] (slides)
   Readings: 2012 Session Track overview, 2011 Session Track overview, Pitt, Georgetown
6. Jan 31, Session Track [Yubin] (slides)
   Readings: BUPT PRIS, CAS ICT, CWI
7. Feb 5, Microblog Track [Juan] (slides)
   Readings: 2012 Microblog Track overview, 2011 Microblog Track overview, Harbin, Waterloo
8. Feb 7, Microblog Track [Yitong] (slides)
   Readings: CMU, Glasgow, CSIRO
9. Feb 12, Contextual Suggestions Track [Lu]
   Readings: 2012 Contextual Suggestions overview, Georgetown, Delaware
10. Feb 14, Contextual Suggestions Track [Anuja] (slides)
    Readings: Toulouse, TNO and Radboud, Amsterdam
11. Feb 19, Project meeting: Presentation of initial project plans
    Reading: Project meeting guidelines
    [Yubin, Anuja, Yitong, Shifa]
12. Feb 21, Project meeting: Presentation of initial project plans
    [Juan, Zi, Lu]
13. Feb 26, Knowledge Base Acceleration Track [Zi]
    Readings: 2012 KBA Track overview, Johns Hopkins, Delware
14. Feb 28, Knowledge Base Acceleration Track [Jamie] (slides)
    Readings: Avignon, Amsterdam, UMass
15. Mar 7, Project meeting
16. Mar 12, Spring break
17. Mar 14, Spring break
18. Mar 21, Project meeting
19. Mar 28, Project meeting
20. Apr 4, No class
21. Apr 11, Project meeting
22. Apr 18, Mid-semester break
23. Apr 25, Project meeting
24. Apr 30, Project presentations
25. May 2, Project presentations
    Optional: First draft of project reports due at 9:00am
26. May 10, Final project reports due at 9:00am