Experimental Information Retrieval
(The TREC Seminar):
11-744
|
|
Experimental Information Retrieval
|
|
|
Description: |
This seminar studies the experimental evaluation of information retrieval systems in community-wide evaluation forums such as TREC, CLEF, NTCIR, INEX, TAC, and other annual research evaluations. The content changes from year to year, but the general format is an in-depth introduction to the evaluation forum; its tracks or tasks, test collections, evaluation methodologies, and metrics; and several of the most competitive or interesting systems in each track or task. Class discussions explore and develop new methods that might be expected to be competitive. The seminar includes a significant project component in which small teams develop systems intended to be competive with the best recent systems. Students are not required to participate in actual TREC, CLEF, etc., evaluations, however some students may wish to do so. A specific goal of the seminar is to prepare students to compete effectively in such evaluations. The course meets twice a week during the first half of the semester. This part of the course is a combination of seminar-style presentations and brainstorming sessions about how to build competitive systems. The course meets once a week during the second half of the semester, when students are doing their projects. This part of the class is essentially weekly progress reports about student projects. The Spring 2013 offering will focus on TREC, and more specifically on the Contextual Suggestions, Knowledge Base Acceleration, Microblog, Session, and Web tracks, all of which will be repeated in the TREC 2013 evaluations (Summer 2013). |
|
Class size: |
8-12 students. |
|
Units: |
This is a 12-unit course. The course satisfies the IR lab requirement for LTI students. |
|
Prerequisites: |
11-741, Information Retrieval, 11-641, Search Engines and Web Mining, or consent of the instructor |
|
Location & Time: |
The class meets in WEH 5328 Tu/Th, 1:30-2:50 during the first half of the semester, and Th, 1:30-2:50 during the second half of the semester. |
|
Instructor: |
|
|
Instructional Materials: |
TREC 2013 notebook papers, and past TREC conference papers, Project Resources web page. |
|
Workload: |
In the first half of the semester the workload consists of reading 3-5 (usually short) papers for each class, and actively participating in discussions. Each person is expected to lead or co-lead the discussion of one track (good practice for public speaking and managing meetings). In the second half of the semester your time is spent working on your project, and attending a weekly meeting to report on your progress. Half of the grade is for the project, so expect to spend significant time developing a good TREC system. (You are free to use open-source or other tools.) A TREC- or SIGIR-style paper describing the project is required at the end of the semester. |
|
Grading: |
Class participation: 30% |
Tentative schedule
|