ClueWeb09 Wiki

The ClueWeb09 dataset was created by the Language Technologies Institute at Carnegie Mellon University to support research on information retrieval and related human language technologies. The dataset consists of 1 billion web pages, in ten languages, collected in January and February 2009. The dataset is used by several tracks of the TREC conference.

• ClueWeb09 Dataset and Licensing Home Page : The official ClueWeb09 licensing information page

• Dataset Information : Information on the structure of the dataset on disk, the formatting of the data and extra information.

• Language Identification : How language identification was performed on the dataset

• Page Encodings : How the character encodings for the dataset are formatted

• Web Graph : Information on the web graph of nodes and oulinks for the dataset

• Redirects : Redirect Information for the Category B dataset

• Indexing the ClueWeb09 Dataset : Notes on indexing the ClueWeb09 Dataset using Indri and the Lemur Toolkit

• Working with WARC Files : Information on working with the WARC files in the ClueWeb09 dataset

• Sample Files : Sample files in various languages from the ClueWeb09 dataset

• Sample Indri Search Page (Category B) : A sample search interface to the ClueWeb09 Category B dataset using the Indri Search Engine
• Sample Indri Search Page (Category A - English) : A sample search interface to the English only portion of the ClueWeb09 Category A dataset using the Indri Search Engine
• More information about using the CGI Web interface to display document contents etc.

• ClueWeb09 Mailing List : The ClueWeb09 mailing list

---

Acknowledgements

The creation of the ClueWeb09 dataset was sponsored by National Science Foundation grant IIS-0841275, under its Cluster Exploratory program. We thank Google and IBM for the use of the CluE computer cluster. We thank Nick Craswell, Dennis Fetterly, Don Metzler, NIST's ITL Retrieval Group, and Yahoo! for their assistance and advice. We thank the Wikimedia Foundation for enabling the inclusion of the English Wikipedia. Any opinions, findings, conclusions or recommendations expressed on this Web site are those of the author(s) of this site, and do not necessarily reflect those of the sponsors.

---