The Sapphire Web Crawler - Crawl Statistics

Current Crawl Statistics as of: Mon Mar 9 16:25:32 2009

For information about the Sapphire Crawl, please see the Frequently Asked Questions page.

Crawl is currently complete

Total Pages Downloaded:1268866864
Total GB Downloaded (HTML only):29114.20
Total GB Downloaded (Page Resources):0.00
Total GB Downloaded (HTML+Resources):29114.20
Average Fetch Throughput (doc/sec):755.31
Average Overall Throughput (doc/sec):431.26
Days Remaining to 1 Billion:n/a (complete)
Days Remaining to 1 Billion:n/a (complete)
Days Remaining to 1.2 Billion:n/a (complete)
Days Remaining to 1.2 Billion:n/a (complete)


Known Language Distribution

Note: this is the known languages for the dataset - guesses are not included

(Rows marked with a + indicates we have reached our limit for that language and are no longer collecting pages for that language)

English (+)499633932 (47.63 %)
Chinese (+)177977879 (16.97 %)
Spanish (+)79521976 (7.58 %)
Japanese (+)67565792 (6.44 %)
German (+)49844623 (4.75 %)
French (+)50930502 (4.86 %)
Korean18165066 (1.73 %)
Italian (+)27321276 (2.60 %)
Portuguese (+)37690229 (3.59 %)
Arabic29223001 (2.79 %)

(Rows marked with a * indicate seed-only iterations)

  Downloaded Found Throughput
 
Timestamp
 
Pages
HTML
(GB)
Page Resources
(GB)
HTML + Resources
(GB)
 
URLs
Fetch Only Througput
(Pages / Second)
Iteration Throughput
(Pages / Second)
*1 Tue Jan 6 22:57:02 2009 924413 26.70 0.00 26.70 27797194 239.42 198.16 Stats
*2 Wed Jan 7 02:15:12 2009 513206 15.15 0.00 15.15 36655047 208.45 143.31 Stats
*3 Wed Jan 7 03:14:53 2009 504171 14.70 0.00 14.70 44740336 207.48 132.78 Stats
*4 Wed Jan 7 04:18:10 2009 536082 15.73 0.00 15.73 52851721 215.12 139.82 Stats
*5 Wed Jan 7 05:22:04 2009 518564 15.18 0.00 15.18 60265071 208.01 135.96 Stats
*6 Wed Jan 7 06:25:39 2009 529618 15.48 0.00 15.48 67600538 213.56 131.16 Stats
*7 Wed Jan 7 07:32:57 2009 536827 15.69 0.00 15.69 74778817 214.39 133.21 Stats
*8 Wed Jan 7 08:40:07 2009 551928 16.16 0.00 16.16 81891309 222.37 136.38 Stats
*9 Wed Jan 7 11:47:46 2009 993247 28.81 0.00 28.81 94035930 260.76 179.48 Stats
*10 Wed Jan 7 13:20:00 2009 953247 28.00 0.00 28.00 105303535 238.91 156.94 Stats
*11 Wed Jan 7 15:01:14 2009 488706 15.19 0.00 15.19 111279529 144.63 91.84 Stats
*12 Wed Jan 7 16:29:55 2009 222611 7.41 0.00 7.41 114048166 68.83 44.15 Stats
*13 Wed Jan 7 18:03:23 2009 96273 3.48 0.00 3.48 115245214 30.86 20.00 Stats
*14 Wed Jan 7 19:28:59 2009 67514 2.63 0.00 2.63 116058146 21.67 13.76 Stats
15 Wed Jan 7 20:58:05 2009 1610135 41.79 0.00 41.79 128000031 400.53 289.07 Stats
16 Wed Jan 7 22:39:57 2009 1382148 37.01 0.00 37.01 137644817 341.36 235.02 Stats
17 Thu Jan 8 00:17:58 2009 1389258 37.16 0.00 37.16 146914479 342.35 230.93 Stats
18 Thu Jan 8 01:58:14 2009 1591286 37.74 0.00 37.74 158029550 389.45 258.91 Stats
19 Thu Jan 8 03:40:41 2009 1522701 37.04 0.00 37.04 168189135 371.57 250.98 Stats
20 Thu Jan 8 05:21:48 2009 1537572 36.51 0.00 36.51 178101568 376.30 245.97 Stats
21 Thu Jan 8 07:05:59 2009 1505170 36.09 0.00 36.09 187702019 381.83 249.49 Stats
22 Thu Jan 8 08:46:33 2009 1509435 36.57 0.00 36.57 197209000 373.44 232.54 Stats
23 Thu Jan 8 10:34:44 2009 1510426 36.64 0.00 36.64 206796920 380.08 245.96 Stats
24 Thu Jan 8 12:17:05 2009 1512948 35.80 0.00 35.80 216441665 374.40 238.00 Stats
25 Thu Jan 8 14:03:02 2009 1450490 34.37 0.00 34.37 226616753 226.43 169.59 Stats
26 Thu Jan 8 16:25:35 2009 1415112 34.16 0.00 34.16 236320243 367.18 233.32 Stats
27 Thu Jan 8 18:06:40 2009 1385497 33.97 0.00 33.97 245581024 193.10 149.91 Stats
28 Thu Jan 8 20:40:43 2009 1283133 31.95 0.00 31.95 254021460 323.78 204.88 Stats
29 Thu Jan 8 22:27:37 2009 1182563 30.24 0.00 30.24 261770169 307.88 193.67 Stats
30 Fri Jan 9 00:09:24 2009 1107924 28.80 0.00 28.80 269007484 291.25 182.01 Stats
31 Fri Jan 9 01:50:51 2009 1123481 29.69 0.00 29.69 276380086 286.02 179.41 Stats
32 Fri Jan 9 03:35:13 2009 1055422 28.21 0.00 28.21 283856688 273.85 168.89 Stats
33 Fri Jan 9 05:19:22 2009 1001100 26.69 0.00 26.69 290770348 269.11 61.33 Stats
34 Fri Jan 9 09:51:24 2009 911537 24.31 0.00 24.31 296994510 192.88 124.26 Stats
35 Fri Jan 9 11:53:40 2009 1375302 35.29 0.00 35.29 305879291 355.93 213.16 Stats
36 Fri Jan 9 13:41:12 2009 1295000 34.28 0.00 34.28 314452168 327.10 188.04 Stats
37 Fri Jan 9 16:31:58 2009 2261327 57.53 0.00 57.53 326188702 185.86 154.37 Stats
38 Fri Jan 9 20:38:10 2009 1404826 34.03 0.00 34.03 333786065 351.82 229.92 Stats
39 Fri Jan 9 22:20:29 2009 1383142 34.19 0.00 34.19 341230229 343.98 199.36 Stats
40 Sat Jan 10 00:16:09 2009 1314156 32.98 0.00 32.98 348320683 340.90 196.76 Stats
41 Sat Jan 10 02:39:41 2009 1229054 31.24 0.00 31.24 354866044 306.34 179.66 Stats
42 Sat Jan 10 04:33:44 2009 1174896 30.23 0.00 30.23 361199129 287.40 168.95 Stats
43 Sat Jan 10 06:29:41 2009 1234326 31.56 0.00 31.56 367794057 315.77 183.93 Stats
44 Sat Jan 10 08:21:34 2009 1151240 28.90 0.00 28.90 374006437 299.41 168.88 Stats
45 Sat Jan 10 10:15:13 2009 1132370 27.68 0.00 27.68 380582919 294.58 161.51 Stats
46 Sat Jan 10 12:12:06 2009 1048021 25.54 0.00 25.54 386694462 278.14 156.63 Stats
47 Sat Jan 10 14:03:41 2009 1527150 36.98 0.00 36.98 395083463 376.42 214.79 Stats
48 Sat Jan 10 16:09:54 2009 1465451 35.61 0.00 35.61 0 358.04 Stats
49 Sat Jan 10 18:30:46 2009 1480437 36.34 0.00 36.34 411148709 368.27 202.86 Stats
50 Sat Jan 10 20:32:26 2009 1360438 34.09 0.00 34.09 418813836 348.65 187.08 Stats
51 Sat Jan 10 22:33:40 2009 1370872 33.86 0.00 33.86 426377961 342.12 187.30 Stats
52 Sun Jan 11 00:35:41 2009 1339318 33.57 0.00 33.57 433772349 333.66 178.01 Stats
53 Sun Jan 11 02:41:07 2009 1384089 34.54 0.00 34.54 441397254 350.14 183.54 Stats
54 Sun Jan 11 04:46:50 2009 1382319 34.60 0.00 34.60 448953227 348.72 184.21 Stats
55 Sun Jan 11 06:51:56 2009 1375257 34.79 0.00 34.79 456665558 341.51 76.22 Stats
56 Sun Jan 11 11:52:42 2009 1332948 34.17 0.00 34.17 464414300 330.92 167.73 Stats
57 Sun Jan 11 14:05:11 2009 1292695 33.27 0.00 33.27 471852298 167.75 110.91 Stats
58 Sun Jan 11 19:36:32 2009 1499632 37.57 0.00 37.57 480023528 373.88 183.98 Stats
59 Mon Jan 12 09:23:56 2009 1389651 35.38 0.00 35.38 487781566 343.89 175.48 Stats
60 Mon Jan 12 12:31:17 2009 1481138 37.52 0.00 37.52 495734873 352.65 178.26 Stats
61 Mon Jan 12 14:49:48 2009 1472987 37.58 0.00 37.58 503883059 357.87 178.70 Stats
62 Mon Jan 12 17:13:19 2009 3333883 85.36 0.00 85.36 521134363 356.15 242.62 Stats
63 Mon Jan 12 21:05:06 2009 2476808 61.86 0.00 61.86 533629088 302.34 195.44 Stats
64 Tue Jan 13 00:44:36 2009 2042713 50.27 0.00 50.27 544276993 295.62 86.98 Stats
65 Tue Jan 13 11:16:31 2009 5115894 115.30 0.00 115.30 568503896 502.99 350.16 Stats
66 Tue Jan 13 15:20:28 2009 5473979 120.83 0.00 120.83 592887919 502.89 348.15 Stats
67 Tue Jan 13 19:56:31 2009 5223685 115.82 0.00 115.82 617705293 488.56 336.53 Stats
68 Wed Jan 14 09:25:33 2009 5122555 110.55 0.00 110.55 641221772 480.54 326.96 Stats
69 Wed Jan 14 14:37:34 2009 4515334 106.24 0.00 106.24 665783000 375.96 263.27 Stats
70 Wed Jan 14 19:32:09 2009 4004567 97.40 0.00 97.40 687421031 334.69 226.08 Stats
71 Thu Jan 15 01:19:44 2009 3535099 91.65 0.00 91.65 707883153 328.17 210.38 Stats
72 Thu Jan 15 05:59:48 2009 2871036 75.42 0.00 75.42 724939783 289.04 184.04 Stats
73 Thu Jan 15 10:21:59 2009 3245666 92.27 0.00 92.27 744756909 305.36 196.66 Stats
74 Thu Jan 15 15:03:12 2009 3037979 88.19 0.00 88.19 763002878 282.81 180.86 Stats
75 Thu Jan 15 21:42:36 2009 2283961 65.32 0.00 65.32 775553160 209.98 138.11 Stats
76 Fri Jan 16 02:19:47 2009 3725297 99.87 0.00 99.87 794273865 353.98 225.57 Stats
77 Fri Jan 16 06:55:04 2009 3366217 85.62 0.00 85.62 810249673 353.41 216.07 Stats
78 Fri Jan 16 11:14:45 2009 2555257 64.83 0.00 64.83 822541182 349.94 193.27 Stats
79 Fri Jan 16 14:55:08 2009 1917601 52.97 0.00 52.97 832310109 790.44 227.23 Stats
80 Fri Jan 16 17:19:53 2009 8257381 215.47 0.00 215.47 886792715 842.16 477.19 Stats
81 Sat Jan 17 01:22:51 2009 5727795 152.58 0.00 152.58 924246567 630.54 325.65 Stats
82 Sat Jan 17 06:16:02 2009 6806152 179.24 0.00 179.24 965465331 705.81 370.06 Stats
83 Sat Jan 17 11:22:37 2009 5961203 157.07 0.00 157.07 1001466403 641.75 326.30 Stats
84 Sat Jan 17 16:27:47 2009 9503544 247.82 0.00 247.82 1054309141 925.37 477.47 Stats
85 Sat Jan 17 21:59:34 2009 5325332 142.00 0.00 142.00 1085295190 566.83 277.88 Stats
86 Sun Jan 18 13:23:15 2009 2352935 64.17 0.00 64.17 1100378259 286.14 136.73 Stats
87 Sun Jan 18 23:38:40 2009 10127570 259.50 0.00 259.50 1152975046 885.35 481.99 Stats
88 Mon Jan 19 06:12:11 2009 6833199 176.69 0.00 176.69 1188621147 747.21 356.79 Stats
89 Mon Jan 19 11:31:25 2009 8680967 221.24 0.00 221.24 1231952794 875.27 425.70 Stats
90 Mon Jan 19 17:13:05 2009 7209163 185.72 0.00 185.72 1268038449 745.52 354.21 Stats
91 Mon Jan 19 22:52:20 2009 3528127 93.26 0.00 93.26 1287005674 407.55 187.89 Stats
92 Tue Jan 20 04:05:21 2009 2594364 68.97 0.00 68.97 1301411004 317.86 141.68 Stats
93 Tue Jan 20 09:10:35 2009 2441170 64.65 0.00 64.65 1314793454 300.82 118.47 Stats
94 Tue Jan 20 14:54:05 2009 4181224 108.38 0.00 108.38 1336068472 482.26 210.56 Stats
95 Tue Jan 20 20:43:49 2009 5230655 134.99 0.00 134.99 1359860472 592.98 270.75 Stats
96 Wed Jan 21 02:13:10 2009 13051043 329.20 0.00 329.20 1419129368 908.53 499.70 Stats
97 Wed Jan 21 16:50:45 2009 6101845 156.54 0.00 156.54 1448935477 476.15 247.30 Stats
98 Wed Jan 21 23:43:02 2009 15663437 389.60 0.00 389.60 1515680545 1390.45 649.42 Stats
99 Thu Jan 22 06:25:03 2009 12375507 302.06 0.00 302.06 1567217599 1232.13 530.16 Stats
100 Thu Jan 22 12:54:08 2009 12746960 323.39 0.00 323.39 1622925621 863.38 442.11 Stats
101 Thu Jan 22 21:05:33 2009 13656698 317.68 0.00 317.68 1673302168 1107.42 529.14 Stats
102 Fri Jan 23 04:15:44 2009 13347169 311.91 0.00 311.91 1721681154 1036.91 508.70 Stats
103 Fri Jan 23 11:33:04 2009 13084824 305.93 0.00 305.93 1768472891 995.27 490.66 Stats
104 Fri Jan 23 18:58:56 2009 12688125 295.05 0.00 295.05 1812758543 993.90 477.97 Stats
105 Sat Jan 24 02:21:24 2009 14329790 309.72 0.00 309.72 1865427720 1212.95 549.75 Stats
106 Sat Jan 24 09:35:52 2009 13275087 312.76 0.00 312.76 1916495786 1104.32 490.69 Stats
107 Sat Jan 24 17:06:48 2009 12875621 312.09 0.00 312.09 1966594207 1019.21 463.30 Stats
108 Sun Jan 25 00:50:02 2009 12247840 298.11 0.00 298.11 2014155732 928.43 417.35 Stats
109 Sun Jan 25 08:59:11 2009 12351637 301.18 0.00 301.18 2061916060 933.47 428.00 Stats
110 Sun Jan 25 17:39:02 2009 12109592 296.05 0.00 296.05 2108353110 1034.92 431.75 Stats
111 Mon Jan 26 01:27:46 2009 10183666 244.84 0.00 244.84 2145810302 918.19 366.42 Stats
112 Mon Jan 26 09:11:00 2009 17243712 411.67 0.00 411.67 2207871057 1076.92 494.83 Stats
113 Mon Jan 26 18:51:50 2009 16622365 402.24 0.00 402.24 2267168949 1060.44 493.89 Stats
114 Tue Jan 27 04:12:49 2009 14085461 340.70 0.00 340.70 2316974316 956.24 428.47 Stats
115 Wed Jan 28 00:09:12 2009 17859227 423.95 0.00 423.95 2378234143 1082.90 510.85 Stats
116 Wed Jan 28 09:51:54 2009 20464082 478.39 0.00 478.39 2447002837 1062.08 524.92 Stats
117 Thu Jan 29 08:24:27 2009 21907266 503.88 0.00 503.88 2518004251 974.39 537.64 Stats
118 Thu Jan 29 19:43:36 2009 24574337 565.04 0.00 565.04 2597277292 1070.50 591.74 Stats
119 Fri Jan 30 07:15:47 2009 24541645 559.13 0.00 559.13 2674896998 935.56 541.64 Stats
120 Fri Jan 30 19:50:59 2009 23094408 527.62 0.00 527.62 2746922943 1023.92 550.40 Stats
121 Sun Feb 1 18:34:02 2009 18370347 398.19 0.00 398.19 2819057918 655.99 379.15 Stats
122 Mon Feb 2 08:01:36 2009 21835934 472.48 0.00 472.48 2901848802 1009.10 513.41 Stats
123 Mon Feb 2 19:50:29 2009 25040311 554.59 0.00 554.59 2992770386 1119.77 578.39 Stats
124 Thu Feb 5 12:25:40 2009 3405746 72.13 0.00 72.13 3005456634 402.24 120.41 Stats
125 Thu Feb 5 20:17:10 2009 25279274 566.23 0.00 566.23 3095507341 1113.53 559.04 Stats
126 Fri Feb 6 08:50:51 2009 25108705 556.08 0.00 556.08 3181628028 1094.63 546.01 Stats
127 Fri Feb 6 21:41:54 2009 3448704 78.39 0.00 78.39 3194476141 426.45 113.29 Stats
128 Sat Feb 7 06:09:18 2009 6641096 149.59 0.00 149.59 3218392627 549.35 190.24 Stats
129 Sat Feb 7 15:51:09 2009 27976185 610.19 0.00 610.19 3309490866 1042.41 543.13 Stats
130 Sun Feb 8 06:09:40 2009 32772067 719.10 0.00 719.10 3409037552 1084.66 585.04 Stats
131 Sun Feb 8 21:43:19 2009 31931662 705.40 0.00 705.40 3505273589 1068.45 568.21 Stats
132 Mon Feb 9 13:19:58 2009 31246691 687.73 0.00 687.73 3597783993 1011.06 554.52 Stats
133 Tue Feb 10 07:57:28 2009 32664287 709.56 0.00 709.56 3689496840 767.88 469.59 Stats
134 Wed Feb 11 03:16:49 2009 31411098 687.60 0.00 687.60 3776725709 1053.14 547.06 Stats
135 Wed Feb 11 19:13:49 2009 28267388 630.71 0.00 630.71 3855504374 851.71 465.65 Stats
136 Thu Feb 12 12:05:36 2009 13697613 317.70 0.00 317.70 3895382175 710.53 297.20 Stats
137 Fri Feb 13 09:50:17 2009 36897789 815.85 0.00 815.85 4026579733 1101.85 571.09 Stats
138 Mon Feb 16 17:31:55 2009 13792555 307.42 0.00 307.42 0 338.14 Stats
139 Tue Feb 17 17:23:05 2009 33579285 757.91 0.00 757.91 4184943908 1124.29 545.33 Stats
140 Wed Feb 18 10:29:24 2009 31767986 723.34 0.00 723.34 0 1037.86 Stats
141 Thu Feb 19 09:43:08 2009 37557382 836.58 0.00 836.58 4399980755 1046.46 533.17 Stats
142 Mon Feb 23 16:10:37 2009 34283567 764.16 0.00 764.16 4501754074 732.35 Stats
143 Tue Feb 24 18:57:50 2009 32436712 725.23 0.00 725.23 4596327721 988.89 479.36 Stats
144 Tue Feb 25 13:57:50 2009 21600667 371.19 0.00 371.19 4644594355 658.54 Stats
145 Thu Feb 26 11:54:15 2009 32335210 597.42 0.00 597.42 4732347524 1177.11 519.72 Stats
146 Fri Feb 27 05:11:14 2009 24847160 400.82 0.00 400.82 4791442213 707.57 359.29 Stats

This research is sponsored in part by National Science Foundation grant IIS-0841275. Any opinions, findings, conclusions or recommendations expressed on this Web site are those of the author(s), and do not necessarily reflect those of the sponsors.


Updated on October 1, 2008.
Maintained by Jamie Callan