抓取流程－小结
从之前 的抓取結果来分析各阶段的情况。其中蓝色表示未修改但要注意的，红色表示前后已经修改的。
?
?
injector：只有二个seed urls（ 这里没有列出csdn数据）
http://www.163.com/??? Version: 7??? ??? ??? ??? #7为当前nutch的修改版本
Status: 1 (db_unfetched)??? ??? ??? ??? ??? #see CrawlDatum.STATUS_DB_UNFETCHED
Fetch time: Mon Jul 04 14:57:19 CST 2011
Modified time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 1.0??? #seed url为1.0
Signature: null??? ??? #page md5摘要,未抓取,所以为空
Metadata:
?
generator：同样只有二个urls
http://www.163.com/??? Version: 7
Status: 1 (db_unfetched)
Fetch time: Mon Jul 04 14:57:19 CST 2011
Modified time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 1.0
Signature: null
Metadata: _ngt_: 1309887693964
?
?
fetcher：
－－－－－－－
crawl_fetch：
http://www.163.com/??? Version: 7
Status: 33 (fetch_success)
Fetch time: Sat Jul 09 15:14:02 CST 2011
Modified time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 1.0
Signature: null
Metadata: _ngt_: 1309933252318_pst_: success(1), lastModified=0
?
crawl_parse：
http://www.163.com/??? Version: 7
Status: 65 (signature)
Fetch time: Sat Jul 09 15:14:08 CST 2011
Modified time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry interval: 0 seconds (0 days)
Score: 1.0
Signature: 989844cdb45e225db2b2731315cb5342
Metadata:?
//其它情况
http://www.163.com/rss/??? Version: 7
Status: 67 (linked)
Fetch time: Sat Jul 09 15:14:08 CST 2011??? ／／未fetched的以parsed的时间记录
Modified time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 0.01
Signature: null
Metadata:
－－－－－－－
?
updatedb（crawldb，可以看出，这个文件存放的是所有历史urls，即global link map）：
http://www.163.com/??? Version: 7
Status: 2 (db_fetched)??????
Fetch time: Mon Aug 08 15:14:02 CST 2011??? ／／已经更新为1个月后的fetch time，表明下次就不要再fetch了
Modified time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 1.0
Signature: 989844cdb45e225db2b2731315cb5342?? ／／与crawl_parse一样，即没有修改，即整个html的md5值
Metadata: _pst_: success(1), lastModified=0
／／其它情况如同在injector阶段一样，以为generator准备
http://www.163.com/rss/??? Version: 7
Status: 1 (db_unfetched)
Fetch time: Tue Jul 12 23:49:27 CST 2011???? ／／未fetched的更新为update时的时间
Modified time: Thu Jan 01 08:00:00 CST 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 0.01
Signature: null
Metadata:
?
** 关于如何保证fetched过的urlds不再fetch，参阅updatedb
**修改crawldb/current下数据的只有：
* injector
* generator中generate.update.crawldb参数为true时进行
* updatedb
?
?
