nutch + hadoop 运行爬取的时候出错
[root@node0 opt]# nutch crawl -dir d -threads 3 -depth 3 -topN 100
solrUrl is not set, indexing will be skipped...
Exception in thread "main" java.net.ConnectException: Call to node0/59.69.74.42:
9000 failed on connection exception: java.net.ConnectException: Connection refus
ed
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1099)
at org.apache.hadoop.ipc.Client.call(Client.java:1075)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119
)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFi
leSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386
)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:100)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574
)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout
.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:4
34)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:56
0)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
at org.apache.hadoop.ipc.Client.call(Client.java:1050)
... 16 more
Nutch??hadoop?? Hadoop 异常
[解决办法]
1、确认/etc/hosts 文件
59.69.74.42 node0 #这个配置要确认，所有HADOOP节点都要确认
2、确认core-site.xml 的配置
<property>
<name>fs.default.name</name>
<value>hdfs://node0:9000</value>
<description></description>
</property>
3、/etc/sysconfig/network
确认你的HADOOP-Namenode配置的HOSTS应该是node0
最后
telnet node0 9000
或者telnet 59.69.74.42 9000
看看通不通
