Hadoop Oozie学习笔记(三) Oozie不支持Sqoop问题解决
本博客属原创文章,转载请注明出处:http://guoyunsky.iteye.com/blog/1254537
欢迎加入Hadoop超级群:?180941958? ?
?
? ? ?搭建起了hadoop＋oozie+Sqoop,准备写个Oozie工作流,里面通过Sqoop从DB里导数据再写到HDFS.一切就绪,但却报错了.报错如下:
? ?Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sqoop'.
? ? 再去看日志,好家伙,错误一大把,但已经能否发现些什么.异常如下:
2011-11-11 11:39:44,658 WARN V1JobsServlet:539 - USER[?] GROUP[users] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs?action=start] error[E0701], E0701: XML schema error, cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sqoop'.org.apache.oozie.servlet.XServletException: E0701: XML schema error, cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sqoop'.at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:163)at org.apache.oozie.servlet.V1JobsServlet.submitJob(V1JobsServlet.java:74)at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:92)at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)at java.lang.Thread.run(Thread.java:662)Caused by: org.apache.oozie.DagEngineException: E0701: XML schema error, cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sqoop'.at org.apache.oozie.DagEngine.submitJob(DagEngine.java:137)at org.apache.oozie.servlet.V1JobsServlet.submitWorkflowJob(V1JobsServlet.java:159)... 17 moreCaused by: org.apache.oozie.command.CommandException: E0701: XML schema error, cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sqoop'.at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:185)at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:61)at org.apache.oozie.command.XCommand.call(XCommand.java:257)at org.apache.oozie.DagEngine.submitJob(DagEngine.java:125)... 18 moreCaused by: org.apache.oozie.workflow.WorkflowException: E0701: XML schema error, cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sqoop'.at org.apache.oozie.workflow.lite.LiteWorkflowAppParser.validateAndParse(LiteWorkflowAppParser.java:120)at org.apache.oozie.workflow.lite.LiteWorkflowLib.parseDef(LiteWorkflowLib.java:47)at org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:46)at org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:41)at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:95)... 21 moreCaused by: org.xml.sax.SAXParseException: cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'sqoop'.at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:195)at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:131)at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:384)at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:318)at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(XMLSchemaValidator.java:417)at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.reportSchemaError(XMLSchemaValidator.java:3182)at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:1927)at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(XMLSchemaValidator.java:705)at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:400)at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2755)at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:511)at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)at com.sun.org.apache.xerces.internal.jaxp.validation.StreamValidatorHelper.validate(StreamValidatorHelper.java:144)at com.sun.org.apache.xerces.internal.jaxp.validation.ValidatorImpl.validate(ValidatorImpl.java:111)at javax.xml.validation.Validator.validate(Validator.java:127)at org.apache.oozie.workflow.lite.LiteWorkflowAppParser.validateAndParse(LiteWorkflowAppParser.java:106)... 25 more
? ?错误大概就是xml不支持自定义标签sqoop.这里我在Oozie中需要使用Sqoop从DB中导入数据,所以在workflow.xml中进行了如下配置:
? ?<action name="db_export">
<sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${job_tracker}</job-tracker> <name-node>${name_node}</name-node> <prepare> <delete path="${wf_job_base_path}/${wf:id()}/db_export"/> </prepare> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queue_name}</value> </property> </configuration> <arg>import</arg> <arg>-D</arg> <arg>mapred.output.compress=false</arg> <arg>--connect</arg> <arg>jdbc:mysql://${db_hostname}/${db_name}</arg> <arg>--query</arg> <arg>${db_banner_query}</arg> <arg>--target-dir</arg> <arg>${wf_job_base_path}/${wf:id()}/db_export</arg> <arg>--num-mappers</arg> <arg>${sqoop_export_mappers_num}</arg> <arg>--username</arg> <arg>${db_user}</arg> <arg>--password</arg> <arg>${db_password}</arg> <arg>--as-sequencefile</arg> <arg>--class-name</arg> <arg>${db_class_name}</arg> </sqoop> <ok to="sqoop_export_done"/> <error to="fail"/> </action>? ?常识告诉我,自定义标签需要进行相关命名.看这边sqoop引入的是uri:oozie:sqoop-action:0.2,所以应该要有oozie-sqoop-action-0.2.xsd文件.于是去$OOZIE_HOME/lib/oozie-clinet-x.x.jar中
并未看到该文件.于是按照http://archive.cloudera.com/cdh/3/oozie/DG_SqoopActionExtension.html的配置新建该文件并放入该jar中.继续运行,但还是报错.同时随着对自定义标签的了解以及看了这篇文章后http://www.infoq.com/cn/articles/ExtendingOozie,我发现需要相关的ActionSuppoert支持,而在oozie-core中并未找到org.apache.oozie.action.hadoop.SqoopActionExecutor这个类.然后通过搜索才发现,之前是下载yahoo的oozie,而只有
cloudera才对oozie支持使用sqoop.于是去cloudera下载了oozie-2.3-cdhu30.tar.gz,并从中找到相关的xsd和ActionSupport.再直接运行,OK,问题解决!看来用上了cloudera,以后啥都要先围着它去转了.下载地址:http://archive.cloudera.com/cdh/3/oozie-2.3.2-cdh3u2.tar.gz
?
1 楼 Merrygrass 2012-05-18 楼主请问你做过用oozie java api 调用 sqoop 任务吗？