关于lucene的分词（三）
到此为止这个简单的但是功能强大的分词器就写完了,下面咱们可以尝试写一个功能更强大的分词器.
如何DIY一个功能更加强大Analyzer
譬如你有词典,然后你根据正向最大匹配法或者逆向最大匹配法写了一个分词方法,却想在Lucene中应用,很简单
你只要把他们包装成Lucene的TokenStream就好了.下边我以调用中科院写的ICTCLAS接口为例,进行演示.你去中科院
网站可以拿到此接口的free版本,谁叫你没钱呢,有钱,你就可以购买了.哈哈
好,由于ICTCLAS进行分词之后,在Java中,中间会以两个空格隔开!too easy,我们直接使用继承Lucene的
WhiteSpaceTokenizer就好了.
所以TjuChineseTokenizer 看起来像是这样.
public class TjuChineseTokenizer extends WhitespaceTokenizer
{
public TjuChineseTokenizer(Reader readerInput)
{
??? super(readerInput);
}
}
而TjuChineseAnalyzer看起来象是这样
public final class TjuChineseAnalyzer
??? extends Analyzer
{
private Set stopWords;
?
/** An array containing some common English words that are not usually useful
??? for searching. */
/*
???? public static final String[] CHINESE_ENGLISH_STOP_WORDS =
????? {
????? "a", "an", "and", "are", "as", "at", "be", "but", "by",
????? "for", "if", "in", "into", "is", "it",
????? "no", "not", "of", "on", "or", "s", "such",
????? "t", "that", "the", "their", "then", "there", "these",
????? "they", "this", "to", "was", "will", "with",
????? "我", "我们"
???? };
?? */
/** Builds an analyzer which removes words in ENGLISH_STOP_WORDS. */
public TjuChineseAnalyzer()
{
??? stopWords = StopFilter.makeStopSet(StopWords.SMART_CHINESE_ENGLISH_STOP_WORDS);
}
?
/** Builds an analyzer which removes words in the provided array. */
//提供独自的stopwords
public TjuChineseAnalyzer(String[] stopWords)
{
??? this.stopWords = StopFilter.makeStopSet(stopWords);
}
?
/** Filters LowerCaseTokenizer with StopFilter. */
public TokenStream tokenStream(String fieldName, Reader reader)
{
??? try
??? {
????? ICTCLAS splitWord = new ICTCLAS();
????? String inputString = FileIO.readerToString(reader);
????? //分词中间加入了空格
????? String resultString = splitWord.paragraphProcess(inputString);
????? System.out.println(resultString);
????? TokenStream result = new TjuChineseTokenizer(new StringReader(resultString));
?
????? result = new LowerCaseFilter(result);
????? //使用stopWords进行过滤
???? result = new StopFilter(result, stopWords);
????? //使用p-stemming算法进行过滤
???? result = new PorterStemFilter(result);
????? return result;
?
??? }
??? catch (IOException e)
??? {
????? System.out.println("转换出错");
????? return null;
??? }
}
?
public static void main(String[] args)
{
??? String string = "我爱中国人民";
??? Analyzer analyzer = new TjuChineseAnalyzer();
??? TokenStream ts = analyzer.tokenStream("dummy", new StringReader(string));
??? Token token;
??? System.out.println("Tokens:");
??? try
??? {
????? int n=0;
????? while ( (token = ts.next()) != null)
????? {
??????? System.out.println((n++)+"->"+token.toString());
????? }
??? }
??? catch (IOException ioe)
??? {
???? ioe.printStackTrace();
??? }
}
}
对于此程序的输出接口可以看一下
0->Token's (termText,startOffset,endOffset,type,positionIncrement) is:(爱,3,4,word,1)
1->Token's (termText,startOffset,endOffset,type,positionIncrement) is:(中国,6,8,word,1)
2->Token's (termText,startOffset,endOffset,type,positionIncrement) is:(人民,10,12,word,1)
?
OK,经过这样一番讲解,你已经对Lucene的Analysis包认识的比较好了,当然如果你想更加了解,还是认真读读源码才好,
呵呵,源码说明一切!
