lucene基本
Lucene索引中有几个最基础的概念,索引(index),文档(document),域(field),和项(或者译为语词term)?字串4?
其中Index为Document的序列?
字串6
?????Document为Field的序列?字串8?
?????Field为Term的序列?
字串7
?????Term就是一个子串.?字串7?
存在于不同的Field中的同一个子串被认为是不同的Term.因此Term实际上是用一对子串表示的,第一个子串为Field的name,第二个为Field中的子串.既然Term这么重要,我们先来认识一下Term.?
字串5
认识Term?字??
最好的方法就是看其源码表示.?
字串7
public?final?class?Term?implements?Comparable,?java.io.Serializable?{?字串8?
?String?field;?字串8?
?String?text;?字串1?
?public?Term(String?fld,?String?txt)?{this(fld,?txt,?true);}?
字串6
?public?final?String?field()?{?return?field;?}?
字串8
?public?final?String?text()?{?return?text;?}?
字串4
//overwrite?equals()?
字串9
?public?final?boolean?equals(Object?o)?{?}?
字串3
//overwrite?hashCode()?
字串5
?public?final?int?hashCode()?{return?field.hashCode()?+?text.hashCode();?字串5?
?}?字串3?
?
?
?
?public?int?compareTo(Object?other)?{return?compareTo((Term)other);}?字串5?
?public?final?int?compareTo(Term?other)?字串8?
?final?void?set(String?fld,?String?txt)?public?final?String?toString()?{?return?field?+?":"?+?text;?}?
字串6
?private?void?readObject(java.io.ObjectInputStream?in)?
字串7
?}?字串4?
从代码中我们可以大体看出Tern其实是一个二元组<FieldName,text>?字串8?
倒排索引
为了使得基于项的搜索更有效率，索引中项是静态存储的。Lucene的索引属于索引方式中的倒排索引，因为对于一个项这种索引可以列出包含它的文档。这刚好是文档与项自然联系的倒置。?字串5?
Field的类型?
Lucene中，Field的文本可能以逐字的非倒排的方式存储在索引中。而倒排过的Field称为被索引过了。Field也可能同时被存储和被索引。Field的文本可能被分解许多Term而被索引，或者就被用作一个Term而被索引。大多数的Field是被分解过的，但是有些时候某些标识符域被当做一个Term索引是很有用的。?字串2?
Index包中的每个类解析?字串3?
CompoundFileReader?
字串4
???????提供读取.cfs文件的方法.?字串9?
CompoundFileWriter?
字串3
???????用来构建.cfs文件,从Lucene1.4开始,会将下面要提到的各类文件,譬如.tii,.tis等合并成一个.cfs文件!?
字串4
???????其结构如下?
字串1
Compound?(.cfs)?-->?FileCount,?<DataOffset,?FileName>FileCount,?FileDataFileCount?
字串1
FileCount?-->?VInt?
字串2
DataOffset?-->?Long?字串6?
FileName?-->?String?
字串2
FileData?-->?raw?file?data?字串9?
DocumentWriter?
字串4
?????构建.frq,.prx,.f文件????
字串7
1．FreqFile?(.frq)?-->?<TermFreqs,?SkipData>TermCount?
字串9
TermFreqs?-->?<TermFreq>DocFreq?
字串4
TermFreq?-->?DocDelta,?Freq??字串2?
SkipData?-->?<SkipDatum>DocFreq/SkipInterval?字串9?
SkipDatum?-->?DocSkip,FreqSkip,ProxSkip?字串4?
DocDelta,Freq,DocSkip,FreqSkip,ProxSkip?-->?VInt?
字串5
?
?
?
?
字串5
??
字串9
??
字串6
2．The?.prx?file?contains?the?lists?of?positions?that?each?term?occurs?at?within?documents.?字串3?
ProxFile?(.prx)?-->?<TermPositions>TermCount?
字串3
TermPositions?-->?<Positions>DocFreq?字串8?
Positions?-->?<PositionDelta>Freq?字串5?
PositionDelta?-->?VInt??
字串7
?
字串7?
?
?
?
3．There′s?a?norm?file?for?each?indexed?field?with?a?byte?for?each?document.?The?.f[0-9]*?file?contains,?for?each?document,?a?byte?that?encodes?a?value?that?is?multiplied?into?the?score?for?hits?on?that?field:?字串4?
Norms?(.f[0-9]*)?-->?<Byte>SegSize?
字串3
Each?byte?encodes?a?floating?point?value.?Bits?0-2?contain?the?3-bit?mantissa,?and?bits?3-8?contain?the?5-bit?exponent.?字串3?
These?are?converted?to?an?IEEE?single?float?value?as?follows:?
字串3
1.????If?the?byte?is?zero,?use?a?zero?float.?
字串1
2.????Otherwise,?set?the?sign?bit?of?the?float?to?zero;?字串7?
3.????add?48?to?the?exponent?and?use?this?as?the?float′s?exponent;?字串4?
4.????map?the?mantissa?to?the?high-order?3?bits?of?the?float′s?mantissa;?and?字串7?
5.????set?the?low-order?21?bits?of?the?float′s?mantissa?to?zero.?字串6?
?
?
?
FieldInfo?
字串3
??????里边有Field的部分信息,是一个四元组<name,isIndexed,num,?storeTermVector>?字串1?
FieldInfos?字串5?
?????此类用来描述Document的fields是否被索引.每个Segment有一个单独的FieldInfo?文件.对于多线程,此类的对象为线程安全的.但是某一时刻,只允许一个线程添加document.别的reader和writer不允许进入.此类维护两个容器ArrayList和HashMap,这两个容器都不是synchronized,何言线程安全,不解???字串4?
观察write函数可知?.fnm文件的构成为?字串1?
?????FieldInfos?(.fnm)?-->?FieldsCount,?<FieldName,?FieldBits>FieldsCount?
????????????????????????FieldsCount?-->?VInt?
字串8
????????????????????????FieldName?-->?String?
字串7
????????????????????????FieldBits?-->?Byte?
字串3
FieldReader?
字串6
????用来读取.fdx文件和.fdt文件?
字串4
FieldWriter?字串3?
?????此类创建两个文件.fdx和.fdt文件?
字串1
?????FieldIndex(.fdx)对于每一个Document,里面都含有一个指向Field的指针(其实是整数)?字串1?
<FieldValuesPosition>SegSize?字串3?
FieldValuesPosition?-->?Uint64?字串4?
?????????????则第n个document的Field?pointer为n*8?字串9?
????FieldData(.fdt)里面包含了每一个文档包含的存储的field信息.内容如下:?字串9?
<DocFieldData>SegSize?字串1?
DocFieldData?-->?FieldCount,?<FieldNum,?Bits,?Value>FieldCount?
字串9
FieldCount?-->?VInt?字串6?
FieldNum?-->?VInt?字串2?
Lucene?<=?1.4:?
字串9
Bits?-->?Byte?字串7?
Value?-->?String?字串1?
Only?the?low-order?bit?of?Bits?is?used.?It?is?one?for?tokenized?fields,?and?zero?for?non-tokenized?fields.?字串8?
FilterIndexReader?
字串2
?????扩展自IndexReader,提供了具体的方法.?字串9?
IndexReader?字串7?
?????为abstract?class!用来读取建完索引的Directory,并可以返回各种信息,譬如Term,TermPosition等等.?字串7?
IndexWriter?字串4?
????IndexWriter用来创建和维护索引。?字串4?
???IndexWriter构造函数中的第三个参数决定一个新的索引是否被创建，或者一个存在的索引是否开放给欲新加入的新的document?字串3?
???通过addDocument()0函数加入新的documents，当添加完document之后，调用close()函数?
字串1
???如果一个Index没有document需要加入并且需要优化查询性能。则在索引close()之前，调用optimize()函数进行优化。?
字串5
????Deleteable文件结构：?
字串8
????A?file?named?"deletable"?contains?the?names?of?files?that?are?no?longer?used?by?the?index,?but?which?could?not?be?deleted.?This?is?only?used?on?Win32,?where?a?file?may?not?be?deleted?while?it?is?still?open.?On?other?platforms?the?file?contains?only?null?bytes.?字串4?
Deletable?-->?DeletableCount,?<DelableName>DeletableCount?
字串7
DeletableCount?-->?UInt32?字串6?
DeletableName?-->?String?字串1?
MultipleTermPositions?
字串2
专门用于search包中的PhrasePrefixQuery?字串2?
MultiReader?字串8?
扩展自IndexReader，用来读取多个索引！添加他们的内容?
字串4
SegmentInfo?
字串1
?????一些关于Segment的信息,是一个三元组<segmentname,docCount,dir>?
字串8
SegmentInfos?字串6?
?????扩展自Vector,就是一个向量组,其中任意成员为SegmentInfo!用来构建segments文件,每个Index有且只有一个这样的文件,此类提供了read和write的方法.?
字串2
?????其内容如下:?
字串5
?????Segments?-->?Format,?Version,?NameCounter,?SegCount,?<SegName,?SegSize>SegCount?
字串2
Format,?NameCounter,?SegCount,?SegSize?-->?UInt32?字串1?
Version?-->?UInt64?字串1?
SegName?-->?String?字串1?
Format?is?-1?in?Lucene?1.4.?字串2?
Version?counts?how?often?the?index?has?been?changed?by?adding?or?deleting?documents.?
字串5
NameCounter?is?used?to?generate?names?for?new?segment?files.?字串4?
SegName?is?the?name?of?the?segment,?and?is?used?as?the?file?name?prefix?for?all?of?the?files?that?compose?the?segment′s?index.?
字串8
SegSize?is?the?number?of?documents?contained?in?the?segment?index.?
字串1
?
?
?
SegmentMergeInfo?
字串9
????用来记录segment合并信息.?字串5?
SegmentMergeQueue?
字串5
????扩展自PriorityQueue(按升序排列)?
字串7
SegmentMerger?字串2?
此类合并多个Segment为一个Segment，被IndexWriter.addIndexes()创建此类对象?字串8?
如果compoundFile为True即可以合并了,创建.cfs文件，并且把其余的几乎所有文件全部合并到.cfs文件中！?
字串2
SegmentReader?字串3?
扩展自IndexReader，提供了很多读取Index的方法?
字串1
SegmentTermDocs?
字串7
扩展自TermDocs?
字串8
SegmentTermEnum?
字串8
??扩展自TermEnum?
字串3
SegmentTermPositions?
字串9
???扩展自TermPositions?
字串8
SegmentTermVector?字串7?
?扩展自TermFreqVector?
字串3
Term?
字串9
?????Term是一个<fieldName,text>对.而Field由于分多种,但是至少都含有<fieldName,fieldValue>这样二者就可以建立关联了.Term是一个搜索单元.Term的text都是诸如dates,email?address,urls等等.?字串7?
TermDocs?字串1?
?????TermDocs是一个Interface.?TermDocs提供一个接口,用来列举<document,frequency>,以共Term使用?字串4?
?????在<document,frequency>对中,document部分给每一个含有term的document命名.document根据其document?number进行标引.frequency部分列举在每一个document中term的数量.<document,frequency>对根据document?number排序.?
字串2
TermEnum?字串3?
?????此类为抽象类,用来enumerate?term.Term?enumerations?由Term.compareTo()进行排序此enumeration中的每一个term都要大于所有在此enumeration之前的term.?字串6?
TermFreqVector?字串8?
?????此Interface用来访问一个document的Field的Term?Vector?
字串6
TermInfo?字串7?
?????此类主要用来存储Term信息.其可以说为一个五元组<Term,docFreq,freqPointer,proxPointer,skipOffset>?字串4?
TermInfoReader?字串9?
?????未细读,待读完SegmentTermEnum?字串7?
TermInfoWriter?
字串9
?????此类用来构建(.tis)和(.tii)文件.这些构成了term?dictionary?
字串9
1.?????The?term?infos,?or?tis?file.?
字串6
TermInfoFile?(.tis)-->?TIVersion,?TermCount,?IndexInterval,?SkipInterval,?TermInfos?字串4?
TIVersion?-->?UInt32?
字串5
TermCount?-->?UInt64?字串1?
IndexInterval?-->?UInt32?
字串9
SkipInterval?-->?UInt32?字串4?
TermInfos?-->?<TermInfo>TermCount?
字串2
TermInfo?-->?<Term,?DocFreq,?FreqDelta,?ProxDelta,?SkipDelta>?字串3?
Term?-->?<PrefixLength,?Suffix,?FieldNum>?字串5?
Suffix?-->?String?字串2?
PrefixLength,?DocFreq,FreqDelta,?ProxDelta,?SkipDelta
-->?VInt?
字串2
This?file?is?sorted?by?Term.?Terms?are?ordered?first?lexicographically?by?the?term′s?field?name,?and?within?that?lexicographically?by?the?term′s?text.?
字串3
TIVersion?names?the?version?of?the?format?of?this?file?and?is?-2?in?Lucene?1.4.?
字串8
Term?text?prefixes?are?shared.?The?PrefixLength?is?the?number?of?initial?characters?from?the?previous?term?which?must?be?pre-pended?to?a?term′s?suffix?in?order?to?form?the?term′s?text.?Thus,?if?the?previous?term′s?text?was?"bone"?and?the?term?is?"boy",?the?PrefixLength?is?two?and?the?suffix?is?"y".?
字串8
FieldNumber?determines?the?term′s?field,?whose?name?is?stored?in?the?.fdt?file.?
字串6
DocFreq?is?the?count?of?documents?which?contain?the?term.?
字串4
FreqDelta?determines?the?position?of?this?term′s?TermFreqs?within?the?.frq?file.?In?particular,?it?is?the?difference?between?the?position?of?this?term′s?data?in?that?file?and?the?position?of?the?previous?term′s?data?(or?zero,?for?the?first?term?in?the?file).?字串5?
ProxDelta?determines?the?position?of?this?term′s?TermPositions?within?the?.prx?file.?In?particular,?it?is?the?difference?between?the?position?of?this?term′s?data?in?that?file?and?the?position?of?the?previous?term′s?data?(or?zero,?for?the?first?term?in?the?file.?字串6?
SkipDelta?determines?the?position?of?this?term′s?SkipData?within?the?.frq?file.?In?particular,?it?is?the?number?of?bytes?after?TermFreqs?that?the?SkipData?starts.?In?other?words,?it?is?the?length?of?the?TermFreq?data.?字串1?
2.?????The?term?info?index,?or?.tii?file.?
字串8
This?contains?every?IndexIntervalth?entry?from?the?.tis?file,?along?with?its?location?in?the?"tis"?file.?This?is?designed?to?be?read?entirely?into?memory?and?used?to?provide?random?access?to?the?"tis"?file.?
字串8
The?structure?of?this?file?is?very?similar?to?the?.tis?file,?with?the?addition?of?one?item?per?record,?the?IndexDelta.?字串3?
TermInfoIndex?(.tii)-->?TIVersion,?IndexTermCount,?IndexInterval,?SkipInterval,?TermIndices?
字串3
TIVersion?-->?UInt32?
字串4
IndexTermCount?-->?UInt64?字串8?
IndexInterval?-->?UInt32?字串9?
SkipInterval?-->?UInt32?
字串6
TermIndices?-->?<TermInfo,?IndexDelta>IndexTermCount?
字串9
IndexDelta?-->?VLong?
字串7
IndexDelta?determines?the?position?of?this?term′s?TermInfo?within?the?.tis?file.?In?particular,?it?is?the?difference?between?the?position?of?this?term′s?entry?in?that?file?and?the?position?of?the?previous?term′s?entry.?字串4?
TODO:?document?skipInterval?information?
字串7
?????????????其中IndexDelta是.tii文件,比之.tis文件多的东西.?
字串4
TermPosition?
字串3
??????此类扩展自TermDocs,是一个Interface,用来enumerate<document,frequency,<position>*>三元组,?
字串3
以供term使用.在此三元组中document和frequency于TernDocs中的相同.postions部分列出了在一个document中,一个term每次出现的顺序位置此三元组为倒排文档的事件表表示.?字串1?
TermPositionVector?字串4?
??????扩展自TermFreqVector.比之TermFreqVector扩展了功能,可以提供term所在的位置?
字串1
TermVectorReader?字串9?
??????用来读取.tvd,.tvf.tvx三个文件.?字串9?
TermVectorWriter?字串7?
??????用于构建.tvd,?.tvf,.tvx文件,这三个文件构成TermVector?字串4?
1.????The?Document?Index?or?.tvx?file.?字串6?
This?contains,?for?each?document,?a?pointer?to?the?document?data?in?the?Document?(.tvd)?file.?字串7?
DocumentIndex?(.tvx)?-->?TVXVersion<DocumentPosition>NumDocs?字串8?
TVXVersion?-->?Int?字串8?
DocumentPosition?-->?UInt64?
字串2
This?is?used?to?find?the?position?of?the?Document?in?the?.tvd?file.?
字串3
2.????The?Document?or?.tvd?file.?字串3?
This?contains,?for?each?document,?the?number?of?fields,?a?list?of?the?fields?with?term?vector?info?and?finally?a?list?of?pointers?to?the?field?information?in?the?.tvf?(Term?Vector?Fields)?file.?字串9?
Document?(.tvd)?-->?TVDVersion<NumFields,?FieldNums,?FieldPositions,>NumDocs?字串2?
TVDVersion?-->?Int?字串8?
NumFields?-->?VInt?
字串7
FieldNums?-->?<FieldNumDelta>NumFields?
字串5
FieldNumDelta?-->?VInt?字串8?
FieldPositions?-->?<FieldPosition>NumFields?字串2?
FieldPosition?-->?VLong?
字串1
The?.tvd?file?is?used?to?map?out?the?fields?that?have?term?vectors?stored?and?where?the?field?information?is?in?the?.tvf?file.?字串8?
3.????The?Field?or?.tvf?file.?字串5?
This?file?contains,?for?each?field?that?has?a?term?vector?stored,?a?list?of?the?terms?and?their?frequencies.?字串6?
Field?(.tvf)?-->?TVFVersion<NumTerms,?NumDistinct,?TermFreqs>NumFields?字串3?
TVFVersion?-->?Int?字串7?
NumTerms?-->?VInt?字串9?
NumDistinct?-->?VInt?--?Future?Use?
字串1
TermFreqs?-->?<TermText,?TermFreq>NumTerms?
字串7
TermText?-->?<PrefixLength,?Suffix>?字串4?
PrefixLength?-->?VInt?
字串6
Suffix?-->?String?字串5?
TermFreq?-->?VInt?
字串2
Term?text?prefixes?are?shared.?The?PrefixLength?is?the?number?of?initial?characters?from?the?previous?term?which?must?be?pre-pended?to?a?term′s?suffix?in?order?to?form?the?term′s?text.?Thus,?if?the?previous?term′s?text?was?"bone"?and?the?term?is?"boy",?the?PrefixLength?is?two?and?the?suffix?is?"y".?字串4?
?
字串8
好的，整个Index包所有类都讲解了，下边咱们开始来编码重新审视一下！?字串5?
下边来编制一个程序来结束本章的讨论。?
字串7
package?org.apache.lucene.index;?字串7?
import?org.apache.lucene.analysis.*;?字串1?
import?org.apache.lucene.analysis.standard.*;?
字串3
import?org.apache.lucene.store.*;?
字串6
import?org.apache.lucene.document.*;?字串8?
import?org.apache.lucene.demo.*;?字串9?
import?org.apache.lucene.search.*;?字串8?
import?java.io.*;?
字串2
/**在使用此程序时，会尽量用到Lucene?Index中的每一个类，尽量将其展示个大家?
字串4
?*使用的Index包中类有?字串9?
?*DocumentWriter(提供给用用户使用的为IndexWriter）?字串7?
?*FieldInfo(和FieldInfos）?字串9?
?*?SegmentDocs（扩展自TermDocs）?
字串1
?*SegmentReader（扩展自IndexReader，提供给用户使用的是IndexReader）?
字串4
?*SegmentMerger?
字串2
?*segmentTermEnum(扩展自TermEnum）?字串2?
?*segmentTermPositions(扩展自TermPositions）?
字串2
?*segmentTermVector(扩展自TermFreqVector）?字串5?
*/?字串8?
?
?
?
public?class?TestIndexPackage?
字串9
{?字串1?
?//用于将Document加入索引?字串8?
?public?static?void?indexDocument(String?segment,String?fileName)?throws?Exception?
字串8
?{?字串8?
????//第二个参数用来控制，如果获得不了目录是否创建?字串5?
????Directory?directory?=?FSDirectory.getDirectory("testIndexPackage",false);?字串3?
????Analyzer?analyzer?=?new?SimpleAnalyzer();?字串8?
????//第三个参数为每一个Field最多拥有的Token个数?
字串5
???DocumentWriter?writer?=?new?DocumentWriter(directory,analyzer,Similarity.getDefault(),1000);?字串4?
????File?file?=?new?File(fileName);?
字串7
????//由于使用FileDocument将file包装成了Docuement，会在document中创建三个field（path，modified，contents）?字串5?
????Document?doc?=?FileDocument.Document(file);?字串7?
????writer.addDocument(segment,doc);?字串9?
????directory.close();?
字串8
?}?
字串8
?//将多个segment进行合并?字串4?
?public?static?void?merge(String?segment1,String?segment2,String?segmentMerged)throws?Exception?字串8?
?{?
字串2
????Directory?directory?=?FSDirectory.getDirectory("testIndexPackage",false);?字串5?
????SegmentReader?segmentReader1=new?SegmentReader(new?SegmentInfo(segment1,1,directory));?字串6?
????SegmentReader?segmentReader2=new?SegmentReader(new?SegmentInfo(segment2,1,directory));?字串8?
????//第三个参数为是否创建.cfs文件?
字串2
????SegmentMerger?segmentMerger?=new?SegmentMerger(directory,segmentMerged,false);?
字串7
????segmentMerger.add(segmentReader1);?字串7?
????segmentMerger.add(segmentReader2);?
字串8
????segmentMerger.merge();?
字串3
????segmentMerger.closeReaders();?
字串4
????directory.close();?字串7?
?}?字串7?
?//将segment即Index的子索引的所有内容展示给你看。?字串4?
?public?static?void?printSegment(String?segment)?throws?Exception?字串8?
?{?字串2?
????Directory?directory?=FSDirectory.getDirectory("testIndexPackage",false);?
字串4
????SegmentReader?segmentReader?=?new?SegmentReader(new?SegmentInfo(segment,1,directory));?字串2?
????//display?documents?
字串8
???for(int?i=0;i<segmentReader.numDocs();i++)?
字串2
??????System.out.println(segmentReader.document(i));?字串3?
????TermEnum?termEnum?=?segmentReader.terms();//此处实际为SegmentTermEnum?
字串6
????//display?term?and?term?positions,termDocs?字串5?
????while(termEnum.next())?
字串6
????{?字串1?
??????System.out.print(termEnum.term().toString2());?字串2?
??????System.out.println("?DocumentFrequency="?+?termEnum.docFreq());?字串2?
??????TermPositions?termPositions=?segmentReader.termPositions(termEnum.term());?字串3?
??????int?i=0;?字串4?
??????while(termPositions.next())?字串5?
??????{?
字串1?
