Apache UIMA Lucene CAS Indexer Documentation

As mentioned above, the text covered by annotation objects represents one possible source for Lucene token values. The following example creates an index with one title field which contains covered texts from all token annotations which are stored in the title sofa.

<fields>
  <field name=“title” index=“yes”>
     <annotations>
         <annotation sofa=“title” type=“de.julielab.types.Token”/>
     </annotations>
   </field>
</fields>

The feature values of annotation objects are another source for token values. Consider the example below.

<fields>
  <field name=“cells” index=“yes”>
     <annotations>
         <annotation sofa=“text” type=“de.julielab.types.Cell”>
             <features>
                <feature name=“specificType”>
             </features>             
         </annotation>         
     </annotations>
   </field>
</fields>

The field cells contains a token stream generated from the annotation index of type de.julielab.types.Cell . Each generated token will contain the value of the feature specificType of the enclosing annotation object.

The next example illustrates how multiple feature values can be combined by using a featureValueDelimiterString . If no featureValueDelimiterString is provided, a single token is generated from each feature value.

<fields>
  <field name=“authors” index=“no” stored="yes">
     <annotations>
         <annotation sofa=“text” type=“de.julielab.types.Author” 
                                 featureValueDelimiterString=", ">
              <features>
                <feature name=“firstname”/>
                <feature name=“lastname”/>
             </features>             
         </annotation>         
     </annotations>
   </field>
</fields>

Since annotation objects may reference other feature structures, it may be desirable to use these feature structures as source for Lucene token values. To achieve this, we utilize feature paths to address these feature structures. Consider the example below.

<fields>
  <field name=“cities” index=“yes”>
     <annotations>
         <annotation sofa=“text” type=“de.julielab.types.Author” 
                                 featurePath="affiliation.address">
             <features>
                <feature name=“city”>
             </features>             
         </annotation>         
     </annotations>
   </field>
</fields>

The type de.julielab.types.Author has a feature affiliation which points to a affiliation feature structure. This affiliation feature structure in turn has a feature address which points to an address feature structure. This path of references is expressed as the feature path affiliation.address . A feature path consists of feature names separated by colons ("."). Please consider that the city feature is a feature of the "address" feature structure and not of the author annotation object.

Currently, not all feature types are supported. Supported feature types are the following:

Consider that you need to provide a number format string if you want to use number types.

For scenarios where the built in token filters where not sufficient, you can provide your own token filters. Simple token filters which do not need any further parameterization, are required to define a public constructor, which takes a token stream as the only parameter. The next example shows how a such a token filter is referenced in the mapping file.

<fields>
  <field name=“text” index=“yes”>
	<annotations>
		<annotation sofa=“text” type=“de.julielab.types.Cell”>
			<filters>
				<filter className="org.example.MyFilter"/>
			</filters>
			<features>
				<feature name=“specificType”>
			</features>
		</annotation>
	</annotations>
  </field>
</fields>

The attribute className must reference the canonical class name of the the filter. In cases where the token filter has parameters we need to provide a factory for it. This factory must implement the org.apache.uima.indexer.analysis.TokenFilterFactory interface. This interface defines a method createTokenFilter which takes a token stream and a java.util.Properties object as parameters. The properties object will include all attribute names as keys and their values which are additionally defined in the filter tag. Consider the example below for a demonstration.

<fields>
  <field name=“text” index=“yes”>
	<annotations>
		<annotation sofa=“text” type=“de.julielab.types.Cell”>
			<filters>
				<filter factoryClassName="org.example.MyTokenFilterFactory" 
		   	            parameter1="value1" parameter2="value2"/>
			</filters>
			<features>
				<feature name=“specificType”>
			</features>
		</annotation>
	</annotations>
  </field>
</fields>

In the example above the token filter factory is new instantiated for every occurrence in the mapping file. In scenarios where token filters use large resources, this will be a waste of memory and time. To reuse a factory instance we need to provide a name and a reuse attribute. The example below demonstrate how we can reuse a factory instance.

<fields>
  <field name=“text” index=“yes”>
	<annotations>
		<annotation sofa=“text” type=“de.julielab.types.Cell”>
			<filters>
				<filter factoryClassName="org.example.MyTokenFilterFactory" 
		   	            name="myFactory" reuse="true" 
		   	            myResourceFilePath="pathToResource"/>
			</filters>
			<features>
				<feature name=“specificType”>
			</features>
		</annotation>
     </annotations>
  </field>
</fields>

Adds suffixes or prefixes to tokens.

<filter name="addition" prefix="PRE_"/>

Adds hypernyms of a token with the same offset and position increment 0.

<filter name="hypernyms" filePath="/path/to/myFile.txt"/>

Allows to select only the first or the last token of a token stream, all other tokens are discarded.

<filter name="position" position="last"/>

Allows to replace token texts.

<filter name="replace" filePath="/path/to/myFile.txt"/>

Integration of the Lucene snowball filter

<filter name="snowball" stemmerName="German"/>

Splits tokens at a certain string.

<filter name="splitter" splitString=","/>

Concatenates token texts with a certain delimiter string.

<filter name="concatenate" concatString=";"/>

Integration of the Lucene stop filter

<filter name="stopwords" filePath="/path/to/myStopwords.txt"/>

Filters tokens with the same token text. The resulting token stream contains only tokens with unique texts.

<filter name="unique"/>

Turns the text of each token into upper case.

<filter name="uppercase"/>

Turns the text of each token into lower case.

<filter name="lowercase"/>