Apache UIMA™ - References

You can also create a named "user library" in Eclipse containing the UIMA Jars, and attach the Javadocs (or optionally, the sources); this named library is saved in the Eclipse workspace. Once created, it can be added to the classpath of newly created Eclipse projects.

Use the menu option Project → Properties → Java Build Path, and then pick the Libraries_- tab, and click the Add Library button. Then select User Libraries, click Next__, and pick the library you created for the UIMA Jars.

To create this library in the workspace, use the same menu picks as above, but after you select the User Libraries and click "Next", you can click the "New Library…​" button to define your new library. You use the "Add Jars" button and multi-select all the Jars in the lib directory of the UIMA binary distribution. Then you add the Javadoc attachment for each Jar. The path to use is file:/ — insert the path to your install of UIMA — /docs/api. After you do this for the first Jar, you can copy this string to the clipboard and paste it into the rest of the Jars.

This chapter uses an informal notation to specify the syntax of Component Descriptors. The formal syntax is defined by an XML schema definition, which is contained in the file resourceSpecifierSchema.xsd, located in the uima-core.jar file.

The notation used in this chapter is:

The UIMA SDK defines a particular syntax for XML descriptors to import information from other XML files. When one of the following appears in an XML descriptor:

it indicates that information from a separate XML file is being imported. Note that imports are allowed only in certain places in the descriptor. In the remainder of this chapter, it will be indicated at which points imports are allowed.

If an import specifies a location attribute, the value of that attribute specifies the URL at which the XML file to import will be found. This can be a relative URL, which will be resolved relative to the descriptor containing the import element, or an absolute URL. Relative URLs can be written without a protocol/scheme (e.g., “file:”), and without a host machine name. In this case the relative URL might look something like org/apache/myproj/MyTypeSystem.xml.

An absolute URL is written with one of the following prefixes, followed by a path such as org/apache/myproj/MyTypeSystem.xml:

For more information about URLs, please read the javadoc information for the Java class “URL”.

If an import specifies a name attribute, the value of that attribute should take the form of a Java-style dotted name (e.g. org.apache.myproj.MyTypeSystem). An .xml file with this name will be searched for in the classpath or datapath (described below). As in Java, the dots in the name will be converted to file path separators. So an import specifying the example name in this paragraph will result in a search for org/apache/myproj/MyTypeSystem.xml in the classpath or datapath.

The datapath works similarly to the classpath but can be set programmatically through the resource manager API. Application developers can specify a datapath during initialization, using the following code:

The default datapath for the entire JVM can be set via the uima.datapath Java system property, but this feature should only be used for standalone applications that don’t need to run in the same JVM as other code that may need a different datapath.

The value of a name or location attribute may be parameterized with references to external override variables using the ${variable-name} syntax.

If a variable is undefined the value is left unmodified and a warning message identifies the missing variable.

Previous versions of UIMA also supported XInclude. That support didn’t work in many situations, and it is no longer supported. To include other files, please use <import>.

A Type System Descriptor is used to define the types and features that can be represented in the CAS. A Type System Descriptor can be imported into an Analysis Engine or Collection Processing Component Descriptor.

The basic structure of a Type System Descriptor is as follows:

All of the subelements are optional.

Analysis Engine (AE) descriptors completely describe Analysis Engines. There are two basic types of Analysis Engines — Primitive and Aggregate. A Primitive Analysis Engine is a container for a single annotator, where as an Aggregate Analysis Engine is composed of a collection of other Analysis Engines. (For more information on this and other terminology, see the Conceptual Overview.

Both Primitive and Aggregate Analysis Engines have descriptors, and the two types of descriptors have some similarities and some differences. Section 2.4.1 discusses Primitive Analysis Engine descriptors. Section 2.4.2 then describes how Aggregate Analysis Engine descriptors are different.

The basic structure of a Flow Controller Descriptor is as follows:

The frameworkImplementation element must always be set to the value org.apache.uima.java.

The implementationName element must contain the fully-qualified class name of the Flow Controller implementation. This must name a class that implements the FlowController interface.

The processingResourceMetaData element contains essentially the same information as a Primitive Analysis Engine Descriptor’s analysisEngineMetaData element, described in Section 2.4.1.2.

The externalResourceDependencies and resourceManagerConfiguration elements are exactly the same as in Primitive Analysis Engine Descriptors (see Section 2.4.1.8 and Section 2.4.1.9).

There are three types of Collection Processing Components –Collection Readers, CAS Initializers (deprecated as of UIMA Version 2), and CAS Consumers. Each type of component has a corresponding descriptor. The structure of these descriptors is very similar to that of primitive Analysis Engine Descriptors.

Service Client Descriptors specify only a location of a remote service. They are therefore much simpler in structure. In the UIMA SDK, a Service Client Descriptor that refers to a valid Analysis Engine or CAS Consumer service can be used in place of the actual Analysis Engine or CAS Consumer Descriptor. The UIMA SDK will handle the details of calling the remote service. (For details on deploying an Analysis Engine or CAS Consumer as a service, see Working with Remote Services.

The UIMA SDK is extensible to support different types of remote services. In future versions, there may be different variations of service client descriptors that cater to different types of services. For now, the only type of service client descriptor is the uriSpecifier, which supports the Vinci protocol.

The resourceType element is required for new descriptors, but is currently allowed to be omitted for backward compatibility. It specifies the type of component (Analysis Engine or CAS Consumer) that is implemented by the service endpoint described by this descriptor.

The uri element contains the URI for the web service. (Note that in the case of Vinci, this will be the service name, which is looked up in the Vinci Naming Service.)

The protocol element may be set to Vinci; other protocols may be added later. These specify the particular data transport format that will be used.

The timeout element is optional. If present, it specifies the number of milliseconds to wait for a request to be processed before an exception is thrown. A value of zero or less will wait forever. If no timeout is specified, a default value (currently 60 seconds) will be used.

The parameters element is optional. If present, it can specify values for each of the following:

If the VNS_HOST and VNS_PORT are not specified in the descriptor, the values used for these comes from parameters passed on the Java command line using the -DVNS_HOST=<host> and/or -DVNS_PORT=<port> system arguments. If not present, and a system argument is also not present, the values for these default to localhost for the VNS_HOST and 9000 for the VNS_PORT.

For details on how to deploy and call Analysis Engine and CAS Consumer services, see Working with Remote Services.

A Custom Resource Specifier allows you to plug in your own Java class as a UIMA Resource. For example you can support a new service protocol by plugging in a Java class that implements the UIMA AnalysisEngine interface and communicates with the remote service.

A Custom Resource Specifier has the following format:

The resourceClassName element must contain the fully-qualified name of a Java class that can be found in the classpath (including the UIMA extension classpath, if you have specified one using the ResourceManager.setExtensionClassPath method). This class must implement the UIMA Resource interface.

When an application calls the UIMAFramework.produceResource method and passes a CustomResourceSpecifier, the UIMA framework will load the named class and call its initialize(ResourceSpecifier,Map) method, passing the CustomResourceSpecifier as the first argument. Your class can override the initialize method and use the CustomResourceSpecifier API to get access to the parameter names and values specified in the XML.

If you are using a custom resource specifier to plug in a class that implements a new service protocol, your class must also implement the AnalysisEngine interface. Generally it should also extend AnalysisEngineImplBase. The key methods that should be implemented are getMetaData, processAndOutputNewCASes, collectionProcessComplete, and destroy.

An illustration of the CPE runtime is shown in Figure 1. Some of the CPE components, such as the queues and processing pipelines, are internal to the CPE, but their behavior and deployment may be configured using the CPE Descriptor. Other CPE components, such as the Collection Reader and CAS Processors, are defined and configured externally from the CPE and then plugged in to the CPE to create the overall engine. The parts of a CPE are:

CPE Descriptors are XML files. This chapter uses an informal notation to specify the syntax of CPE Descriptors.

The notation used in this chapter is:

Which elements are optional and which are required is specified in prose, not in the syntax definition.

As of version 2.2, a CPE Descriptor can use the same import mechanism as other component descriptors. This allows referring to component descriptors using either relative paths (resolved relative to the location of the CPE descriptor) or the classpath/datapath.

The follwing older syntax is still supported, but not recommended:

The [URL or File] attribute is a URL or a filename for the descriptor of the incorporated component. The argument is first attempted to be resolved as a URL.

Relative paths in an include are resolved relative to the current working directory (NOT the CPE descriptor location as is the case for import). A filename relative to another directory can be specified using the CPM_HOME variable, e.g.,

java -DCPM_HOME="C:/Program Files/apache/uima/cpm" …​

<?xml version="1.0"?> <cpeDescription> <collectionReader> …​ </collectionReader> <casProcessors> …​ </casProcessors> <cpeConfig> …​ </cpeConfig> <resourceManagerConfiguration> …​ </resourceManagerConfiguration> </cpeDescription>

<collectionReader> <collectionIterator> <descriptor> <import …​> | <include …​/> </descriptor> <configurationParameterSettings>…​</configurationParameterSettings> <sofaNameMappings>…​</sofaNameMappings> </collectionIterator> </collectionReader>

<casProcessors dropCasOnException="true|false" casPoolSize="[Number]" processingUnitThreadCount="[Number]">

<casProcessor deployment="local|remote|integrated" name="[String]" > <descriptor> <import …​> | <include …​/> </descriptor> <configurationParameterSettings>…​</configurationParameterSettings> <sofaNameMappings>…​</sofaNameMappings> <runInSeparateProcess>…​</runInSeparateProcess> <deploymentParameters>…​</deploymentParameters> <filter/> <errorHandling>…​</errorHandling> <checkpoint batch="Number"/> </casProcessor>

<configurationParameterSettings> <nameValuePair> <name>CivilianTitles</name> <value> <array> <string>Mr.</string> <string>Ms.</string> <string>Mrs.</string> <string>Dr.</string> </array> </value> </nameValuePair> …​ </configurationParameterSettings>

<sofaNameMappings> <sofaNameMapping cpeSofaName="a_CPE_name" componentSofaName="a_component_Name"/> …​ </sofaNameMappings>

<runInSeparateProcess> <exec dir="[String]" executable="[String]"> <env key="[String]" value ="[String]"/> …​ <arg>[String]</arg> …​ </exec> </runInSeparateProcess>

<env key="CLASSPATH" value="C:Javalib"/>

<runInSeparateProcess> <exec executable="java"> <arg>-DVNS_HOST=localhost</arg> <arg>-DVNS_PORT=9099</arg> <arg>org.apache.uima.reference_impl.analysis_engine.service. vinci.VinciAnalysisEngineService_impl</arg> <arg>C:uimadescdeployCasProcessor.xml</arg> </exec> <runInSeparateProcess>

java -DVNS_HOST=localhost -DVNS_PORT=9099 org.apache.uima.reference_impl.analysis_engine.service.vinci.\\ VinciAnalysisEngineService_impl C:uimadescdeployCasProcessor.xml

<arg>${descriptor}</arg>

<deploymentParameters> <parameter name="[String]" value="…​" type="string|integer" /> …​ </deploymentParameters>

<deploymentParameters> <parameter name="service-access" value="exclusive" type="string"/> </deploymentParameters>

<errorHandling> <maxConsecutiveRestarts action="continue|disable|terminate" value="[Number]"/> <errorRateThreshold action="continue|disable|terminate" value="[Rate]"/> <timeout max="[Number]"/> </errorHandling>

<errorRateThreshold value="3/1000" action="disable"/>

<checkpoint batch="1000">

<cpeConfig> <startAt>[NumberOrID]</startAt>

</cpeConfig>

<outputQueue dequeueTimeout="100000" queueClass= "org.apache.uima.collection.impl.cpm.engine.SequencedQueue"/> ---- where the mandatory queueClass attribute defines the name of the class and the second mandatory attribute, dequeueTimeout specifies the maximum number of milliseconds to wait for the expected chunk.

If the chunk doesn’t arrive in the configured time window, the entire document is presumed to be invalid and the CAS is dropped from further processing. This action occurs regardless of any other error action specification. The SequencedQueue invalidate the document, adding the offending document’s metadata to a local cache of invalid documents.

If the time out occurs, the CPM notifies all registered listeners by calling entityProcessComplete(). As part of this call, the SequencedQueue will pass null instead of a CAS as the first argument, and a special exception — CPMChunkTimeoutException. The reason for passing null as the first argument is because the time out occurs due to the fact that the chunk has not been received in the configured timeout window, so there is no CAS available when the timeout event occurs.

The CPMChunkTimeoutException object includes an API that allows the listener to retrieve the offending document id as well as the other metadata attributes as defined above. These attributes are part of each chunk’s metadata and are added by the Collection Reader.

Each chunk that SequencedQueue works on is subjected to a test to determine if the chunk belongs to an invalid document. This test checks the chunk’s metadata against the data in the local cache. If there is a match, the chunk is dropped. This check is only performed for chunks and complete documents are not subject to this check.

If there is an exception during the processing of a chunk, the CPM sends a notification to all registered listeners. The notification includes the CAS and an exception. When the listener notification is completed, the CPM also sends separate notifications, containing the CAS, to the Artifact Producer and the SequencedQueue. The intent is to stop adding new chunks to the Work Queue that belong to an invalid document and also to deal with chunks that are en-route, being processed by the processing threads.

In response to the notification, the Artifact Producer will drop and release back to the CAS Pool all CASes that belong to an “invalid” document. Currently, there is no support in the CollectionReader’s API to tell it to stop generating chunks. The CollectionReader keeps producing the chunks but the Artifact Producer immediately drops/releases them to the CAS Pool. Before the CAS is released back to the CAS Pool, the Artifact Producer sends notification to all registered listeners. This notification includes the CAS and an exception — SkipCasException.

In response to the notification of an exception involving a chunk, the SequencedQueue retrieves from the CAS the metadata and adds it to its local cache of invalid documents. All chunks de-queued from the OutputQueue and belonging to invalid documents will be dropped and released back to the CAS Pool. Before dropping the CAS, the CPM sends notification to all registered listeners. The notification includes the CAS and SkipCasException.

The <checkpoint> element is an optional element. It specifies a CPE checkpoint file, checkpoint frequency, and strategy for checkpoints (time or count based). At checkpoint time, the CPM saves status information and statistics to the checkpoint file. The checkpoint file is specified in the file attribute, which has the same form as the href attribute of the <include> element described in Section 3.3. The time attribute indicates that a checkpoint should be taken every [Number] seconds, and the batch attribute indicates that a checkpoint should be taken every [Number] batches.

The <timerImpl> element is optional. It is used to identify a custom timer plug-in class to generate time stamps during the CPM execution. The value of the element is a Java class name.

The <deployAs> element indicates the type of CPM deployment. Valid contents for this element include:

External resource bindings for the CPE may optionally be specified in an element:

In the resourceManagerConfiguration element, the value of the href attribute refers to another file that contains definitions and bindings for the external resources used by the CPE. The format of this file is the same as for Aggregate Analysis Engines. For example, in a CPE containing an aggregate analysis engine with two annotators, and a CAS Consumer, the following resource manager configuration file would bind external resource dependencies in all three components to the same physical resource:

In this example, MyAE and MyCasConsumer are the names of the Analysis Engine and CAS Consumer, as specified by the name attributes of the CPE’s <casProcessor> elements. annotator1 and annotator2 are the annotator keys specified within the Aggregate AE Descriptor, and myResourceKey, someResourceKey, and otherResourceKey are the keys of the resource dependencies declared in the individual annotator and CAS Consumer descriptors.

The subdirectory docs/api contains the documentation details of all the classes, methods, and constants for the APIs discussed here. Please refer to this for details on the methods, classes and constants, specifically in the packages org.apache.uima.cas.*.

There are three^[4] main parts to the CAS: the type system, data creation and manipulation, and indexing. We will start with a brief description of these components.

The CAS has two kinds of built-in types –primitive and non-primitive. The primitive types are:

The Byte, Short, Integer, and Long are all signed integer types, of length 8, 16, 32, and 64 bits. The Double type is 64 bit floating point. The String type can be subtyped to create sets of allowed values; see String Subtypes. These types can be used to specify the range of a String-valued feature. They act like Strings, but have additional checking to insure the setting of values into them conforms to one of the allowed values, or to null (which is the value if it is not set). Note that the other primitive types cannot be used as a supertype for another type definition; only uima.cas.String can be sub-typed.

The non-primitive types exist in a type hierarchy; the top of the hierarchy is the type uima.cas.TOP. All other non-primitive types inherit from some supertype.

There are 9 built-in array types. These arrays have a size specified when they are created; the size is fixed at creation time. They are named:

The uima.cas.FSArray type is an array whose elements are arbitrary other feature structures (instances of non-primitive types).

The JCas cover classes for the array types support the Iterable API, so you may write extended for loops over instances of these. For example:

There are 3 built-in types associated with the artifact being analyzed:

The AnnotationBase type defines one system-used feature which specifies for an annotation the subject of analysis (Sofa) to which it refers. The Annotation type extends from this and defines 2 features, taking uima.cas.Integer values, called begin and end. The begin feature typically identifies the start of a span of text the annotation covers; the end feature identifies the end. The values refer to character offsets; the starting index is 0. An annotation of the word “CAS” in a text “CAS Reference” would have a start index of 0, and an end index of 3; the difference between end and start is the length of the span the annotation refers to.

Annotations are always with respect to some Sofa (Subject of Analysis –see Annotations, Artifacts, and Sofas.

The DocumentAnnotation type has one special instance. It is a subtype of the Annotation type, and the built-in definition defines one feature, language, which is a string indicating the language of the document in the CAS. The value of this language feature is used by the system to control flow among annotators when the “CapabilityLanguageFlow” mode is used, allowing the flow to skip over annotators that don’t process particular languages. Users may extend this type by adding additional features to it, using the XML Descriptor element for defining a type.

Each CAS view has a different associated instance of the DocumentAnnotation type. On the CAS, use getDocumentationAnnotation() to access the DocumentAnnotation.

There are also built-in types supporting linked lists, similar to the ones available in Java and other programming languages. Their use is constrained by the usual properties of linked lists: not very space efficient, no (efficient) random access, but an easy choice if you don’t know how long your list will be ahead of time. The implementation is type specific; there are different list building objects for each of the primitive types, plus one for general feature structures. Here are the type names:

For the primitive types Float, Integer, String and FeatureStructure, there is a base type, for instance, uima.cas.FloatList. For each of these, there are two subtypes, corresponding to a non-empty element, and a marker that serves to indicate the end of the list, or an empty list. The non-empty types define two features –head and tail. The head feature holds the particular value for that part of the list. The tail refers to the next list object (either a non-empty one or the empty version to indicate the end of the list).

For JCas users, the new operator for the NonEmptyXyzList classes includes a 3 argument version where you may specify the head and tail values as part of the constructor. The JCas cover classes for these implement a push(item) method which creates a new non-empty node, sets the head value to item, and the tail to the node it is called on, and returns the new node. These classes also implement Iterable, so you can use the enhanced Java for operator. The iterator stops when it gets to the end of the list, determined by either the tail being null or the element being one of the EmptyXXXList elements. Here’s a StringList example:

There are no other built-in types. Users are free to define their own type systems, building upon these types.

During annotator processing, or outside an annotator, access the type system by calling CAS.getTypeSystem().

However, CAS annotators implement an additional method, typeSystemInit(), which is called by the UIMA framework before the annotator’s process method. This method, implemented by the annotator writer, is passed a reference to the CAS’s type system metadata. The method typically uses the type system APIs to obtain type and feature objects corresponding to all the types and features the annotator will be using in its process method. This initialization step should not be done during an annotator’s initialize method since the type system can change after the initialize method is called; it should not be done during the process method, since this is presumably work that is identical for each incoming document, and so should be performed only when the type system changes (which will be a rare event). The UIMA framework guarantees it will call the typeSystemInit method of an annotator whenever the type system changes, before calling the annotator’s process() method.

The initialization done by typeSystemInit() is done by the UIMA framework when you use the JCas APIs; you only need to provide a typeSystemInit() method, as described here, when you are not using the JCas approach.

To create feature structures in JCas, we use the Java “new” operator. In the CAS, we use one of several different API methods on the CAS object, depending on which of the 10 basic kinds of feature structures we are creating (a plain feature structure, or an instance of the built-in primitive type arrays or FSArray). There are is also a method to create an instance of a uima.tcas.Annotation, setting the begin and end values.

Once a feature structure is created, it needs to be added to the CAS indexes (unless it will be accessed via some reference from another accessible feature structure). The CAS provides this API: Assuming aCAS holds a reference to a CAS, and token holds a reference to a newly created feature structure, here’s the code to add that feature structure to all the relevant CAS indexes:

There is also a corresponding removeFsFromIndexes(token) method on CAS objects.

As of version 2.4.1, there are two methods you can use on an index repository to efficiently bulk-remove all instances of particular types of feature structures from a particular view. One of these, aCas.getIndexRepository().removeAllIncludingSubtypes(aType) removes all instances of a particular type, including instances which are subtypes of the specified type. The other, aCas.getIndexRepository().removeAllExcludingSubtypes(aType) remove all instances of a particular type, only. In both cases, the removal is done from the particular view of the CAS referenced by aCas.

Values of individual features for a feature structure can be set or referenced, using a set of methods that depend on the type of value that feature is declared to have. There are methods on FeatureStructure for this: getBooleanValue, getByteValue, getShortValue, getIntValue, getLongValue, getFloatValue, getDoubleValue, getStringValue, and getFeatureValue (which means to get a value which in turn is a reference to a feature structure). There are corresponding “setter” methods, as well. These methods on the feature structure object take as arguments the feature object retrieved earlier in the typeSystemInit method.

Using the previous example, with the type system initialized with type personType and feature lastNameFeature, here’s a sample code fragment that gets and sets that feature:

The getters and setters for each of the primitive types are defined in the Javadocs as methods of the FeatureStructure interface.

Each CAS can have many indexes associated with it; each CAS View contains a complete set of instantiations of the indexes. Each index is represented by an instance of the type org.apache.uima.cas.FSIndex. You use the object org.apache.uima.cas.FSIndexRepository, accessible via a method on a CAS object, to retrieve instances of indexes. There are methods that let you select the index by name, by type, or by both name and type. Since each index is already associated with a type, passing both a name and a type is valid only if the type passed in is the same type or a subtype of the one declared in the index specification for the named index. If you pass in a subtype, the returned FSIndex object refers to an index that will return only items belonging to that subtype (or subtypes of that subtype).

The returned FSIndex objects are used, in turn, to create iterators. There is also a method on the Index Repository, getAllIndexedFS, which will return an iterator over all indexed Feature Structures (for that CAS View), in no particular order. The iterators created can be used like common Java iterators, to sequentially retrieve items indexed. If the index represents a sorted index, the items are returned in a sorted order, where the sort order is specified in the XML index definition. This XML is part of the Component Descriptor, see Index Definition.

In UIMA V3, Feature structures may be added to or removed from indexes while iterating over them. If this happens, any iterators already created will continue to operate over the before-modification version of the index, unless or until the iterator is re-synchronized with the current value of the index via one of the following specific 3 iterator API calls: moveToFirst, moveToLast, or moveTo(FeatureStructure). ConcurrentModificationException is no longer thrown in UIMA v3.

Feature structures being iterated over may have features which are used as the "keys" of an index, updated. If this is done, UIMA will protect the indexes (to prevent index corruption) by automatically removing the Feature Structure from the indexes, updating the field, and adding the FS back to the index (possibly in a new position). This automatic remove / add-back operation no longer makes the iterator throw a ConcurrentModificationException (as it did in UIMA Version 2) if the iterator is incremented or decremented; existing iterators will continue to operate as if no index modification occurred.

The CAS APIs are organized into 3 Java packages: cas, cas.impl, and cas.text. Most of the APIs described here are in the cas package. The cas.impl package contains classes used in serializing and deserializing (reading and writing external representations) the CAS in various formats, for transporting the CAS among local and remote annotators, or for storing the CAS in permanent storage. The cas.text contains the APIs that extend the CAS to support artifact (including “text”) analysis.

When annotators are combined in an aggregate, their defined type systems are merged. This is designed to support independent development of annotator components. The merge results in a single defined type system for CASes that flow through a particular set of annotators.

The basic operation of a type system merge is to iterate through all the defined types, and if two annotators define the same fully qualified type name, to take the features defined for those types and form a logical union of those features. This operation requires that same-named features have the same range type names. The resulting type system has features comprising the union of all features over all the various definitions for this type in different annotators.

Feature merging checks that for all features having the same name in a type, that the range type is identical; otherwise an error is signaled.

Types are combined for merging when their fully qualified names are the same. Two different definitions can be merged even if their supertype definitions do not match, if one supertype subsumes the other supertype; otherwise an error is signaled. Likewise, two types with the same name can be merged only if their features can be merged.

Some applications may find it useful to scale up pipelines and run these in parallel.

Generally, CASs are not threadsafe, and only one thread at a time may operate on it. In many scenarios, a CAS may be initialized and then filled with Feature Structures, and after some point, no more updates to that particular CAS will be done.

If a CAS is no longer going to be changed, it is possible to access it on multiple threads in a read-only mode, simultaneously, with some limitations. Limitations arise because some UIMA Framework activities may update internal CAS data structures.

Operational data is updated while running a pipeline when a PEAR is entered or exited, because PEARs establish new class loaders and can potentially switch the JCas classes being used (This happens because the class loaders might define different JCas cover classes implementing the same UIMA type). Because of this, you cannot have multiple pipelines accessing a CAS in read-only mode if one or more of those pipelines contains a PEAR. There are other edge cases where this may happen as well; for example, if you are running a pipeline with an Extension Class Loader, and have a callback routine loaded under a different class loader, UIMA will switch the JCas classes when calling the callback.

Full Type names consist of a “namespace” prefix dotted with a simple name. Namespaces are used like packages to avoid collisions between types that are defined by different people at different times. The namespace is used as the Java package name for generated Java files.

Type names used in the CAS correspond to the generated Java classes directly. If the CAS name is com.myCompany.myProject.ExampleClass, the generated Java class is in the package com.myCompany.myProject, and the class is ExampleClass.

An exception to this rule is the built-in types starting with uima.cas and uima.tcas; these names are mapped to Java packages named org.apache.uima.jcas.cas and org.apache.uima.jcas.tcas.

Each XML type specification can have <description …​ > tags. The description for a type will be copied into the generated Java code, as a Javadoc style comment for the class. When writing these descriptions in the XML type specification file, you might want to use html tags, as allowed in Javadocs.

If you use the Component Description Editor, you can write the html tags normally, for instance, “<h1>My Title</h1>”. The Component Descriptor Editor will take care of coverting the actual descriptor source so that it has the leading “<” character written as “<”, to avoid confusing the XML type specification. For example, <p> would be written in the source of the descriptor as <p>. Any characters used in the Javadoc comment must of course be from the character set allowed by the XML type specification. These specifications often start with the line <?xml version=“1.0” encoding=“UTF-8” ?>, which means you can use any of the UTF-8 characters.

The built-in primitive CAS types map to Java types as follows:

The Java Class Models generated for each type can be augmented by the user. Typical augmentations include adding additional (non-CAS) fields and methods, and import statements that might be needed to support these. Commonly added methods include additional constructors (having different parameter signatures), and implementations of toString().

To augment the code, just edit the generated Java source code for the class named the same as the CAS type. Here’s an example of an additional method you might add; the various getter methods are retrieving values from the instance:

Type definitions are merged by the framework from all the components being run together.

To use JCas within an annotator, you must include the generated Java classes output from JCasGen in the class path.

An annotator written using JCas is built by defining a class for the annotator that extends JCasAnnotator_ImplBase. The process method for this annotator is written

The process method is passed the JCas instance to use as a parameter.

The JCas reference is used throughout the annotator to refer to the particular JCas instance being worked on. In pooled or multi-threaded implementations, there will be a separate JCas for each thread being (simultaneously) worked on.

You can do several kinds of operations using the JCas APIs: create new feature structures (instances of CAS types) (using the new operator), access existing feature structures passed to your annotator in the JCas (for example, by using the next method of an iterator over the feature structures), get and set the fields of a particular instance of a feature structure, and add and remove feature structure instances from the CAS indexes. To support iteration, there are also functions to get and use indexes and iterators over the instances in a JCas.

The JCas Java classes generated by JCasGen are typically compiled and put into a JAR file, which, in turn, is put into the application’s class path.

This JAR file must be generated from the application’s merged type system. This is most conveniently done by opening the top level descriptor used by the application in the Component Descriptor Editor tool, and pressing the Run-JCasGen button on the Type System Definition page.

As of version 2.2, the framework supports component descriptors which are PEAR descriptors. These descriptors define components plus include information on the class path needed to run them. The framework uses the class path information to set up a localized class path, just for code running within the PEAR context. This allows PEAR files requiring different versions of common code to work well together, even if the class names in the different versions have the same names.

The mechanism used to switch the class loaders when entering a PEAR-packaged annotator in a flow depends on the framework knowing if JCas is being used within that annotator code. The framework will know this if the particular view being passed has had a previous call to getJCas(), or if the particular annotator is marked as a JCas-using one (by having it extend the class JCasAnnotator_ImplBase).

For the purpose of describing the process of creating a PEAR file and its internal structure, this section describes the steps used to package a UIMA component as a valid PEAR file. The PEAR packaging process consists of the following steps:

The installation of a PEAR package can be done using the PEAR installer tool (PEAR Installer User’s Guide, or by an application using the PEAR APIs, directly.

During the PEAR installation the PEAR file is extracted to the installation directory and the PEAR macros in the descriptors are updated with the corresponding path. At the end of the installation the PEAR verification is called to check if the installed PEAR package can be started successfully. The PEAR verification use the classpath, datapath and the system property settings of the PEAR package to verify the PEAR content. Necessary Java library path settings for native libararies, PATH variable settings or system environment variables cannot be recognized automatically and the use must take care of that manually.

To run an installed PEAR package directly in the UIMA framework the pearSpecifier XML descriptor can be used. Typically during the PEAR installation such an specifier is automatically generated and contains all the necessary information to run the installed PEAR package. Settings for system environment variables, system PATH settings or Java library path settings cannot be recognized automatically and must be set manually when the JVM is started.

The generated PEAR descriptor is located in the component root directory of the installed PEAR package and has a filename like <componentID>_pear.xml.

The PEAR package descriptor looks like:

The pearPath setting in the descriptor must point to the component root directory of the installed PEAR package.

The optional pearParameters section, if used, specifies parameter values, which are used to customize / override parameter values in the PEAR descriptor. The format for parameter values used here is the same as in component parameters. External Settings overrides continue to work for PEAR descriptors, and have precedence, if specified.

Additionally, there can be a parameters section. This section supports only string-valued parameters. This way of specifying parameters is deprecated and should no longer be used. Support for will eventually be removed in a future version of Apache UIMA. Parameters set in the pearParameters have precedence over parameters defined in parameters section. For the time being, both sections can be present simultaneously in a PEAR specifier.

The outermost tag is <XMI> and must include a version number and XML namespace attribute:

XML namespaces^[7] are used throughout. The “xmi” namespace prefix is used to identify elements and attributes that are defined by the XMI specification. The XMI document will also define one namespace prefix for each CAS namespace, as described in the next section.

UIMA Feature Structures are mapped to XML elements. The name of the element is formed from the CAS type name, making use of XML namespaces as follows.

The CAS type namespace is converted to an XML namespace URI by the following rule: replace all dots with slashes, prepend http:///, and append .ecore.

This mapping was chosen because it is the default mapping used by the Eclipse Modeling Framework (EMF)^[8] to create namespace URIs from Java package names. The use of the http scheme is a common convention, and does not imply any HTTP communication. The .ecore suffix is due to the fact that the recommended type system definition for a namespace is an ECore model.

Consider the CAS type name org.myproj.Foo. The CAS namespace (org.myorg.) is converted to the XML namespace URI is http:///org/myproj.ecore.

The XML element name is then formed by concatenating the XML namespace prefix (which is an arbitrary token, but typically we use the last component of the CAS namespace) with the type name (excluding the namespace).

So the example org.myproj.Foo Feature Structure is written to XMI as:

The xmi:id attribute is only required if this object will be referred to from elsewhere in the XMI document. If provided, the xmi:id must be unique for each feature.

All namespace prefixes (e.g. myproj) in this example must be bound to URIs using the xmlns:…​ attribute, as defined by the XML namespaces specification.

CAS features of primitive types (String, Boolean, Byte, Short, Integer, Long , Float, or Double) can be mapped either to XML attributes or XML elements. For example, a CAS FeatureStructure of type org.myproj.Foo, with features:

could be mapped to:

or equivalently:

The attribute serialization is preferred for compactness, but either representation is allowable. Mixing the two styles is allowed; some features can be represented as attributes and others as elements.

CAS features that are references to other feature structures (excluding arrays and lists, which are handled separately) are serialized as ID references.

If we add to the previous CAS example a feature structure of type org.myproj.Baz, with feature “myFoo” that is a reference to the Foo object, the serialization would be:

As with primitive-valued features, it is permitted to use an element rather than an attribute. However, the syntax is slightly different:

Note that in the attribute representation, a reference feature is indistinguishable from an integer-valued feature, so the meaning cannot be determined without prior knowledge of the type system. The element representation is unambiguous.

For a CAS feature whose range type is one of the CAS array or list types, the XMI serialization depends on the setting of the “multipleReferencesAllowed” attribute for that feature in the UIMA Type System Description.

An array or list with multipleReferencesAllowed = false (the default) is serialized as a multi-valued property in XMI. An array or list with multipleReferencesAllowed = true is serialized as a first-class object. Details are described below.

A UIMA CAS contain one or more subjects of analysis (Sofas). These are serialized no differently from any other feature structure. For example:

Each Sofa defines a separate View. Feature Structures in the CAS can be members of one or more views. (A Feature Structure that is a member of a view is indexed in its IndexRepository, but that is an implementation detail.)

In the XMI serialization, views will be represented as first-class objects. Each View has an (optional) “sofa” feature, which references a sofa, and multi-valued reference to the members of the View. For example:

Here the integers 3, 7, 21, 39, and 61 refer to the xmi:id fields of the objects that are members of this view.

If the CAS Type System has been saved to an Ecore file, it is possible to store a link from an XMI document to that Ecore type system. This is done using an xsi:schemaLocation attribute on the root XMI element.

The xsi:schemaLocation attribute is a space-separated list that represents a mapping from namespace URI (e.g. http:///org/myproj.ecore) to the physical URI of the .ecore file containing the type system for that namespace. For example:

would indicate that the definition for the org.myproj CAS types is contained in the file c:/typesystems/myproj.ecore. You can specify a different mapping for each of your CAS namespaces, using a space separated list. For details see Budinsky et al. Eclipse Modeling Framework.

The Delta CAS XMI serialization format is designed primarily to reduce the overhead serialization when calling annotators configured as services. Only Feature Structures and Views that are new or modified by the service are serialized and returned by the service.

The classes org.apache.uima.cas.impl.XmiCasSerializer and org.apache.uima.cas.impl.XmiCasDeserializer support serialization of only the modifications to the CAS. A caller is expected to set a marker to indicate the point from which changes to the CAS are to be tracked.

A Delta CAS XMI document contains only the Feature Structures and Views that have been added or modified. The new and modified Feature Structures are represented in exactly the format as in a complete CAS serialization. The ` cas:View ` element has been extended with three additional attributes to represent modifications to View membership. These new attributes are added_members, deleted_members and reindexed_members. For example:

Here the integers 63, 77 represent xmi:id fields of the objects that have been newly added members to this View, 7 and 61 are xmi:id fields of the objects that have been removed from this view and 39 is the xmi:id of an object to be reindexed in this view.

UIMA has a proprietary binary serialization format, used internally for several things, including communicating with embedded C++ annotators using UIMA-CPP. This binary format is also selectable for use with UIMA-AS. Its use requires that the source and target systems implement the identical type system (because the type system is not sent, and internal coding is used within the format that is keyed to the particular type system).

Starting with version 2.4.1, two additional forms of binary serialization are added. Both compress the data being serialized; typical size ratios can approach 50 : 1, depending on the exact contents of the CAS, when compared with normal binary serialization.

The two forms are called 4 and 6, for historical/internal reasons. The serialized forms of both of these is fixed, but not currently standardized, and the form being used is encoded in the header so that the appropriate deserializer can be chosen. Both forms include support for Delta CAS being returned from a service.

Form 6 builds on form 4, and adds: serializing only those feature structures which are reachable (that is, in some index, or referenced by other reachable feature structures), and type filtering.

Type filtering takes a source type system and a target type system, and for serializing (source to target), sends the binary representation of reachable feature structures in the target’s type system. For deserializing (reading a target into a source), the filtering takes the specification being read as being encoded using the target’s type system, and translates that into the source’s type system. In this process, types which exist in the source but not the target are skipped (when serializing); types which exist in the target, but not the source are skipped when deserializing. Features that exist in some source type but not in the version of the same type in the target are skipped (when serializing) or set to default values (i.e., 0 or null) when being deserialized.

There are two main use cases for using compressed forms. The first one is for communicating with UIMA-AS remote services (not yet implemented).

The second use case is for saving compressed representations of CASes to other media, such as disk files, where they can be deserialized later for use in other UIMA applications.

The main user interface for serializing a CAS using compression is to use one of the static methods named serializeWithCompression in Serialization. If you pass a Type System argument representing a target type system, then form 6 compression is used; otherwise form 4 is used. To get the benefit of only serializing reachable Feature Structure instances, without type mapping (which is only in form 6), pass a type system argument which is null.

To deserialize into a CAS without type mapping, use one of the deserialize method in Serialization. There are multiple forms of this method, depending on the arguments. The forms which take extra arguments include a ReuseInfo may only be used with serialized forms created with form 6 compression. The plain form of deserialize works with all forms of binary serialization, compressed and non-compressed, by examining a common header which identifies the form of binary serialization used; however, for form 6, since it requires additional arguments, it will fail - and you need to use the other deserialize form.

Form 6 has an additional object, ReuseInfo, which holds information which is required for subsequent Delta CAS format serializations / deserializations. It can speed up subsequent serializations of the same CAS (before it is further updated), for instance, if an application is sending the CAS to multiple services in parallel. The serializeWithCompression method returns this object when form 6 is being used.

In addition, the CasIOUtils class offers static load and save methods, which can be used with the SerialFormat enum to serialize and deserialize to URLs or streams; see the Javadocs for details.

Use Form 4 for this, because form 6 supports delta CAS but requires that at the time of deserialization of a CAS (on the receiver side) which will later be delta serialized back to the sender, an instance of the ReuseInfo must be saved, and that same instance then used for delta serialization; furthermore, the original serialization (on the sender side) also must save an instance of the ReuseInfo and use this when deserializing the delta CAS.

Form 4 may not be as efficient as form 6 in that it does not filter the CASes either by type systems nor by only sending reachable Feature Structure instances. But, it doesn’t require a ReuseInfo object when doing delta serialization or deserialization, so it may be more convenient to use when saving delta CASes to files (as opposed to the other use case of a remote service returning delta CASes to a remote client).

Here are some use cases, together with a suggested approach and example of how to use the APIs.

Save a CAS to an output stream, using form 4 (no type system filtering):

Deserialize from a stream into an existing CAS:

Note that the deserializeCAS(cas, inputStream) method is a general way to deserialize into a CAS from an inputStream for all forms of binary serialized data (with exceptions as noted above). The method reads a common header, and based on what it finds, selects the appropriate deserialization routine.

Serialize to an output stream, filtering out some types and/or features:

To do this, an additional input specifying the Type System of the target must be supplied; this Type System should be a subset of the source CAS’s. The out parameter may be an OutputStream, a DataOutputStream, or a File.

Deserialize with type filtering:

There are 2 type systems involved here: one is the receiving CAS, and the other is the type system used to decode the serialized form. This may optionally be stored with the serialized form:

and/or it can be supplied at load time. Here’s two examples of suppling this at load time:

The reuseInfo should be null unless deserializing a delta CAS, in which case, it must be the reuse info captured when the original CAS was serialized out. If the target type system is identical to the one in the CAS, you may pass null for it. If a delta cas is not being received, you must pass null for the reuseInfo.

Applications are moving to the "cloud", and new applications are being rapidly developed that are hooking things up using various mashup techniques. New standards and conventions are emerging to support this kind of application development, such as REST services. JSON is now a popular way for services to communicate; its popularity is rising (in 2014) while XML is falling.

Starting with version 2.7.0, JSON style serialization (but not (yet) deserialization) for CASs and UIMA descriptions is supported. The exact format of the serialization is configurable in several aspects. The implementation is built on top of the Jackson JSON generation library.

The next section discusses serialization for CASes, while a later section describes serialization of description objects, such as type system descriptions.

CASs primarily consist of collections of Feature Structures (FSs). Similar to XMI serialization, JSON serialization skips serializing unreachable FSs, outputting only those FSs that are found in the indexes (these are called roots), plus all of the FSs that are referenced via some chain of references, from the roots.

To support the kinds of things users do with FSs, the serialized form may be augmented to include additional information beyond the FSs.

For traditional UIMA implementations, the serialized formats mostly assumed that the receivers had access to a type system description, which specified details of the types of each feature value. For JSON serialization, some of this information can be including directly in the serialization.

This abbreviated type system information is one kind of additional information that can be included; here’s a summary list of the various kinds of additional information you can add to the serialization:

Simple JSON serialization does not have a convention for supporting these, but many extensions do. We borrow some of the concepts in the JSON-LD (linked data) standard in providing this additional information.

The set of all FSs being serialized is divided into two parts. The first part represents all FSs that are root FSs, in that they were in one or more indexes at the time of serialization. The second part represents feature structures that are multiply referenced, or are referenced via a chain of references from the root FSs. The same feature structure can appear in both lists. The elements in the second part are actual serialized FSs, whereas, the elements in the first part are either references to the corresponding FSs in the second part, if they exist, or the actual embedded serialized FSs. Actual embedded serialized FSs only exist once in the two parts.

The first part map is made up of multiple maps, one for each separate CAS View. The outer map is keyed by the id of the corresponding SofaFS (or 0, if there is no corresponding SofaFS). For each view, the value is a map whose key is a used Type, and the values are an array of instances of FSs of that type which were found in some index; these are the "root" FSs. Only root instances of a particular type are included in this array.

The second part map has keys which are the id value of the FSs, and values which are a map of key-value pairs corresponding to the feature-values of that FS. In this case, the _type extra feature is added to record the type.

The _views map, keyed by view and type name, has all the FSs (as an JSON array) for that type that were in one or more indexes in any View. If a FS in this array is not multiply referenced (using dynamic mode), then it is embedded here. Otherwise, only the reference (a simple number representing the id of that FS) is serialized for that FS.

JSON serialization also supports several additional features, including:

See the Javadocs for JsonCasSerializer for details.

The support is built on top the Jackson JSON serialization package. We follow Jackson conventions for configuring.

The serialization APIs are in the JsonCasSerializer class.

Although there are some static short-cut methods for common use cases, the basic operations needed to serialize a CAS as JSON are:

Here’s an example:

The JsonCasSerializer class also has some static convenience methods for JSON serialization, for the most common configuration cases; please see the Javadocs for details. These are named jsonSerialize, to distinguish them from the non-static serialize methods.

Many of the common configuration methods generally return the instance, so they can be chained together. For example, if jcs is an instance of the JsonCasSerializer, you can write jcs.setPrettyPrint(true).setOmit0values(true); to configure both of these.

UIMA descriptors are things like analysis engine descriptors, type system descriptors, etc. UIMA has an internal form for these, typically named UIMA descriptions; these can be serialized out as XML using a toXML method. JSON support adds the ability to serialize these a JSON objects, as well. It may be of use, for example, to have the full type system description for a UIMA pipeline available in JSON notation.

The class JsonMetaDataSerializer defines a set of static methods that serialize UIMA description objects using a toJson method that takes as an argument the description object to be serialized, and the standard set of serialiization targets that Jackson supports (File, Writer, or OutputStream). There is also an optional prettyprint flag (default is no prettyprinting).

The resulting JSON serialization is just a straight-forward serialization of the description object, having the same fields as the XML serialization of it.

Here’s what a small TypeSystem description looks like, serialized:

Here’s a sample of code to serialize a UIMA description object held in the variable tsd, with and without pretty printing:

Some updates change UIMA’s behavior between released versions. For example, sometimes an error check is enhanced, and this can cause something that previously incorrect but not checked, to now signal an error. Often, users will want these kinds of things to be ignored, at least for a while, to give them time to analyze and correct the issues.

To enable users to gradually address these issues, there are some global JVM properties for UIMA that can restore earlier behaviors, in some cases. These are detailed in the table below. Additionally, there are other JVM properties that can be used in checking and optimizing some performance trade-offs, such as the automatic index protection. For the most part, you don’t need to assign any values to these properties, just define them. For example to disable the enhanced check that insures you don’t add a subtype of AnnotationBase to the wrong View, you could disable this by adding the JVM argument -Duima.disable_enhanced_check_wrong_add_to_index. This would remove the enhanced checking for this, added in version 2.7.0 (the previously existing partial checking is still there, though).

A new feature in version 2.7.0 optionally can include checking for invalid feature updates which could corrupt indexes. Because this checking can slightly slow down performance, there are global JVM properties to control it. The suggested way to operation with these is as follows.

One additional JVM property, -Duima.throw_exception_when_fs_update_corrupts_index, is intended to be used in automated build / testing configurations. It causes the framework to throw a UIMARuntimeException if an update outside of a protectIndexes block occurs that could corrupt the indexes, rather than "recovering" this.

This table describes the various JVM defined properties; specify these on the Java command line using -Dxxxxxx, where the xxxxxx is one of the properties starting with uima. from the table below.

Some additional global flags intended for helping v3 migration are documented in the V3 user’s guide.

UIMA uses the term Resource to describe all UIMA components that can be acquired by an application or by other resources.

There are many kinds of resources; here’s a list of the main kinds:

UIMA applications run one or more UIMA Pipelines. Each pipeline has a top-level Analysis Engine, which may be an aggregation of many other Analysis Engine components. The UIMA framework instantiates Annotator resources as specified to configure the pipelines.

Sometimes, many identical pipelines are created (for example, in order to exploit multi-core hardware by processing multiple CASes in parallel). In this case, the framework would produce multiple instances of those Annotation resources; these are implemented as multiple instances of the same Java class.

Sets of External Resources plus a CAS Pool and UIMA Extension ClassLoader are set up and kept, per instance of a ResourceManager; this instance serves to allow sharing of these items across one or more pipelines.

When setting up a pipeline, the UIMA Framework’s produceResource or one of its specialized variants is called, and a new ResourceManager being created and used for that pipeline. However, in many cases, it may be advantageous to share the same Resources across multiple pipelines; this is easily doable by passing a common instance of the ResourceManager to the pipeline creation methods (using the additional parameters of the produceResource method).

To handle additional use cases, the ResourceManager has a copy() method which creates a copy of the Resource Manager instance. The new instance is created with a null CAS Manager; if you want to share the the CAS Pool, you have to copy the CAS Manager: newRM.setCasManager(originalRM.getCasManager()). You also may set the Extension Class Loader in the new instance (PEAR wrappers use this to allow PEARs to have their own classpath). See the Javadocs for details.

A typical external resource gets a single instantiation, shared with all users of a particular ResourceManager. Sometimes, multiple instantiations may be useful (of the same resource). The framework supports this for ParameterizedDataResources. There’s one kind supplied with UIMA - the fileLanguageResourceSpecifier. This works by having each call to getResource(name, extra_keys[]) use the extra keys to select a particular instance. On the first call for a particular instance, the named resource uses the extra keys to initialize a new instance by calling its load method with a data resource derived from the extra keys by the named resource.

For example, the fileLanguageResourceSpecifier uses the language code and goes through a process with lots of defaulting and fall back to find a resource to load, based on the language code.