Distributed UIMA Cluster Computing

License and Disclaimer The ASF licenses this documentation to you under the Apache License, Version 2.0 (the ”License”); you may not use this documentation except in compliance with the License. You may obtain a copy of the License at

Unless required by applicable law or agreed to in writing, this documentation and its contents are distributed under the License on an ”AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Trademarks All terms mentioned in the text that are known to be trademarks or service marks have been appropriately capitalized. Use of such terms in this book should not be regarded as affecting the validity of the the trademark or service mark.

List of Figures

Part I
DUCC Concepts

Chapter 1
DUCC Overview

1.1 What is DUCC?

DUCC stands for Distributed UIMA Cluster Computing. DUCC is a cluster management system providing tooling, management, and scheduling facilities to automate the scale-out of applications written to the UIMA framework.

Core UIMA provides a generalized framework for applications that process unstructured information such as human language, but does not provide a scale-out mechanism. UIMA-AS provides a scale-out mechanism to distribute UIMA pipelines over a cluster of computing resources, but does not provide job or cluster management of the resources. DUCC defines a formal job model that closely maps to a standard UIMA pipeline. Around this job model DUCC provides cluster management services to automate the scale-out of UIMA pipelines over computing clusters.

As of DUCC version 3.0.0 both UIMAv2 and UIMAv3 applications are supported. Although DUCC distributes a UIMA-AS runtime, it only uses UIMA classes specified by the user’s application classpath.

1.2 DUCC Job Model

The Job Model defines the steps necessary to scale-up a UIMA pipeline using DUCC. The goal of DUCC is to scale-up any UIMA pipeline, including pipelines that must be deployed across multiple machines using shared services.

The DUCC Job model consists of standard UIMA components: a Collection Reader (CR), a CAS Multiplier (CM), application logic as implemented one or more Analysis Engines (AE), and a CAS Consumer (CC).

The Collection Reader builds input CASs and forwards them to the UIMA pipelines. In the DUCC model, the CR is run in a process separate from the rest of the pipeline. In fact, in all but the smallest clusters it is run on a different physical machine than the rest of the pipeline. To achieve scalability, the CR must create very small CASs that do not contain application data, but which contain references to data; for instance, file names. Ideally, the CR should be runnable in a process not much larger than the smallest Java virtual machine. Later sections demonstrate methods for achieving this.

Each pipeline must contain at least one CAS Multiplier which receives the CASs from the CR. The CMs encapsulate the knowledge of how to receive the data references in the small CASs received from the CRs and deliver the referenced data to the application pipeline. DUCC packages the CM, AE(s), and CC into a single process, multiple instances of which are then deployed over the cluster.

A DUCC job therefore consists of a small specification containing the following items:

On job submission, DUCC creates a single process executing the CR and one or more processes containing the analysis pipeline.

1.3 DUCC From UIMA to Full Scale-out

In this section we demonstrate the progression of a simple UIMA pipeline to a fully scaled-out job running under DUCC.

UIMA Pipelines A normal UIMA pipeline contains a Collection Reader (CR), one or more Analysis Engines (AE) connected in a pipeline, and a CAS Consumer (CC) as shown in Figure 1.1.

UIMA-AS Scaled Pipeline With UIMA-AS the CR is separated into a discrete process and a CAS Multiplier (CM) is introduced into the pipeline as an interface between the CR and the pipeline, as shown in Figure 1.2 below. Multiple pipelines are serviced by the CR and are scaled-out over a computing cluster. The difficulty with this model is that each user is individually responsible for finding and scheduling computing nodes, installing communication software such as ActiveMQ, and generally managing the distributed job and associated hardware.

UIMA Pipeline Scaled By DUCC DUCC is a UIMA and UIMA-AS-aware cluster manager. To scale out work under DUCC the developer tells DUCC what the parts of the application are, and DUCC does the work to build the scale-out via UIMA/AS, to find and schedule resources, to deploy the parts of the application over the cluster, and to manage the jobs while it executes.

On job submission, the CR is wrapped with a DUCC main class and launched as a Job Driver (or JD). The DUCC main class establishes communication with other DUCC components and instantiates the CR. If the CR initializes successfully, and indicates that there are greater than 0 work items to process, the specified CM, AE and CC components are assembled into an aggregate, wrapped with a DUCC main class, and launched as a Job Process (or JP).

The JP will replicate the aggregate as many times as specified, each aggregate instance running in a single thread. When the aggregate initializes, and whenever an aggregate thread needs work, the JP wrapper will fetch the next work item from the JD, as shown in Figure 1.3 below.

UIMA Pipeline with User-Supplied DD Scaled By DUCC Application programmers may supply their own Deployment Descriptors to control intra-process threading and scale-out. If a DD is specified in the job parameters, DUCC will launch each JP with the specified UIMA-AS service instantiated in-process, as depicted in Figure 1.4 below. In this case the user can still specify how many work items to deliver to the service concurrently.

1.4 Error Management

1.5 Cluster and Job Management

DUCC supports management of multiple jobs and multiple users in a distributed cluster:

1.6 Security Measures

1.6.1 ducc_ling

ducc_ling contains the following functions, which the security-conscious may verify by examining the source in $DUCC_HOME/duccling. All sensitive operations are performed only AFTER switching userids, to prevent unauthorized root access to the system.

1.7 Security Issues

Chapter 2
Glossary

Part II
Ducc Users Guide

Chapter 3
Command Line Interface

Overview The DUCC CLI is the primary means of communication with DUCC. Work is submitted, work is canceled, work is monitored, and work is queried with this interface.

All parameters may be passed to all the CLI commands in the form of Unix-like “long-form” (key, value) pairs, in which the key is proceeded by the characters “--”. As well, the parameters may be saved in a standard Java Properties file, without the leading “--” characters. Both a properties file and command-line parameters may be passed to each CLI. When both are present, the parameters on the command line take precedence. Take, for example the following simple job properties file, call it 1.job, where the environment variable “DH” has been set to the location of $DUCC_HOME.

3.1 The DUCC Job Descriptor

The DUCC Job Descriptor includes properties to enable automated management and scale-out over large computing clusters. The job descriptor includes

3.2 Operating System Limit Support

The CLI supports specification of operating system limits applied to the various job processes. To specify a limit, pass the name of the limit and its value in the environment specified in the job. Limits are named with the string “DUCC_RLIMIT_name” where “name” is the name of a specific limit. Supported limits include:

See the Linux documentation for details on the meanings of these limits and their values.

For example, to set the maximum number of open files allowed in any job process, specify an environment similar to this when submitting the job:

3.3 Command Line Forms

When using the scripts the full execution environment is established silently. When invoking a command’s class directly, the java CLASSPATH must include the uima-ducc-cli.jar, as illustrated in the wrapper scripts.

3.4 DUCC Commands

3.5 ducc_submit

Description: The submit CLI is used to submit work for execution by DUCC. DUCC assigns a unique id to the job and schedules it for execution. The submitter may optionally request that the progress of the job is monitored, in which case the state of the job as it progresses through its lifetime is printed on the console.

Notes: When searching for UIMA XML resource files such as descriptors, DUCC searches either the filesystem or Java classpath according to the following rules:

3.6 ducc_cancel

Description: The cancel CLI is used to cancel a job that has previously been submitted but which has not yet completed.

3.7 ducc_reserve

Description: The reserve CLI is used to request a reservation of resources. Reservations can be for machines based on memory requirements. All reservations are persistent: the resources remain dedicated to the requester until explicitly returned. All reservations are performed on an ”all-or-nothing” basis: either the entire set of requested resources is reserved, or the reservation request fails.

All forms of ducc_reserve block until the reservation is complete (or fails) at which point the DUCC ID of the reservation and the names of the reserved nodes are printed to the console and the command returns.

Notes: Reservations must be for entire machines, in a job class implementing the RESERVE scheduling policy. The default DUCC distribution configures class reserve for entire machine reservations. A reservation request will be queued if there is no available machine in that class matching the requested size (after rounding up), or up to ducc.rm.reserve_overage larger than the request (after rounding up). The user may cancel the request with ducc_unreserve or with CTRL-C if --cancel_on_interrupt was specified.

3.8 ducc_unreserve

3.9 ducc_process_submit

Description: Use ducc_process_submit to submit a Managed Reservation, also known as an arbitrary process to DUCC. The intention of this function is an alternative to utilities such as ssh, in order to allow the spawned processes to be fully managed by DUCC. This allows the DUCC scheduler to allocate the necessary resources (and prevent over-allocation), and the DUCC run-time environment to manage process lifetime.

If attach_console is specified, Stdin, Stderr, and Stdout of the remote process are redirected to the submitting console. It is thus possible to run interactive sessions with remote processes where the resources are managed by DUCC.

3.10 ducc_process_cancel

Description: The cancel CLI is used to cancel a process that has previously been submitted but which has not yet completed.

3.11 ducc_services

Description: The ducc_services CLI is used to manage service registration. It has a number of functions as listed below.

3.11.1 Common Options

3.11.2 ducc_services –register [specification file] [options]

The register function submits a service specification to DUCC. DUCC stores this information until it is unregistered. Once registered, a service may be started, stopped, etc.

The specification file is optional. If designated, it is a Java properties file containing other registration options, minus the leading “–”. If both a specification file and command-line options are designated, the command-line options override those in the specification.

3.11.3 ducc_services –start options

The start function instructs DUCC to allocate resources for a service and to start it in those resources. The service remains running until explicitly stopped. DUCC will attempt to keep the service instances running if they should fail. The start function is also used to increase the number of running service instances if desired.

3.11.4 ducc_services –stop options

The stop function instructs DUCC to stop some number of service instances. If no specific number is specified, all instances are stopped.

3.11.5 ducc_services –enable options

The enable function removes the disabled flag and allows a service to resume spawning new instances according to its management policy.

3.11.6 ducc_services –disable options

The disable function prevents the service from starting new instances. Existing instances are not affected. Use the ducc_services –enable command to reset.

3.11.7 ducc_services –observe_references options

If the service is not autostarted and has active instances, this instructs the Service Manager to track references to the service, and when the last referencing service exits, stop all instances. The registered linger time is observed after the last reference exits before stopping the service. See the management policy section for more information.

3.11.8 ducc_services –ignore_references options

If the service is manually started and has active instances, this instructs the Service Manager to NOT stop the service when the last referencing job has exited. It transforms a manually-started service into a reference-started service. See the management policy section for more information.

3.11.9 ducc_services –modify options

The modify function dynamically updates some of the attributes of a registered service. All service options as described under --register other than the service_endpoint and process_descriptor_DD may be modified wihtout re-registering the service. In most cases the service will need to be stopped and restarted for the update to apply.

The following modifications take place immediately without the need to restart the service:

Modifying the following registration options causes the service pinger to be stopped and started, without affecting any of the service instances themselves. The pinger is restarted even if the modification value is the same as the old value. (A good way to restart a possibly errant pinger is to modify it’s service_ping_dolog from “true” to “true” or from “false” to “false”.)

3.11.10 ducc_services –query options

The query function returns details about all known services of all types and classes, including the DUCC ids of the service instances (for submitted and registered services), the DUCC ids of the jobs using each service, and a summary of each service’s queue and performance statistics, when available.

All information returned by ducc_services --query is also available via the Services Page of the Web Server as well as the DUCC Service API (see the JavaDoc).

3.12 viaducc and java_viaducc

Description: Viaducc is a small script wrapper around the ducc_process_submit CLI to facilitate launching processes on DUCC-managed machines, either from the command line or from an Eclipse run configuration.

When run from the command line as “viaducc”, the arguments are bundled into the form expected by ducc_process_submit and submitted to DUCC. By default the remote stdin and stdout of the deployed process are mapped back to the command line terminal.

If a symbolic link to the viaducc script is created with the name “java_viaducc” and used from the command line, the arguments are assumed to be a Java classname and its arguments. The java process will be executed using DUCC’s default JRE, or optionally, a specific JRE supplied by the user with a -D argument.

If the “java_viaducc” symbolic link is installed in a JRE/bin directory, DUCC will use the java executable from the same directory. More interestingly, it may be specified as an alternative to the “java” command in an eclipse launcher. The remote stdin and stdout of the deployed DUCC process are redirected to the Eclipse console. This provides essentially transparent execution of code in an Eclipse workspaces on DUCC-managed resources.

The “defines” are described below. The “command and parameters” are either any command (with full path) and it’s arguments, or a Java class (with a “main”) and its arguments (including the classpath if necessary.)

Defines The arguments are specified in the syntax of Java “-D” system properties, to be more consistent with execution under Eclipse.

3.13 ducc_status

Description: The status tool is used to obtain the status of the various DUCC daemons.

3.14 ducc_watcher

Description: The watcher tool is used to monitor the status of the various DUCC daemons. It can additionally be used monitor the status of the Job Driver allocation.

Notes: This command is meant to monitor DUCC daemons status and report only when there is a change, e.g. from up to down.

crontab: Below is a sample crontab to check the state of DUCC head node daemons as well as the JobDriver allocation every 5 minutes, and to send e-mail when there is a status change.

Chapter 4
The DUCC Public API

4.1 Overview Of The DUCC API

The DUCC API provides a simple programmatic (Java) interface to DUCC for submission and cancellation of work. (Note that the DUCC CLI is implemented using the API and provides a model for how to use the API.)

All the API objects are instantiated using the same arguments as the CLI. The API provides three variants for supplying arguments:

After instantiation of an API object, the boolean execute() method is called. This method transmits the arguments to DUCC. If DUCC receives and accepts the args, the method returns “true”, otherwise it returns “false. Methods are provided to retrieve relevant information when the execute() returns such as IDs, messages, etc.

In the case of jobs and managed reservations, if the specification requested debug, console attachment, or “wait for completion”, the API provides methods to block waiting for completion.

In the case of jobs and managed reservations, a callback object may also be passed to the constructor. The callback object provides a means to direct messages to the API user. If the callback is not provided, messages are written to standard output.

The API is thread-safe, so developers may manage multiple, simultaneous requests to DUCC.

Below is the “main()” method of DuccJobSubmit, demonstrating the use of the API:

4.2 Compiling and Running With the DUCC API

A single DUCC jar file is required for both compilation and execution of the DUCC API, uima-ducc-cli.jar. This jar is found in $DUCC_HOME/lib.

4.3 Java API

Chapter 5
Service Management

5.1 Overview.

A service is usually a UIMA-AS service, but DUCC supports any arbitrary process as a service.

When work enters the system with a declared dependency on a service, one of the following actions is taken:

5.2 Service Types.

DUCC also supports services that are not managed by DUCC. These are known as ping-only services. The registration for a ping-only service contains only keywords needed to support a pinger, which communicates with the non-DUCC service. Ping-only services must be defined as custom services; there is no default pinger provided for ping-only services.

5.3 Service Instance IDs

DUCC 2.0.0 introduces support for constant service instance IDs. As a service is being started, the SM assigns monotonically increasing IDs to each service instance, starting with ID 0, up the the maximum number of instances started.

If an instance exits unexpectedly, the SM re-spawns it (unless a failure threshold has been exceeded). The new instance is assigned the same instance ID as the instance it replaces. This insures that, for example, instance “three” is always started as instance “three”, maintained constant over failures and SM restarts.

The instance ID is communicated to the process through the environment with the key DUCC_SERVICE_INSTANCE. This key may also be used in service registrations if it is desired to pass the instance ID via parameters of some sort. For example:

5.4 Service References and Endpoints

Services are identified by an entity called a service endpoint. Jobs and other services use the registered service endpoint to indicate dependencies on specific services.

The unique id is any string needed to ensure the service is uniquely named. For UIMA-AS services, the unique ID must be the same as the service endpoint specified in service’s DD XML descriptor. The UIMA-AS service endpoint is always of the form:

where queue-name is the name of the ActiveMQ queue used by the service, and broker-url is the ActiveMQ broker URL. Sample DUCC Service endpoints:

Jobs or other services may register dependencies on specific services by listing one or more service endpoints int their specifications. See the job and services CLI descriptions for details.

A service is registered with DUCC using the ducc_services API/CLI. Service registrations are persisted by DUCC and last over DUCC and cluster restarts.

5.5 Application Broker for UIMA-AS Services

UIMA-AS services require a separate ActiveMQ broker. An application broker can be started using a script in the UIMA-AS distribution bundled with DUCC.

This uses the default configuration of port 61616 for tcp access and 1099 for JMX access (used by the default UIMA-AS pinger.) The internal DUCC broker uses different ports so both may run on the same node.

5.6 Service Management Policies

Dynamically Changing Service Policies A service may be stopped; that is, no instances are running. This state can occur if the service has experienced too many errors within its failure window, in which case the service is disabled, or because the service is not autostarted or referenced by other work.

If a manual stop is issued the service will be automatically disabled to insure it cannot be restarted (by reference or at boot with autostart) without manual intervention.

In all cases, if a service is disabled, it must be manually enabled using the CLI.

It is possible, via the CLI, to dynamically switch any service from any management policy to any other policy, as shown in the following table.

See the Service CLI reference for details on the various commands described in this section.

5.7 Service Pingers

A service pinger is a small program that queries a service on behalf of the DUCC Service Manager. A default pinger is provided for UIMA-AS services and provides the following functions:

Users may supply their own pingers. The following additional functions are available for pingers. Note that a custom pinger MAY be supplied for UIMA-AS services, and MUST be supplied for CUSTOM services. Custom pingers use the Service Manager’s “pinger” API to perform the following tasks:

5.7.1 The Pinger API

Pingers are passed static information about the service at pinger-initialization time, and subsequently, current state of the service is provided on each call (ping).

Information provided at initialization follows. Most of this is provided in fields in the AServicePing base class. See the Javadoc for specific field names and types.

Pinger Initialization Data

Pinger Dynamic Data

5.7.2 Declaring a Pinger in A Service

The following registration options are used for declaring and configuring pingers. Any of these may be dynamically modified with the service CLI’s --modify option. Dynamically changing these causes the current pinger to be terminated and restarted with the new configuration. See ducc_services for details of the options:

5.7.3 Implementing a Pinger

Pingers must implement the class org.apache.uima.ducc.cli.AServicePing. See the Javadoc for the details of this class.

Below is a sample CUSTOM pinger for a hypothetical service that returns four integers in response to a ping. It illustrates simple use of the three required methods, init(), stop(), and getStatistics().

5.7.4 Building And Testing Your Pinger

This section provides the information needed to use the pinger API and build a custom pinger.

1. Establish a compilation CLASSPATH One DUCC jar is required in the CLASSPATH to build your pinger:

This provides the definition for the AServicePing and ServiceStatistics classes.

2. Create a registration Next, create a service registration for the pinger. While debugging, it is useful set the directive

This will log any output from System.out.println() to the declared log directory for the service. If not specified in the reqistration, this directory is:

If your pinger requires a different version of Java than is used by DUCC, include a setting for the JAVA_HOME variable in the environment option.

A sample service registration may look something like the following. Note that you do not need to include any of the DUCC jars in the classpath for the pinger. DUCC will add the jars it requires to interact with the pinger automatically. (However you may need other jars to provide UIMA, UIMA-AS, ActiveMQ, Spring, or other function.)

3. Register and start the service and pinger Start up your custom service so the pinger with the registration containing lines similar to those above. As soon as the service instance is in DUCC state Running the SM starts the pinger.

Check the web server to make sure the service “comes alive”. Check your pinger’s debugging log if it doesn’t. Once registered, you can dynamically modify and restart the pinger at any time without re-registering the service or restarting the service by use of the --modify option of the ducc_services CLI:

4. If all else fails ... If your pinger does not work and you cannot determine the reason, be sure you enable service_ping_dolog and look in your log directory, as most problems with pingers are reflected there. As a last resort, you can inspect the the Service Manager’s log in

5.7.5 Globally Registered Pingers

A user-built pinger may be registered with DUCC so that it can be globally used by any DUCC service. To do this, a registration file containing only pinger-specific parameters is created in DUCC’s run-time directory. Such a pinger may then be designated for a service by using its registered filename instead of its class in the service_ping_class field of a registration. There is no API or CLI to register such a pinger; only a DUCC administrator may create a global ping registration.

A globally-registered pinger may then be designated to run as a thread inside the SM or as a process spawned and managed by the SM. A pinger that runs in a thread in the SM is called an internal pinger, and one that runs in a process is called an external pinger. An internal pinger generally has nearly unmeasurable impact on the system, whereas external pingers will occupy full JVMs with processes of 50-100MB or more.

A service may override any of the options of a globally-registered external pinger, thus allowing significant reuse of existing code. Only the service_ping_arguments of an internal pinger may be overridden however.

Globally registered pingers use a special boolean property, not supported by the ducc_services API/CLI, “internal”, to determine whether the pinger is to be run internally to SM or as an external process. Only the DUCC administrator may update a global pinger’s registration to “internal”, to insure such pingers are properly vetted and approved by the installation.

More Details of registering global pingers is found in the Administration section of this document.

5.8 Sample Pinger

A sample custom UIMA-AS pinger is provided in the Examples directory shipped with DUCC in

This pinger increases or decreases the number of service instances based on the queue statistics found by querying ActiveMQ. The goal of this pinger is to maintain the ActiveMQ “enqueued time” to be no more than some multiple of the average service time for a single item. The factor used is a parameter passed in with the argument string.

5.8.1 Using the Sample Pinger

The following arguments may be specified to use the sample pinger with any UIMA-AS service. The service_ping_arguments are specific to this pinger.

The following arguments are accepted by this pinger and may be specified in a single single comma-delimited string containing the following initialization parameters:

5.8.2 Understanding Sample Pinger

The best way to understand this pinger is to examine the code itself in the Examples directory. Here we provide a brief line-by-line synopsis of the code.

void init(String args, String ep) This required method examines the service arguments and endpoint and establishes a monitor to issue get-meta calls to the service and JMS calls to the ActiveMq broker. The argument string args is described above. The endpoint ep is the service endpoint used to register the service.

Lines 100-119 These lines parse the endpoint ep its components comprising the UIMA-AS queue name and the URL to the service broker.

Lines 121-125 These lines disable most UIMA-AS logging as these messages can be quite numerous. However, during debugging it may be desired to change the logging levels here.

Lines 130-172 These lines parse the service argument string args into its constituent parts and places the values in variables. They initialize the expansion and deletion window and normalize it to one slot per minute, regardless of the actual ping rate.

The window normalization uses the DUCC-supplied value monitor_rate to determine the number of slots in the windows.

Lines 176-177 These lines initialize the DUCC-supplied UimaAsServiceMonitor that queries the UIMA-AS queues, and it resets the queue statistics via JMX so the monitor can make accurate measurements.

Lines 181-187 These lines implement the required stop method which is invoked when the Service Manager needs to stop the pinger for any reason. They stop the ActiveMQ queue monitor and emit a shutdown message.

Lines 191-240 These lines define the required getStatistics method. This method collects ActiveMQ statistics, issue get-meta to the service to see if it is responding, sets the formatted information string into the ping reply, and invokes the code to calculate a potential redeployment of service instances.

Lines 245-248 These lines override the optional getLastUse method which simply returns the time of last known use of the service. The actual value is calculated in the pinger-specific calculateNewDeployment method, described below.

Lines 253-298 These lines define the pinger-specific calculateNewDeployment method. This is invoked after get-meta is called and after the UIMA-AS queue has been queried in ActiveMQ. This is the key method of this pinger. It uses information passed in on the last ping from the Service Manager in conjunction with information in the ActiveMQ queue to determine if more, or fewer service instances are needed to meet the performance goals. If fewer instances are needed, it selects specific instances to stop. The method is described in detail below.

Lines 407-410 These lines override the optional getAdditions method. The method returns the number of new service instances required to meet performance goals, as calculated in calculateNewDeployment.

Regardless of what this method returns, the Service Manager may choose not to start new instances, based on its configured maximum, ducc.sm.max.instances as defined in ducc.properties.

Lines 416-419 These lines override the optional getDeletions method. This method returns the specific service instances to be stopped, if any.

The DUCC-assigned unique IDs of all service instances are passed in to the pinger on each ping. These instances are monotonically increasing over time so pingers may assume that lower numbers represent older instances.

Lines 429-480 These lines define a class used as a call-back on the UIMA-AS get-meta requests to determine the host and PID of the service instance responding to the get-meta. If the get-meta request should timeout, this information can be used to help identify ailing or overloaded service instances.

5.8.3 Calculating New Deployments in the Pinger

his section details the use of ActiveMQ queue statistics in conjunction with the Service Monitor data to calculate the number of service instances to increase or decrease.

It is important that this code be very careful about “smoothing” the performance statistics to keep growth and shrinkage stable. Things to take into consideration include:

To handle this data smoothing, the SamplePing classes uses two time-based windows, one for growth, and one for shrinkage, to keep growth and shrinkage stable. The window size is defined in the service ping argument window. Each window period, if more services are needed, a mark is made in the current slot of the expansion window; otherwise the current slot is cleared. Similarly, each period, if fewer services are needed, a mark is made in the shrinkage window; otherwise, the current slot is cleared.

After the marks are made, if the expansion window has all slots filled, a request for new processes is made; thus, a short period of increased does not destabilize the system with a request for services that may be of little use. Additionally, when a request is made, the number of new processes requested is capped by the ping argument max-growth to insure that the service grows smoothly. And finally, if the service is already at some configured maximum number of instances, defined by the max parameter, no additional instances are requested.

Similarly, the shrinkage window is used to govern shrinkage. All slots must be filled, indicating the service has been over-provisioned for a while, before a request is made to delete instances. The number of instances is never reduced below the configured min value. As well, this particular pinger never shrinks by more than a single instance at a time, on the reasoning that it is more costly to start a new service than to maintain one for too long. Only if there is no long-term use of the extra instances are they reduced (as determined by the window).

The code then gets the DUCC IDs of all the currently started service instances, and the number of instances that are started but still in their “initialization” phase. This is important because instances that are still initializing are not servicing the queue, but will soon start to do so. The current ActiveMQ statistics reflect do NOT yet reflect this however, they reflect only the instances that are actually serving.

Finally, if there are service producers, we note the time of day to return to the SM as the last known use of this service by some process.

Lines 267 This line calculates the number of Java threads per service instance, needed to calculate the maximum capacity of the service in its current deployment.

(Note that in each UIMA-AS service, UIMA-AS itself occupies one thread, used to manage the service, and this thread manifests itself as a consumer on the queue.)

Line 301 This declares new_ni, the number of additional instances, if any. At the end of this method, new_ni will either be 0 or >0.

Lines 303-312 If the current queue depth is 0 (Q == 0), we know a number of things:

There is nothing else that can be said about a service if its current queue depth is 0.

Lines 312-360 If the queue depth is non-zero we are able to calculate the total service capacity and the amount each instance contributes to the total capacity. From this we can determine

Lines 314 and 315 The average time a single instance takes to serve a single request, Ti is given by the simple formula

where goal is given by the ping arguments. The current ratio of actual service time to desired is then given by

Because we know that the DUCC job driver will never over-commit; that is, we know the current demand will remain constant unless the jobs using the service expand or contract (which are relatively rare events), we can state that the number of service instances required is directly proportional to r.

If r > 1 we may need more instances to meet our goal and if r < 1 we may be over-provisioned.

Lines 325-347 If r > 1 we may be over-provisioned. We calculate the number of required instances by multiplying the current instances by r and rounding down. We account for instances that we know are starting but not yet started, cap on max instances per service, and again on max growth per cycle.

If we still require additions, we make a mark in the expansion window, otherwise we clear the expansion window.

Lines 349-360 If r < 1 we need to calculate shrinkage. Because starting instances is expensive we conservatively use r < .5 instead and make a mark in the shrinkage window.

Lines 367-396 Finally we sum across the shrinkage and expansion windows. If either window is full, we schedule growth (line 375, set the variable additions) or shrinkage (line 388, set deletions).

Note that to schedule shrinkage, we must choose a specific instance. In this case we choose the newest instance, i.e. the one with the largest DUCC ID, as it is most likely not to have initialized, or perhaps not to have “warmed up” (i.e. caches filled, etc.). We could choose more than one but this pinger is conservative and only shrinks by one instance each time.

5.8.4 Summary of Sample Pinger

This pinger illustrates these functions over-and above the functions provided by the default UIMA-AS pinger:

It illustrates one mechanism for smoothing growth and shrinkage of a service to prevent thrashing in your system.

It illustrates one mechanism for determining the actual performance of a service by analyzing ActiveMQ queueing statistics.

Chapter 6
Job Logs

The ”user logs” are the Job Driver (JD) and Job Process (JP) logs. There is one JP log for each process of a job. These hold the redirected stdout and stderr messages from user code as well as some DUCC-generated messages. These DUCC messages are managed by Apache log4j using the configuration file $DUCC_HOME/resources/log4j.xml. It is not in the scope of this document to describe log4j or its configuration mechanism. Details on log4j can be found at http://logging.apache.org/log4j.

Contents of the Log Directory A number of other useful files are written to the log directory:

Job Process Logs The Job Process logs are written to the configured log directory. There is one job process log for every job processes started for the job. The log names are of the following form:

Job Driver Logs There are several Job Driver logs. 988-JD-agent86-1-58087.log jd.out.log jd.err.log

Sample Log Directory This shows the contents a sample log directory for a small job that consisted of two processes.

Chapter 7
Job Error Handler

Overview The ErrorHandler allows for the per Job customized handling of runtime anomalies.

Operation The Job Driver comes with a built-in ErrorHandler. Its purpose is to instruct the Job Driver on what action(s) to take when a work item error is encountered.

Programmability The Job Driver built-in (or custom) ErrorHandler behavior can be modified according to the driver_exception_handler_arguments string in the Job Specification. Currently recognized are:

Chapter 8
DUCC Web Server

The DUCC Web Server default address is accessed from the URL http://[DUCC-HOST]:42133. The [DUCC-HOST] is the hostname where the local installation has installed the DUCC Web Server.

The Webserver is designed to be mostly self-documenting. The design is intentionally simple and contains a link to this document. Most of the interesting fields and column headers have “mouse hovers” which display a short description if you hover your mouse pointer over it for a moment.

Normally, the Web Server automatically fetches new data from DUCC and updates the display. This is controlled by setting one of the two refresh modes:

Refresh Refresh causes the current data on the page to be updated with the most current information in the Webserver’s possession. This is performed when the refresh button is clicked.

Reload Reload occurs when the enter key is pressed. Reload causes not just the data to be updated but rather the entire page is replaced.

Scroll Mode When scroll table style is the preference, a scroll bar is shown to the right, within the main window. The scroll bar allows scrolling to be restricted to the data display, leaving column and DUCC headers in place. In this mode any column may be sorted simply by clicking on it.

With respect to sorting, any specified sort is remembered for refresh but forgotten for reload. Sorting is permitted when either manual or automatic refresh mode is selected.

Note that not all pages have a scroll version - some only have a classic version.

Classic Mode When classic table style is the preference, the main data may extend below the bottom of the page and it will be necessary to use the browser’s scroller on the right to access it. The column headers and DUCC header scrolls off when doing this. Columns may be sorted in this mode but it is necessary to first switch to “Manual” refresh mode to prevent browser refreshes during sorting and display of data.

With respect to sorting, any specified sort is forgotten for refresh and reload. Sorting is only permitted when manual refresh mode is selected.

8.1 Common Links

Every page contains a common header containing links and controls. The links permit navigation to other content at the site. The controls provide page-wise configuration of the content at that page.

8.2 Login

Access for monitor and update capability via the Web Server is controlled by login. Logs, files and other data are accessed by the Linux permissions of the logged-in user.

Specifications for Jobs and Managed Reservations are kept in the database. Read access is granted for a logged-in user via site-security-directory/.ducc/db.access file read permission.

For the Services Registry in particular, the following attributes give the Web Server browsing user read or write access:

8.3 Jobs Page

The Web Server’s home page is also the Jobs page. This page has links to all the rest of the content at the site and shows the status of all the jobs in the system.

8.4 Job Details Page

This page shows details of all the processes that run in support of a job. The information is divided among five tabs:

8.4.1 Processes

8.4.2 Work Items

8.4.3 Performance

This tab shows performance summaries of all the pipeline components. The statistics are aggregated over all instances of each component in each process of the job.

8.4.4 Specification

This tab shows the full job specification in the form of a Java Properties file. This will include all the parameters specified by the user, plus those filled in by DUCC.

8.4.5 Files

8.5 Reservations Page

This page shows details of all reservations. There are two types of reservations: managed and unmanaged.

A managed reservation is a reservation whose process is fully managed by DUCC. This process is any arbitrary process and is submitted with the ducc_process_submit CLI. The lifetime of the reservation starts at the time DUCC assigns a unique ID, and ends when the process terminates for any reason.

An unmanaged reservation is essentially a sandbox for the user. DUCC starts no processes in the reservation and manages none of the processes which run on that host. The lifetime of the reservation starts at the time DUCC assigns a unique ID, and ends when the submitter or system administrator cancels it.

8.6 Managed Reservation Details Page

This page shows details of the processes which run in a managed reservation. The information is divided between three tabs:

8.6.1 Processes

8.6.2 Specification

This tab shows the full managed reservation specification in the form of a Java Properties file. This will include all the parameters specified by the user, plus those filled in by DUCC.

8.6.3 Files

8.7 Services Page

8.8 Service Details Page

8.8.1 Deployments

8.8.2 Registry

This tab shows the full service specification in the form of a Java Properties file. This will include all the parameters specified by the user, plus those filled in by DUCC.

8.8.3 Files

8.8.4 History

8.9 System Pages

8.9.1 Administration

8.9.2 Broker

This page shows selective information for the system’s broker. Information includes host, port, version, uptime, memory used, threads, load average, topics and queues.

8.9.3 Classes

This page shows the definitions of the DUCC scheduling classes. The scheduling classes are discussed in more detail in the Resource Manager section.

8.9.4 Daemons

This page shows the current state of all DUCC processes. By default, only the administrative processes, Broker, Database, Orchestrator, ProcessManager, ResourceManager, ServiceManager, and Webserver are shown. A button in the upper left of the page titled “Show Agents” enables display of the status of all the DUCC agents as well. (Agents are suppressed by default because the page is expensive to render for large systems.)

8.9.5 Machines

8.10 Visualization

This page shows a visualization of all scheduled work. Every host is represented by a square whose area is proportional to the amount of memory on the host. If work is scheduled to a host, it is represented by a rectangle whose area is proportional to the amount of memory that is scheduled for the work. In a multi-user environment, each userid is mapped into a different color, making it possible to see the usage per-user.

Hovers are provided to show the real memory size of each host, the schedulable memory for each host, and the amount of memory scheduled for each bit of work.

If multiple allocations are made on a single host for the same job or service, the rectangles are combined into a single rectangle, reducing clutter and better showing the actual usage of the job (or service).

Clicking on any box representing scheduled work sends the browser to the details page for the corresponding work.

The screenshot below shows a visualization with a handful of 127GB hosts, 48GB hosts, and 32GB hosts. Regular UIMA-AS jobs show as untextured boxes; for example, job 6080, owned by user Hilaria, running in a 37GB allocation in host bluej291-41 which is a 127GB host.

Hosts bluej291-45 and 291-46 are running Managed Reservations, which are shown with crosshatches from lower-left to upper right.

Hosts bluej291-37 and bluej291-40 are running Unmanaged Reservations, shown with vertical-horizontal crosshatches.

Below bluej291-34, bluej291-36, bluej293-49, and bluej293-60 are running DUCC-managed services, shown by crosshatching from upper-left to lower-right.

The host representations may be sorted by clicking on the “size” or the “name” text near the top of the display.

8.11 JSON

Part III
Programming Model And Applications

Chapter 9
Building and Testing Jobs

9.1 Overview

A DUCC job consists of two process types, a Job Driver (JD) and one or more Job Processes (JPs). These processes communicate via HTTP. The Job Driver process wraps the job’s Collection Reader (CR). The CR function is to define the set of Work Items to be processed. The Collection Reader provides a small CAS for each Work Item containing a reference to the Work Item data. The Job Processes request Work Item CASes from the Job Driver via HTTP and present them to the job’s analytic pipeline.

A basic job’s analytic pipeline consists of an Aggregate Analysis Engine comprised by the user specified CAS Multiplier (CM), Analysis Engine (AE) and CAS Consumer (CC) components, along with a built-in DUCC Flow Controller. The Work Item CAS is typically sent only to the CM and returned by the Job Process when all child CASes produced by the CM have completed processing; optionally the CR can configure Work Item CAS flow to go to the CC or to the AE & CC to complete all processing for that Work Item.

9.1.1 Basic Job Process Threading Model

In addition to the pipeline definition of explicitly named CM, AE and CC components, the job specification also includes the number of pipeline threads to run in each Job Process (using the job specification parameter: process_pipeline_count). Each pipeline thread receives Work Items independently.

DUCC creates an aggregate descriptor for the pipeline, and then creates a Deployment Descriptor for the Job Process which specifies the number of synchronous pipelines.

9.1.2 Alternate Pipeline Threading Model

Alternately a Job Process can be fully specified by a user submitted UIMA-AS Deployment Descriptor. Thus any UIMA-AS service deployment can be used as a Job Process. Here the parameter process_pipeline_count just defines how many Work Items CASes will be sent to each Job Process concurrently.

9.1.3 Overriding UIMA Configuration Parameters

UIMA configuration parameters in the CR, CM, AE or CC components can be overridden using job specification parameters: driver_descriptor_CR_overrides, process_descriptor_CM_overrides, process_descriptor_AE_overrides and process_descriptor_CC_overrides, respectively.

Another approach is to use the External Configuration Parameter Overrides mechanism in core UIMA. External overrides is the only approach available for jobs submitted with a Deployment Descriptor.

9.2 Collection Segmentation and Artifact Extraction

UIMA is built around artifact processing. A classic UIMA pipeline starts with a Collection Reader (CR) that defines collection segmentation, extracts the artifacts to be analyzed and puts them into the CASes to be delivered to subsequent analytic components. A CR designed for a specific data collection is highly reusable for many different analytic scenarios.

A single CR supplying artifacts to a large number of analysis pipelines would be a bottleneck. Not only would artifact data need to be transported twice across the compute cluster, but analysis results would be uselessly returned to the Job Driver. To solve both of these problems, in a DUCC job the CR only sends a reference to the artifacts in the Work Item CAS, and artifact data is read directly by the analysis pipeline.

In DUCC collection processing the role of collection segmentation is implemented by the CR run in the Job Driver, while artifact extraction and CAS initialization are implemented in the Cas Multiplier (CM) run in the Job Process. The combination of a CR and associated CM should be highly reusable.

9.3 CAS Consumer Changes for DUCC

CAS Consumers in a UIMA pipeline may require changes for scale out into DUCC jobs, to avoid scale out bottlenecks, to preserve collection level processing, or to flush results at end-of-work-item processing.

9.4 Job Development for an Existing Pipeline Design

Assuming that an existing job input-output design (CR, CM, CC) is to be reused, job development is focused on the Analysis Engine (AE) to be plugged in. Before deploying a new AE in a multithreaded Job Process it is best to run it single threaded (process_pipeline_count=1) to separate basic logic errors from threading problems.

To debug a Job Process with eclipse, first create a debug configuration for a ”remote java application”, specifying ”Connection Type = Socket Listen” on some free port P. Start the debug configuration and confirm it is listening on the specified port. Then add to the job specification process_debug=port, where port is the value P used in the running debug configuration.

When the process_debug parameter is specified, DUCC will only run a single Job Process that will connect back to the eclipse debug configuration.

9.5 Job Development for a New Pipeline Design

A DUCC job is a UIMA application comprised of user code broken into a Collection Reader running in the Job Driver and an Aggregate Analysis Engine (analysis pipeline) running in one or more Job Processes. Each Job Process may run multiple instances of the pipeline, each in a different thread. The major components of the basic Job Process application are as follows:

9.5.1 Collection Reader (CR) Characteristics

A DUCC Job CR sends Work Item CASes to the Job Processes. These CASes contain references to the data to be read by the Job Processes. Typically the CR Type System will be very small; in the DUCC sample applications the CR Type System only contains the Workitem Feature Structure described below.

9.5.2 DUCC built-in Flow Controller

This flow controller provides separate flows for Work Item CASes and for CASes produced by the CM and/or AE. Its behavior is controlled by the existence of a CM component, and then further specified by the org.apache.uima.ducc.Workitem feature structure in the Work Item CAS.

When no CM is defined the Work Item CAS is simply delivered to the AE, and then to the CC if defined. Any CASes created by the AE will be routed to the CC.

With a defined CM, the Work Item CAS is delivered only to the CM, and then returned from the JP when processing of all child CASes created by the CM and AE has completed. Work Item CAS flow can be further refined by the CR by creating a org.apache.uima.ducc.Workitem feature structure and setting the setSendToLast feature to true, or by setting the setSendToAll feature to true.

9.5.3 Workitem Feature Structure

In addition to Work Item CAS flow control features, the WorkItem feature structure includes other features that are useful for a DUCC job application. Here is the complete list of features:

9.5.4 Deployment Descriptor (DD) Jobs

Job Processes with arbitrary aggregate hierarchy, flow control and threading can be fully specified via a UIMA AS Deployment Descriptor. The Job Process uses a container which employs a UIMA-AS client to send a JMS message comprising each CAS to the service via an internal broker.

9.5.5 Debugging

It is best to develop and debug the interactions between job application components as one, single-threaded UIMA aggregate. DUCC provides an easy way to accomplish this, for both basic and DD job models, using the all_in_one specification parameter.

To debug an all_in_one job with eclipse, first create a debug configuration for a ”remote java application”, specifying ”Connection Type = Socket Listen” on some free port P. Start the debug configuration and confirm it is listening on the specified port. Then, before submitting the all_in_one job, add the argument process_debug=port, where port is the value P used in the running debug configuration.

Chapter 10
Sample Application: Raw Text Processing

10.1 Application Function and Design

This application expects as input a directory containing one or more flat text files, uses paragraph boundaries to segment the text into separate artifacts, processes each artifact with the OpenNlpTextAnalyzer, and writes the results as compressed UIMA CASes packaged in zip files. Paragraph boundaries are defined as two or more consecutive newline characters.

By default each input file is a Work Item. In order to facilitate processing scale out, an optional blocksize parameter can be specified that will be used to break larger files into multiple Work Items. Paragraphs that cross block boundaries are processed in the block where they started. An error is thrown if a paragraph crosses two block boundaries.

An output zip file is created for each Work Item. The CAS compression format is selectable as either ZIP compressed XmiCas or UIMA compressed binary form 6 format. When compressed binary is used, each zip file also contains the full UIMA Type System in ZIP compressed text. CASes in UIMA compressed binary form 6 format have the same flexibility as an XmiCas in that they can be deserialized into a CAS with a different, but compatible Type System.

By default any previously completed output files found in the output directory are preserved. While Work Item processing is in progress the associated output files have ”_temp” appended to their filenames, and any such incomplete output files are always ignored for subsequent jobs.

10.2 Configuration Parameters

The Collection Reader for this job is the DuccJobTextCR. It has the following configuration parameters:

The CAS Consumer is the DuccCasCC and has the following configuration parameters:

10.3 Set up a working directory

For this and the following sample program, create a working directory in a writable filesystem.

Copy a UIMA logger configuration file that suppresses tons of output from OpenNLP:

Copy the executable code and resources for the DUCC sample application components. If using UIMA v2.x:

If using UIMA v3.x, update the example JCas classes and then copy the updated uima-ducc-examples*.jar from the /tmp folder to lib, as below:

For reference the source code for DUCC sample applications is in $DUCC_HOME/examples/src, with descriptors in $DUCC_HOME/examples/sampleapps/descriptors.

10.4 Download and Install OpenNLP

Download the OpenNLP source distribution from http://opennlp.apache.org and follow the directions in the UIMA Integration section of the included documentation to build the UIMA pear file. Then install the UIMA pear file in a directory (which we will refer to below as $OPENNLP_HOME) with the runPearInstaller script and test it with the UIMA Cas Visual Debugger application.

A small modification of the installed OpenNLP descriptor file is necessary for DUCC to run the component multithreaded. Edit opennlp.uima.OpenNlpTextAnalyzer/desc/OpenNlpTextAnalyzer.xml and change the setting for multipleDeploymentAllowed from false to true.

10.5 Get some Input Text

Choose one or more flat text files in UTF8 format that only use newline characters, not CR-LF sequences. The text should be big enough to see the impact of DUCC job scale out. We used test data from gutenberg.org at

downloading ’Plain Text UTF-8’ versions of Moby Dick, War and Peace and The Complete Works of William Shakespeare as flat text files in a subdirectory ‘Books’, and removing all ’CR’ characters (0xD) as well as extraneous text.

10.6 Run the Job

The job specification, DuccRawTextSpec.job, uses placeholders to reference the working directory and various operational components located there. As run below the placeholders are resolved from environmental variables.

The total size of the three txt files is 9.4Mbytes and with a blocksize of 100000 there are 100 Work Items. Each Job Process is configured to run 8 parallel OpenNLP pipelines. To examine the performance of processing with just a single Job Process, the job can be submitted as:

10.7 Job Output

There will be an output zipfile for every Work Item, with zipfiles containing a compressed CAS for each document (paragraph) found in a Work Item. If UseBinaryCompression=true each zipfile will also contain the TypeSystem for the CASes. This is needed when deserializing these CASes into a different TypeSystem.

DuccTextCM finds 19245 paragraphs in the three text files. If the output CASes are stored as 19245 uncompressed XMI files, the total size is 911MB. Using the default ZIP compressed XMI format and packed into 100 Work Item zip files, the total size is 165MB, a 5.5x compression. Using UIMA binary compressed format further reduces total size to 62MB.

This output data will be used as input data for the following CAS input processing sample application.

10.8 Job Performance Details

DUCC captures a number of process performance metrics. Figure 10.1 shows details on the JD and single JP processes. The %CPU time shown, 728, is lower than the actual because the Job Process was idle for some time before it received the first Work Item and also idle between finishing the last Work Item and being shut down. DUCC shows the JVM spent a total of 58 seconds in GC (garbage collection), had no major page faults or page space, and used a max of 2.1GB of RSS.

On the Performance tab, DUCC shows the breakdown of clock time spent in each primitive UIMA component running in the Job Process. See Figure 10.2. Processing time was dominated by the Parser component at 76.7%. The time spent compressing and writing out CASes was 0.5%, and the time reading the input text files well below 0.1%.

Chapter 11
Sample Application: CAS Input Processing

11.1 Application Function and Design

The main purpose of this application is to demonstrate the overhead of processing a collection of CASes grouped into zipfiles and stored as ZIP compressed XmiCas or with UIMA compressed binary form 6 format.

11.2 Configuration Parameters

The Collection Reader for this job is the DuccJobCasCR. It has the following configuration parameters:

The CAS Consumer is the DuccCasCC and has the following configuration parameters:

11.3 Run the Job

The job specification, DuccCasInputSpec.job, uses placeholders to reference the working directory and various operational components located there. As run below the placeholders will be resolved from environmental variables.

11.4 Job Performance Details

Figure 11.1 shows the component breakdown using binary CAS compression. Reading and deserializing took 38% vs the 60% spent serializing and writing. Using 8 pipeline threads in one process the 19245 CASes output from the last application were read and re-written in 9 seconds.

11.5 Limiting Job Resources

Although this 8-threaded Job Process was primarily CPU bound doing serialization work, it is possible to become I/O bound with enough threads banging on a shared filesystem. DuccCasInputSpec.job demonstrates how to limit the total number of processing threads to 32 using the combination of process_pipeline_count=8 and process_deployments_max=4.

I/O vs CPU bottlenecks can be detected using the detailed performance job data reported by DUCC and comparing results with various levels of scale out.

Part IV
Ducc Administrators Guide

Chapter 12
Installation, Configuration, and Verification

12.1 Overview

DUCC is a multi-user, multi-system distributed application. For first-time users a staged installation/verification methodology is recommended, roughly as follows:

Nominally user ducc runs the DUCC software. However another user, for example duccster, could be chosen. In this document references to user ducc will mean to the user that you have chosen, be it ducc or duccster or some other. The chosen user is the OS user installing DUCC.

When upgrading from an existing installation the ducc_update script may be used to replace the system files while leaving the site-specific configuration files in place. For more information see ducc_update.

Since with this release the persistence data about completed work is stored in a database, additional upgrade steps are required to convert the older file-based data in order to preserve information about past work. For more information see db_create and db_loader.

DUCC is distributed as a compressed tar file. If building from source, this file will be created in your svn trunk/target directory. The distribution file is in the form

where [version] is the DUCC version; for example, uima-ducc-2.1.0-bin.tar.gz. This document will refer to the distribution file as the “<distribution.file>”.

12.2 Software Prerequisites

Additional requirements for running multiple user processes with their own credentials.

The DUCC webserver server optionally supports direct “jconsole” attach to DUCC job processes. To install this, the following is required:

12.3 Building from Source

To build from source, ensure you have Subversion and Maven installed. Extract the source from the SVN repository named above.

Then from your extract directory into the root directory (usually current-directory/trunk), and run the command

if you have LaTeX installed and wish to do the optional build of documentation. The build-duccdocs profile can also activated if the environment valiable BUILD_DUCCDOCS is set true.

An alternative approach for modifying DUCCDOCS is to use the LaTeX commands manually. Not only is the build much faster, it also exposes LaTeX error messages lost by the suppression. To build duccbook.html manually and leave the output in the same directory:

Note: The manual command needs to be run twice if modifications have been made that affect internal book references such as the table of contents. Also, temporary files created by LaTeX will need to be cleaned up manually.

If this is your first Maven build it may take quite a while as Maven downloads all the open-source pre-requisites. (The pre-requisites are stored in the Maven repository, usually your $HOME/.m2).

When build is complete, a tarball is placed in your current-directory/trunk/target directory.

12.4 Documentation

After installation the DUCC documentation is found (in both PDF and HTML format) in the directory ducc_runtime/docs. As well, the DUCC webserver contains a link to the full documentation on each major page. The API is documented only via JavaDoc, distributed in the webserver’s root directory $DUCC_HOME/webserver/root/doc/api.

12.5 Single System Installation and Verification

Any user ID can be used to run a single-system DUCC, but “ducc” userid is recommended. This user will employ cgroups as well as launching and running processes with the credentials of the submitting user.

If multiple nodes are going to be added later, the ducc runtime tree should be installed on a shared filesystem so that it can be mounted on the additional nodes.

Verification submits a very simple UIMA pipeline for execution under DUCC. Once this is shown to be working, one may proceed installing additional features.

12.6 Minimal Hardware Requirements for Single System Installation

Please note: DUCC is intended for scaling out memory-intensive UIMA applications over computing clusters consisting of multiple nodes with large (16GB-256GB or more) memory. The minimal requirements are for initial test and evaluation purposes, but will not be sufficient to run actual workloads.

12.7 Single System Installation

That’s it, DUCC is installed and ready to run. (If errors were displayed during ducc_post_install they must be corrected before continuing.)

12.8 Initial System Verification

Here we verify the system configuration, start DUCC, run a test Job, and then shutdown DUCC.

Start_ducc will first perform a number of consistency checks. It then starts the ActiveMQ broker, the DUCC control processes, and a single DUCC agent on the local node.

Now open a browser and go to the DUCC webserver’s url, http://<hostname>:42133 where <hostname> is the name of the host where DUCC is started. Navigate to the Reservations page via the links in the upper-left corner. You should see the DUCC JobDriver reservation in state WaitingForResources. In a few minutes this should change to Assigned. Now jobs can be submitted.

12.8.1 Submitting a test job

Open the browser in the DUCC jobs page. You should see the job progress through a series of transitions: Waiting For Driver, Waiting For Services, Waiting For Resources, Initializing, and finally, Running. You’ll see the number of work items submitted (15) and the number of work items completed grow from 0 to 15. Finally, the job will move into Completing and then Completed..

Since this example does not specify a log directory DUCC will create a log directory in your HOME directory under

In this directory, you will find a log for the sample job’s JobDriver (JD), JobProcess (JP), and a number of other files relating to the job.

This is a good time to explore the DUCC web pages. Notice that the job id is a link to a set of pages with details about the execution of the job.

Notice also, in the upper-right corner is a link to the full DUCC documentation, the “DuccBook”.

12.8.2 Registering a test service

First start an application broker using the UIMA-AS script included in the DUCC installation, see UIMA-AS broker. Then in another shell define the broker URL using the name of the host the broker was started on, and register the service. On your browser select the Services page and wait for the service to become Available.

12.8.3 To stop DUCC

12.9 Add additional nodes to the DUCC cluster

$DUCC_HOME must be on a shared filesystem and mounted at the same location on all DUCC nodes.

If user’s home directories are on local filesystems the location for user logfiles should be specified to be on a shared filesystem.

Addional nodes are normally added to a worker node group. Note that the DUCC head node does not have to be a worker node. In addition, the webserver node can be separate from the DUCC head node (see webserver configuration options in ducc.properties).

For worker nodes DUCC needs to know what node group each machine belongs to, and what nodes need an Agent process to be started on.

The configuration shipped with DUCC have all nodes in the same ”default” node pool. Worker nodes are listed in the file

During initial installation, this file was initialized with the node DUCC is installed on. Additional nodes may be added to the file using a text editor to increase the size of the DUCC cluster.

12.10 Ducc_ling Configuration - Running with credentials of submitting user

DUCC launches user processes through ducc_ling, a small native C application. By default the resultant process runs with the credentials of the user ID of the DUCC application. It is possible for multiple users to submit work to DUCC in this configuration, but it requires that the user ID running DUCC has write access to all directories to which the user process outputs data. By configuring the ducc user ID and ducc_ling correctly, work submitted by all users will run with their own credentials.

ducc_ling is designed to be a setuid-root program whose function is to run user processes with the identity of the submitting user. This must be installed correctly; incorrect installation can prevent jobs from running as their submitters, and in the worse case, can introduce security problems into the system.

ducc_ling can either be installed on a local disk on every system in the DUCC cluster, or on a shared-filesystem that does not suppress setuid-root permissions on client nodes. The path to ducc_ling must be the same on each DUCC node. The default path configuration is $DUCC_HOME/admin/${os.arch}/ in order to handle clusters with mixed OS platforms. ${os.arch} is the architecture specific value of the Java system property with that name; examples are amd64 and ppc64.

The steps are: build ducc_ling for each node architecture to be added to the cluster, copy ducc_ling to the desired location, and then configure ducc_ling to give user ducc the ability to spawn a process as a different user.

As user ducc, build ducc_ling for necessary architectures (this is done automatically for the DUCC head machine by the ducc_post_install script). For each unique OS platform:

Then, as user root on the shared filesystem, cd $DUCC_HOME/admin, and for each unique OS architecture:

If these steps are correctly performed, ONLY user ducc may use the ducc_ling program in a privileged way. ducc_ling contains checks to prevent even user root from using it for privileged operations.

If a different location is chosen for ducc_ling the new path needs to be specified for ducc.agent.launcher.ducc_spawn_path in $DUCC_HOME/resources/site.ducc.properties. For more information see Properties merging.

12.11 CGroups Installation and Configuration

Note: if CGroups is not installed on a machine the DUCC Agent will detect this and not attempt to use the feature. CGroups can also be disabled for all machines or for individual machines. Use ducc.agent.launcher.cgroups.enable and ducc.agent.exclusion.file respectively in site.ducc.properties.

12.12 Full DUCC Verification

This is identical to initial verification, with the one difference that the job “1.job” should be submitted as any user other than ducc. Watch the webserver and check that the job executes under the correct identity. Once this completes, DUCC is installed and verified.

12.13 Enable DUCC webserver login

This step is optional. As shipped, the webserver is disabled for logins. This can be seen by hovering over the Login text located in the upper right of most webserver pages:

To enable logins, a Java-based authenticator must be plugged-in and the login feature must be enabled in the ducc.properties file by the DUCC administrator. Also, ducc_ling should be properly deployed (see Ducc_ling Installation section above).

A beta version of a Linux-based authentication plug-in is shipped with DUCC. It can be found in the source tree:

The Linux-based authentication plug-in will attempt to validate webserver login requests by appealing to the host OS. The user who wishes to login provides a userid and password to the webserver via https, which in-turn are handed-off to the OS for a success/failure reply.

To have the webserver employ the beta Linux-based authentication plug-in, the DUCC administrator should perform the following as user ducc:

Note: The beta Linux-based authentication plug-in has limited testing. In particular, it was tested using:

A beta version of a File-based authentication plug-in is shipped with DUCC. It can be found in the source tree:

The File-based authentication plug-in will attempt to validate webserver login requests by checking againist a file that only the user can view, either in the security directory specified in ducc.properties, else in the user home directory. The user who wishes to login provides a userid and password to the webserver via https, which in-turn are checked against the file-based password for a success/failure reply.

Each time a login is attempted by the user, a new password is generated in the secure file. Thus, each login password is single use.

To have the webserver employ the beta File-based authentication plug-in, the DUCC administrator should perform the following as user ducc:

12.14 DUCC webserver user data restricted access

To configure for user data secure access, use the following settings (port numbers can be any available, but normally as shown below):

With these settings, users will be required to login to the DUCC Web Server to view user data (e.g. user log files). In addition, user data will be transported over https only.

12.15 DUCC daemons monitoring and notification

Chapter 13
Administration

13.1 WebServer Authentication

By default, DUCC is configured such that there is effectively no authentication enforcement by the WebServer. No password entry is permitted on the Login panel and any userid specified is accepted whether it exists or not.

To enable your own authentication measures, you should perform the following steps:

Note: When a user clicks on the WebServer Login link, the login dialog is shown. On that dialog panel is shown the authenticator: version, which is supplied by your authentication manager implementation’s getVersion() method. Also shown are boxes for userid and password entry. If your authentication manager implemenation’s isPasswordChecked() method returns true then the password box will accept input, otherwise it will be disabled.

13.1.1 Example Implementation

Shown below is an example implementation which can be used as a template for coding protection by means of interfacing with your site’s security measures.

In this example, the SiteSecurity Java class is presumed to be existing and available code at your installation.

13.1.2 IAuthenticationManager

Shown below is the interface which must be implemented by your authentication manager.

13.1.3 IAuthenticationResult

Shown below is the interface which must be returned by the required authentication methods in your authentication manager.

13.1.4 Example ANT script to build jar

Shown below is an example ANT script to build a ducc-authenticator.jar file. The resulting jar file should be placed user DUCC’s lib directory along with any dependency jars, and defined in ducc.properties file.

13.1.5 Example ducc.properties entries

Shown here is a snippet of the ducc.properties file defining the class to be used for authentication and the administrator created folder site-security, which should contain the ducc-authenticator.jar you built plus any jar files upon which it depends.

Note: the site-security directory must be located within DUCC’s lib directory.

13.1.6 Example ducc.administrators

Example contents of ducc.administrators file located within DUCC’s resources directory. Only userids listed here can assume the Administrator role when performing operations via the WebServer.

13.2 Properties

Public properties are in a primary configuration file is called ducc.properties and always resides in the directory ducc_runtime/resources.

Private properties are in a secondary configuration file call ducc.private.properties and always resides in the directory ducc_runtime/resources/private.

13.3 Properties merging

With DUCC 2.0.0 the shipped DUCC properties file is designed to be read-only. Installations create a local properties file which is automatically merged with the default properties file as part of system startup.

The shipped DUCC properties file is called default.ducc.properties. This file should never be edited or modified.

The local site override properties file is called site.ducc.properties. This is a normal Java properties file containing override and additional properties. An initial site.ducc.properties is created on installation of DUCC 2.0.0 by ducc_post_install.

On startup (start_ducc), verification (check_ducc), and RM reconfiguration (rm_reconfigure), the two properties files are merged, with site.ducc.properties taking preference, to create the operational file, ducc.properties, which is used by all DUCC components. This file should not be edited as it will be over-written whenever start_ducc or check_ducc is run.

13.4 ducc.properties

Some of the properties in ducc.properties are intended as the ”glue” that brings the various DUCC components together and lets then run as a coherent whole. These types of properties should be modified only by developers of DUCC itself.

Some of the properties are tuning parameters: timeouts, heartbeat intervals, and so on. These may be modified by DUCC administrators, but only after experience is gained with DUCC, and only to solve specific performance problems. The default tuning parameters have been chosen by the DUCC system developers to provide ”best” operation under most reasonable situations.

Some of the properties describe the local cluster configuration: the location of the ActiveMQ broker, the location of the Java JRE, port numbers, etc. These should be modified by the DUCC administrators to configure DUCC to each individual installation.

13.5 default.ducc.properties

# -----------------------------------------------------------------------
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
# -----------------------------------------------------------------------

# WARNING: DO NOT EDIT THIS FILE.
# All customizations must be created in a file "site.ducc.properties" that is in
# your ducc_runtime/resources.  DUCC’s startup procedures will merge this file and
# your site.ducc.propeties into ducc.properties which is what DUCC will use.
# WARNING DO NOT EDIT THIS FILE.

# ====================================================================================
#   This configuration file contains most of the specifications for DUCC.
#
#   Two other important customizable files define the classes and nodes.
#
#   The class definition file is specified by the property ’ducc.rm.class.definitions’
#   and a sample called ducc.classes has been provided in DUCC_HOME/resources.
#
#   Node definitions are by default taken from ducc.nodes but this may be overridden
#   with the -n option on start_ducc.
#
#   The first two entries here are customized in site.ducc.properties by ducc_post_install.
#
# ====================================================================================

# +==================================================================================================+
# | General                                                                                          |
# +==================================================================================================+

# The name of the node where DUCC runs.
# This property declares the node where the DUCC administrative processes run (Orchestrator,
# Resource Manager, Process Manager, Service Manager).  This property is required and MUST be
# configured in new installation.  The installation script ducc_post_install initializes this
# property to the node the script is executed on.
# Reliable DUCC: if running reliably, then this value must resolve to the same ip address
# specified for the virtual_ipaddress in /etc/keepalived/keepalived.conf for master and
# backup nodes.  DUCC CLI and Agents employ this value to connect to the current reliable
# DUCC head node.
ducc.head = <head-node>

# Reliable DUCC: if running reliably, then this value must comprise the blank delimited list
# of nodes that are eligible to become the DUCC head node.  Admin commands start_ducc and
# stop_ducc are only allowed on the ducc.head node or any node in the ducc.head.reliable.list.
# An empty ducc.head.reliable.list indicates that DUCC is not running in reliably.
ducc.head.reliable.list =

# The full name of the Java command.
# This specifies the full path to the JVM to be used by the DUCC processes. This MUST be
# configured.  The installation script ducc_post_install initializes this property to
# full path to java in the installer’s environment.  (If the java command cannot be found,
# ducc_post_install exits with error.)
ducc.jvm  = <full-path-to-java-command>

# The name of the cluster as shown by the Web Server.
# This is a string used in the Web Server banner to identify the local cluster. It is used
# for informational purposes only and may be set to anything desired.
ducc.cluster.name=Apache UIMA-DUCC

# Specify location of private resources directory.  UIMA-3892
ducc.private.resources = ${DUCC_HOME}/resources.private

# Location of security home directory.  When non-empty the userid is appended and it replaces the
# default of <user.home> as the location of the keys used in request validation.
ducc.security.home =

# Name any site-local jars.
# This may be used to list site-specific jars that are required by local
# customizations, e.g. authentication.  The value must be a blank-delimited list of jars
# relative to the DUCC_HOME/lib directory.
#ducc.local.jars =

# Declare the type of middleware providing the JMS service used by DUCC.
ducc.jms.provider=activemq

# +==================================================================================================+
# | Broker                                                                                           |
# +==================================================================================================+

#Declare the wire protocol used to communicate with ActiveMQ.
ducc.broker.protocol=tcp

# This declares the port on which the ActiveMQ broker is listening for messages.
ducc.broker.port=61617

# The broker *client* url decoration (ie - DUCC daemons).
ducc.broker.url.decoration=jms.useCompression=true&jms.prefetchPolicy.all=0

# The Broker’s name must match the actual broker name in the broker config.
# This is the internal name of the broker, used to locate Broker’s MBean in
# JMX Registry. It is NOT related to any node name. When using the ActiveMQ
# distribution supplied with DUCC it should always be set to localhost.
ducc.broker.name=localhost

# The Broker’s jmx port.
# This is the port used to make JMX connections to the broker.  This should only
# be changed by administrators familiar with ActiveMQ configuration.
ducc.broker.jmx.port=1100

# If set to true, DUCC will start and stop the ActiveMQ broker as part of its normal
# start/stop scripting.
ducc.broker.automanage = true

# This is the ActiveMQ credentials file used to authenticate DUCC daemons with the
# broker.
ducc.broker.credentials.file=${ducc.private.resources}/ducc-broker-credentials.properties

# This sets the heap size for the broker.
ducc.broker.memory.options = -Xmx1G

# This is the ActiveMQ configuration file to use.  The path
# must be specified relative to the ActiveMQ installation directory.
ducc.broker.configuration = conf/activemq-ducc.xml

# This names the location where ActiveMQ is installed.
ducc.broker.home = ${DUCC_HOME}/apache-uima/apache-activemq

# The broker "server" URL decoration,
#   = 45 second timeout on write operations to the socket.
ducc.broker.server.url.decoration = transport.soWriteTimeout=45000

# +==================================================================================================+
# | Transport                                                                                        |
# +==================================================================================================+

# To enable tracing of RM messages arriving in OR and NodeMetrics arriving in WS.
#ducc.transport.trace = orchestrator:RmStateDuccEvent webserver:NodeMetricsUpdateDuccEvent

# +==================================================================================================+
# | Authentication                                                                                   |
# +==================================================================================================+

# This specifies the class used for Web Server session authentication.
# If unconfigured, the Web Server enforces no authentication.
#ducc.authentication.implementer=org.apache.uima.ducc.ws.authentication.LinuxAuthenticationManager
#ducc.authentication.implementer=org.apache.uima.ducc.ws.authentication.SecureFileAuthenticator
#ducc.authentication.implementer=org.apache.uima.ducc.ws.authentication.GSAAuthenticator

# Specify users allowed to log in to the web server.
#ducc.authentication.users.include = user1 user2

# Specify users not allowed to login (default is all users can login; if user is in
# both include and exclude lists, then exclude is enforced)
#ducc.authentication.users.exclude = user1 user3

# Specify groups allowed to login (default is all groups can login)
#ducc.authentication.groups.include = groupA groupB

# Specify groups not allowed to login (default is all groups can login; if group is in
# both include and exclude lists, then exclude is enforced)
#ducc.authentication.groups.exclude = groupA groupC

# +==================================================================================================+
# | Language                                                                                         |
# +==================================================================================================+

# Establish the language for national language support of messages.
# Currently only "en" is supported.
ducc.locale.language=en
# Establish the country for National Language Support of messages.
# Currently only "us" is supported.
ducc.locale.country=us

# +==================================================================================================+
# | Daemon Administration                                                                            |
# +==================================================================================================+

# This is the JMS endpoint name used for DUCC administration messages.
ducc.admin.endpoint=ducc.admin.channel

# This is the JMS message type used for DUCC administration messages.
# Only "topic" is supported.
ducc.admin.endpoint.type=topic

# JMX port number for Ducc process. Each DUCC process will attempt
# to use this port for its JMX Connector. If the port is not available
# port+1 will be used until an available port is found.
# Every process started by DUCC has JMX enabled by default.
# The DUCC WebSever’s System.Daemons page is used to find the JMX URL that gets assigned to
# each of the DUCC management processes. The Web Server’s Job.Details page for each job is
# used to find the JMX URL that is assigned to each JP.
ducc.jmx.port=2099

ducc.agent.jvm.args        = -Xmx500M
ducc.orchestrator.jvm.args = -Xmx1G
ducc.rm.jvm.args           = -Xmx1G
ducc.pm.jvm.args           = -Xmx1G
ducc.sm.jvm.args           = -Xmx1G

# use the following flag under IBM Java 8 to allow Chromium to visit
# w/o getting ERR_SSL_VERSION_OR_CIPHER_MISMATCH
# -Dcom.ibm.jsse2.overrideDefaultTLS=true
# see https://issues.apache.org/jira/browse/UIMA-5475

ducc.ws.jvm.args           = -Xmx2G -Djava.util.Arrays.useLegacyMergeSort=true -Dcom.ibm.jsse2.overrideDefaultTLS=true

# +==================================================================================================+
# | Node Administration                                                                              |
# +==================================================================================================+

# Specify a minimum amount of free swap space available on a node.
# If an agent detects free swap space dipping below the value defined
# below, it will find the fattest (in terms of memory) process in its
# inventory and kill it. The value of the parameter below is expressed
# in bytes.
# Initially disabled by setting the threshold at 0.
ducc.node.min.swap.threshold=0

# +==================================================================================================+
# | Job Administration                                                                               |
# +==================================================================================================+

# Max number of work-item CASes for each job (default is "unlimited")
# Note: formerly known as ducc.threads.limit
# This enforces a maximum number of pipelines per job, over all its processes. No
# job will have more active work-items than this dispatched.
# The value is related to the size of the Job Driver heap and the real memory consumed by JD.
# If the JD is consuming too much memory, try reducing this value.
ducc.job.max.pipelines.count = 5000

# +==================================================================================================+
# | CLI Administration                                                                               |
# +==================================================================================================+

# These environment values are included on job/service/AP submissions
ducc.environment.propagated = USER HOME LANG DUCC_SERVICE_INSTANCE

# No timeout on CLI requests
ducc.cli.httpclient.sotimeout=0
#------------------------------------------------------------------------------
# When set, the CLI signs each request so the Orchestrator can be sure the
# requestor is actually who he claims to be.
# off,  // CLI submit and cancel signature enforcement disabled
# on,   // CLI submit and cancel signature enforcement enabled (default)
ducc.signature.required=on
#------------------------------------------------------------------------------

# +==================================================================================================+
# | Web Server                                                                                       |
# +==================================================================================================+

# The name of the pluggable java class used to implement the Web Server.
ducc.ws.configuration.class=org.apache.uima.ducc.ws.config.WebServerConfiguration

# This endpoint is used for Web Server self test to determine viability of broker.
# After 3 missed messages to self via broker, Web Server considers broker to be down.
ducc.ws.state.update.endpoint=ducc.ws.state

# This is the JMS endpoint type used for the state messages sent by the Web Server.
ducc.ws.state.update.endpoint.type=topic

# The interval in milliseconds between Web Server publications of its state.
ducc.ws.state.publish.rate=5000

# The elapsed time in milliseconds between monitored head-node daemons’ publications
# that if exceeded indicates "down". Default = 120000 (two minutes).
ducc.ws.monitored.daemon.down.millis.expiry=120000

# Optionally configure the Web Server to run on a non-head node
# This is the name of the node the web server is started on. If not specified,
# the web server is started on ${ducc.head}.
# ducc.ws.node = my.node.com

# Optionally configure the Web Server IP address
# In multi-homed systems it may be necessary to specify to which of the multiple addresses
# the Web Server listens for requests. This property is an IP address that specifies to which
# address the Web Server listens.
# ducc.ws.ipaddress = <fill in an IP address>

# Optionally configure the Web Server IP port for HTTP requests, default is 42133
ducc.ws.port = 42133

# Optionally configure the Web Server IP port for HTTPS requests, default is 42155
ducc.ws.port.ssl = 42155

# Optionally configure the Web Server welcome page, default is index.html (which forwards to jobs.html)
ducc.ws.welcome.page = index.html

# Optionally configure the Web Server job automatic cancel timeout, default is 10. To disable
# feature specify 0. Employed when user specifies --wait_for_completion flag on job submission,
# in which case the job monitor program must visit
# http://<host>:<port>/ducc-servlet/proxy-job-status?id=<job-id> within this expiry time
# else the job will be automatically canceled (unless feature disabled) by the Web Server
# acting as the administrator ducc (which must be specified in the ducc.administrators file).
ducc.ws.automatic.cancel.minutes = 5

# Optionally configure the Web Server max cached (and thus available for display)
# history entries for each of Jobs/Reservations/Services
ducc.ws.max.history.entries = 4096

# Specify login enabled (default is true)
ducc.ws.login.enabled = false

# For node visualization - if true, strip domain names from labels for cleaner visuals
ducc.ws.visualization.strip.domain = true

# Optionally configure the Web Server request log, default is 0 (meaning no request logging)
# Logs are written to DUCC_HOME/logs/webserver
ducc.ws.requestLog.RetainDays = 30

# Specify one of { unrestricted, encrypted, blocked } to control
# requests to the Web Server with responses containing user data.
# When "unrestricted" requests for user data via http or https are honored.
# When "encrypted" requests for user data only via https are honored.
# When "blocked" requests for user data are not honored.
ducc.ws.user.data.access = unrestricted

# Note: to employ "encrypted" use the following settings:
#   ducc.ws.port = 42133
#   ducc.ws.port.ssl = 42155
#   ducc.ws.login.enabled = true
#   ducc.ws.user.data.access = encrypted
# See documentation for further information.

# --------------------------------------------------------------
#    name: ducc.ws.banner.message
# purpose: display banner message on all main pages
# choices: default=none
#  change: effective immediately
#ducc.ws.banner.message = Do not adjust your set.  This is a message from your DUCC administrator.

# +==================================================================================================+
# | Job Driver                                                                                       |
# +==================================================================================================+

# The name of the pluggable java class used to implement the Job Driver (JD).
ducc.jd.configuration.class=org.apache.uima.ducc.transport.configuration.jd.JobDriverConfiguration

# This is the JMS endpoint name by the Job Driver to send state to the Orchestrator.
ducc.jd.state.update.endpoint=ducc.jd.state

# This is the JMS message type used to send state to the Orchestrator.
ducc.jd.state.update.endpoint.type=topic

# The interval in milliseconds between JD state publications to the Orchestrator.
# A higher rate (smaller number) may slightly increase system response but will
# increase network load. A lower rate will somewhat decrease system response and
# lower network load.
ducc.jd.state.publish.rate=15000

# This is a human-readable string used to form queue names for the JMS queues used to pass
# CASs from the Job Driver to the Job Processes. The completed queue named comprises the prefix
# concatenated with the DUCC assigned Job number.
ducc.jd.queue.prefix=ducc.jd.queue.

# After dispatching a work item to UIMA-AS client for processing, the number of minutes that the
# Job Driver will wait for two callbacks (queued and assigned) before considering the work item
# lost. The elapsed time for the callbacks is normally sub-second. Intermittent network problems
# may cause unusual spikes. If not specified, default value is 5 minutes.
ducc.jd.queue.timeout.minutes=5

# If not specified, default value is 24 hrs (24*60 minutes)
# This property specifies the default value for the time, in minutes, that the JD should
# wait for a work-item to be processed. If processing has not completed in this time the
# process is terminated and the job’s error handler decides whether to retry the
# work-item or not.
ducc.default.process.per.item.time.max = 1440

# If not specified, default max time in minutes allowed for AE initialization.
# This property specifies the default value for the time, in minutes, that the agent should
# wait for a JP to complete initialization. If initialization is not completed in this time
# the process is terminated and and InitializationTimeout status is sent to the JD
# which decides whether to retry the process or terminate the job.
ducc.default.process.init.time.max = 120

# The following 5 values comprise the specification used by the DUCC Orchestrator daemon to
# request an allocation from the DUCC Resource Manager for Job Driver use. The values given
# below are the defaults.
ducc.jd.host.class=JobDriver
ducc.jd.host.description=Job Driver
ducc.jd.host.memory.size=2GB
ducc.jd.host.number.of.machines=1
ducc.jd.host.user=System

# For a newly started Job, the number of JP UIMA initialization failures
# allowed until at least one JP succeeds - otherwise, the Job self-destructs.
# Default is 1.
ducc.jd.startup.initialization.error.limit=1

# The next 4 values are related - each JD is assigned a piece of the Job Driver host memory
# which, # along with the size of the CR’s type system, limits the number of active work-item
# CASes in a job. To avoid swapping the max heap size should also be restricted.

# Memory size in MB allocated for each JD (default 300)
# When CGroups are enabled, this is the RSS, in MB, that is reserved for each JD process,
# and enforced by the CGroup support.  Larger JDs are permitted, but the CGroup support will
# force the excess RSS onto swap.  This potentially slows the performance of that JD, but
# preserves the resources for other, better-behaved, JDs.
ducc.jd.share.quantum = 400

#   The number of "slices" of size "jd.share.quantum" kept in reserve.
#   The Orchestrator makes Reservation requests to RM to get Reservations
#   (Job Driver hosts) each of which is then subdivided into "slices", one
#   per JD.  This number specifies the number of unused "slices" that should
#   be kept on-hand in anticipation of newly submitted jobs (default 2).
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
#   If the value specified is 0 then no JD allocation will take place
#   and all submitted jobs will be rejected.
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ducc.jd.share.quantum.reserve.count = 3

# The maximum length of a work-item name returned by CAS.getDocumentText().
# Truncation is enforced beyond this limit.
# If not specified, there is no limit.
ducc.jd.workitem.name.maximum.length = 64

# Extra JVM args to be appended to any user-supplied "driver_jvm_args"
# Dynamic: changes to this property immediately affect new submitted work
# Flag: -DUimaAsCasTracking will add to logs:
#        UIMA-AS client & service trigger internal tracing including RefId,
#        UIMA-AS client onBeforeMessageSendHandler callbacks including RefId, and
#        UIMA-AS client onBeforeProcessCAS callbacks including RefId
# Flag: -DUimaAsClientTracking will add to jd.out.log:
#        core pool size changes,
#        UIMA-AS client sendAndReceive invocations,
#        UIMA-AS client onBeforeMessageSendHandler callbacks, and
#        UIMA-AS client onBeforeProcessCAS callbacks
# Note: should contain an Xmx a little below the "ducc.jd.share.quantum" value
#       the Xmx entry is ignored if the user has specified one
ducc.driver.jvm.args  = -Xmx300M

# Max number of threads in Jetty thread pool servicing incoming
# HTTP requests
# ducc.driver.jetty.max.threads = 100

# Max idle time for jetty threads (in millis). When a thread exceeds
# its idle time it will be terminated.
# ducc.driver.jetty.thread.idletime = 60000

# Extra JVM args to be appended to any user-supplied "process_jvm_args"
# Dynamic: changes to this property immediately affect new submitted work
# ducc.process.jvm.args=-XX:+HeapDumpOnOutOfMemoryError

# --------------------------------------------------------------
#    name: ducc.jd.communications.scheme
# purpose: specify communications scheme between JD and JPs
# choices: [http,https] default=https
#  change: effective immediately for new jobs
ducc.jd.communications.scheme=https

# --------------------------------------------------------------
#    name: ducc.jd.error.handler.class
# purpose: specify error handler
# default: org.apache.uima.ducc.ErrorHandler
#  change: effective immediately for new jobs
# details: The error handler class is employed when work items fail or time-out in order
#          to determine what to do next in three dimensions:
#          job kill or continue, process kill or continue, work item kill or continue.
ducc.jd.error.handler.class = org.apache.uima.ducc.ErrorHandler

# --------------------------------------------------------------
#    name: ducc.jd.error.handler.args
# purpose: configure error handler
# choices: max_job_errors=<e> max_timeout_retrys_per_workitem=<r>
#  change: effective immediately for new jobs
# details: The error handler args passed to the error handler class (see above).
#          These values, if any, are combined with any specified by the user.
#          Individual user specified --driver_exception_handler_arguments <args>
#          prevail.
ducc.jd.error.handler.args = max_job_errors=15 max_timeout_retrys_per_workitem=0

# +==================================================================================================+
# | Service Manager                                                                                  |
# +==================================================================================================+

# This is the name of the pluggable java class used to implement the Service Manager.
ducc.sm.configuration.class=org.apache.uima.ducc.sm.config.ServiceManagerConfiguration

# This is the JMS endpoint name used for API messages received by the Service Manager.
ducc.sm.api.endpoint=ducc.sm.api

# This is the JMS message type used for API messages received by the Service Manager.
ducc.sm.api.endpoint.type=queue

# This is the JMS endpoint name used for state messages sent by the Service Manager.
ducc.sm.state.update.endpoint=ducc.sm.state

# This is the JMS message type used for state messages sent by the Service Manager.
ducc.sm.state.update.endpoint.type=topic

# Default pinger
# This is the name of the default UIMA-AS ping/monitor class.  The default class issues
# get-meta to a service and uses JMX to fetch queue statistics for presentation in
# the Web Server.
# This name is either:
#    - The fully qualified name of the class to use as the default UIMA-AS pinger. It may
#        be necessary to include the class or jar file in the classpath used to start the SM.
#        (The recommended way to do this is add an entry to the ducc.local.jars property
#        in ducc.properties.
#    - The name of a pinger registration file.  This is the recommended way to
#     provide installation-customized pingers.  In short, it resides in ducc.properties
#     and contains the full set of ping-related properties needed to run a pinger.
ducc.sm.default.monitor.class = org.apache.uima.ducc.sm.UimaAsPing

# This is the maximum number of consecutive failures of service instance initialization
# permitted before DUCC stops creating new instances.  When this cap is hit the SM
# will disable autostart for the service.  It may be overridden by the service
# registration’s instance_failures_limit parameter.
ducc.sm.instance.failure.max 5

# backward compatibility
ducc.sm.instance.failure.limit ${ducc.sm.instance.failure.max}

# This specifies a window of time in minutes over which some number of service instance
# failures are tolerated.  If the maximum number of tolerated failures is
# exceeded within this time window the Service Manager ceases to restart
# instances automatically.  The maximum tolerated failures is defined in
# ducc.sm.instance.failure.max.
# This may be overridden by individual service pingers using the registration
# property instance_failures_window.
ducc.sm.instance.failure.window 30

# max consecutive instance init failures before we stop trying to start things
ducc.sm.init.failure.limit  1

# This is the time, in milliseconds, between pings by the Service Manager
# to each known, running service.
ducc.sm.meta.ping.rate = 60000

# This is the number of consecutive pings that may be missed before a
# service is considered unavailable.
ducc.sm.meta.ping.stability =  10

#  This is the time in milliseconds the SM waits for a response to a ping. If the service does
#  not respond within this time the ping is accounted for as a "missed" ping.
ducc.sm.meta.ping.timeout =  15000

# This is the HTTP port used by the Service Manager to field requests from the CLI / API.
# *****
# NOTE: for backward CLI / API compatibility (2.2.0 and before) this port is the same as
# ducc.orchestrator.http.port.  This entry can otherwise be deleted post-2.2.0 as it is no
# longer used by the current CLI / API.
# *****
ducc.sm.http.port=${ducc.orchestrator.http.port}

# This is the node where the Service Manager runs. It MUST be ${ducc.head}.
ducc.sm.http.node=${ducc.head}

# This is the length of time, in milliseconds, that the SM allows a service to remain alive after
# all jobs that reference it have exited. If no new job referencing it enters the system before this
# time has expired, the SM stops the service.
ducc.sm.default.linger=300000

# +==================================================================================================+
# | Orchestrator                                                                                     |
# +==================================================================================================+

# This is the name of the pluggable java class used to implement the DUCC Orchestrator.
ducc.orchestrator.configuration.class=org.apache.uima.ducc.orchestrator.config.OrchestratorConfiguration

# This indicates the level of recovery to be taken on restarting a
# system. There are three levels of startup:
#     [cold] All reservations are canceled, all currently running
#            jobs (if any) are terminated. All services are terminated. The
#            system starts with no jobs, reservations, or services active.
#     [warm] All active work is continued.
#------------------------------------------------------------------------------
# cold, // Recover: All active are forced to Completed     JD host: employ new
#   warm, // Recover: All Jobs+Services+Reservations (default)  JD host: employ current
ducc.orchestrator.start.type=warm
#------------------------------------------------------------------------------

# This is the name of the JMS endpoint through which the Orchestrator broadcasts its
# state messages. These messages include full job information and can be relatively
# large, though they are now compressed.
ducc.orchestrator.state.update.endpoint=ducc.orchestrator.state

# This is the JMS endpoint type used for the state messages sent by the Orchestrator.
ducc.orchestrator.state.update.endpoint.type=topic

# The interval in milliseconds between Orchestrator publications of its state.
ducc.orchestrator.state.publish.rate=10000

#------------------------------------------------------------------------------
# How long between between maintenance cycles (in milliseconds)
# Each maintenance cycle the orchestrator removes obsolete JD queues (MqReaper)
# and performs health related activities (HealthMonitor) including: capping JPs
# for a Jobs that have too many initialization failures and terminating Jobs whose
# JDs have failed.
ducc.orchestrator.maintenance.rate=60000

#------------------------------------------------------------------------------
# This is the HTTP port used by the Orchestrator to field requests from the CLI / API.
ducc.orchestrator.http.port=19988

# Node where Orchestrator is running. It MUST be ${ducc.head}.
ducc.orchestrator.http.node=${ducc.head}

#------------------------------------------------------------------------------
# Specify if reserving an entire machine is allowed
# Allowing could be a bad idea when a cluster has very few, very large machines.
ducc.orchestrator.unmanaged.reservations.accepted=true

#------------------------------------------------------------------------------
# This is the name of the JMS endpoint through which the daemons broadcast state changes
ducc.daemons.state.change.endpoint=ducc.daemons.state.change

# This is the JMS endpoint type used for the state change messages sent by the daemons.
ducc.daemons.state.change.endpoint.type=queue

# +==================================================================================================+
# | Resource Manager (aka Scheduler)                                                                 |
# +==================================================================================================+

# This is the name of the pluggable java class used to implement the DUCC Resource
# Manager.
ducc.rm.configuration.class=org.apache.uima.ducc.rm.config.ResourceManagerConfiguration

# This is the name of the JMS endpoint through which the Resource Manager broadcasts its
# state.
ducc.rm.state.update.endpoint=ducc.rm.state

# This is the JMS endpoint type used for state messages sent by the Resource Manager.
ducc.rm.state.update.endpoint.type=topic

# This specifies the frequency of RM schedules, relative to the number of Orchestrator publications.
# If the value is set to 1, RM runs and publishes a schedule immediately on receipt of OR state.
# If set to some number N, RM runs a schedule after receipt of every N Orchestrator publications.
ducc.rm.state.publish.ratio = 1

# This specifies the maximum non-preemptable shares any user may be awarded, in GB.  If not configured,
# there is no maximum enforced.  This can be overridden on a per-user basis in the user registry.
#ducc.rm.global_allotment = 360

# The share quantum is the smallest amount of RAM that is schedulable for jobs, in GB.
# Jobs are scheduled based entirely on their memory requirements. Memory is allocated in
# multiples of the share quantum.
ducc.rm.share.quantum = 1

# The component that implements the scheduling algorithm is pluggable. This specifies the
# name of that class.
ducc.rm.scheduler = org.apache.uima.ducc.rm.scheduler.NodepoolScheduler

# File defining the scheduler classes - found in DUCC_HOME/resources
ducc.rm.class.definitions = ducc.classes

# File defining the user registry.  Optional, need not exist.  Found in DUCC_HOME/resources.
# The registry is used only to override the global allotments.  The registry entries may also
# be placed in the ducc.classes file if desired.
ducc.rm.user.registry  = ducc.users

# default memory, in GB, if not specified
ducc.rm.default.memory = 4

# Number of node metrics heartbeats to wait for before rm starts up.
# During DUCC initialization the Resource Manager must wait some period of time for
# all the nodes in the cluster to check-in via their "heartbeats". If the RM were to start
# scheduling too soon there would be a period of significant "churn" as the perceived cluster
# configurations changes rapidly. As well, it would be impossible to recover work in a warm
# or hot start if the affected nodes had not yet checked in.
# The init.stability property indicates how many heartbeat intervals the RM must wait before
# it starts scheduling after initialization.
ducc.rm.init.stability = 2

# Number of missed node metrics updates to consider node down.
# The RM receives regular "heartbeats" from the DUCC agents in order to know what
# nodes are available for scheduling. The node.stability property configures the number of
# consecutive heartbeats that may be missed before the Resource Manager considers the
# node to be inoperative.
# If a node becomes inoperative, the Resource Manager deallocates all processes on that
# node and attempts to reallocate them on other nodes. The node is marked offline and is
# unusable until its heartbeats start up again.
# The default configuration declares the agent heartbeats to occur at 1 minute intervals.
# Therefore heartbeats must be missed for five minutes before the Resource Manager takes
# corrective action.
ducc.rm.node.stability = 5

# Which policy to use when shrinking/evicting shares - alternatively, SHRINK_BY_MACHINE.
# The eviction.policy is a heuristic to choose which processes of a job to preempt because of
# competition from other jobs.
# The SHRINK_BY_INVESTMENT (default) policy attempts to preempt processes such that the
# least amount of work is lost. It chooses candidates for eviction in order of:
# - Processes still initializing, with the smallest time spent in the initializing step.
# - Processes whose currently active work items have been executing for the shortest time.
# The SHRINK_BY_MACHINE policy attempts to preempt processes so as to minimize
# fragmentation on machines with large memories that can contain multiple job processes.
# No consideration of execution time or initialization time is made.
ducc.rm.eviction.policy = SHRINK_BY_INVESTMENT

# Max nodes to initially allocate until init is complete.
# The type of jobs supported by DUCC generally have very long and often fragile
# initialization periods. Errors in the applications and other problems such is missing or
# errant services can cause processes to fail during this phase.
# To avoid preempting running jobs and allocating a large number of resources to jobs only
# to fail during initialization, the Resource Manager schedules a small number of processes
# until it is determined that the initialization phase will succeed.
# The initialization.cap determines the maximum number of processes allocated to a job
# until at least one process successfully initializes. Once any process initializes the Resource
# Manager will proceed to allocate the job its full fair share of processes.
# The initialization cap can be overridden on a class basis by configuration via ducc.classes.
ducc.rm.initialization.cap = 1

# When true, jobs expand not all at once after init, but a bit slower, doubling each epoch
# until max fair-share is set.  If false, jobs increase immediately to their fair share,
# at the cost of mass evictions.
# Expand.by.doubling can be overridden on a class basis by configuration via ducc.classes.
ducc.rm.expand.by.doubling = true

# Predict when a job will end and avoid expanding if not needed.
# Because initialization time may be very long, it may be the case that a job that might be
# eligible for expansion will be able to complete in the currently assigned shares before any
# new processes are able to complete their initialization. In this case expansion results in
# waste of resources and potential eviction of processes that need not be evicted.
# The Resource Manager monitors the rate of task completion and attempts to predict the
# maximum number of processes that will be needed at a time in the future based on the
# known process initialization time. If it is determined that expansion is unnecessary then it
# is not done for the job.
# Prediction can be overridden on a class basis by configuration via ducc.classes.
ducc.rm.prediction = true

# Add this fudge factor (milliseconds) to the expansion target when using prediction.
# When ducc.rm.prediction is enabled, the known initialization time of a job’s processes plus
# some "fudge" factor is used to predict the number of future resources needed. The "fudge"
# is specified in milliseconds.
# The default "fudge" is very conservative. Experience and site policy should be used to set a
# more practical number.
# Prediction.fudge can be overridden on a class basis by configuration via ducc.classes.
ducc.rm.prediction.fudge = 120000

# What is minimum number of processes for a job before we do defrag? If a job has less than
# this amount RM may attempt defragmentation to bring the processes up to this value.
# If enabled, limited defragmentation of resources is performed by the Resource Manager to
# create sufficient space to schedule work that has insufficient resources (new jobs, for
# example.).  The term "insufficient" is defined as "needing more processes than the
# defragmentation # threshold, but currently having fewer processes than the defragmentation
# threshold."  These are called "needy" jobs.  Additionally, the Resource Manager
# will never evict processes from ‘‘needy’’ jobs for the purpose of defragmentation.
# This property allows installations to customize the value used to determine if a
# job is "needy".  Jobs with fewer processes than this are potentially needed, and
# jobs with more processes are never needy.
ducc.rm.fragmentation.threshold = 8

# This JMS endpoint used for RM administrative requests.
ducc.rm.admin.endpoint = ducc.rm.admin.channel

# This is the JMS endpoint type used for RM administrative requests.
ducc.rm.admin.endpoint.type = queue

# This JMS endpoint used for RM administrative requests.
ducc.rm.via.or.admin.endpoint = ducc.rm.via.or.admin.channel

# This is the JMS endpoint type used for RM administrative requests.
ducc.rm.via.or.admin.endpoint.type = queue

# Unmanaged Reservations are usually assigned to machines whose size exactly matches the
# requested size, after rounding up to a multiple of the node pool’s quantum.
# When this property is positive, machines up to this many GB larger than the rounded up
# size will be considered.
ducc.rm.reserve_overage = 0

# +==================================================================================================+
# | Agents                                                                                           |
# +==================================================================================================+

# This is the name of the pluggable java class used to implement the DUCC Agents.
ducc.agent.configuration.class=org.apache.uima.ducc.agent.config.AgentConfiguration

# This is the JMS endpoint through which agents receive state from the Process Manager.
ducc.agent.request.endpoint=ducc.agent

# This is the JMS endpoint type used for state messages sent by the Process Manager.
ducc.agent.request.endpoint.type=topic

# This is the JMS endpoint used to communicate
# from the managed process to the Agent (Job Process).
ducc.agent.managed.process.state.update.endpoint=ducc.managed.process.state.update

# This is the JMS endpoint type used to communicate
# from the managed process (Job Process) to the Agent.
ducc.agent.managed.process.state.update.endpoint.type=socket

# Endpoint parameters which are transport specific.
# For socket transport params are:
#  - transferExchange=true - include Camel Exchange wrapper in a message
#  - synch=false - use socket transport for one-way messaging (no replies neeeded)
ducc.agent.managed.process.state.update.endpoint.params=transferExchange=true&sync=false

# ***** Note: the Resource Manager uses the data in the node metrics for scheduling.

# The interval in milliseconds between node metric publications.
# Every agent publishes its updates at this rate.  On large clusters, a high rate (small
# interval) can be a burden on the network.
ducc.agent.node.metrics.publish.rate=30000

# This is the JMS endpoint used to send node metrics updates to listeners. Listeners
# are usually the Resource Manager and Web Server. These messages serve as node
# "heartbeats". As well, the node metrics heartbeats contain the amount of RAM on the node
# and the number of processors.
ducc.agent.node.metrics.endpoint=ducc.node.metrics

# This is the JMS endpoint type used to send node metrics updates from the agents.
ducc.agent.node.metrics.endpoint.type=topic

# The interval in milliseconds between node inventory publications.
# If the inventory has not changed since the last update the agent bypasses sending the
# update, up to a maximum of ducc.agent.node.inventory.publish.rate.skip times.
ducc.agent.node.inventory.publish.rate=10000

# This is the number of times the agent will bypass publishing its node inventory if the
# inventory has not changed.
ducc.agent.node.inventory.publish.rate.skip=3

# This is the JMS endpoint used to send node inventory messages to listeners. Listeners are
# usually the Orchestrator and Web Server. Information in these messages include a map of
# processes being managed on the node.
ducc.agent.node.inventory.endpoint=ducc.node.inventory

# This is the JMS endpoint type used to send node inventory updates from the agents.
ducc.agent.node.inventory.endpoint.type=topic

# ducc.agent.launcher.thread.pool.size - Deprecated
#ducc.agent.launcher.thread.pool.size=10

# enable/disable use of ducc_ling
# ducc.agent.launcher.use.ducc_spawn: true, use ducc_ling. Default: false
ducc.agent.launcher.use.ducc_spawn=true

# Specify location of ducc_ling in the filesystem.
# This property specifies the full path to the ducc_ling utility. During installation ducc_ling
# is normally moved to local disk and given setuid-root privileges. Use this property to tell
# the DUCC agents the location of the installed ducc_ling.  The default location is within
# an architecture dependent subdiretory of DUCC_HOME/admin.
# The arcitecture is derived from the JRE property os.arch.  During DUCC installation
# the ducc_ling utility is compiled for the architecture of the host where DUCC is installed.
# In heterogeneous clusters, the system administrator should run the utility build_duccling
# once on a machine of each architecture to insure this utility gets correctly installed.
ducc.agent.launcher.ducc_spawn_path=${DUCC_HOME}/admin/${os.arch}/ducc_ling

# Max amount of time (in millis) agent allows the process to stop before issuing kill -9.
# This property specifies the time, in milliseconds, the agent should wait before forcibly
# terminating a job process (JP) after an attempted graceful shutdown. If the child process
# does not terminate in the specified time, it is forcibly terminated with kill -9.
# This type of stop can occur because of preemption or system shutdown.
ducc.agent.launcher.process.stop.timeout=60000

# Max time in millis allowed for AE initialization. Default 2 hours 7200000. - Deprecated
#ducc.agent.launcher.process.init.timeout=7200000

# Exclude the following user ids while detecting rogue processes
# The DUCC Agents scan nodes for processes that should not be running; for example,
# a job may have left a ’rogue’ process alive when it exits, or a user may log in to a node
# unexpectedly. These processes are reported to the administrators via the webserver for
# possible action.
# This configuration parameter enumerates userids which are ignored by the rogue-process scan.
# default = root,posstfix,ntp,nobody,daemon,100
ducc.agent.rogue.process.user.exclusion.filter=

# Max UID reserved by OS. This is used to detect rogue processes and to report
# available memory on a node.
# The ducc.agent.node.metrics.sys.uid.max property is droped in favor of
# ducc.agent.rogue.process.sys.user.max, which seems more descriptive
#
# ducc.agent.node.metrics.sys.uid.max=500
ducc.agent.rogue.process.sys.uid.max=500

# Exclude the following processes while detecting rogue processes
# The DUCC Agents scan nodes for processes that should not be running; for example,
# a job may have left a ’rogue’ process alive when it exits, or a user may log in to a node
# unexpectedly. These processes are reported to the administrators via the webserver for
# possible action.
# This configuration parameter enumerates processes by name which are ignored by the
# rogue process detector.
# default = sshd:,-bash,-sh,/bin/sh,/bin/bash,grep,ps
ducc.agent.rogue.process.exclusion.filter=sshd:,-bash,-sh,/bin/sh,/bin/bash,grep,ps

# Fudge Factor (in terms of percantage) that agent uses to multiply a share size when
# determining if a JP exceeds its alloted memory, which is calculated as follows
# (fudge factor/100)*share size + share size. If this number exceeds JPs RSS, the agent
# kills the process.
# The DUCC agent monitors the size of the resident memory of its spawned processes. If a
# process exceeds its declared memory size by any significant amount it is terminated and
# a ShareSizeExceeded message is sent. The Job Driver counts this towards the maximum
# errors for the job and will eventually terminate the job if excessive such errors occur.
# This property defines the percentage over the declared memory size that a process is
# allowed to grow to before being terminated.
# To disable this feature, set the value to -1.
ducc.agent.share.size.fudge.factor=5

# Enable/Disable CGroup support.
# If CGroups are not installed on a specific machine, this is ignored.
# With CGroups the RSS for a managed process (plus any children processes it may spawn) is
# limited to the allocated share size. Additional memory use goes to swap space. DUCC
# monitors and limits swap use to the same proportion of total swap space as allocated
# share size is to total RAM. If a process exceeds its allowed swap space it is terminated
# and a ShareSizeExceeded message is sent to the Job Driver.
# Nodes not using CGroups fall back to the ducc.agent.share.size.fudge.factor.
ducc.agent.launcher.cgroups.enable=true

# Define location of cgroups utils like cgexec. For mixed clusters with different
# flavors of unix, the utilities may be in different places. In such case, add a
# a comma separated list of paths like this:
# ducc.agent.launcher.cgroups.utils.dir=/usr/bin,<OS 1 path>,<OS 2 path>
ducc.agent.launcher.cgroups.utils.dir=/usr/bin,/bin

# Set cgroup memory.swappiness
ducc.agent.launcher.cgroups.swappiness=10

# Number of retries to use when cgcreate fails
ducc.agent.launcher.cgroups.max.retry.count=1

# Amount by which to increase delay (in msecs) between each retry
# Defines a delay factor in millis an agent will use to determine how long it
# should sleep between cgroup create retries. Each time cgcreate fails
# an agent will increment the delay by adding the delay factor to a previous
# delay. If the delay factor is 2000, on first cgcreate failure an agent will
# wait 2000 ms, on second failure the wait time will be 4000, on third 6000, and
# so on.
ducc.agent.launcher.cgroups.retry.delay.factor=2000

# Exclusion file to enable node based exclusion for cgroups and aps
# syntax:  <node>=cgroups,ap
# the above will exclude node from using cgroups and/or prevent deployment of APs
ducc.agent.exclusion.file=${DUCC_HOME}/resources/exclusion.nodes

# Define script which will collect total swap used by a given process. This
# script is launched by an agent via duccling and running as the owner
# of the process.
ducc.agent.swap.usage.script=${DUCC_HOME}/admin/ducc_get_process_swap_usage.sh

# Uncomment the following line to support auto reaping of rogue processes by Ducc’s Agent
# ducc.agent.rogue.process.reaper.script=${DUCC_HOME}/admin/ducc_reap_rogues.sh

# +==================================================================================================+
# | Process Manager                                                                                  |
# +==================================================================================================+

# This is the name of the pluggable java class used to implement the DUCC Process Manager.
ducc.pm.configuration.class=org.apache.uima.ducc.pm.config.ProcessManagerConfiguration

# This is the endpoint through which process manager receive state from the Orchestrator.
ducc.pm.request.endpoint=ducc.pm

# This is the JMS endpoint type used for state messages sent by the Orchestrator.
ducc.pm.request.endpoint.type=queue

# This is the endpoint through which process manager sends its heartbeat. The main receiver
# is the Web Server for it’s daemon status page.
ducc.pm.state.update.endpoint=ducc.pm.state

# This is the JMS endpoint type used for process manager heartbeats. The primary receiver
# is the Web Server for its daemon status page.
ducc.pm.state.update.endpoint.type=topic

# The interval in milliseconds between process manager heartbeat publications.
ducc.pm.state.publish.rate=15000

# +==================================================================================================+
# | Job Process                                                                                      |
# +==================================================================================================+

# This is the name of the pluggable java class that implements
# the shell for Agent launched user processes.
ducc.uima-as.configuration.class=org.apache.uima.ducc.transport.configuration.jp.JobProcessConfiguration
ducc.job-process.configuration.class=org.apache.uima.ducc.transport.configuration.jp.JobProcessConfiguration
ducc.service.configuration.class=org.apache.uima.ducc.transport.configuration.service.ServiceConfiguration

# This is the endpoint through which job processes (JPs) receive messages from the Agents.
ducc.uima-as.endpoint=ducc.job.managed.service

# This is the JMS endpoint type used for messages sent to the JPs from the Agents.
ducc.uima-as.endpoint.type=socket

# Endpoint parameters which are transport specific.
# For socket transport params are:
#  - transferExchange=true - include Camel Exchange wrapper in a message
#  - synch=false - use socket transport for one-way messaging (no replies neeeded)
ducc.uima-as.endpoint.params=transferExchange=true&sync=false

# saxon8.jar - Saxon is an XSLT and XQuery processor. It is used by UIMA-AS
#              when deploying a service.
ducc.uima-as.saxon.jar.path=file:${DUCC_HOME}/apache-uima/saxon/saxon8.jar

# dd2spring.xsl - contains a set of rules used when transforming
#                 UIMA-AS deployment descriptor into SpringFramework’s
#                 context file used to deploy UIMA-AS based service.
ducc.uima-as.dd2spring.xsl.path=${DUCC_HOME}/apache-uima/bin/dd2spring.xsl

# Maximum amount of time to wait for a response from the JD. This value
# is used by the JP when sending requests to the JD.
ducc.process.request.timeout=30000

# Define process container class for DD jobs to instantiate and invoke via reflection.
# The container provides classpath  isolation for user defined analytics.
# The container is instantiated with classes from a System classloader.
ducc.process.uima.as.container.class = org.apache.uima.ducc.user.jp.UimaASProcessContainer

# Define process container class for non-DD jobs to instantiate and invoke via reflection.
# The container provides classpath  isolation for user defined analytics.
# The container is instantiated with classes from a System classloader.
ducc.process.uima.container.class = org.apache.uima.ducc.user.jp.UimaProcessContainer

# Define the sleep time (millis) for JP to use when JD sends empty CAS. In this case the
# JD’s CR has processed its collection. The JP threads need to slow down sending
# requests
ducc.process.thread.sleep.time=60000

# Custom Flow Controller to use for Ducc Job Processes that don’t provide a process_DD descriptor
ducc.flow-controller.specifier=org.apache.uima.ducc.FlowController

# +==================================================================================================+
# | Database                                                                                         |
# +==================================================================================================+

# Name of the database host(s) - initially disabled until database is created.
# Overridden in site.ducc.properties by ducc_post_install or db_create.
# This is the name of the host(s) where the database is run.  It usually defaults to the
# same host as the ducc.head.  Those knowledgable of the database can install the
# database elsewhere.  Use this parameter to specify that location(s).
# To disable use of the database, set this parameter to the string --disabled--
ducc.database.host.list       = --disabled--

# Database JMX host. Default is to allow only local JMX access.  Specify the real name for remote access.
ducc.database.jmx.host = localhost
# Database JMX port.
ducc.database.jmx.port = 7199

# If set to true, DUCC will start and stop the Cassandra database as part of its normal
# start/stop scripting.
ducc.database.automanage = true

# Path to Cassandra database home
ducc.database.home = ${DUCC_HOME}/cassandra-server

#-----------------------------------
# DATABASE MEMORY TUNING
#    WARNING: if you’re not sure what these do you should probably not change them as incorrect
#             values can adversly affect the database, the database node, or both.
#    Given that.  If you’re running on a mostly small system, e.g. 8GB or less, probably you
#    should set ducc.database.mem.heap to 1 or 2 GB and ducc.database.mem.new to 100M
#
#    By default we allow cassandra to calculate these for us.
#
# IMPORTANT: if you set one of these you MUST set both, or cassandra will refuse to start.
#-----------------------------------

# Database xmx.  If not set, cassandra will calculate it for you.
# This is the value used to set {\em Xmx and Xms} when the database starts.  The
# Cassandra database makes an attempt to determine the best value of this.  The
# default is one-half of real memory, up to a maximum of 8G.  It is recommended that
# the default be used.  However, small installations may reduce this to as little
# as 512M.  Note that both Xmx and Xms are set.
#ducc.database.mem.heap = 4G

# Database setting for ’young’ generation.  If not set, cassandra will calculate it for you.
#ducc.database.mem.new = 100M

#-----------------------------------
k# END OF DATABASE MEMORY TUNING
#-----------------------------------

# This class implementes the persistence interface for the services registry
#ducc.service.persistence.impl = org.apache.uima.ducc.database.StateServicesDb
ducc.service.persistence.impl  = org.apache.uima.ducc.common.persistence.services.StateServices

# This class implements the persistence interface for Job, Reservation, Service, and AP instances, plus
# the Orchestrator checkpoint.
#ducc.job.history.impl         = org.apache.uima.ducc.database.HistoryManagerDb
ducc.job.history.impl          = org.apache.uima.ducc.transport.event.common.history.HistoryPersistenceManager

#ducc.rm.persistence.impl     = org.apache.uima.ducc.database.RmStatePersistence
ducc.rm.persistence.impl      = org.apache.uima.ducc.common.persistence.rm.NullRmStatePersistence

# end database things -------------------------------------------------------------------------------

13.6 Resource Manager Configuration: Classes and Nodepools

The class configuration file is used by the Resource Manager configures the rules used for job scheduling. See the Resource Manager chapter for a detailed description of the DUCC scheduler, scheduling classes, and how classes are used to configure the scheduling process.

The scheduler configuration file is specified in ducc.properties. The default name is ducc.classes and is specified by the property ducc.rm.class.definitions.

13.6.1 Nodepools

Overview

A nodepool is a grouping of a subset of the physical nodes to allow differing scheduling policies to be applied to different nodes in the system. Some typical nodepool groupings might include:

A Nodepool is a subset of some larger collection of nodes. Nodepools themselves may be further subdivided. Nodepools may not overlap: every node belongs to exactly one nodepool. During system start-up the consistency of nodepool definition is checked and the system will refuse to start if the configuration is incorrect.

NOTE: The administrative command check_ducc -c may be used to verify and validate you class configration before attemping to start DUCC. check_ducc -cv may be used to additionally “pretty-print” the ducc.classes configuration to the console to revel class nesting and inheritance.

For example, the diagram below is an abstract representation of all the nodes in a system. There are five nodepools defined:

Multiple “top-level” nodepools are allowed. A “top-level” nodepool has no containing pool. Multiple top-level pools logically divide a cluster of machines into multiple independent clusters from the standpoint of the scheduler. Work scheduled over one pool in no way affects work scheduled over the other pool. The figure below shows an abstract nodepool configuration with two top-level nodepools, “Top-NP1” and “Top-NP2”.

Scheduling considerations

A primary goal of the scheduler is to insure that no resources are left idle if there is pending work that is able to use those resources. Therefore, work scheduled to a class defined over a specific nodepool (say, NpAllOfThem), may be scheduled on nodes in any of the nodepools contained within NpAllOfThem. If work defined over a subpool (such as NP1) arrives, processes on nodes in NP1 that were scheduled for NpAllOfThem are considered squatters and are the most likely candidates for eviction. (Processes assigned to their proper nodepools are considered residents and are evicted only after all squatters have been evicted.) The scheduler strives to avoid creating squatters.

Because non-preemptable allocations can’t be preempted, work submitted to a class implementing one of the non-preemptable policies (FIXED or RESERVE) are never allowed to “squat” in other nodepools and are only scheduled on nodes in their proper nodepool.

In the case of multiple top-level nodepools: these nodepools and their sub-pools form independent scheduling groups. Specifically,

Most installations will want to assign the majority of nodes to a single top-level nodepool (or its subpools), using other top-level pools for nodes that cannot be shared with other work.

Configuration

DUCC uses simple named stanzas containing key/value pairs to configure nodepools.

At least one nodepool definition is required. This nodepool need not have any subpools or node definitions. The first top-level nodepool is considered the “default” nodepool. Any node not named specifically in one of the node files which checks in with DUCC is assigned to this first, default nodepool.

Thus, if only one nodepool is defined with no other attributes, all nodes are assigned to that pool.

A nodepool definition consists of the token “Nodepool” followed by the name of the nodepool, followed by a block delimited with “curly” braces { and }. This block contains the attributes of the nodepool as key/value pairs. Lineneds are ignored. A semicolon “;” may optionally be used to delimit key/value pairs for readability, and an equals sign “=” may optionally be used to delimit keys from values, also just for readability. See the below.

The following example shows an entry utilizing the .regex feature, followed by the content of the specifed file comprising a regular expression.

13.6.2 Class Definitions

A simple inheritance (or “template”) scheme is supported for classes. Any class may be configured to “derive” from any other class. In this case, the child class acquires all the attributes of the parent class, any of which may be selectively overridden. Multiple inheritance is not supported but nested inheritance is; that is, class A may inherit from class B which inherits from class C and so on. In this way, generalized templates for the site’s class structure may be defined.

The general form of a class definition consists of the keyword Class, followed by the name of the class, and then optionally by the name of a “parent” class whose characteristics it inherits. Following the name (and optionally parent class name) are the attributes of the class, also within a { block } as for nodepools, and with lines and key/value pairs optionally delimited by “;” and “=”, respectively. See the sample below.

The following figure illustrates a representative class configuration for a large cluster, consisting of mixed Intel and Power nodes. This class definition assumes the nodepool configuration shown above. FAIR_SHARE, FIXED_SHARE, and RESERVE classes are defined over each machine architecture, Intel and Power, and over the combined pool.

13.6.3 Validation

The administrative command, check_ducc may be used to validate a configuration, with the -c and v options. This reads the entire configuration and nodefiles, validates consistency of the definitions and insures the nodepools do not overlap.

The start_ducc command always runs full validation, and if the configuration is found to be incorrect, the cluster is not started.

Configuration checking is done internally by the DUCC java utility org.apache.uima.ducc.common.NodeConfiguration. This utility contains a public API as described in the Javadoc. It may be invoked from the command line as follows:

13.7 Ducc Node Definitions

The DUCC node list is used to configure the nodes used to run jobs and assign reservations. When DUCC is started, the nodelist is read an a DUCC Agent is started on every node in the list.

The node list can be composed of multiple node lists to assist organization of the DUCC cluster. All the administrative commands operate upon node lists. By carefully organized these lists it is possible to administer portions of a cluster independently.

In particular, it is highly recommended that the nodelists reflect the nodepool structure. In this way, the configuration used to start DUCC is guaranteed to match the nodeppool definitions.

The example below shows a partial, hypothetical node configuration corresponding to the nodepool configuration above.

13.8 Ducc User Definitions

The DUCC user registry provides user-specific overrides of various constraints DUCC might impose.

As of 2.0.0, the only constraint override is allotment for non-preemptable requests.

The syntax of the user registry is the same as that used in ducc.classes, and in fact, the user registry may be embedded directly in that file, rather than specified externally.

The registry consists of multiple entries, one for affected user. Any user of the system NOT in the registry acquires the system defaults.

A user definition consists of the token “User” followed by the id of the user, followed by a block delimited with “curly” braces { and }. This block contains the attributes of the nodepool as key/value pairs. Lineneds are ignored. A semicolon “;” may optionally be used to delimit key/value pairs for readability, and an equals sign “=” may optionally be used to delimit keys from values, also just for readability.

13.9 DUCC Database Integration

As of Version 2.1.0, DUCC uses the Apache Cassandra database instead of the filesystem to manage history and the service registry. Additionally, the Resource Manager maintains current scheduling and node state in the database.

13.9.1 Overview

During first-time installation, the ducc_post_install utility randomly generates a (database) super-user password, which is kept in the protected file DUCC_HOME/resources.private/ducc.private.properties. The utility proceeds to configure the database and install the schema.

If DUCC is being upgraded, generally ducc_post_install is not used, in which case, again, db_create and db_loader may be used to convert the older file-based state to the database.

Orchestrator use of the Database

Service Manager use of the Database

The service manager uses the database to store the service registry and all state of active services. Prior to the database, this data was saved in Java properties files in the directory DUCC_HOME/state/services.

When a service is “unregistered” it is not physically removed from the database. Instead, a bit is set indicating the service is no long active. These registrations may be recovered if needed by querying the database. Prior to the database, this data was saved in DUCC_HOME/history/service-registry.

Resource Manager use of the Database

The resource manager saves its entire runtime state in the database. Prior to the database, this dynamnic state was not saved or directly accessible.

Webserver use of the Database

The web server uses the database in read-only mode to fetch work history, service registrations, and node status. Previously to the database most of this information was fetched from the filesystem. Node status was inferred using the Agent publications; with the database, the webserver has direct access to the Resource Manager’s view of the DUCC nodes, providing a much more accurate picture of the system.

13.9.2 Database Scripting Utilities

Database support is fully integrated with the DUCC start, stop, and check utilities as well as the post installation scripting.

In addition two utilities are supplied to enable migration of older installations to enable the database:

13.9.3 Database Configuration

Most database configuration is accomplished by setting appropriate values into your local site.ducc.properties.

For existing installations, the db_create utility installs the database scheme and updates your site.ducc.properties with reasonable defaults.

13.10 Administrative Commands

The administrative commands include a command to start DUCC, one to stop it, and one to verify the configuration and query the state of the cluster.

Note: The scripting that supports some of these functions runs (by default) in multi-threaded mode so large clusters can be started, stopped, and queried quickly. If DUCC is running on an older system, the threading may not work right, in which case the scripts detect this and run single-threaded. As well, all these commands support a “–nothreading” option to manually disable the threading.

13.10.1 autostart.py

Description

The command $DUCC_HOME/admin/autostart.py is used to monitor and start DUCC daemons. It is run individually by each node on the cluster, nominally invoked via crontab. It takes the following actions:

Example crontab entry:

13.10.2 db_autostart_delete.py

Description

The command $DUCC_HOME/admin/db_autostart_delete.py is used to remove entries from the autostart database table.

Usage

Options:

Example:

13.10.3 db_autostart_query.py

Description

The command $DUCC_HOME/admin/db_autostart_query.py is used to list entries in the autostart database table.

Example:

13.10.4 start_ducc

Description

The command $DUCC_HOME/admin/start_ducc is used to start DUCC processes. It must be run from the head node. If run with no parameters it takes the following actions:

Usage

Options:

Notes:

A different nodelist may be used to specify where Agents are started. As well multiple node lists may be specified, in which case Agents are started on all the nodes in the multiple node lists.

To start only agents, run start_ducc specifying a nodelist explicitly. Note that the broker must have already been started.

To start a specific management process, run start_ducc with the -c component parameter, specify the component that should be started.

Examples:

Debugging:

Sometimes something will not start and it can be difficult to understand why. To diagnose, it is helpful to know that start_ducc is simply a wrapper around a lower-level bit of scripting that does the actual work. That lower-level code can be invoked stand-alone, in which case console messages that start_ducc will have suppressed are presented to the console.

The lower-level script is called ducc.py and accepts the same -c component flag as start_ducc. If some component will not start, try running ducc.py -c component directly. It will start in the foreground and usually the cause of the problem becomes evident from the console.

13.10.5 stop_ducc

Description:

Usage:

Options:

Notes:

Examples:

13.10.6 check_ducc

Description:

Check_ducc is used to verify the integrity of the DUCC installation and to find and report on DUCC processes. It identifies processes owned by ducc (management processes, agents, and job processes), and processes started by DUCC on behalf of users.

Usage:

Options:

Notes:

13.10.7 build_duccling

13.10.8 ducc_disk_info

Description:

13.10.9 ducc_gather_logs

Description:

Purpose: gather problem determination information comprising DUCC logs and configuration data into a gz file.

13.10.10 ducc_post_install

Description: The post-installation script must be run only after the first installation of DUCC. When updating an existing installation use the ducc_update script. ducc_post_install performs these tasks:

Once the script completes successfully start_ducc will run a single-user/unprivileged DUCC.

Notes: If the script is rerun it will rename the previously created files so any customizations applied can be recovered.

13.10.11 ducc_update

Description: This command is used to unpack a new release of DUCC and create a new installion or update an existing one. For a new installation it simply unpacks the tar file with the appropriate permissions. The setup must be completed by running ducc_post_install.

The site-specific files, those holding customizations such as node and class definitions as well as logs and job history, are left in place, while all replaced files are archived under a folder called ducc_archives so the previous installation can be restored if necessary.

Note that the update does not create the database. After updating from a version prior to 2.1.0 using the file-based persistent scheme the database should be created with db_create and the files holding state such as job history and service registrations loaded into the database with db_loader. If this conversion is omitted DUCC will continue to use the file-based scheme but with some loss of functionality that the database design would provide.

Usage: This command takes two parameters, a pointer to the DUCC_HOME to be updated or created, and the name of the tar file containing the new build.

Example: If this is the first installation or the current installation does not contain the ducc_update script you can first extract it from the tar file.

13.10.12 rm_reconfigure

Description:

Rm_reconfigure is used to force the Resource Manager (RM) to reread all its configuration files and reconfigure itself accordingly, without the need to fully stop and restart RM. This is generally much faster than RM restart and avoids losing most state messages from the other DUCC processes.

RM then validates the new configuration, and if no errors are found, saves certain information such as current node online-offline status. It then rereads the following configuration files and rebuilds its internal structures accordingly:

The saved configuration is then restored into the newly configured structures. On receipt of the next Orchestrator state, the RM fully rebuilds its state from the current DUCC load and scheduling restarts.

Depending on the nature of the new configuration, the current load may be adjusted; for example, if the weight of a fair-share class is changed, preemptions or extra allocations may be performed.

If the new configuration is not consistent with the current load, a number of more drastic adjustments will be performed:

In short, the RM makes every effort to avoid disturbing existing allocations, and blacklists hosts that are no longer consistently configured for the current load, until such time as the allocations on those hosts are released.

Usage:

13.10.13 rm_qoccupancy

Description:

Rm_qoccupancy provides a list of all known hosts to the RM, and for each host, the following information:

The following example shows seven hosts, one with a preemptable share in the –default– nodepool (on bluej290-5), and one with a non-preemptable share in the jobdriver nodepool.

Usage:

13.10.14 vary_off

Description:

Vary_off is used to remove a host from scheduling and to evict the preemptable work that is running on it. This allows for graceful clearance of a host so the host can be take offline for maintenance, or any other purpose (such as sharing the host with other applications.) The DUCC agent is NOT stoppped; use stop_ducc to stop the agent. Managed and unmanaged reservations are not canceled by vary_off.

Only the userid that started DUCC may issue vary_off; attempts from other userids are rejected.

Usage:

13.10.15 vary_on

Description:

Vary_on is used to restore a host to scheduling by DUCC. If the agent is still alive the host becomes immediately available. The agent is not started by vary_on; use use start_ducc to start the agent if needed.

Only the userid that started DUCC may issue vary_on; attempts from other userids are rejected.

Usage:

13.10.16 ducc_properties_manager

Description: This CLI is used to manually merge or difference two properties files.

Normally, the DUCC scripts start_ducc, check_ducc,, and rm_configure automatically merge the file default.ducc.properties and site.ducc.properties when invoked.

13.10.17 db_create

Description: This command is used to initialize the database. Normally the database is initialized during ducc_post_install but if this is an existing DUCC installation that is being migrated from a version that does not use the database, it will be necessary to initialize the database with this command.

NOTE: The database user and password are NOT RELATED to any login ID on the system, they are used and maintained by the database only.

13.10.18 db_loader

Description: This command is used to copy the data from DUCC’s older (pre 2.1.0) file-based persistence into the database. The database schema must already exist, created either with ducc_post_install or with db_create.

When the command exits, DUCC should be ready to run with all its state in the database.

This command takes two parameters, a pointer to the DUCC_HOME you want to load from, and a flag to disable the rename of the file-based state.

Notes: The console shows progress of the loader. Full details of the load are written to a log db-loader-log in the usual DUCC log directory, for reference and potential problem determination if something goes wrong.

13.10.19 db_tool

13.10.20 ducc_get_process_swap_usage

Description:

The script is run by an agent to obtain swap space used by a process with a given PID.

13.11 Administrative Tasks

The administrative tasks comprise one or several DUCC commands and/or file editing to achieve a desired system state.

It may be inconvenient to stop DUCC to make changes if there are, for example, services running that take a long time to initialize after a DUCC re-start. These administrative tasks are performed with DUCC running, but carefully! Making an invalid change may cause the unexpected. It is safer to shutdown DUCC, make changes, run check_ducc to verify, then re-start DUCC.

13.11.1 Add Node

Description

The node should start appearing in the results from the rm_qoccupancy¹ command, perhaps with some delay due to system latency.

13.11.2 Remove Node

Description

The node should stop appearing in the results from the rm_qoccupancy¹ command, perhaps with some delay due to system latency.

If running without the database, then stop and start the DUCC Web Server to cause the node to be removed from the Machines page by performing these additional steps:

13.11.3 Notes

¹ rm_qoccupancy is supported when system is configured to employ database only.

Chapter 14
Resource Management

14.1 Overview

The DUCC Resource Manager is responsible for allocating cluster resources among the various requests for work in the system. DUCC recognizes several categories of work:

To apportion the cumulative memory resource among requests the Resource Manager defines some minimum unit of memory and allocates machines such that a ”fair” number of ”memory units” are awarded to every user of the system. This minimum quantity is called a share quantum, or simply, a share. The scheduling goal is to award an equitable number of memory shares to every user of the system. The memory shares in a system are divided equally among all the users who have work in the system. Once an allocation is assigned to a user, that user’s jobs are then also assigned an equal number of shares, out of the user’s allocation. Finally, the Resource Manager maps the share allotments to physical resources. To map a share allotment to physical resources, the Resource Manager considers the amount of memory that each job declares it requires for each process. That per-process memory requirement is translated into the minimum number of co-located quantum shares required for the process to run.

To compute the memory requirements for a job, the declared memory is rounded up to the nearest multiple of the share quantum. The total number of quantum shares for the job is calculated, and then divided by the number of quantum shares declared for the job to arrive at the number of processes to allocate. The output of each scheduling cycle is always in terms of processes, where each process is allowed to occupy some number of shares. The DUCC agents implement a mechanism to ensure that no user’s job processes exceed their allocated memory assignments.

For example, suppose the share quantum is 15GB. A job that declares it requires 14GB per process is assigned one quantum share per process. If that job is assigned 20 shares, it will be allocated 20 processes across the cluster. A job that declares 28GB per process would be assigned two quanta per process. If that job is assigned 20 shares, it is allocated 10 processes across the cluster. Both jobs occupy the same amount of memory; they consume the same level of system resources. The second job does so in half as many processes.

Some work may be deemed to be more ”important” than other work. To accommodate this, the RM implements a weighted fair-share scheduler. During the fair share calculations, jobs with higher weights are assigned more shares proportional to their weights; jobs with lower weights are assigned proportionally fewer shares. Jobs with equal weights are assigned an equal number of shares.

The abstraction used to organized jobs by fair-share weight is the job class or simply class. All job submissions must included a declared job class; if none is declared, a default class is chosen by DUCC. As jobs enter the system they are grouped with other jobs of the same class weight. The class abstraction and its attributes are described in subsequent sections.

All non-preemptable allocations are restricted to one allocation per request. If space is available, the request succeeds immediately. If space can be made for the request through preemptions, the preemptions are scheduled and the reservation is deferred until space is available. If space cannot be found by means of preemption, the reservation remains pending until it either succeeds (by cancelation of other non-preemptive work, by adding resources to the system, or by increasing the user’s non-preemptive allotment), or until it is canceled by the user or an administrator.

14.2 Preemption vs Eviction

Preemption occurs only as a result of fair-share calculations or defragmentation. Preemption is the process of deallocating shares from jobs belonging to users whose current allocation exceeds their fair-share, and conversely, only processes belonging to fair-share jobs can be preempted. This is generally dynamic: more jobs in the system result in a smaller fair-share for any given user, and fewer jobs result in a higher fair-share allocation.

Eviction occurs only as a result of system-detected errors, changes in node configuration, or changes in class configuration. Eviction may affect both preemptable work and some types of non-preemptable work.

Work that is non-preemptable, but restartable can be evicted. Such work consists of service processes (which are automatically resubmitted by the Service Manager), and managed reservations, which can be resubmitted by the user.

Unmanaged reservations are never evicted for any reason. If something occurs that would result in the reservation being (fatally) misplaced, the node is marked unschedulable and remains as such until the condition is corrected or the reservation is canceled. Once the condition is repaired (either the reservation is canceled, or the problem is corrected), the node becomes schedulable again.

14.3 Scheduling Policies

The Resource Manager implements three scheduling policies. Scheduling policies are associated with classes.

14.4 Allotment

Allotment is a new concept introduced with DUCC 2.0.0 to prevent non-preemptable requests from dominating a cluster. This replaces the DUCC version 1 class policies of max-processes and max-machines.

It is possible to associate a maximum share allotment with any non-preemptable class. Allotment is assigned per user and is global across all non-preemptable classes. It is configured ducc.properties with ducc.rm.global_allotment.

A simple user registry provides per-user overrides of the global allotment as needed. The registry may be included in the class definition file (specified in ducc.properties under ducc.rm.class.definitions), or in a separate file, specified in ducc.properties as ducc.rm.user.registry.

14.5 Priority vs Weight

It is possible that the various policies may interfere with each other. It is also possible that the fair share weights are not sufficient to guarantee sufficient resources are allocated to high importance jobs. Class-based priorities are used to resolve these conflicts.

Simply: priority is used to specify the order of evaluation of the job classes. Weight is used to proportionally allocate the number of shares to all classes of the same priority under the weighted fair-share policies.

Priority. When a scheduling cycle starts, the scheduling classes are ordered from ”best” to ”worst” priority. The scheduler then attempts to allocate ALL of the system’s resources to the ”best” priority class. If any resources are left, the scheduler proceeds to schedule classes in the next best priority, and so on, until either all the resources are exhausted or there is no more work to schedule.

It is possible to have multiple job classes of the same priority. What this means is that resources are allocated for the set of job classes from the same set of resources at the same time, usually under weighted fair-share. (It would be unusual to have multiple non-preemptable classes at the same priority. If this is configured, the class requests are filled arbitrarily with no attempt to divide the resources fairly or equitably). Resources for higher priority classes will have already been allocated, resources for lower priority classes may never become available.

To constrain high priority jobs from completely monopolizing the system, fair-share weights are used for FAIR_SHARE classes, and allotment is used for non-preemptable classes.

Weight. Weight is used to determine the relative importance of jobs in a set of job classes of the same priority when doing fair-share allocation. All job classes of the same priority are assigned shares from the full set of available resources according to their weights using weighted fair-share. Weights are used only for fair-share allocation.

14.6 Node Pools

It may be desired or necessary to constrain certain types of resource allocations to a specific subset of the resources. Some nodes may have special hardware, or perhaps it is desired to prevent certain types of jobs from being scheduled on some specific set of machines. Nodepools are designed to provide this function.

Nodepools impose hierarchical partitioning on the set of available machines. A nodepool is a subset of the full set of machines in the cluster. Nodepools may not overlap. A nodepool may itself contain non-overlapping subpools.

Job classes are associated with nodepools. The scheduler treates preemptable work and non-preemptable work differently with regards to nodepools:

14.7 Scheduling Classes

The primary abstraction to control and configure the scheduler is the class. A class is simply a set of rules used to parametrize how resources are assigned to work requests. Every request that enters the system is associated with a single class.

Chapter 15
Service Management

A globally-registered service pinger is a properties file that contains only service registraton options pertaining to pingers. This file must be placed in DUCC’s runtime/resources/service_monitors directory. It may be given any name but “best practices” would suggest it be named the same as the service_ping_class. Services then use this pinger by specifying its filename in their service_ping_class option.

Globally-registered pingers may be run internally as threads within the SM, or externally as processes. To specify that a pinger be run internally, add the property

The “internal” option is flagged as in illegal option when specified in service registrations and all pingers not specified as “internal” are run as external processes managed by the SM.

Best practices dictate that the filename of an external pinger contain the postfix .external to clearly identify it as external.

As an example, the default UIMA-AS pinger is supplied in the global registry under the two names:

Note that users may override any of the properties in globally-registered external pingers, but only the service_ping_arguments of an internal pinger to protect its integrity by specify that argument in their own service registrations.

By default external pingers are launched with the same JRE as used by the DUCC daemons but if the registration contains a entry for the JAVA_HOME environment variable, that JRE will be used instead.

For backwards compatibility the default UIMA-AS pinger is also registered under its old name:

Chapter 16
DUCC Web Server Customization

This chapter describes how to take advantage of DUCC Web Server plug-in capabilities in order to add local modifications.

Why would you want to do this? Perhaps you have some related information that your DUCC Web Server could display to the user community. There are considerations for both the server and client sides.

The following discussion is related to the downloaded DUCC source code, specifically the project uima-ducc-web.

16.1 Server Side

In package org.apache.uima.ducc.ws you will find DuccPlugins.java which you can modify or extend.

16.2 Client Side

In folder /src/main/webapp/root/js/ you will find ducc.local.js which you can modify.

16.3 Build and Install

Build a new uima-ducc-web-[version].jar comprising the revised DuccPlugins.class and any additional dependent classes. Replace the vanilla $DUCC_HOME/lib/uima-ducc/uima-ducc-web-[version].jar with the one containing your modifications.

Copy your new ducc.local.js to the installed Web Server’s $DUCC_HOME/webserver/root/js directory.

Chapter 17
Reliable DUCC

17.1 Introduction

DUCC can be configured to run reliably by having multiple head nodes, comprising one master and one or more backup head nodes. DUCC exploits Linux keepalived virtual IP addressing to enable this capability.

The advantages are that if the master DUCC host becomes unusable, the backup DUCC can take over seamlessly such that active distributed Jobs, Reservations, Managed Reservations and Services continue uninterrupted. Take over also facilitates continued acceptance of new submissions and monitoring of new and existing submissions without interruption.

17.2 Configuring Host Machines

In this example, the IP address 9.59.193.8 is managed by the keepalived daemons. It is assigned to the current DUCC master. If can float to another host when the presently assigned master is no longer viable.

determines the viability of DUCC on the present host, returning 0 when viable or 1 otherwise. Note that in this example the value of “ducc_runtime” is hard-coded.

17.3 Configuring DUCC

To configure DUCC to run reliable, one required property must be configured in the site.ducc.properties file. Example:

Use the virtual IP address configured for your host machines keepalived. Use of the DNS name is also supported.

17.4 Webserver

The master DUCC Webserver will display all pages normally with additional information in the heading upper left:

The backup DUCC Webserver will display some pages normally with additional information in the heading upper left:

Hovering over reliable will yield the following information: Click to visit master

17.5 Database

Configure the database to be on a separate machine from the reliable DUCC head nodes. In site.ducc.properties specify:

The existing administrator commands start_ducc and stop_ducc will honor the value specified for ducc.database.automanage in site.ducc.properties.

Chapter 18
Simulation and System Testing

DUCC is shipped with support for simulating large clusters of arbitrarily configured nodes. A simple control file describes some number of simulated nodes of arbitrary memory sizes. DUCC’s design allows multiples of these to be spawned on a single node, or on a small set of nodes with multiple simulated nodes apiece. The standard testing configuration used for most of the development of DUCC consisted of four physical 32-GB machines running 52 simulated nodes of varying memory sizes from 32 to 128-GB each.

To simulate job loads, a simple UIMA-AS job that sleeps for some easily configured length of time was constructed. Another control file is used to generate job specifications requesting randomly-chosen job parameters such as memory requirements, service dependencies, scheduling classes, and so on.

The test suite contains a simple UIMA Analysis Engine called FixedSleepAE, and a simple Collection Reader called FixedSleepCR. The CR reads a set of sleep times, creates CASs, and ships them to the AEs via DUCC’s Job Driver. The CAS contains the time to sleep and various parameters regarding error injection.

The AE receives a CAS, performs error injection if requested, and sleeps the indicated period of time, simulating actual computation but requiring very few physical resources. Hence, many of these may be run simultaneously on relatively modest hardware.

Developers may construct arbitrary jobs by creating a file with sleep times designed to exercise what ever is necessary. DUCC ships with the three primary job collections (test suites) used during initial development. The suites are based on actual workloads and have shown to be very robust for proving the correctness of the DUCC code under stress.

The cluster simulator has been also been run on a 4GB iMac with 8 simulated Agents, an 8GB MacBook with the same configuration, a 32GB iMac with up to 40 simulated Agents. It has also been scaled up to run on 8 45GB Intel nodes running Linux, simulating 20TB of memory.

18.1 Cluster Simulation

18.1.1 Overview

Cluster-based tools such as DUCC are very hard to test and debug because all interesting problems occur only when the system is under stress. Acquisition of a cluster of sufficient size to expose the interesting problems is usually not practical.

DUCC’s design divorces all the DUCC processes from specific IP addresses or node names. ActiveMQ is used as a nameserver and packet router so that all messages can be delivered by name, irrespective of the physical hardware the destination process may reside upon.

Normally, the DUCC Agents report the name, IP address, and physical memory of the node they actually do reside upon. This is simply for convenience. It is possible to parameterize the DUCC Agents to report any arbitrary name and address to the DUCC. DUCC components that need to know about Node Agents establish subscriptions to the Agent publications with ActiveMQ and build up their internal structures from the node identities in the Agent publications. Processes which normally establish agent listeners are are the RM, PM, and WS.

It is also possible to parameterize a DUCC agent to cause it to report any arbitrary memory size. Thus, an agent running on a 2GB machine can be started so that it reports 32GB of memory. This parameterization is specifically for testing, of course.

The ability to parameterize agent identities and memory sizes is what enables cluster simulation. A control file is used by start-up scripting to spawn multiple agents per node, each with unique identities.

18.1.2 Node Configuration

A Java properties file is used to configure simulated nodes. There are three types o entries in this file:

For example, the following simulated cluster configuration defines twenty (20) simulated nodes, all to be run on the single physical machine called agentn. The simulated nodes contain a mix of 31GB, 47GB, and 79GB memory sizes. There are 7 31GB nodes, 7 47GB nodes, and 6 79GB nodes.

The nodenames generated by this means are the name of the physical node where the agent is spawned, and a numeric id appended, for example,

18.1.3 Setting up Test Mode

During simulation and testing it is desirable and usually required that DUCC run in unprivileged mode, with all processes belonging to a single userid. Unfortunately, this does not exercise any of the multi-user code paths, especially in the Resource Manager.

To accommodate this, DUCC can be configured to run in “test mode”, such that work is submitted under “simulated” userid which DUCC treats as discrete IDs. All actual work is executed under the ownership of the tester however.

In test mode, jobs may specify what simulated userid is to be used. Most of DUCC does not pay any attention to the user so this works fine, and the parts that do care about the user are bypassed when ducc.runmode=Test is configured.

18.1.4 Starting a Simulated Cluster

DUCC provides a start-up script in the directory $DUCC_HOME/examples/systemtest called start_sim.

WARNING: Cluster simulation is intended for DUCC testing, including error injection. It is similar to flying a high-performance fighter jet. It is intentionally twitchy. Very little checking is done and processes may be started multiple time regardless of whether is sane to do this.

To start a simulated cluster, use the start_sim script. The DUCC broker will be started without any access restrictions so that it can be used as an application broker for any UIMA-AS services used in the simulation tests.

18.1.5 Stopping a Simulated Cluster

Note that check_ducc is found in $DUCC_HOME/admin. The stop_sim script is found in duccruntime/examples/systemtest.

The start_sim script creates a file called sim.pids containing the physical node name, Unix process ID (PID), and component ID (ws, sm, or, pm, rm) of each started DUCC component. In the case of agents, each agent is assigned a number as a unique id. These ids are used with stop_sim to affect specific processes. If the cluster is stopped without using stop_sim, or if it simply crashes, this PID file will get out of date. Fly more carefully next time!

Description The stop_sim script is used to stop some or all of a simulated cluster.

18.2 Job Simulation

18.2.1 Overview

“Real” jobs are highly memory and CPU intensive. For testing and simulation purposes, the jobs need not use anywhere close to their declared memory, and need not consume any CPU at all. The FixedSleepAE is a UIMA analytic that is given a time, in milliseconds, and all it does is sleep for that period of time and then exit. By running many of these in a simulated cluster it is possible to get all the DUCC administrative processes to behave as if there is a real load on the system when in fact all the nodes and jobs are taking minimal resources.

The FixedSleepAE is delivered CASs by the FixedSleepCR. This CR reads a standard Java properties file, using the property “elapsed” to derive the set of sleep times. On each call to the CR’s “getNext()” method, the next integer from “elapsed” is fetched, packaged into a CAS, and shipped to ActiveMQ where it is picked up by the next available FixedSleepAE.

The test driver is given a control file with the names of all the jobs to be submitted in the current run, and the elapsed time to wait between submission of each job. Each job name corresponds to a file that is not an actual DUCC specification, but rather the description of a DUCC specification. Each description is a simple Java properties file.

To submit a job, the test driver reads the next job description file derive the number of threads, the simulated user, the desired (simulated) memory for the job, (possibly) the service ID, and the scheduling class for the job. From these it constructs a DUCC job specification and submits it to DUCC.

Scripting is used to read the job meta-descriptors and generate a control file that submits the job set with a large set of variations. The same scripting reads each meta-descriptor and modifies it according to the specific parameters of the run, adjust things such as scheduling class, memory size, etc.

18.2.2 Job meta-descriptors

For each simulated job in a run, a meta-descriptor must be constructed. These may be constructed “by hand”, or via local scripting, for example from log analysis. (The packaged meta-descriptors are generated from logs of actual workloads.)

18.2.3 Prepare Descriptors

A prepare descriptor is also a standard Java properties file. This defines where the set of meta descriptors resides, where to place the modified meta-files, how to assign scheduling classes to the jobs, how to apportion memory sizes, how to apportion services, how long the total run should last, and how to compress sleep times.

All parts of the run are randomized, but the randomization can be made deterministic between runs by specifying a seed to the random number generator.

18.2.4 Services

It is possible to run the FixedSleepAE as a UIMA-AS service, with each job specifying a dependency on the service, and the indicated service doing the actual sleeping on behalf of the job.

To use these simulated services, configure a “service boot” file and reference the services from the job generation config file.

Service pseudo IDs DUCC is packaged with 10 pre-configured services that use the FixedSleepAE. All of these services behave identically, the only difference is their endpoints, which allows the simulated runs to activate and use multiple independent services. Because the endpoints are in the various UIMA XML service descriptors, it is necessary to use exactly these IDs when generating a test run. Thus, the only valid pseudo-ids for service configuration are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9.

These service ids are used on the job configuration file to establish a weighted distribution of service use among the jobs.

18.2.5 Generating a Job Set

The prepare script, found in $DUCC_HOME/examples/systemtest is used to generate a test run from the control files described above. To use it, execute

This script reads the meta-specification in the jobs.in directive of the config-file, generates a set of meta-specification files into the jobs.out directory, and creates a control file, job.ctl. The job.ctl file is used by the simulation driver to submit all the jobs.

18.2.6 Running the Test Driver

A test run is driven from the script runducc which resides in the directory $DUCC_HOME/examples/systemtest. This script supports a large number of options intended to inject errors and otherwise perturb a run.

To use the test driver, first create a job collection as described above. This will generate a file called job.ctl in the test directory containing the prepare file.

18.3 Pre-Packaged Tests

Three test suites are provided using the mechanisms described in the previous section:

The configurations are found in the $DUCC_HOME/examples/systemtest directory and are in sub directories called,


Current Mode	Desired Mode	Action	Notes


Autostart	Manual	Use CLI to modify registration to autstart false.	Service does not stop until requested by CLI. Service will not start at DUCC boot.

Autostart	Reference	Use CLI to modify registration to autostart false and observe references.	Service stops after last reference exits, plus linger time.

Autostart	Stopped	Use CLI to stop the service.	The CLI stop will by necessity disable the service to insure it remains stopped.

Reference	Autostart	Use CLI to modify registration to autostart true.	Service continues to run after last reference exits. Service always started at DUCC boot.

Reference	Manual	Use CLI to ignore references.	Service continues to run after last reference exits.

Reference	Stopped	Use CLI to stop the service.	The CLI stop will by necessity disable the service to insure it remains stopped.

Manual	Autostart	Use CLI to modify registraiton to autostart true.	Service will be started on DUCC boots.

Manual	Reference	Use CLI to observe references.	Service will stop after last referencing job exits, plus linger time..

Manual	Stopped	Use CLI to stop the service.	The CLI stop will by necessity disable the service to insure it remains stopped.

Stopped	Autostart	Use CLI to modify registration to autostart true.	Service will start immediately. It may be necessary to enable the service as well.

Stopped	Reference	Submit a job or service that references the service.	It may be necessary to enable the service as well. The service will stop after the last referencing work exits, plus linger.

Stopped	Manual	Use CLI to start the service.	The CLI start will also enable the service if necessary.


Component	Log Name


Resource Manager	rm.log

Service Manager	sm.log

Orchestrator	or.log

Process Manager	pm.log

Web Server	ws.log

Agent	[hostname].agent.log