Building Cylc Workflow Suites

This application provides the user with a YAML-based configuration interface for building Cylc workflow suites. This section provides a simple example for configuring a HELLO_WORLD type Cylc workflow suite.

Configuring the Suite

The following is an example configuration for a Cylc workflow suite.

environment:
  ENVVAR_ONE: 1
  ENVVAR_TWO: 2
  ENVVAR_THREE: 3

experiment:
  EXPERIMENT_NAME: HELLO_WORLD
  INITIAL_CYCLE_POINT: 20000101T000000
  FINAL_CYCLE_POINT: 20000101T120000
  CYCLE_INTERVAL: PT21600S
  NENSMEM: 3

graph: !ENV ${ENGINES_ROOT}/parm/cylc/graph.demo.jinja2

platform:
  SCHEDULER: slurm
  EXEC_RETRIES_COUNT: 3
  EXEC_RETRIES_INTERVAL_SECONDS: 30
  EXEC_RETRIES: "3*PT30S"

schema: !ENV ${ENGINES_ROOT}/parm/cylc/schema/workflow.schema.yaml

schemas:
  directives: !ENV ${ENGINES_ROOT}/parm/cylc/schema/slurm.schema.yaml
  experiment: !ENV ${ENGINES_ROOT}/parm/cylc/schema/experiment.schema.yaml
  platform: !ENV ${ENGINES_ROOT}/parm/cylc/schema/platform.schema.yaml
  slurm: !ENV ${ENGINES_ROOT}/parm/cylc/schema/slurm.schema.yaml

tasks:
  - !ENV ${ENGINES_ROOT}/parm/cylc/tasks/eggs.yaml
  - !ENV ${ENGINES_ROOT}/parm/cylc/tasks/ham.yaml
  - !ENV ${ENGINES_ROOT}/parm/cylc/tasks/SPAM.yaml

templates:
  directives: !ENV ${ENGINES_ROOT}/parm/cylc/templates/slurm.jinja2
  environment: !ENV ${ENGINES_ROOT}/parm/cylc/templates/environment.jinja2
  runtime: !ENV ${ENGINES_ROOT}/parm/cylc/templates/runtime.jinja2
  suite: !ENV ${ENGINES_ROOT}/parm/cylc/templates/suite.jinja2
  tasks: !ENV ${ENGINES_ROOT}/parm/cylc/templates/tasks.jinja2

The configuration variables are described in the following table.

Variable	Description
environment	The global environment variables available to the entire Cylc workflow suite.
experiment	The experiment variables; note that the EXPERIMENT_NAME, INITIAL_CYCLE_POINT, FINAL_CYCLE_POINT, and CYCLE_INTERVAL are mandatory variables; the NENSMEM attribute is used by Cylc family-type tasks described below.
graph	The Cylc workflow suite graph; the graph describes task dependencies and scheduling and is generally specific to a given experiment.
platform	The attributes describing the respective host/platform upon which the respective Cylc workflow suite will be launched; the SCHEDULER variable is mandatory and a full list of available attributes can be found here.
schema	Contains the schema of attributes available for the respective Cylc workflow configuration.
schemas	The schemas for each Cylc workflow component; the user is strongly discouraged to modify the respective schema files.
tasks	YAML-formatted files containin the attributes for each respective Cylc workflow task.
templates	Template files of various formats required to construct the Cylc workflow suite; the user is discouraged from modifying the respective template files.

Building the Suite

The Cylc workflow suite configured above may be constructed as follows.

user@host:$ cd /path/to/ufs_engines/scripts
user@host:$ ./cylc_workflow.py --help

Usage: ./cylc_workflow.py [-h] output yaml

Cylc workflow builder application interface.

Positional Arguments:
  output      The directory-tree path to where the Cylc workflow suite is to be written.
  yaml        The YAML-formatted file containing the Cylc workflow attributes.

Optional Arguments:
  -h, --help  show this help message and exit

user@host:$ ./cylc_workflow.py /path/to/output/directory /path/to/yaml/suite/attributes

Successful execution will result in the Cylc configuration files and the respective suite being written to the path specified by /path/to/output/directory in the example above. The directory tree will appear as follows, and the respective sub-directories are described accordingly.

/path/to/output/directory/
|-- directives
|   |-- SPAM.directives
|   |-- eggs.directives
|   |-- ham.directives
|-- environment
|   |-- SPAM.environment
|   |-- eggs.environment
|   |-- ham.environment
|-- environment.rc
|-- experiment.rc
|-- graph.rc
|-- job
|   |-- SPAM.job
|   |-- eggs.job
|   |-- ham.job
|-- platform.rc
|-- runtime
|   |-- SPAM.runtime
|   |-- eggs.runtime
|   |-- ham.runtime
|-- runtime.rc
|-- suite.rc
|-- tasks
|   |-- SPAM.task
|   |-- eggs.task
|   |-- ham.task
|-- tasks.rc

Path	Description
directives	Configuration-type files containing variables specific to the job scheduler for each respective workflow task.
environment	Jinja2-formatted files containining environment variables for the respective workflow tasks.
job	Configuration-type files containining job attributes such as the respective batch system and job scheduler.
runtime	Cylc workflow templates for the respective workflow tasks.
tasks	Configuration-type files containing the job scheduler attributes for the respective workflow tasks.

Directives

The .directives files for each task are generated automatically from the corresponding .task file contents. An example .directives file is as follows.

ntasks = 1
account = ACCOUNT
partition = PARTITION
time = 3:00

The above example provides the SLURM configuration for a workflow task. Each workflow task requires a corresponding .directives file.

Environment

An example .environment file is as follows.

#!Jinja2
{% set NTASKS_SPAM = 1 %}

The above example provides an .environment configuration containing attributes specific to the SPAM task. A similar example for the ham task is as follows.

#!Jinja2
{% set NTASKS_ham = 1 %}

For environment variable available to the entirety of the Cylc workflow suite, the /path/to/output/directory/environment.rc file should be modified. An example of such a file is as follows.

CYCLE = $(cylc cyclepoint --template=%Y%m%d%H%M%S)
ENVVAR_ONE = 1
ENVVAR_TWO = 2
ENVVAR_THREE = 3

The CYCLE attribute is mandatory and should not be modified. The ENVVAR_ONE, ENVVAR_TWO, and ENVVAR_THREE environment variables are available for all aspects within the respective Cylc workflow suite. The user may place environment variables required beyond specific tasks within this file.

Job

The .job files are typically reserved for batch job declarations. At a minimum, a file must exist for each task and contain the following.

batch system = slurm

This informs the Cylc workflow suite that the SLURM batch system scheduler will be used.

Runtime

The .runtime files contain that instructions for the respective tasks. An example for the ham task is as follows.

[[ham]]
script = sh /home/ufs_engines/jobs/JJOBS_HAM

[[[directives]]]
%include '/path/to/output/directory/directives/ham.directives'

[[[environment]]]
%include '/path/to/output/directory/environment/ham.environment'

[[[job]]]
%include '/path/to/output/directory/job/ham.job'

Note that for each mandatory attribute (e.g., directives, environment, and job), the corresponding files (as described above) are linked using the %include attribute. This informs the Cylc workflow suite where to find the attributes for the respective task. An example for a Cylc family task, in this case SPAM, the .runtime file is as follows.”

[[SPAM]]
{%for IDX in range(0, NENSMEM)%}

[[mem_{{ IDX }}_spam]]
inherit = SPAM
script = sh /home/ufs_engines/jobs/JJOBS_SPAM

[[[directives]]]
%include '/home/ufs_engines/scripts/tests/cylc_workflow_2/directives/SPAM.directives'

[[[environment]]]
%include '/home/ufs_engines/scripts/tests/cylc_workflow_2/environment/SPAM.environment'

[[[job]]]
%include '/home/ufs_engines/scripts/tests/cylc_workflow_2/job/SPAM.job'

{% endfor %}

Note that the Cylc family tasks require a loop of some dimension, in this case, the value for NENSMEM. For each member of the family, the SPAM task attributes will be inherited, as well as the other task attributes, similar to the ham example above. Finally, each task required for the respective Cylc workflow suite must be entered into the /path/to/output/directory/runtime.rc as follows.

%include '/path/to/output/directory/runtime/ham.runtime'
%include '/path/to/output/directory/runtime/SPAM.runtime'

Tasks

The .task files inform the respective job scheduler (SLURM in this example) of the resources required for each task. An example .task file for the ham task is as follows.

--account = "ACCOUNT"
--ntasks = 1
--partition = "PARTITION"
--time = "3:00"

These attributes, as noted above, correspond to the available SLURM attributes. The /path/to/output/directory/tasks.rc is constructed automatically from the respective Cylc workflow .task files.

Processed Suite

Once the Cylc workflow suite has been registered and subsequently launched, a processed Cylc workflow suite consisting of each of the components listed above will be created. An example derived from the above-mentioned and described components is as follows.

#!Jinja2
[cylc]
  UTC mode = True
[scheduling]
  initial cycle point = 20000101T000000
  final cycle point = 20000101T120000
  [[dependencies]]
    [[[R1]]]
      graph = """
             ham: succeed => SPAM
             SPAM: succeed-all => eggs
             """
    [[[PT21600S]]]
      graph = """
             eggs[-PT21600S]: succeed => ham
             ham: succeed => SPAM
             SPAM: succeed-all => eggs
             """
    [[[R1/$]]]
      graph = """
             eggs[-PT21600S]: succeed => ham
             ham: succeed => finish
             """
[runtime]
  [[root]]
    [[[environment]]]
      CYCLE = $(cylc cyclepoint --template=%Y%m%d%H%M%S)
      ENVVAR_ONE = 1
      ENVVAR_TWO = 2
      ENVVAR_THREE = 3

 [[eggs]]
   script = sh /home/ufs_engines/jobs/JJOBS_EGGS
   [[[directives]]]
     ntasks = 1
     account = ACCOUNT
     partition = PARTITION
     time = 10:00
   [[[job]]]
     batch system = slurm

 [[ham]]
   script = sh /home/ufs_engines/jobs/JJOBS_HAM
   [[[directives]]]
     ntasks = 1
     account = ACCOUNT
     partition = PARTITION
     time = 3:00
   [[[job]]]
     batch system = slurm

 [[SPAM]]
 [[mem_0_spam]]
   inherit = SPAM
   script = sh /home/ufs_engines/jobs/JJOBS_SPAM
   [[[directives]]]
     ntasks = 1
     account = ACCOUNT
     partition = PARTITION
     time = 3:00
   [[[job]]]
     batch system = slurm

 [[mem_1_spam]]
   inherit = SPAM
   script = sh /home/ufs_engines/jobs/JJOBS_SPAM
   [[[directives]]]
     ntasks = 1
     account = ACCOUNT
     partition = PARTITION
     time = 3:00
   [[[job]]]
     batch system = slurm

 [[mem_2_spam]]
   inherit = SPAM
   script = sh /home/ufs_engines/jobs/JJOBS_SPAM
   [[[directives]]]
     ntasks = 1
     account = ACCOUNT
     partition = PARTITION
     time = 3:00
   [[[job]]]
     batch system = slurm