XIOS-2

XIOS is the IO management library that is used by NEMO to produce netCDF results files. NEMO-3.6 uses XIOS-2.

XIOS operates as a server process that multiple NEMO calculation processes communicate with to write results to netCDF files. XIOS buffers the output from NEMO in memory while the much slower process of writing to disk happens. That means that the NEMO processes can do calculations almost continuously without having to periodically pause to write results to disk.

In many cases, especially on the Alliance HPC clusters with large memory per node, it is possible to configure NEMO runs with tens to a hundred or more processors to use a single XIOS process. The XIOS process must be started in the same mpirun command as NEMO. Fortunately, the NEMO Command Processor and SalishSeaCast NEMO Command Processor tools take care of doing that for you.

Getting XIOS-2

The MOAD group maintains our own Mercurial repositories on GitHub of the XIOS-2 code and build configuration files for building XIOS-2 on the compute platforms that we use. Our XIOS-2 code repo is accessible only by members of the MOAD group so as to respect the sign-up requirement of the upstream XIOS repository. The architecture file repo is public so that other researchers can make use of the build system options that we have figured out for various system.

To get the XIOS-2 code and build configuration files repos you need to clone them into your research project directory, typically one of MEOPAR/. Here are the commands to do that on the 2 platforms that we use most.

`graham`

cd $PROJECT/$USER/MEOPAR/
git clone git@github.com:SalishSeaCast/XIOS-2.git
git clone git@github.com:SalishSeaCast/XIOS-ARCH.git

`salish`

cd /data/$USER/MEOPAR/
git clone git@github.com:SalishSeaCast/XIOS-2.git
git clone git@github.com:SalishSeaCast/XIOS-ARCH.git

Building XIOS-2

First symlink the XIOS-2 build configuration files for the machine that you are working on from the XIOS-ARCH repo clone into the XIOS-2/arch/ directory, then compile and link XIOS-2.

`MEOPAR` on `graham`

cd $PROJECT/$USER/MEOPAR/XIOS-2/arch
ln -sf $PROJECT/$USER/MEOPAR/XIOS-ARCH/ALLIANCE/arch-X64_GRAHAM.env
ln -sf $PROJECT/$USER/MEOPAR/XIOS-ARCH/ALLIANCE/arch-X64_GRAHAM.fcm
ln -sf $PROJECT/$USER/MEOPAR/XIOS-ARCH/ALLIANCE/arch-X64_GRAHAM.path
cd $PROJECT/$USER/MEOPAR/XIOS-2
./make_xios --full --arch X64_GRAHAM --job 8

It typically takes about 5 minutes to build XIOS-2 on graham.

To build NEMO you will need an environment variable named XIOS_HOME whose value is the absolute path to you XIOS-2 directory. For the MEOPAR project it is:

XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2

You can add the line:

export XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2

to your $HOME/.bash_profile file if you want the XIOS_HOME environment variable to be set automatically whenever you ssh in.

`MEOPAR` on `salish`

cd /data/$USER/MEOPAR/XIOS-2/arch
ln -sf /data/$USER/MEOPAR/XIOS-ARCH/UBC-EOAS/arch-GCC_SALISH.fcm
ln -sf /data/$USER/MEOPAR/XIOS-ARCH/UBC-EOAS/arch-GCC_SALISH.path
cd /data/$USER/MEOPAR/XIOS-2
./make_xios --full --arch GCC_SALISH --netcdf_lib netcdf4_seq --job 8

To build NEMO you will need an environment variable named XIOS_HOME whose value is the absolute path to you XIOS-2 directory. For the MEOPAR project it is:

XIOS_HOME=/data/$USER/MEOPAR/XIOS-2

You can add the line:

export XIOS_HOME=/data/$USER/MEOPAR/XIOS-2

to your $HOME/.bash_profile file if you want the XIOS_HOME environment variable to be set automatically whenever you ssh in.

XIOS-2 Configuration Files

To use XIOS-2 with NEMO, four configuration files written in XML are required:

field_def.xml defines the variables that can be output and the grids on which they are defined. Field definition elements may (and generally should) also contain metadata attributes such as long name, standard name, and units. Please see the field_def.xml section below for more information about the structure and contents of field_def.xml files.
domain_def.xml defines “zoomed” sub-domains of the model domain and the grids on which they are defined. The “zooms” are defined on the i-j (x-y) directions, regardless of the depth of the sub-domain. Please see the domain_def.xml section below for more information about the structure and contents of domain_def.xml files.
iodef.xml defines the vertical extent of output grids in the axis elements, and the output grids. It also contains a separate context element for xios in which a few settings that control XIOS-2 are declared.
file_def.xml defines the files into which field variables are output and the frequency of output of those files. Variable names can be transformed from the internal NEMO names to more user friendly names in the field elements in this file. This is also where on-the-fly deflation of output files is enabled via the compression_level="4" attribute of file_group elements.

Warning

XML syntax is very exacting, so care is required when you edit XML files to ensure that tags are correctly closed, attribute values are correctly quoted, etc.

Annoyingly, NEMO will fail with no diagnostic messages if your XML files contain errors. If you suspect that you have made an error in editing an XML file, one way of checking is to use an online validator like https://www.xmlvalidation.com/.

Customizing XML Files

The NEMO-3.6-code repositories contains sample XIOS-2 configuration files in the NEMOGCM/CONFIG/SHARED/ and some of the NEMOGCM/CONFIG/*/EXP00/ directories.

Warning

Please do not modify and commit those files. Doing so will cause conflicts when changes to NEMO are pulled in from the upstream repository, and your changes will be overwritten. Instead, put copies of the XML files that you want to change under version control in your runs configuration repo (for example, the SS-run-sets repo for people working on MEOPAR).

Command Processors and XML Files

The NEMO Command Processor and SalishSeaCast NEMO Command Processor tools provide a way, via YAML run description files, to map XML files with arbitrary file names and directory paths on to the file names that NEMO requires in the directory from which NEMO is executed.

The output section of the YAML description file is where the XML file mappings and other XIOS-2 settings are specified. Please see the salishsea YAML file output section docs if you are working on the Salish Sea configurations of NEMO, or the nemo YAML file output section docs if you use another NEMO configuration. There are also examples of complete YAML run description files in those docs.

The simplest possible YAML file output section is:

output:
  XIOS servers: 1
  separate XIOS server: True
  filedefs: file_def.xml
  domaindefs: domain_def.xml
  fielddefs: field_def.xml
  iodefs: iodef.xml

In this case, the XML files are all in the same directory as the YAML file. If you use relative paths, they have to be relative to the directory where the YAML file is.

A more complicated example is:

output:
  XIOS servers: 1
  separate XIOS server: True
  filedefs: $PROJECT/$USER/SS-run-sets/v201905/tuning/tuning_file_def.xml
  domaindefs: ../domain_def.xml
  fielddefs: $PROJECT/SS-run-sets/v201905/field_def.xml
  iodefs: iodef.xml

Note the use of:

A relative path for domaindefs
Absolute paths containing the environment variables like $USER and $PROJECT for filedefs and fielddefs. Other environment variables like $HOME and $SCRATCH can also be used in XML file paths.
The more descriptive file name tuning_file_def.xml for filedefs

`field_def.xml`

field_def.xml defines the variables that can be output and the grids on which they are defined. Field definition elements may (and generally should) also contain metadata attributes such as long name, standard name, and units.

This section provides some information about the structure and contents of a field_def.xml file. This is not an exhaustive reference guide for all of the possible attribute values; for that, please see chapter 3 of the XIOS User Guide.

NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml is the reference version of the file that is provided with the NEMO code. In many cases, you can use that reference file by putting its path as the value of the filedefs element in the output section of your run description YAML file (see Command Processors and XML Files). Reasons why you might want to create your own customized version (see Customizing XML Files) of field_def.xml include:

Adding new variable(s) to NEMO that you want to include in your output files
Adjusting/correcting the values of variable field attributes such as long_name, standard_name, unit, etc. Those attributes provide variable-level metadata items in output files.

Here is an example fragment of a field_def.xml file:

<field_definition level="1" prec="4" operation="average" enabled=".TRUE." default_value="1.e20">
 <field_group id="grid_T" grid_ref="grid_T_2D">
   <field id="sst" long_name="sea surface temperature" standard_name="sea_surface_temperature" unit="degC"/>
   <field id="toce" long_name="temperature" standard_name="sea_water_conservative_temperature" unit="degC" grid_ref="grid_T_3D"/>

   <field id="sss" long_name="sea surface salinity" standard_name="sea_surface_reference_salinity" unit="g kg-1"/>
   <field id="soce" long_name="salinity" standard_name="sea_water_reference_salinity" unit="g kg-1" grid_ref="grid_T_3D"/>

   <field id="sst2" long_name="square of sea surface temperature" standard_name="square_of_sea_surface_temperature" unit="degC2">
     sst * sst
   </field >

   <field id="sstmax" long_name="max of sea surface temperature" field_ref="sst" operation="maximum"/>
   ...
 </field_group>
 ...
</field_definition>

field_def.xml files contain 3 types of tags:

field_definition
field_group
field

field tags must be contained within a field_group tag, which must be contained within a field_definition tag.

Attributes included in a tag apply to all contained tags unless they are explicitly overridden in a contained tag. So the operation="average" attribute in:

<field_definition level="1" prec="4" operation="average" enabled=".TRUE." default_value="1.e20">

means that all field values will be averaged over the output time interval unless a different operation is specified in the field tag, for example:

<field id="sstmax" long_name="max of sea surface temperature" field_ref="sst" operation="maximum"/>

in which case the maximum value over the output time interval of the sst field (specified by the field_ref attribute) will be calculated by XIOS.

The operation attribute enables the burden of calculating various temporal quantities on field variables to be shifted from NEMO to XIOS. Please see section 3.2 of the XIOS User Guide for details.

Another way of doing field operations in XIOS is to specify them in the field tag, for example:

<field id="sst2" long_name="square of sea surface temperature" standard_name="square_of_sea_surface_temperature" unit="degC2">
  sst * sst
</field >

Here again, the burden of declaration, memory allocation, and calculation of the sst2 variable is shifted from NEMO to XIOS. This form of field calculation can be useful for calculating fluxes.

field_group tags specify the default grid on which the contained field tags are defined via the grid_ref attribute. That attribute can, of course, be overridden in the contained field tags.

All field tags should have the following attributes:

long_name
standard_name
unit

Those attributes are passed through to the netCDF output files as field variable metadata.

Values for the standard_name attribute should be chosen from the CF conventions standard names table. Standard names are written in “snake case” (words separated by _ characters). That table also provides canonical units that should be used at the value of the unit attribute.

The value of the long_name attribute can be more free-from and descriptive. It is typically used for plot axis labels, table headings, etc.

In addition to NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml, there are examples of field_def.xml files in the SS-run-sets/v201702/ directory tree.

`domain_def.xml`

domain_def.xml defines “zoomed” sub-domains of the model domain and the grids on which they are defined. The “zooms” are defined on the i-j (x-y) directions, regardless of the depth of the sub-domain.

This section provides some information about the structure and contents of a domain_def.xml file. This is not an exhaustive reference guide for all of the possible attribute values; for that, please see chapter 5 of the XIOS User Guide.

NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/domain_def.xml is the reference version of the file that is provided with the NEMO code. In many cases, you can use that reference file by putting its path as the value of the domaindefs element in the output section of your run description YAML file (see Command Processors and XML Files). The main reason why you might want to create your own customized version (see Customizing XML Files) of domain_def.xml is to define your own “zoomed” sub-domain of the model domain. Assuming that your “zoomed” sub-domain is significantly smaller than the full model domain, the output files you produce form it will be significantly smaller than full domain files. Examples of uses of “zoomed” sub-domains in the SalishSeaCast NEMO configuration are:

single point sub-domains for sea surface height output at tide gauge station locations
single point sub-domains for model output at the ONC VENUS instrument platform locations
sub-domains encompassing the southern Strait of Georgia for velocity fields outputs to compare against drifter tracks
sub-domains encompassing the Baynes Sound AGRIF sub-grid for tracer outputs for visualization of the transition between the full domain grid and the AGRIF sub-grid
sub-domains that provide boundary condition fields for the Vancouver Harbour and Lower Fraser River FVCOM model

Here is an example fragment of a domain_def.xml file:

<domain_definition>
  <domain_group id="grid_T">
    <domain id="grid_T" long_name="grid T"></domain>

    <!-- Tide Gauge Stations -->
    <domain id="PortRenfrew" domain_ref="grid_T">
      <zoom_domain ibegin="61" jbegin="401" ni="1" nj="1"/>
    </domain>
    ...
  </domain_group>
  ...
</domain_definition>

domain_def.xml files contain 4 types of tags:

domain_definition
domain_group
domain
zoom_domain

domain tags must be contained within a domain_group tag, which must be contained within a domain_definition tag. zoom_domain tags must be contained within a domain tag.

A minimal, complete domain_def.xml file would contain domain definitions for the full domain T, U, V, and W grids:

<domain_definition>
  <domain_group id="grid_T">
    <domain id="grid_T" long_name="grid T"></domain>
  </domain_group>

  <domain_group id="grid_U">
    <domain id="grid_U" long_name="grid U"></domain>
  </domain_group>

  <domain_group id="grid_V">
    <domain id="grid_V" long_name="grid V"></domain>
  </domain_group>

  <domain_group id="grid_W">
    <domain id="grid_W" long_name="grid W"></domain>
  </domain_group>
</domain_definition>

Zoomed sub-domains are defined by adding a domain tag that contains a zoom_domain tag. The domain tag for the sub-domain must be contained within the domain_groug tag with the appropriate id attribute, and the domain tag must have a domain_ref attribute whose value matches the domain_group id value. So, since sea surface height is calculated on the T grid, we add a tide gauge station sub-domain to the grid_T domain_group tag:

<domain_group id="grid_T">
  ...
  <!-- Tide Gauge Stations -->
  <domain id="PortRenfrew" domain_ref="grid_T">
    <zoom_domain ibegin="61" jbegin="401" ni="1" nj="1"/>
  </domain>
  ...
</domain_group>

The zoom_domain tab defines the lower left corner of the sub-domain with grid point numbers in its ibegin and jbegin attributes. The extent of the sub-domain is defined by counts of grid points in the ni and nj attributes.

In addition to NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/domain_def.xml, there are examples of domain_def.xml files in the SS-run-sets/v201702/ directory tree.

Switching from XIOS-1 to XIOS-2

The main changes when switching from XIOS-1 to XIOS-2 are to the XML configuration files. These changes are described in the sections below. In addition, you will need to add “key_xios2” to your list of cpp keys in your NEMO configuration, and if you are using NEMO-cmd, you will need to link the location of your file_def.xml and XIOS-2 folder in your config.yaml.

Changes to iodef.xml

First, remove the file definition section from iodef.xml and move it to a new file named file_def.xml (see the following section for more information). The file definition will now be loaded similar to domain_def.xml and field_def.xml. To do this, add the following lines to iodef.xml:

<file_definition src="./file_def.xml"/>

The formatting of the grids within the grid definition section will also need to be changed. As an example, in XIOS-1 grid_T is defined as:

<grid id="grid_T_2D" domain_ref="grid_T"/>
<grid id="grid_T_3D" domain_ref="grid_T" axis_ref="deptht"/>

While, in XIOS-2 it becomes:

<grid id="grid_T_2D"> <domain domain_ref="grid_T"> </domain> </grid>
<grid id="grid_T_3D"> <domain domain_ref="grid_T"> </domain> <axis id="deptht"> </axis> </grid>

Another difference is that XIOS-2 calculates buffersize, compared to XIOS-1 where it is user-specified. The following lines are changed/added in XIOS-2 to specify variables to do with the buffersize:

<context id="xios">
  <variable_definition>
    <variable id="optimal_buffer_size"       type="string">performance</variable>
    <variable id="buffer_size_factor"        type="double">1.0</variable>
    <variable id="info_level"                type="int" >10</variable>
  </variable_definition>
</context>

Create file_def.xml

The content of the file_definition section of iodef.xml in XIOS-1 is moved to a seperate file: file_def.xml in XIOS-2. In addition, the file definition needs to be changed from:

<file_definition type="multiple_files" name="@expname@_@freq@_@startdate@_@enddate@" sync_freq="1d" min_digits="4">

to:

<file_definition type="one_file" name="@expname@_@freq@_@startdate@_@enddate@" sync_freq="1d" min_digits="4">

For each file group, you will want to specify a compression level:

<file_group id="1ts" output_freq="1ts" output_level="10" compression_level="4" enabled=".TRUE."> </file_group>

Changes to domain_def.xml

The only changes to domain_def.xml occur in the domain statements which need to be reformatted for XIOS-2. For example, for grid_T in XIOS-1 we had:

<domain_group id="grid_T">
        <domain id="grid_T" long_name="grid T"/>
        <domain id="test_T" domain_ref="grid_T"/>
</domain_group>

In XIOS-2 this becomes:

<domain_group id="grid_T">
        <domain id="grid_T" long_name="grid T"/>
        <domain id="test_T" domain_ref="grid_T"> </domain>
</domain_group>

XIOS-2

Getting XIOS-2

graham

salish