XIOS-2
XIOS is the IO management library that is used by NEMO to produce netCDF results files. NEMO-3.6 uses XIOS-2.
XIOS operates as a server process that multiple NEMO calculation processes communicate with to write results to netCDF files. XIOS buffers the output from NEMO in memory while the much slower process of writing to disk happens. That means that the NEMO processes can do calculations almost continuously without having to periodically pause to write results to disk.
In many cases, especially on the Alliance HPC clusters with large memory per node, it is possible to configure NEMO runs with tens to a hundred or more processors to use a single XIOS process. The XIOS process must be started in the same mpirun command as NEMO. Fortunately, the NEMO Command Processor and SalishSeaCast NEMO Command Processor tools take care of doing that for you.
Getting XIOS-2
The MOAD group maintains our own Mercurial repositories on GitHub of the XIOS-2 code and build configuration files for building XIOS-2 on the compute platforms that we use. Our XIOS-2 code repo is accessible only by members of the MOAD group so as to respect the sign-up requirement of the upstream XIOS repository. The architecture file repo is public so that other researchers can make use of the build system options that we have figured out for various system.
To get the XIOS-2 code and build configuration files repos you need to clone them into your
research project directory,
typically one of MEOPAR/
.
Here are the commands to do that on the 2 platforms that we use most.
graham
cd $PROJECT/$USER/MEOPAR/
git clone git@github.com:SalishSeaCast/XIOS-2.git
git clone git@github.com:SalishSeaCast/XIOS-ARCH.git
salish
cd /data/$USER/MEOPAR/
git clone git@github.com:SalishSeaCast/XIOS-2.git
git clone git@github.com:SalishSeaCast/XIOS-ARCH.git
Building XIOS-2
First symlink the XIOS-2 build configuration files for the machine that you are working on
from the XIOS-ARCH
repo clone into the XIOS-2/arch/
directory,
then compile and link XIOS-2.
MEOPAR
on graham
cd $PROJECT/$USER/MEOPAR/XIOS-2/arch
ln -sf $PROJECT/$USER/MEOPAR/XIOS-ARCH/ALLIANCE/arch-X64_GRAHAM.env
ln -sf $PROJECT/$USER/MEOPAR/XIOS-ARCH/ALLIANCE/arch-X64_GRAHAM.fcm
ln -sf $PROJECT/$USER/MEOPAR/XIOS-ARCH/ALLIANCE/arch-X64_GRAHAM.path
cd $PROJECT/$USER/MEOPAR/XIOS-2
./make_xios --full --arch X64_GRAHAM --job 8
It typically takes about 5 minutes to build XIOS-2 on graham
.
To build NEMO you will need an environment variable named XIOS_HOME
whose value is the absolute path to you XIOS-2
directory.
For the MEOPAR
project it is:
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2
You can add the line:
export XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2
to your $HOME/.bash_profile
file if you want the XIOS_HOME
environment variable
to be set automatically whenever you ssh in.
MEOPAR
on salish
cd /data/$USER/MEOPAR/XIOS-2/arch
ln -sf /data/$USER/MEOPAR/XIOS-ARCH/UBC-EOAS/arch-GCC_SALISH.fcm
ln -sf /data/$USER/MEOPAR/XIOS-ARCH/UBC-EOAS/arch-GCC_SALISH.path
cd /data/$USER/MEOPAR/XIOS-2
./make_xios --full --arch GCC_SALISH --netcdf_lib netcdf4_seq --job 8
To build NEMO you will need an environment variable named XIOS_HOME
whose value is the absolute path to you XIOS-2
directory.
For the MEOPAR
project it is:
XIOS_HOME=/data/$USER/MEOPAR/XIOS-2
You can add the line:
export XIOS_HOME=/data/$USER/MEOPAR/XIOS-2
to your $HOME/.bash_profile
file if you want the XIOS_HOME
environment variable
to be set automatically whenever you ssh in.
XIOS-2 Configuration Files
To use XIOS-2 with NEMO, four configuration files written in XML are required:
field_def.xml
defines the variables that can be output and the grids on which they are defined. Field definition elements may (and generally should) also contain metadata attributes such as long name, standard name, and units. Please see the field_def.xml section below for more information about the structure and contents offield_def.xml
files.domain_def.xml
defines “zoomed” sub-domains of the model domain and the grids on which they are defined. The “zooms” are defined on the i-j (x-y) directions, regardless of the depth of the sub-domain. Please see the domain_def.xml section below for more information about the structure and contents ofdomain_def.xml
files.iodef.xml
defines the vertical extent of output grids in theaxis
elements, and the output grids. It also contains a separatecontext
element forxios
in which a few settings that control XIOS-2 are declared.file_def.xml
defines the files into which field variables are output and the frequency of output of those files. Variable names can be transformed from the internal NEMO names to more user friendly names in thefield
elements in this file. This is also where on-the-fly deflation of output files is enabled via thecompression_level="4"
attribute offile_group
elements.
Warning
XML syntax is very exacting, so care is required when you edit XML files to ensure that tags are correctly closed, attribute values are correctly quoted, etc.
Annoyingly, NEMO will fail with no diagnostic messages if your XML files contain errors. If you suspect that you have made an error in editing an XML file, one way of checking is to use an online validator like https://www.xmlvalidation.com/.
Customizing XML Files
The NEMO-3.6-code repositories contains sample XIOS-2 configuration files in the
NEMOGCM/CONFIG/SHARED/
and some of the NEMOGCM/CONFIG/*/EXP00/
directories.
Warning
Please do not modify and commit those files. Doing so will cause conflicts when changes to NEMO are pulled in from the upstream repository, and your changes will be overwritten. Instead, put copies of the XML files that you want to change under version control in your runs configuration repo (for example, the SS-run-sets repo for people working on MEOPAR).
Command Processors and XML Files
The NEMO Command Processor and SalishSeaCast NEMO Command Processor tools provide a way, via YAML run description files, to map XML files with arbitrary file names and directory paths on to the file names that NEMO requires in the directory from which NEMO is executed.
The output
section of the YAML description file is where the XML file mappings and
other XIOS-2 settings are specified.
Please see the salishsea YAML file output section docs if you are working on the
Salish Sea configurations of NEMO,
or the nemo YAML file output section docs if you use another NEMO configuration.
There are also examples of complete YAML run description files in those docs.
The simplest possible YAML file output
section is:
output:
XIOS servers: 1
separate XIOS server: True
filedefs: file_def.xml
domaindefs: domain_def.xml
fielddefs: field_def.xml
iodefs: iodef.xml
In this case, the XML files are all in the same directory as the YAML file. If you use relative paths, they have to be relative to the directory where the YAML file is.
A more complicated example is:
output:
XIOS servers: 1
separate XIOS server: True
filedefs: $PROJECT/$USER/SS-run-sets/v201905/tuning/tuning_file_def.xml
domaindefs: ../domain_def.xml
fielddefs: $PROJECT/SS-run-sets/v201905/field_def.xml
iodefs: iodef.xml
Note the use of:
A relative path for
domaindefs
Absolute paths containing the environment variables like
$USER
and$PROJECT
forfiledefs
andfielddefs
. Other environment variables like$HOME
and$SCRATCH
can also be used in XML file paths.The more descriptive file name
tuning_file_def.xml
forfiledefs
field_def.xml
field_def.xml
defines the variables that can be output and the grids on which they are defined.
Field definition elements may
(and generally should)
also contain metadata attributes such as long name,
standard name,
and units.
This section provides some information about the structure and contents of a field_def.xml
file.
This is not an exhaustive reference guide for all of the possible attribute values;
for that,
please see chapter 3 of the XIOS User Guide.
NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml
is the reference version of the file
that is provided with the NEMO code.
In many cases,
you can use that reference file by putting its path as the value of the filedefs
element
in the output
section of your run description YAML file
(see Command Processors and XML Files).
Reasons why you might want to create your own customized version
(see Customizing XML Files)
of field_def.xml
include:
Adding new variable(s) to NEMO that you want to include in your output files
Adjusting/correcting the values of variable field attributes such as
long_name
,standard_name
,unit
, etc. Those attributes provide variable-level metadata items in output files.
Here is an example fragment of a field_def.xml
file:
<field_definition level="1" prec="4" operation="average" enabled=".TRUE." default_value="1.e20">
<field_group id="grid_T" grid_ref="grid_T_2D">
<field id="sst" long_name="sea surface temperature" standard_name="sea_surface_temperature" unit="degC"/>
<field id="toce" long_name="temperature" standard_name="sea_water_conservative_temperature" unit="degC" grid_ref="grid_T_3D"/>
<field id="sss" long_name="sea surface salinity" standard_name="sea_surface_reference_salinity" unit="g kg-1"/>
<field id="soce" long_name="salinity" standard_name="sea_water_reference_salinity" unit="g kg-1" grid_ref="grid_T_3D"/>
<field id="sst2" long_name="square of sea surface temperature" standard_name="square_of_sea_surface_temperature" unit="degC2">
sst * sst
</field >
<field id="sstmax" long_name="max of sea surface temperature" field_ref="sst" operation="maximum"/>
...
</field_group>
...
</field_definition>
field_def.xml
files contain 3 types of tags:
field_definition
field_group
field
field
tags must be contained within a field_group
tag,
which must be contained within a field_definition
tag.
Attributes included in a tag apply to all contained tags unless they are explicitly overridden
in a contained tag.
So the operation="average"
attribute in:
<field_definition level="1" prec="4" operation="average" enabled=".TRUE." default_value="1.e20">
means that all field values will be averaged over the output time interval unless a
different operation
is specified in the field
tag,
for example:
<field id="sstmax" long_name="max of sea surface temperature" field_ref="sst" operation="maximum"/>
in which case the maximum value over the output time interval of the sst
field
(specified by the field_ref
attribute)
will be calculated by XIOS.
The operation
attribute enables the burden of calculating various temporal quantities on
field variables to be shifted from NEMO to XIOS.
Please see section 3.2 of the XIOS User Guide for details.
Another way of doing field operations in XIOS is to specify them in the field
tag,
for example:
<field id="sst2" long_name="square of sea surface temperature" standard_name="square_of_sea_surface_temperature" unit="degC2">
sst * sst
</field >
Here again,
the burden of declaration,
memory allocation,
and calculation of the sst2
variable is shifted from NEMO to XIOS.
This form of field calculation can be useful for calculating fluxes.
field_group
tags specify the default grid on which the contained field
tags are
defined via the grid_ref
attribute.
That attribute can,
of course,
be overridden in the contained field
tags.
All field
tags should have the following attributes:
long_name
standard_name
unit
Those attributes are passed through to the netCDF output files as field variable metadata.
Values for the standard_name
attribute should be chosen from the CF conventions standard names table.
Standard names are written in “snake case”
(words separated by _
characters).
That table also provides canonical units that should be used at the value of the unit
attribute.
The value of the long_name
attribute can be more free-from and descriptive.
It is typically used for plot axis labels,
table headings,
etc.
In addition to NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml
,
there are examples of field_def.xml
files in the SS-run-sets/v201702/ directory tree.
domain_def.xml
domain_def.xml
defines “zoomed” sub-domains of the model domain and the grids on which
they are defined.
The “zooms” are defined on the i-j (x-y) directions,
regardless of the depth of the sub-domain.
This section provides some information about the structure and contents of a domain_def.xml
file.
This is not an exhaustive reference guide for all of the possible attribute values;
for that,
please see chapter 5 of the XIOS User Guide.
NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/domain_def.xml
is the reference version of the file
that is provided with the NEMO code.
In many cases,
you can use that reference file by putting its path as the value of the domaindefs
element
in the output
section of your run description YAML file
(see Command Processors and XML Files).
The main reason why you might want to create your own customized version
(see Customizing XML Files)
of domain_def.xml
is to define your own “zoomed” sub-domain of the model domain.
Assuming that your “zoomed” sub-domain is significantly smaller than the full model domain,
the output files you produce form it will be significantly smaller than full domain files.
Examples of uses of “zoomed” sub-domains in the SalishSeaCast NEMO configuration are:
single point sub-domains for sea surface height output at tide gauge station locations
single point sub-domains for model output at the ONC VENUS instrument platform locations
sub-domains encompassing the southern Strait of Georgia for velocity fields outputs to compare against drifter tracks
sub-domains encompassing the Baynes Sound AGRIF sub-grid for tracer outputs for visualization of the transition between the full domain grid and the AGRIF sub-grid
sub-domains that provide boundary condition fields for the Vancouver Harbour and Lower Fraser River FVCOM model
Here is an example fragment of a domain_def.xml
file:
<domain_definition>
<domain_group id="grid_T">
<domain id="grid_T" long_name="grid T"></domain>
<!-- Tide Gauge Stations -->
<domain id="PortRenfrew" domain_ref="grid_T">
<zoom_domain ibegin="61" jbegin="401" ni="1" nj="1"/>
</domain>
...
</domain_group>
...
</domain_definition>
domain_def.xml
files contain 4 types of tags:
domain_definition
domain_group
domain
zoom_domain
domain
tags must be contained within a domain_group
tag,
which must be contained within a domain_definition
tag.
zoom_domain
tags must be contained within a domain
tag.
A minimal, complete domain_def.xml
file would contain domain definitions for the
full domain T, U, V, and W grids:
<domain_definition>
<domain_group id="grid_T">
<domain id="grid_T" long_name="grid T"></domain>
</domain_group>
<domain_group id="grid_U">
<domain id="grid_U" long_name="grid U"></domain>
</domain_group>
<domain_group id="grid_V">
<domain id="grid_V" long_name="grid V"></domain>
</domain_group>
<domain_group id="grid_W">
<domain id="grid_W" long_name="grid W"></domain>
</domain_group>
</domain_definition>
Zoomed sub-domains are defined by adding a domain
tag that contains a zoom_domain
tag.
The domain
tag for the sub-domain must be contained within the domain_groug
tag with the
appropriate id
attribute,
and the domain
tag must have a domain_ref
attribute whose value matches the domain_group
id value.
So,
since sea surface height is calculated on the T grid,
we add a tide gauge station sub-domain to the grid_T
domain_group
tag:
<domain_group id="grid_T">
...
<!-- Tide Gauge Stations -->
<domain id="PortRenfrew" domain_ref="grid_T">
<zoom_domain ibegin="61" jbegin="401" ni="1" nj="1"/>
</domain>
...
</domain_group>
The zoom_domain
tab defines the lower left corner of the sub-domain with grid point numbers
in its ibegin
and jbegin
attributes.
The extent of the sub-domain is defined by counts of grid points in the ni
and nj
attributes.
In addition to NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/domain_def.xml
,
there are examples of domain_def.xml
files in the SS-run-sets/v201702/ directory tree.
Switching from XIOS-1 to XIOS-2
The main changes when switching from XIOS-1 to XIOS-2 are to the XML configuration files.
These changes are described in the sections below.
In addition, you will need to add “key_xios2” to your list of cpp keys in your NEMO configuration,
and if you are using NEMO-cmd, you will need to link the location of your file_def.xml
and XIOS-2 folder in your config.yaml
.
Changes to iodef.xml
First, remove the file definition section from iodef.xml
and move it to a new file named
file_def.xml
(see the following section for more information).
The file definition will now be loaded similar to domain_def.xml
and field_def.xml
.
To do this, add the following lines to iodef.xml
:
<file_definition src="./file_def.xml"/>
The formatting of the grids within the grid definition section will also need to be changed. As an example, in XIOS-1 grid_T is defined as:
<grid id="grid_T_2D" domain_ref="grid_T"/>
<grid id="grid_T_3D" domain_ref="grid_T" axis_ref="deptht"/>
While, in XIOS-2 it becomes:
<grid id="grid_T_2D"> <domain domain_ref="grid_T"> </domain> </grid>
<grid id="grid_T_3D"> <domain domain_ref="grid_T"> </domain> <axis id="deptht"> </axis> </grid>
Another difference is that XIOS-2 calculates buffersize, compared to XIOS-1 where it is user-specified. The following lines are changed/added in XIOS-2 to specify variables to do with the buffersize:
<context id="xios">
<variable_definition>
<variable id="optimal_buffer_size" type="string">performance</variable>
<variable id="buffer_size_factor" type="double">1.0</variable>
<variable id="info_level" type="int" >10</variable>
</variable_definition>
</context>
Create file_def.xml
The content of the file_definition section of iodef.xml
in XIOS-1 is moved to a seperate file:
file_def.xml
in XIOS-2.
In addition,
the file definition needs to be changed from:
<file_definition type="multiple_files" name="@expname@_@freq@_@startdate@_@enddate@" sync_freq="1d" min_digits="4">
to:
<file_definition type="one_file" name="@expname@_@freq@_@startdate@_@enddate@" sync_freq="1d" min_digits="4">
For each file group, you will want to specify a compression level:
<file_group id="1ts" output_freq="1ts" output_level="10" compression_level="4" enabled=".TRUE."> </file_group>
Changes to domain_def.xml
The only changes to domain_def.xml
occur in the domain statements which need to be
reformatted for XIOS-2.
For example, for grid_T in XIOS-1 we had:
<domain_group id="grid_T">
<domain id="grid_T" long_name="grid T"/>
<domain id="test_T" domain_ref="grid_T"/>
</domain_group>
In XIOS-2 this becomes:
<domain_group id="grid_T">
<domain id="grid_T" long_name="grid T"/>
<domain id="test_T" domain_ref="grid_T"> </domain>
</domain_group>