Probtest

Probtest is a suite of python scripts to test various aspects of weather and climate models. Even though it is mainly developed for the ICON model, it does not contain ICON-specific definitions in the source code but allows to model specifics in a configuration script.

Table of contents

1. Commands
2. Quick start guide
3. Developing probtest
4. Probtest container

Commands

The suite consists out of the following scripts:

probtest

This script is the entry point to the suite that manages all the other scripts and sets up command line arguments with click.

init

To work with probtest, a configuration file is needed. This file contains all the options that are used for the various commands of probtest and can be generated with this command.

perturb

Creates a set of netcdf files that can be used as input for a perturbed model ensemble.

run-ensemble

Reads and modifies model namelists to run a perturbed model ensemble.

stats

Generates a csv file containing the min, max and mean values for each of the selected fields on each model level and for each time step.

tolerance

Computes the maximum spread in each of the selected variables for each time step within a perturbed model ensemble. This step is applied to both files generated with stats and fof files from each perturbed ensemble.

check

Compares two files generated with stats and/or two fof files under the tolerance ranges generated by tolerance.

select-members

Uses all given stats files generated for a model ensemble with stats. From those stats files, randomly selects a specified number of members to generate the tolerances with tolerance. Repeats this process by iteratively increasing the number of selected members and the tolerance factor until finding a selection for which all other members pass the tolerance check.

check-plot

Creates a plot visualizing the results of the comparison conducted with check.

cdo-table

Often people like to compare model output using cdo diff. This tool compares each point of each variable and time step and reports the largest relative error for each. This is different from stats which computes differences on statistics (min, max, mean) of each model level and time step. However, the perturbed ensemble can be used to estimate expected errors with cdo diff by applying the same algorithm and storing the output in a database.

cdo-table-reader

The output of cdo-table is written in a Pandas dataframe. To ease reading, cdo-table-reader allows to filter the database by variable name, output file and time step.

performance

Note: This script is ICON-specific because it parses the ICON logfile with hard-coded regex. Read the timing output from the model. These timings are stored in a tree format in the case of nested timings. It can either create a new database or append to an existing one. It produces three databases:

• tree: the relationship between the individual timers
• meta: some metadata storing information about the model run
• data: the actual timer data

performance-check

Compares two performance databases generated with performance and checks if the current one is too slow compared to the reference.

performance-meta-data

Reads some data from a series of performance databases generated with performance and produces a graph showing the performance over time.

performance-plot

Visualizes the performance database generated with performance.

fof-compare

Compares two fof files variable by variable and line by line and indicates whether the files are consistent. If they are not consistent, indicates the percentage of inconsistent data. There are also options to print the lines with errors or save them in a separate file.

Quick start guide

Even though probtest is used exclusively with ICON at the moment, it does not contain any information about the model or its directory structure. This makes it very flexible and applicable to any circumstance (e.g. usable by Buildbot, Jenkins and human users alike). However, it also requires a lot of information about the model and the data to be processed upon invocation. Since a typical probtest usage involves multiple commands (e.g. run-ensemble -> stats -> tolerance -> check) this leads to a lot of redundancy in the invocation. Therefore, probtest can read commonly used input variables (e.g. the model output directory, the experiment name, the name of the submit script, ...) from a configuration file in json format. To further ease the process, these configuration files can be created from templates using the init command. A template for ICON is contained in this repository in the templates subdirectory.

Setup probtest

All requirements for using probtest can be easily installed with pip or poetry.

Setup with pip

To set up the virtual environment and install the required dependencies:

# Create a Python 3.10 virtual environment
python3.10 -m venv venv

# Activate the virtual environment
source venv/bin/activate

# Upgrade pip to the latest version
pip install --upgrade pip

# Install the project dependencies
pip install -r requirements.txt

Note: This setup uses the requirements.txt file generated by Poetry, which ensures all dependencies are compatible and locked to tested versions.

Setup with Poetry

This setup is required if you want to change or add dependencies. It assumes you have Poetry installed and are using Python 3.10.

# Use Python 3.10 for the Poetry virtual environment
poetry env use python3.10

# Install all project dependencies
poetry install

To verify that everything is set up correctly and tests are passing:

poetry run pytest -v -s --cov --cov-report=term tests/

Tip: To avoid prefixing every command with poetry run, you can activate the Poetry virtual environment directly:

source $(poetry env info --path)/bin/activate

The init command

This command sets up the configuration file. For more help on the command line arguments for init, see

poetry run probtest init --help

The --template-name argument can be used to specify the template from which the configuration file is created. One of the templates provided by probtest is templates/ICON.jinja which is used as the default in case no other template name is provided. The init command replaces all placeholder values in the template with the values given as command line arguments. All other probtest commands can then read from the configuration file. The name of the configuration file to use is read from the PROBTEST_CONFIG environment variable. If this is not set explicitly, probtest will look for a file called probtest.json in the current directory.

Setting up the configuration file with init may not be fitted perfectly to where you want your probtest files to be. In that case, you can manually edit the file after creation. Alternatively, you can add arguments for your probtest commands on the command line which take precedence over the configuration file defaults. For more help on the options on a specific command, see

poetry run probtest {command} --help

Example: Check the output of an ICON experiment with a test build compared to a reference build

Objective: Run an exp_name ICON experiment with a test build and check if the output of the test is within a perturbed ensemble of the reference build. This is in particular used to validate a GPU build against a CPU reference.

You need to have set up a proper environment, for example as described in the section Setup probtest.

Initialize probtest

Once set up, probtest can generate the config file according to your needs. Initialized a probtest.json file in your reference build directory, exp_name here should refer to your experiment script:

cd icon-base-dir/reference-build
python ../externals/probtest/probtest.py init --codebase-install $PWD --experiment-name exp_name --reference $PWD --file-id NetCDF "*atm_3d_ml*.nc" --file-id NetCDF "*atm_3d_il*.nc" --file-id NetCDF "*atm_3d_hl*.nc" --file-id NetCDF "*atm_3d_pl*.nc" --file-id latlon "*atm_2d_ll*.nc" --file-id meteogram "Meteogram*.nc"

You might need to update the used account in the json file. The perturbation amplitude may also need to be changed in the json file (buildbot uses 1e-07 for mixed precision and 1e-14 for double precision). Note that to change this you should modify the second entry of rhs_new in probtest.json, which should be set to 1e-14 by default.

Note that, it is important that the file-id are uniquely describing the data with the same structure. Otherwise you might get an error like

packages/pandas/core/indexes/base.py", line 4171, in _validate_can_reindex
    raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels

For examples of proper file-ids have a look in the ICON repo at run/tolerance/set_probtest_file_id.

Now you should have created a probtest.json file in the reference build directory. This file contains all information needed by probtest to process the ICON experiment.

Generate references and tolerances for the reference build

With everything set up properly, the chain of commands can be invoked to run the reference binary (run-ensemble), generate the statistics files used for probtest comparisons (stats) and generate tolerances from these files (tolerance). To run the perturbed experiments and wait for the submitted jobs to finish:

python ../externals/probtest/probtest.py run-ensemble

FYI: if the experiment does not generate all of the files listed in the file-ids above, you you receive a message that certain file-id patterns do not match any file. Those files can remove them from file-ids.

Extract the statistics of your perturbed runs:

python ../externals/probtest/probtest.py stats --ensemble

Note that the --ensemble option which is set to take precedence over the default False from the configuration and make probtest process the model output from each ensemble generated by run-ensemble.

Finally create the tolerance.csv file for the exp_name by analysing those statistics:

python ../externals/probtest/probtest.py tolerance

Alternatively, if FOF files are also to be considered(fof_type must be replaced with the name of the fof file(s) to be analysed):

python ../externals/probtest/probtest.py tolerance --fof-types {fof_type}

These commands will generate a number of files:

• stats_ref.csv: contains the post-processed output from the unperturbed reference run
• stats_{member_id}.csv: contain the post-processed output from the perturbed reference runs (only needed temporarily to generate the tolerance file)
• exp_name_tolerance.csv: contains tolerance ranges computed from the stats-files

Eventually:

• exp_name_tolerance_fof{fof_type}.csv: contains tolerance ranges computed from the fof-files

These can then be used to compare against the output of a test binary (usually a GPU binary). For that, manually run the exp_name.run experiment with the test binary to produce the test output.

Run and check with test build

To then check if your data from the test binary are validating against reference build, first run the experiments with the test build. Run your test simulation without probtest:

cd icon-base-dir/test-build
sbatch run/exp_name.run

Then create the test statistics with:

python ../externals/probtest/probtest.py stats --no-ensemble --model-output-dir icon-base-dir/test-build/experiments/exp_name

Note how --model-output-dir is set to take precedence over the default which points to the reference binary output to now point to the test binary output. This command will generate the following file:

• stats_exp_name.csv: contains the post-processed output of the test binary model output.

Now all files needed to perform a probtest check are available; the reference file stats_ref.csv, the test file stats_exp_name.csv as well as the tolerance range exp_name_tolerance.csv. Providing these files to check will perform the check:

python ../externals/probtest/probtest.py check --reference-files stats_ref.csv --current-files stats_exp_name.csv --factor 5

Since check accepts also lists of filenames, applying it to the FOF files as well simply requires adding:

python ../externals/probtest/probtest.py check --reference-files stats_ref.csv, fof{fof_type}_ref.nc --current-files stats_exp_name.csv, ..experiments/exp/fof{fof_type}.nc --factor 5

This check can be also visualized by:

python ../externals/probtest/probtest.py check-plot --reference-files stats_ref.csv --current-files stats_exp_name.csv --tolerance-files exp_name_tolerance.csv --factor 5 --savedir ./plot_dir

Note that the reference --reference-files and test stats files --current-files need to be set by command line arguments. This is because the default stored in the ICON.jinja template is pointing to two files from the ensemble as a sanity check.

Developing probtest

Testing with pytest

Our tests are executed using pytest, ensuring a consistent and efficient testing process. Each test dynamically generates its necessary test data, allowing for flexible and isolated testing scenarios.

Simply run

poetry run pytest -s -v tests/*

in order to run all tests.

To run only a subset of test run

poetry run pytest -s -v path/to/your/test.py

Reference data, crucial for validating the outcomes of our tests and detecting any deviations in probtests results, is maintained in the tests/data directory. This approach guarantees that our tests are both comprehensive and reliable, safeguarding the integrity of our codebase.

Update references for pytests

Some code changes might require to update the references in tests/data.

1. Check where new references are stored for example:

   pytest -s -v tests
   ....
   New reference data will be stored in /tmp/tmpg92cbw4z
   

2. Move new references to current reference location:

   cp -v /tmp/tmpg92cbw4z/* /path/to/repo/tests/data
   

3. Commit new references.

Formatting probtest source code

The probtest source code is formatted using multiple formatters. Please install the pre-commit hooks (after installing all Python requirements including the pre-commit package):

poetry run pre-commit install

This hook will be executed automatically whenever you commit. It will check your files and format them according to its rules. If files have to be formatted, committing will fail. Just stage and commit again to finalize the commit. You can also run the following command, to trigger the pre-commit action without actually committing:

poetry run pre-commit run --all-files

Updating and Adding Dependencies

Use Poetry to safely add or update dependencies while keeping the lock file and requirements files in sync.

Add a new dependency

Add a production dependency

poetry add requests

Add a development dependency

poetry add --group dev ruff

Add a testing dependency

poetry add --group test pytest-mock

Update dependencies

Update dependencies within the allowed ranges of pyproject.toml and writes a new poetry.lock.

Update all packages

poetry update

Update a specific package

poetry update numpy

Regenerate requirements files

After adding or updating dependencies, run the lock/export script:

./scripts/poetry_lock.sh

This will:

• Update poetry.lock
• Export requirements.txt (production)
• Export requirements_dev.txt (dev/test)

Note: The export command in poetry_lock.sh is provided by the optional plugin poetry-plugin-export. If you see The requested command export does not exist, install the plugin with:

poetry self add poetry-plugin-export

Release a new probtest version

Release versions use two digits only: the first digit represents breaking major changes, and the second digit represents minor, backward-compatible updates.

To prepare a new release, use the provided script:

./scripts/release.sh <new_version>

The script automates creating a release branch, bumping the version in pyproject.toml, and updating the poetry.lock file. Open a PR with the release branch. Once the PR is merged into main, the tag-release.yml GitHub Actions workflow automatically creates the vX.Y Git tag and a corresponding GitHub Release with auto-generated release notes. Pushing that tag also triggers deploy-container-for-release.yml, which builds and publishes the Docker image.

Note:

• The release script automatically updates the version in pyproject.toml and regenerates dependency files. Ensure that Poetry is correctly set up and that the ./scripts/poetry_lock.sh script runs successfully (see Regenerate requirements files).
• Do not include a leading v in the version number — use 1.1, not v1.1.

Probtest Container

Prebuilt Docker images for the latest Probtest releases are available on Docker Hub: c2sm/probtest

Build the Probtest Container via GitHub Actions

You can also build the Probtest Docker image yourself using the GitHub Actions workflow defined in build-container.yml.

Steps:

1. Navigate to the Actions tab of this repository.
2. Select the Build Docker Container workflow from the list.
3. Click the Run workflow button (top-right) and confirm.
4. Wait for the workflow to complete (approximately 50 minutes).
5. Once completed, open the workflow run and scroll down to the Artifacts section.
6. Download the probtest_image.tar artifact.

Load the Image into Docker

To load the downloaded image into your local Docker installation, run:

docker load -i probtest_image.tar