PyFlexPlot

PyFlexPlot is a Python-based tool to visualize FLEXPART dispersion simulation results stored in NetCDF format.

Table of Contents

• Features
• Installation
• Run pyflexplot
  • Examples
• Development (CSCS)
• External Links
• License

Key Features

PDF and PNG Files

Pyflexplot allows to visualize data on a map plot and save the output in either PDF or PNG format. To utilize this feature, simply adjust the outfile variable with the appropriate file extension.

Shape File Generation

Furthermore, Pyflexplot provides the functionality to export data into shape files (.shp) to utilize them in GIS programs such as QGIS 3. The output is a ZIP archive containing the essential components of a shapefile: .shp, .dbf, .shx, .prj, and .shp.xml. Key aspects of this feature include:

• Filtering Zero Values: The tool initially removes zero values from fields (e.g., concentration) before processing.
• Logarithmic Transformation: Field values undergo a log_10 transformation to optimize the visualization of data ranges.
• Precision Handling: The transformed field values are recorded with 15 decimal places, accommodating the precision limitations of some GIS software.
• Metadata Storage: Information, such as details about released materials, are stored within a .shp.xml file as metadata.

Scaling the field values

Another feature is to manipulate the field values by scaling with an arbitrary factor. This factor can be set in the preset with the variable multiplier.

Installation

Prerequisites: Git, Miniforge (or another Conda installation), Poetry

Get source code

git clone git@github.com:MeteoSwiss-APN/pyflexplot.git
cd pyflexplot

Create Conda environment

Create a Conda environment with only the desired Python version and Poetry and activate it

conda create --name=pyflexplot python=$(cat .python-version)
conda activate pyflexplot
conda install poetry

Configure Poetry to not create a new virtual environment. If it detects an already enabled virtual environment (eg a Conda environment) it will install dependencies into it:

poetry config --local virtualenvs.create false

Build the project (Poetry)

Alternative build with mchbuild see further below.

poetry version $(git describe --tags --always | sed 's/-\([0-9]*\)-/.dev\1+g/')
poetry install

Build the project (mchbuild)

Documentation to be added.

Run tests and quality tools

poetry run pytest

If no errors occur, the tests save plots in their respective folders in the

Run quality assurance tools

Run pylint to check code (suggestions are not being followed, Jenkins uses SonarQube):

poetry run pylint src

Run mypy to check typing:

poetry run mypy

Run ruff to check formatting:

poetry run ruff format

Run pyflexplot

The primary command for pyflexplot follows this structure:

pyflexplot [OPTIONS] CONFIG_FILE...

To see the available options, run:

pyflexplot --help

If you want to run the following examples interactively, you may want to allocate parallel resources with the help of SLURM (if available), e.g. 10 cores:

salloc -c 10

To use all allocated cpus, add the --num-procs option to the pyflexplot command (note that for a complete pyflexplot command, the definition of a preset or input and output need to be added, see below):

pyflexplot --num-procs=$SLURM_CPUS_PER_TASK

Important: Free resources when done!

exit

Examples how to run pyflexplot

Example using default input file. This example assumes you are in the pyflexplot directory.

Default input files are searched for in ./data. If you want to use the files as defined in the presets for your tests, link them into the root of the repository. At CSCS on Alps, use:

ln -s /store_new/mch/msopr/pyflexplot_testdata data

There are several default config files available under src/pyflexplot/data/presets/opr.

To produce graphics for a specific FLEXPART output, select the corresponding preset from the table below and define the preset variable accordingly:

Model	Type	Define Preset Variable
FLEXPART-ICON-CH1-CTRL	deterministic	preset=opr/icon-ch1-ctrl/all_pdf
FLEXPART-ICON-CH2-EPS	ensemble	preset=opr/icon-ch2-eps/all_pdf
FLEXPART-IFS	Global det.	preset=opr/ifs-hres/all_pdf
FLEXPART-IFS	Europe det.	preset=opr/ifs-hres-eu/all_pdf

Legacy presets for former COSMO model:

Model	Type	Define Preset Variable
FLEXPART-COSMO-1E-CTRL	deterministic	preset=opr/cosmo-1e-ctrl/all_pdf
FLEXPART-COSMO-2E-CTRL	deterministic	preset=opr/cosmo-2e-ctrl/all_pdf
FLEXPART-COSMO-1E	ensemble	preset=opr/cosmo-1e/all_pdf
FLEXPART-COSMO-2E	ensemble	preset=opr/cosmo-2e/all_pdf

You may use the * wildcard to operate pyflexplot with several presets at once. For example, to run pyflexplot with all presets that produce the graphics in PDF format for a specific NWP model, define the preset variable as one of:

preset='opr/icon*/all_pdf'
preset='opr/ifs*/all_pdf'

Note that the preset in this form requires the respective default input files to be accessible through the ./data directory.

Alternatively, to create all graphics formats from one model, use one of:

preset='opr/icon-ch1-ctrl/all_*'
preset='opr/icon-ch2-eps/all_*'
preset='opr/ifs-hres-eu/all_*'
preset='opr/ifs-hres/all_*'

Define an output directory and create it, if it does not exist, e.g.

nwp=$(echo ${preset} | cut -d/ -f2 | sed 's/*//g') # Extract NWP model name
dest=plot_$nwp
mkdir $dest

After selecting a preset, you may run pyflexplot interactively for the default test data:

pyflexplot --preset "$preset" --merge-pdfs --dest=$dest

To use any FLEXPART output file in NetCDF format as input for pyflexplot, it must be specified with the help of the --setup option. Define the input file

infile=/path/to/your/flexpart_output.nc  # replace the r.h.s. this with your own file path

an run pyflexplot as follows:

pyflexplot --preset "$preset" --merge-pdfs --dest=$dest --setup infile $infile

On the production server at the CSCS, it is highly recommended to run this command as batch job using the batchPP utility:

batchPP -t 2 -T 10 -n pfp_$nwp -- \
  $CONDA_PREFIX/bin/pyflexplot --preset "$preset" \
  --merge-pdfs --dest=$dest  --setup infile $infile \
  --num-procs=\$SLURM_CPUS_PER_TASK

To use a FLEXPART ensemble as input, the placeholder {ens_member:03} may be used within the path of <netcdf-file>. Instead of 03 (for %03d), another C-style field width for the ensemble member field can be used.

Example using operational Flexpart ensemble output based on ICON-CH2-EPS:

# Define preset, model name, base time
preset=opr/icon-ch2-eps/all_pdf  # Preset for ICON-CH2-EPS
nwp=$(echo ${preset} | cut -d/ -f2 | sed 's/*//g')  # Extract NWP model name
basetime=$(date --utc --date="today 00" +%Y%m%d%H)  # Recent base time

# Get name of first input file, insert placeholders for ensemble members
infile000=$(echo /store_new/mch/msopr/osm/ICON-CH2-EPS/FCST${basetime:2:2}/${basetime:2:8}_???/flexpart_c/000/grid_conc_*_BEZ.nc)
infile=${infile000/\/000\//\/\{ens_member:03\}\/}   # Input file definition

# Define and create outut directory
dest=plot_${basetime:2:8}    # Output directory with base time of NWP model
mkdir $dest                  # Create output directory

# Submit job with the help of the batchPP utility
batchPP -t 1 -T 10 -n pfp-$nwp -- \
  $CONDA_PREFIX/bin/pyflexplot --preset $preset \
    --merge-pdfs --setup infile $infile --setup base_time $basetime --dest=$dest \
    --num-procs=\$SLURM_CPUS_PER_TASK

The following examples use FLEXPART output generated with the test-fp script in the flexpart-test repository of MeteoSwiss. Define FP_JOBS as path to the FLEXPART output files that are to be used as input for pyflexplot, e.g.

FP_JOBS=$SCRATCH/flexpart/job

Write output to a location where you have write access, e.g.

FP_OUT=$SCRATCH/flexpart/job

After additionally defining the preset as above and nwp as the job name (directory name below FP_JOBS), create the output directory with

infile=$(echo $FP_JOBS/$nwp/output/*.nc)
basetime=$(cat $FP_JOBS/$nwp/output/plot_info)
dest=$FP_OUT/$nwp/plots
mkdir -p $dest

and submit the job with the batchPP command as above.

For ensembles, the infile needs to be a pattern rather than a single file:

infile000=$(echo $FP_JOBS/$nwp/output/000/*.nc)
infile=${infile000/\/000\//\/\{ens_member:03\}\/}

After job completion, list and visualize results e.g. with evince:

ls $dest/*pdf
evince $dest/*pdf

Running Pyflexplot with S3 input (and output)

In order to download input NETCDF data from S3, and S3 URI can be specified as the setup parameter infile as below (example):

pyflexplot --preset "$preset" --merge-pdfs --dest=$dest --setup infile s3://<s3-bucket-name>/flexpart_cosmo-2e_2021030503_{ens_member:03}_MUE.nc

In order to output the resulting plots to an S3 bucket, specify the S3 bucket name as the --dest. The plots will still be created locally at the dest dir path defined in the config/settings.yaml

pyflexplot --preset "$preset" --merge-pdfs --dest=s3://<s3-bucket-name>

Development

Updating the Test References

Pyflexplot includes a set of functionality tests that compare generated output against predefined reference data. These reference files, which contain summary dicts, begin with ref_ and have the nomal Python file ending .py, and are stored in the directory tests/slow/pyflexplot/test_plots. To update these reference files, uncomment the following line near the end of the file shared.py in the same directory:

_TestBase = _TestCreateReference

Then re-run the (slow) tests to generate the new reference files. After generating the new reference files, comment out the above line again or simply revert the file with git.

External Links

• pyshp - Python module to generate Shapefiles

License

This project is licensed under the terms of the MIT License. The full license text can be found in the LICENSE file. In essence, you are free to use, modify, and distribute the software, provided the associated copyright notice and disclaimers are included.