Calling make_figure() Functions in the make_plots Worker

This section discusses:

  • how to test a website figure module in a test notebook that replicates the nowcast system automation context

  • how to add calls to a figure module’s make_figure() function to the nowcast.workers.make_plots nowcast system worker

  • how to run the make_plots worker in debugging mode to test that it renders a figure correctly

We’ll use the nowcast.figures.research.tracer_thalweg_and_surface figure module as an example to create a nitrate thalweg and surface figure.

You should run your test notebooks and make_plots worker tests in a Nowcast Figures Development Environment. You can activate it with:

$ source activate nowcast-fig-dev

Figure Module Test Notebook

Once you’ve created a website figure module, you should create a notebook that tests it in the nowcast context. The TestTracerThalwegAndSurfaceModule notebook in notebooks/figures/research/ is an example for the nowcast.figures.research.tracer_thalweg_and_surface module.

Registering make_figure() Calls in the make_plots Worker

Website figure modules like nowcast.figures.research.tracer_thalweg_and_surface produce matplotlib.figure.Figure objects. The nowcast.workers.make_plots worker is the element of the nowcast system automation that renders the Figure objects to image files. After a NEMO run has completed the make_plots worker is launched to call the make_figure() functions that are registered for the run type. The figure objects returned by the make_figure() functions are rendered to files in the /results/nowcast-sys/figures/ tree. Files in that tree are served to the web by the nowcast system figure server.

The make_plots worker organizes figures by NEMO run type and plot type. The python -m nowcast.workers.make_plots -h command will show you a list of the run types and plot types. At present the run types are:

  • nowcast

  • nowcast-green

  • forecast

  • forecast2

and the plot types are:

  • research

  • comparison

  • publish

make_plots also accepts a --run-date to specify the date of the daily nowcast system NEMO run to create the figure for. Without --run-date today’s date is used.

The make_plots() function uses paths defined in the nowcast system configuration file (SalishSeaNowcast/config/nowcast.yaml) to set up a collection of commonly used paths and datasets such as:

  • the results storage directory tree

  • the weather forcing directory tree

  • bathymetry and mesh mask files

  • the BC and Washington coastline polygons file

Then make_plots() calls a _prep_*_fig_functions() function for the requested run type and plot type. Those functions open the datasets that will be used to create the figure objects, and return a data structure of information about the make_figure() calls that we want make_plots to execute to render figures. Adding an item to a _prep_*_fig_functions() function data structure is referred to as registering the make_figure() call.

To use the nowcast.figures.research.tracer_thalweg_and_surface to produce a figure that shows nitrate concentration on thalweg and surface slices we will register a call of its make_figure() function in the _prep_nowcast_green_research_fig_functions() function:

def _prep_nowcast_green_research_fig_functions(bathy, mesh_mask, results_dir):
    ptrc_T_hr = _results_dataset('1h', 'ptrc_T', results_dir)
    fig_functions = {
        'nitrate_thalweg_and_surface': {
            'function': tracer_thalweg_and_surface.make_figure,
            'args': (ptrc_T_hr.variables['NO3'], bathy, mesh_mask),
            'kwargs': {'cmap': cmocean.cm.matter, 'depth_integrated': False}
        },
    }
    return fig_functions

That function presently loads only one results dataset, from the hourly SalishSea_*_ptrc_T.nc file from a nowcast-green run. If you wanted to also produce a salinity thalweg and surface figure you would need to add a line to load the corresponding grid_T.nc dataset, something like:

grid_T_hr = _results_dataset('1h', 'grid_T', results_dir)

Each make_figure() call that we want make_plots is described by a dict item in the fig_functions dictionary:

'nitrate_thalweg_and_surface': {
    'function': tracer_thalweg_and_surface.make_figure,
    'args': (ptrc_T_hr.variables['NO3'], bathy, mesh_mask),
    'kwargs': {'cmap': cmocean.cm.matter, 'depth_integrated': False}
}

The key, nitrate_thalweg_and_surface is the the root part of the file name into which the figure will be rendered. If make_plots is run with the command-line options nowcast-green research --run-date 2017-04-29, it will stored the rendered figure with the file name nitrate_thalweg_and_surface_29apr17.svg.

The value is a dict that defines how to call the make_figure() function. It has 2 required key/value pairs, and 2 optional ones.

The function key is required. Its value is the name of the website figure module and the function in it to call (i.e. make_figure()) in dotted notation. Note that the value is a function object, so it is not enclosed in quotes. The website figure module must be imported at the top of the make_plots module; e.g.

from nowcast.figures.research import tracer_thalweg_and_surface

The args key is required. Its value is is a tuple containing the positional arguments that make_figure() is to be called with.

The kwargs key is optional. Its value is a dict containing the keyword arguments and their values that make_plots() is to be called with. If no keyword arguments are needed you can omit kwargs.

The other optional key is format. Its value is the image format to use to store the rendered figure. It defaults to svg, our preferred figure image format because it is a resolution-independent vector format. The format key is provided for the occasional special instances when we want to save images as png bitmap images.

So, the fig_functions item:

'nitrate_thalweg_and_surface': {
    'function': tracer_thalweg_and_surface.make_figure,
    'args': (ptrc_T_hr.variables['NO3'], bathy, mesh_mask),
    'kwargs': {'cmap': cmocean.cm.matter, 'depth_integrated': False}
}

will result make_plots doing the function call:

fig = tracer_thalweg_and_surface.make_figure(
    ptrc_T_hr.variables['NO3'], bathy, mesh_mask,
    cmap=cmocean.cm.matter, depth_integrated=False
)

and storing the rendered fig in /results/nowcast-sys/figures/nowcast-green/ddmmmyy/nitrate_thalweg_and_surface_ddmmmyy.svg.

Running make_plots Worker to Test a Figure

We can test that we have set up the necessary dataset loading and registered our make_figure() function correctly by running the make_plots worker in debug mode to confirm that it renders our figure correctly.

  1. If you haven’t done so already, activate your Nowcast Figures Development Environment, and navigate to your SalishSeaNowcast/ directory:

    $ source activate nowcast-fig-dev
(nowcast-fig-dev)$ cd SalishSeaNowcast/

  2. Set up 2 environment variables that the nowcast system expects to find, and create a temporary logging directory for it to use:

    (nowcast-fig-dev)$ export NOWCAST_LOGS=/tmp/$USER
(nowcast-fig-dev)$ export NOWCAST_ENV=$CONDA_PREFIX
(nowcast-fig-dev)$ mkdir -p $NOWCAST_LOGS

  3. Run the make_plots worker:

    (nowcast-fig-dev)$ python -m nowcast.workers.make_plots config/nowcast.yaml nowcast-green research --debug --test-figure nitrate_thalweg_and_surface --run-date 2017-04-29

    The command line elements are:

    • -m to tell Python to run a module

    • nowcast.workers.make_plots, the module to run

    • config/nowcast.yaml, the path and file name of the nowcast system configuration file

    • `nowcast-green, the run type

    • research, the plots type

    • --debug to send logging output to the terminal and not communicate with the nowcast system manager process (very important)

    • --test-figure to produce a test figure

    • nitrate_thalweg_and_surface, the key of the make_figure() call to test

    • --run-date, to say what date’s run results to render the figure for

    The output of a successful test should look something like:

    2017-05-05 17:11:16,119 INFO [make_plots] running in process 2993
2017-05-05 17:11:16,120 INFO [make_plots] read config from config/nowcast.yaml
2017-05-05 17:11:16,120 DEBUG [make_plots] **debug mode** no connection to manager
2017-05-05 17:11:16,358 DEBUG [make_plots] starting nowcast.figures.research.tracer_thalweg_and_surface.make_figure
2017-05-05 17:11:18,645 INFO [make_plots] /results/nowcast-sys/figures/test/nowcast-green/29apr17/nitrate_thalweg_and_surface_29apr17.svg saved
2017-05-05 17:11:18,646 INFO [make_plots] research plots for 2017-04-29 nowcast-green completed
2017-05-05 17:11:18,647 DEBUG [make_plots] **debug mode** message that would have been sent to manager: (success nowcast-green research nowcast-green research plots produced)
2017-05-05 17:11:18,647 DEBUG [make_plots] shutting down

    It is particularly important that your output contains the line that tells you that your figure was saved:

    INFO [make_plots] /results/nowcast-sys/figures/test/nowcast-green/29apr17/nitrate_thalweg_and_surface_29apr17.svg saved

    You can transform that path into a URL like:

    https://salishsea.eos.ubc.ca/test/nowcast-green/29apr17/nitrate_thalweg_and_surface_29apr17.svg

    and visually check your figure in your browser.

    Alternatively, you can use the Image Viewer program on your workstation to open the file at that path.