`skookum` Deployment

Git Repositories

Clone the following repos into /SalishSeaCast/:

$ cd /SalishSeaCast/
$ git clone git@github.com:SalishSeaCast/grid.git
$ git clone git@github.com:UBC-MOAD/moad_tools.git
$ git clone git@github.com:UBC-MOAD/Reshapr.git
$ git clone git@github.com:SalishSeaCast/NEMO-Cmd.git
$ git clone git@github.com:43ravens/NEMO_Nowcast.git
$ git clone git@github.com:SalishSeaCast/private-tools.git
$ git clone git@github.com:SalishSeaCast/rivers-climatology.git
$ git clone git@github.com:SalishSeaCast/SalishSeaCmd.git
$ git clone git@github.com:SalishSeaCast/SalishSeaNowcast.git
$ git clone git@github.com:SalishSeaCast/salishsea-site.git
$ git clone git@github.com:SalishSeaCast/SS-run-sets.git
$ git clone git@github.com:SalishSeaCast/tidal-predictions.git
$ git clone git@github.com:SalishSeaCast/tides.git
$ git clone git@github.com:SalishSeaCast/tools.git
$ git clone git@github.com:SalishSeaCast/tracers.git
$ git clone git@gitlab.com:mdunphy/FVCOM-VHFR-config.git
$ git clone git@gitlab.com:douglatornell/OPPTools.git

Python Packages

The Python packages that the system depends on are installed in conda environments.

Note

In Mar-2022 the Python environment and package management tool used for the system was changed from Miniconda3 to Mambaforge-pypy3.

For the SalishSeaCast automation system:

$ cd /SalishSeaCast/
$ mamba env create \
    --prefix /SalishSeaCast/nowcast-env \
    -f SalishSeaNowcast/envs/environment-prod.yaml
$ conda activate /SalishSeaCast/nowcast-env
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable NEMO_Nowcast/
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable moad_tools/
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable Reshapr/
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable tools/SalishSeaTools/
(/SalishSeaCast/nowcast-env)$ cd OPPTools/
(/SalishSeaCast/nowcast-env)$ git switch SalishSeaCast-prod
(/SalishSeaCast/nowcast-env)$ cd /SalishSeaCast/
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable OPPTools/
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable NEMO-Cmd/
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable SalishSeaCmd/
(/SalishSeaCast/nowcast-env)$ python3 -m pip install --editable SalishSeaNowcast/

For the sarracenia client that maintains mirrors of the HRDPS forecast files and rivers hydrometric files from the ECCC MSC datamart service:

$ cd /SalishSeaCast/
$ mamba env create \
    --prefix /SalishSeaCast/sarracenia-env \
    -f SalishSeaNowcast/envs/environment-sarracenia.yaml
$ conda activate /SalishSeaCast/sarracenia-env
(/SalishSeaCast/sarracenia-env)$ sr_subscribe edit credentials.conf  # initialize datamart credentials

For the salishsea-site web app that is mounted at https://salishsea.eos.ubc.ca/:

$ cd /SalishSeaCast
$ mamba env create \
    --prefix /SalishSeaCast/salishsea-site-env \
    -f salishsea-site/envs/environment-prod.yaml
$ conda activate /SalishSeaCast/salishsea-site-env
(/SalishSeaCast/salishsea-site-env) $ python3 -m pip install --editable salishsea-site/

Environment Variables

`/SalishSeaCast/nowcast-env`

Add the following files to the /SalishSeaCast/nowcast-env environment to automatically export the environment variables required by the nowcast system when the environment is activated:

$ cd /SalishSeaCast/nowcast-env
$ mkdir -p etc/conda/activate.d
$ cat << EOF > etc/conda/activate.d/envvars.sh
export NOWCAST_ENV=/SalishSeaCast/nowcast-env
export NOWCAST_CONFIG=/SalishSeaCast/SalishSeaNowcast/config
export NOWCAST_YAML=/SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
export NOWCAST_LOGS=/SalishSeaCast/logs/nowcast
export NUMEXPR_MAX_THREADS=6
export ONC_USER_TOKEN=a_valid_ONC_data_API_user_token
export SARRACENIA_ENV=/SalishSeaCast/sarracenia-env
export SARRACENIA_CONFIG=/SalishSeaCast/SalishSeaNowcast/sarracenia
export SENTRY_DSN=a_valid_sentry_dsn_url
export SLACK_SSC_DAILY_PROGRESS=a_valid_slack_incoming_webhook_url
export SLACK_SSC_HINDCAST_PROGRESS=a_valid_slack_incoming_webhook_url
EOF

and unset them when it is deactivated.

$ mkdir -p etc/conda/deactivate.d
$ cat << EOF > etc/conda/deactivate.d/envvars.sh
unset NOWCAST_ENV
unset NOWCAST_CONFIG
unset NOWCAST_YAML
unset NOWCAST_LOGS
unset NUMEXPR_MAX_THREADS
unset ONC_USER_TOKEN
unset SARRACENIA_ENV
unset SARRACENIA_CONFIG
unset SENTRY_DSN
unset SLACK_SSC_DAILY_PROGRESS
unset SLACK_SSC_HINDCAST_PROGRESS
EOF

`/SalishSeaCast/sarracenia-env`

The /SalishSeaCast/sarracenia-env environment variables are included in the SalishSeaNowcast/envs/environment-sarracenia.yaml file so that they are managed by conda to automatically export the environment variables required by the sarracenia client when the environment is activated and unset them when the environment is deactivated. To see the variables and their values:

$ cd /SalishSeaCast/sarracenia-env
$ source activate /SalishSeaCast/salishsea-site-env
(/SalishSeaCast/salishsea-site-env) $ conda env config vars list

`/SalishSeaCast/salishsea-site-env`

Add the following files to the /SalishSeaCast/salishsea-site-env environment to automatically export the environment variables required by the https://salishsea.eos.ubc.ca website app when the environment is activated:

$ cd /SalishSeaCast/salishsea-site-env
$ mkdir -p etc/conda/activate.d
$ cat << EOF > etc/conda/activate.d/envvars.sh
export SALISHSEA_SITE_ENV=/SalishSeaCast/salishsea-site-env
export SALISHSEA_SITE=/SalishSeaCast/salishsea-site
export SALISHSEA_SITE_LOGS=/SalishSeaCast/logs/salishsea-site
export NOWCAST_LOGS=/SalishSeaCast/logs/nowcast
export SENTRY_DSN=a_valid_sentry_dsn_url
EOF

and unset them when it is deactivated.

$ mkdir -p etc/conda/deactivate.d
$ cat << EOF > etc/conda/deactivate.d/envvars.sh
unset SALISHSEA_SITE_ENV
unset SALISHSEA_SITE
unset SALISHSEA_SITE_LOGS
unset NOWCAST_LOGS
unset SENTRY_DSN
EOF

Nowcast Runs Directories

On the hosts where the nowcast system NEMO runs will be executed create a runs/ directory and populate it with:

$ chmod g+ws runs
$ cd runs/
$ mkdir -p LiveOcean NEMO-atmos rivers ssh
$ chmod -R g+s LiveOcean NEMO-atmos rivers ssh
$ cp ../SS-run-sets/v201702/nowcast-green/namelist.time_nowcast_template namelist.time
$ ln -s ../grid
$ ln -s ../rivers-climatology
$ ln -s ../tides
$ ln -s ../tracers

The hosts and their runs directories presently in use are:

arbutus.cloud
See NEMO Runs Directory
orcinus
/home/sallen/MEOPAR/nowcast/

ECCC MSC Datamart Mirror Directories

Create directories on skookum for storage of the HRDPS forecast files and rivers hydrometric files maintained by the sarracenia client:

$ mkdir -p /SalishSeaCast/datamart/hrdps-west
$ mkdir -p /SalishSeaCast/datamart/hydrometric

Logging Directories

Create directories on skookum for storage of the nowcast system and salishsea-site web app log files:

$ mkdir -p /SalishSeaCast/logs/nowcast
$ mkdir -p /SalishSeaCast/logs/salishsea-site

Static Web Site Assets Directories

A collection of static file assets for the salishsea-site web app are stored in the /results/nowcast-sys/figures/ tree. Create the that directory, and the directories for results visualization figures from the NEMO model runs with:

$ mkdir -p /results/nowcast-sys/figures
$ chmod g+ws /results/nowcast-sys/figures
$ mkdir -p /results/nowcast-sys/figures/forecast
$ mkdir -p /results/nowcast-sys/figures/forecast2
$ mkdir -p /results/nowcast-sys/figures/nowcast
$ mkdir -p /results/nowcast-sys/figures/nowcast-agrif
$ mkdir -p /results/nowcast-sys/figures/nowcast-green
$ mkdir -p /results/nowcast-sys/figures/surface_currents/forecast
$ mkdir -p /results/nowcast-sys/figures/surface_currents/forecast2

Create directories for results visualization figures from the FVCOM Vancouver Harbour and Lower Fraser River model runs with:

$ mkdir -p /results/nowcast-sys/figures/fvcom/forecast-x2
$ mkdir -p /results/nowcast-sys/figures/fvcom/nowcast-r12
$ mkdir -p /results/nowcast-sys/figures/fvcom/nowcast-x2

Create directories for results visualization figures from the WaveWatch III® Strait of Georgia amd Juan de Fuca Strait wave model runs with:

$ mkdir -p /results/nowcast-sys/figures/wwatch3/forecast
$ mkdir -p /results/nowcast-sys/figures/wwatch3/forecast2

Create a directory for visualization figures generated during preparation of the forcing files for the NEMO model runs with:

$ mkdir -p /results/nowcast-sys/figures/monitoring

Create a directory for storm surge alert ATOM feed with:

$ mkdir -p /results/nowcast-sys/figures/storm-surge/atom

Finally, create a directory and symlinks for the images used on the index page of https://salishsea.eos.ubc.ca/ with:

$ mkdir -p /results/nowcast-sys/figures/salishsea-site/static/img/index_page
$ cd /results/nowcast-sys/figures/salishsea-site/static/img/index_page
$ ln -s /SalishSeaCast/salishsea-site/salishsea_site/static/img/index_page/about_project.svg
$ ln -s /SalishSeaCast/salishsea-site/salishsea_site/static/img/index_page/biology.svg
$ ln -s /SalishSeaCast/salishsea-site/salishsea_site/static/img/index_page/currents_and_physics.svg
$ ln -s /SalishSeaCast/salishsea-site/salishsea_site/static/img/index_page/diatom_bloom_forecast.svg
$ ln -s /SalishSeaCast/salishsea-site/salishsea_site/static/img/index_page/storm_surge_forecast.svg
$ ln -s /SalishSeaCast/salishsea-site/salishsea_site/static/img/index_page/storm_surge_nowcast.svg

$ mkdir -p /results/nowcast-sys/figures/bloomcast

Persistent Dask Cluster for `make_averaged_dataset` Worker

The make_averaged_dataset worker is launched:

after every nowcast-green run to down-sample hour-average NEMO results files to day-averaged files
after that processing is completed at the end of each month to down-sample day-averaged files to month-averaged files

That means that there are often concurrent instances of the worker. Instead of letting each worker instance spin up its own ad hoc dask cluster, we use a persistent dask cluster on salish that the worker dispatches tasks to.

Create a tmux session on salish for the dask cluster:

$ tmux new -s make_averaged_dataset

In the first tmux terminal, activate the /SalishSeaCast/nowcast-env environment, and launch the dask-scheduler with its serving port on 4386, and its dashboard port on 4387:

$ conda activate /SalishSeaCast/nowcast-env
(/SalishSeaCast/nowcast-env)$ dask scheduler --port 4386 --dashboard-address :4387

Use Control-b , to rename the tmux terminal to dask-scheduler.

Start a second tmux terminal with Control-b c, activate the /SalishSeaCast/nowcast-env environment, and launch the 4 dask worker processes with these properties:

1 thread per worker
64G memory limit per worker
worker files stored on the /tmp/SalishSeaCast/ directory
workers restart every 3600 seconds with 60 second random staggering of their restart times
workers communicate with the scheduler on port 4386

$ conda activate /SalishSeaCast/nowcast-env
(/SalishSeaCast/nowcast-env)$ dask worker --nworkers=4 --nthreads=1 --memory-limit 64G \
  --local-directory /tmp/SalishSeaCast \
  --lifetime 3600 --lifetime-stagger 60 --lifetime-restart \
  localhost:4386

Use Control-b , to rename the tmux terminal to dask-workers.

`ssh` Keys and Configuration

Generate a passphrase-less RSA key pair to use for connections to most remote hosts:

$ ssh-keygen -t rsa -f $HOME/.ssh/SalishSeaNEMO-nowcast_id_rsa -C SalishSeaNEMO-nowcast

Use ssh-copy-id to install the public key on arbutus, optimum, and orcinus; e.g.

$ ssh-copy-id -i $HOME/.ssh/SalishSeaNEMO-nowcast_id_rsa arbutus.cloud

Generate a passphrase-less ED25519 key pair to use for connections to the graham HPC cluster:

ssh-keygen -t ed25519 -f $HOME/.ssh/SalishSeaCast_robot.graham_ed25519 -C "SalishSeaCast robot.graham"

Edit the public key to prefix it with the constraint predicates necessary for automation in the context of multuifactor authentication on the graham cluster. The constraint predicates are:

restrict,from="142.103.36.*",command="/cvmfs/soft.computecanada.ca/custom/bin/computecanada/allowed_commands/transfer_commands.sh"

Use https://ccdb.computecanada.ca/ssh_authorized_keys to install the public key for graham via the Alliance CCDB.

Add the following stanzas to $HOME/.ssh/config on skookum:

Host arbutus.cloud-nowcast
    HostName        <ip-address>
    User            ubuntu
    IdentityFile    ~/.ssh/SalishSeaNEMO-nowcast_id_rsa
    ForwardAgent    no

Host robot.graham
    HostName     robot.graham.alliancecan.ca
    User         <userid>
    IdentityFile    ~/.ssh/SalishSeaCast_robot.graham_ed25519
    ForwardAgent no

Host optimum-hindcast
    HostName optimum.eos.ubc.ca
    User <userid>
    HostKeyAlgorithms=+ssh-rsa
    PubkeyAcceptedKeyTypes=+ssh-rsa
    IdentityFile    ~/.ssh/SalishSeaNEMO-nowcast_id_rsa
    ForwardAgent no

Host orcinus-nowcast-agrif
    HostName     orcinus.westgrid.ca
    User         <userid>
    HostKeyAlgorithms=+ssh-rsa
    PubkeyAcceptedKeyTypes=+ssh-rsa
    IdentityFile    ~/.ssh/SalishSeaNEMO-nowcast_id_rsa
    ForwardAgent no

skookum Deployment