Nowcast System Operations

ssh Hosts and Keys Configuration

Todo

Write this section.

Cold Start

The long-running processes in the nowcast system framework, the log aggregator, the message broker, and the manager, are managed by the supervisor process manager tool. So is the sarracenia client that maintains mirrors of the HRDPS forecast files and rivers hydrometric files from the ECCC MSC datamart service.

Start the nowcast system and sarracenia client with:

$ source activate /results/nowcast-sys/nowcast-env
(/results/nowcast-sys/nowcast-env)$ supervisord --configuration $NOWCAST_CONFIG/supervisord.ini

supervisord monitors the long-running processes and restarts them if they crash or are shutdown accidentally.

The https://salishsea.eos.ubc.ca website app is also managed by supervisor. Start it with:

$ source activate /SalishSeaCast/salishsea-site-env
(/SalishSeaCast/salishsea-site-env)$ supervisord --configuration $SALISHSEA_SITE/supervisord-prod.ini

System Management

supervisorctl is the command-line interface for interacting with the processes that are running under supervisord. Start it with:

$ source activate /results/nowcast-sys/nowcast-env
(/results/nowcast-sys/nowcast-env)$ supervisorctl --configuration $NOWCAST_CONFIG/supervisord.ini

See the supervisorctl docs, or use the help command within supervisorctl to get information on the available commands. A few that are useful:

  • avail to get a list of the processes that supervisord is configured to manage

  • status to see their status

  • stop to stop a process; e.g. stop manager

  • start to start a stopped process; e.g. start manager

  • restart to stop and restart a process; e.g. restart manager

  • signal hup to send a HUP signal to a process, which will cause it to reload its configuration from the NOWCAST_YAML file that the process was started with; e.g. signal hup manager. This is the way to communicate nowcast system configuration changes to the long-running processes.

  • shutdown to stop all of the processes and shutdown supervisord

Use quit or exit to exit from supervisorctl.

sr_subscribe is the command-line interface for interacting with the sarracenia client that maintains mirrors of the HRDPS forecast files and rivers hydrometric files from the ECCC MSC datamart service.

sr_subscribe is run in foreground mode instead of daemonized so that it can be managed by :supervisord. Use supervisorctl to view the sr_subscribe log files:

$ source activate /results/nowcast-sys/nowcast-env
(/results/nowcast-sys/nowcast-env)$ supervisorctl --configuration $NOWCAST_CONFIG/supervisord.ini tail sr_subscribe-hrdps-west

or

(/results/nowcast-sys/nowcast-env)$ supervisorctl --configuration $NOWCAST_CONFIG/supervisord.ini tail sr_subscribe-hydrometric

Use tail -f to follow the logs to view updates as they occur.

Automatic Deployment of Changes to salishsea-site App

A GitHub Actions workflow causes changes to be pulled and updated to /SalishSeaCast/salishsea-site/ and the app to be restarted via supervisorctl whenever changes are pushed to the repo on GitHub.