This document describes the NCWF-2 system as it is configured in the install tar file. No changes made by the AWC are included in this document.
The source code files for the NCWF-2 system are kept in the
ncwf2/apps, ncwf2/libs and
ncwf2/incs directories in the install tar file.
The NCWF-2 configuration files are kept in the
ncwf2/projects/ncwf2 directory.
Documentation files are kept in the ncwf2/docs/ncwf2
directory.
The original source code at NCAR is stored using the Concurrent Versions
System (CVS).
The $NCWF_SRC_DIR/distribs/ncwf_cvs.kc script
was used to retrieve all of the files for the system from CVS in Boulder.
The source directory was then tarred up and sent to
Kansas City, where it was untarred by hand and then compiled and
installed using
the $NCWF_SRC_DIR/distribs/ncwf_make.kc script.
Compiled executables and general purpose scripts were installed
in ~/rap/bin by this script.
Compiled libraries were installed in ~/rap/lib and
include files were installed in ~/rap/include.
Click here to see detailed information about the source code included with the NCWF-2 sytem.
Environment variables are used in the NCWF-2 system for several purposes. First, using environment variables allows the configuration files to be moved to new places with little effort. Second, they allow for simple load balancing. If needed, NCWF-2 subsystems can be easily moved to different hosts to speed up processing. Finally, environment variables allow for easy porting of the executables to different Operating Systems. Click here to see the environment variables defined in the NCWF-2 system
The NCWF-2 system is divided into the following subsystems:
All of the scripts and parameter files for the system are kept under subdirectories based on the logical subsystem to which they belong. As required by auto_restart, every process has a start script. The scripts are named using the convention:
start_executable.instance
where:
executable
instanceThe executable and instance together uniquely identify each process used by the system.
If a process uses a parameter file, the parameter file is named using the following convention:
executable.instance
The directories defined by the
navigation environment variables
contain all of the start scripts and parameter files for the system.
In each of the directories defined by the navigation environment variables,
there is a bin subdirectory
which contains all of the start scripts and a params
subdirectory which
contains all of the parameter files for that subsystem.
All of the processes in the NCWF-2 system produce log files to help in debugging system problems. The log files are found in the $NCWF_LOG_DIR directory. This directory contains subdirectories, named by date, each containing all of the log files produced on that day.
In the operational system, most of the log files will be empty except for messages that are always written hourly to the log files. If a process experiences a detectable error, however, an error message will be written to the appropriate log file. Like the start scripts and parameter files, log files are named using the process executable and instance so that the log information can be easily found. Note that NCWF-2 is a data-driven system so a missing input file for a process is not considered an error condition. Instead, the process will just sit there waiting for new data to appear.
The log files can be very useful when debugging a system problem. If a process in the system doesn't seem to be operating properly, setting the debug flag in the process' parameter file and restarting that process will usually cause useful messages to be put into the log file. Unfortunately, some processes have a debug flag in their parameter file, but no extra information is printed when the flag is turned on. In that case, about the only way to debug the process is to add print statements to the source code.
The data ingest processes are all of the processes needed to retrieve the raw data and convert it to the formats (MDV and SPDB) needed for the algorithms. The following data are currently being ingested by the system:
/data/obs/ascii/ltg/nldn_1m)./data/radar/unisys/nat/vil)./data/radar/unisys/nat/tops)./data/model/grib/ruc20km)./data/model/grib/ruc20km)./data/model/grib/ruc20km).Click here to view the data-flow diagrams for the ingest portion of the NCWF-2 system.
The extrapolation forecast processes are all of the processes that are used to produce a simple extrapolation forecast. The extrapolation forecast is used to supply vectors for extrapolating the final forecast once growth and dissipation have been added. Click here to see the data-flow diagrams for the extrapolation forecast portion of the NCWF-2 system.
The growth forecast processes are all of the processes used to produce a growth forecast. Click here to see the data-flow diagrams for the growth forecast portion of the NCWF-2 system.
Colide is an algorithm developed at NCAR for detecting and extrapolating boundaries. The Colide algorithm is used in the NCWF-2 system to determine the orientation of large-scale features. Although Colide is a part of the growth subsystem, it is implemented as a separate subsystem in NCWF-2 because the processing can be fairly intense. Click here to see the data-flow diagrams for the Colide portion of the NCWF-2 system.
The stability processes are used to detect regions of environmental instability based on the RUC model data. Like Colide, the stability processes are part of the growth subsystem, but are implemented as a separate subsystem in NCWF-2 because the processing can be fairly intense. Click here to see the data-flow diagrams for the stability portion of the NCWF-2 system.
The trending processes compute the current dissipation trends in the system. Click here to see the data-flow diagrams for the trending portion of the NCWF-2 system.
The probability forecast processes take the extrapolation, growth and trending information calculated above and produce 30-minute, 60-minute, 90-minute and 120-minute probability forecasts. Click here to see the data-flow diagrams for the probability forecast portion of the NCWF-2 system.
The data distribution processes are used to produce the datasets that will be distributed to other organizations. Click here to see the data-flow diagrams for the data distribution portion of the NCWF-2 system.
The NCWF data is stored in subdirectories of the $RAP_DATA_DIR directory.
Within the main NCWF data directory, the system data is divided based on the data format. Click here to see a description of all of the datasets used by the NCWF-2 system.
Automatic process monitoring of the NCWF-2 system is done using
a PERL script called auto_restart.
auto_restart compares a given list of processes with the currently
running processes to see if any processes need to be restarted.
auto_restart determines
which processes are currently running by getting a list of current
processes from a process mapper process called procmap.
All realtime processes in the system send heartbeat messages to
the procmap process on their local host so
it can keep track of the current process list.
In the NCWF-2 system, each operational host runs its own instance of auto_restart, which checks with the procmap process on that host to see what's currently running. Currently, the entire NCWF-2 system is set up to run on a single host. If load becomes an issue, some of the processes can be moved to other hosts as described below.
The process list for each host is found in
$NCWF_CONTROL_HOME/runtime/PROCESS_LIST, which is
a symbolic link to one of the process list files in the
$NCWF_CONTROL_HOME/proc_lists directory.
This list is used by auto_restart as an indication
of which processes should be running on the machine.
This list must always be up-to-date for the system to work.
auto_restart is kept running via cron. Periodically, a cron job checks to see if auto_restart is running and restarts it if necessary. This allows the system to automatically recover from hardware reboots and auto_restart process failures. auto_restart maintains a log of process restarts so we can see if a system is having a recurrent problem. This log is found in the file:
$NCWF_LOG_DIR/date/auto_restart.log
cron can also used to run other utilities,
like daily backups, that are run on a periodic basis.
The crontab file for each host is found in
$NCWF_CONTROL_HOME/runtime/CRONTAB_LIST, which is
a symbolic link to one of the crontab files in the
$NCWF_CONTROL_HOME/crontab directory.
Note that it is always better to check the installed crontab using:
crontab -l
rather than relying on the local files when checking for problems. It's possible that someone changed a crontab file without installing it on the system.
The NCWF-2 system is set up so that pieces of the system can be easily moved to other hosts if load becomes an issue. The load balancing environment variables are defined so that all of the processes for a given NCWF-2 subsystem can be run on any host.
To move processes to another host, the first thing you must do is
create new process list files for each host in the system based on
the original process list
that came with the system.
This list includes all of the processes in the NCWF-2 system along with
comments indicating which logical host should run each process.
When load balancing, the new process list files should contain just
the processes that should run on that particular host.
Once the new process list files are created, the
$NCWF_CONTROL_HOME/runtime/PROCESS_LIST link on each
host should be pointed to the correct process list for that host.
If any of the hosts will be doing additional processing controlled
by cron, then new crontab files will need to be created for the hosts.
These files are kept in the
$NCWF_CONTROL_HOME/crontab directory and the correct
crontab file for each host should be linked to
$NCWF_CONTROL_HOME/runtime/CRONTAB_LIST.
If cron will only be used to keep the auto_restart and procmap processes
running, as is done in the original system, then all of the hosts can
point to the original crontab file.
Once the host-specific files have been created and the correct
symbolic links made in the $NCWF_CONTROL_HOME/runtime
directory, the last thing that needs to be done to complete the
load balancing process is to change the definitions of the
load balancing environment
variables
on each of the hosts.
Now the system can be started on each of the hosts.
Author: Nancy Rehak