RAL | | Thunderstorm Auto Nowcasting

Hardware

The Auto-Nowcast Environment contains a set of machines (or hosts) with specific tasks. Multiple machines are required to meet the resources required by the various algorithms in the Auto-Nowcast Environment. The number of machines in a given Auto-Nowcast Environment installation will vary depending on the number and types of algorithms running at that installation and the data available.

The minimal Auto-Nowcast Environment configuration consists of the two machines which have graphical data displays running on them: the Control Host and the TITAN Host. Other Hosts are added to the configuration to support non-graphical, analysis algorithms.

Throughout this document, the Auto-Nowcast Environment machines are referred to by the main function that they serve.

* Control Host
* TITAN Host
* Other hosts, for example, Colide Host, Ingest Host, etc.

In general, the hosts are configured identically with the same CPU, memory, swap space, and OS version. The host must be a high-end UNIX workstation with Linux 2.4.21 or higher installed.

procmap

The minimal recommended configuration for a nowcast host includes a 2.66 GHz processor with 2-3 Gb RAM and 146 Gb SCSI HD. Two ethernet cards are recommended on any machine which ingests a raw radar data from a UDP broadcast stream. This is required mainly to reduce network traffic problems. If radar data is being ingested via an LDM or TCP/IP , then a second card is not necessary.

Software

The main software components of the Auto-Nowcast Environment are:
Data Service
Process Control
Utilities

Data Service

Data in MDV, SPDB, FMQ, and Titan Storm Track formats are available through data servers. In addition, SPDB data is served out in a graphical format, Symprod, suitable for rendering on X-displays. The servers and clients communicate via TCP/IP protocol. This model allows a client (or application) running on one host within the Auto-Nowcast Environment to access data which resides on a different host. Try to co-locate data and algorithms which use the data on machines as much as possible to minimize overhead of data serving across hosts.

ASCII datasets are not served but are read directly from disk files by the various algorithms which use the ASCII data.

Following is a list of some of RAP's data service applications and their functions:

DataService
Application	Function
DataMapper	Contains a map of data sets realtive to $RAP_DATA_DIR, used by Sysview, the system monitor
DsMdvServer	Server for data in MDV format.
DsSpdbServer	Server for data in SPDB format.
DsFmqServer	Server for FMQ messages.
DsTitanServer	Server for titan data.
DsServerMgr	Server manager handles requests from applications and starts necessary servers. (Thus DsServerMgr is the only server which needs to be included in a host process list.)
Janitor	Recurses through the data directory tree and compresses or removes data based on _Janitor param files located with the data.

Back to top

Process Control

Process control referrs to a simple and effective mechanism to keep realtime processes running in the Auto-Nowcast Environment.At the heart of the process control mechanism is the process mapper, a.k.a. PROCMAP, and a program called auto_restart which reads and responds to the output of the process mapper .

procmap

All of the realtime processes within the Auto-Nowcast Environment register with the process mapper on a regular interval. This regular registration, or heartbeat, provides an indicator of the status of each realtime process in the Auto-Nowcast Environment. auto_restart compares the output of the process mapper with a user configured process list and makes sure all of the processes in the list are up and running and registering at proper intervals.

Following is a list of utilities, applications, and scripts used for process control:

Control Utilities, Applications and Scripts
Utility, Application, or Script	Function
cron	cron is a UNIX utility which is a daemon used for executing commands on a timed schedule.
procmap	procmap is an application with which system applications register on a regular basis.
auto_restart	auto_restart is an application which uses procmap and a process list to insure that all processes in the list are running and checking in with procmap at regular intervals. If a process is missing, auto_restart will use the start script in the process list to restart it. If a process is hung, auto_restart will use the 'kill' and 'start' scripts in the process list to kill it and then restart it.
procmap_list_start	procmap_list_start starts all processes in a process list with the start scripts for each process indicated in the list.
start_process.control	start_process.control is a script run by cron every five minutes. If necessary, this script starts procmap, the processes in the process list with procmap_list_start and auto_restart.
host_startup	host_startup executes the start_up sequence for a single host in the system. Here is the sequence: Concatenate relevant $CONTROL_DIR/params/host_env.* files and install in $CONTROL_LOCAL_DIR/runtime/HOST_ENV. This file contains the system hosts and their roles. Install $CONTROL_DIR/params/radar_env.lookup in $CONTROL_LOCAL_DIR/runtime/RADAR_ENV (which is used for text substitution in process list.) Concatenate relevant process lists from $CONTROL_DIR/proc_list based on the host roles in HOST_ENV and install in $CONTROL_LOCAL_DIR/runtime/PROCESS_LIST Concatenate relevant cron files from $CONTROL_DIR/crontab and install in $CONTROL_LOCAL_DIR/runtime/CRONTAB_LIST and start cron. Start start_process.control with procmap_list_start and auto_restart using $CONTROL_LOCAL_DIR/runtime/PROCESS_LIST.
host_shutdown	host_shutdown executes the shutdown sequence for a single host in the system. Here is the sequence: kill cron kill auto_restart kill processes in the process list kill data servers kill procmap remove FMQs if part of entire system shutdown. remove HOST_ENV
niwot_startup	niwot_startup executes host_startup on each host in the system.
niwot_shutdown	niwot_shutdown executes host_shutdown on each host in the system.

For more detailed description of procmap usage and cron use, see Procmap and Cron.

Back to top

Utilities

Following is a list of some handy utilities and their functions:

Utilities
Application	Function
PrintMdv	Print contents of mdv file.
SpdbQuery or XSpdbQuery	Query an SPDB database
print_procmap	prints the processes currently registering with procmap
RadMon	View radar beam data as it is being written to an FMQ
snuff or snuff_inst	kills processes or instances of processes
utime_decode	decodes unix time into UTC and local time
LogFilter	Writes application output in dated subdirectory format. (All of the start scripts for applications pipe the application ouput to LogFilter)
running	Returns 0 if process is running and 1 if it is not. (Most of the start scripts use this utility before starting an application so that more than one instance of an application will not start.)

Back to top

Environment Configuration

The Project Directory: Application Param Files and Start Scripts, Host Config, and Data Config.
Environment Variables
Network Communications

The Project Directory: Application Param Files and Start Scripts, Host Config, and Data Config.

The project directory or $PROJ_DIR contains all project specific information including: "homes" for all of your project specific application parameter files and start scripts, project host configuration and process lists for each host, cvs project checkout script and build scripts, data directories and data param files for each host, and system monitor.

Following are the three subdirectories of $PROJ_DIR that contain all host, data, and software configuration information for a particular installation of the AutoNowcaster.

$PROJ_DIR subdirectories for process control, and host data and software configuration
Directory	Contents
control	crontab -- crontab files for each host params -- email.list for nightly stats host_env.<mode> (host/hostRole pairs) mode.equiv (system startup mode equivalences) radar_env.lookup (radar/host pairs + identifying keyword) proc_list -- process lists for each host runtime -- some runtime files including: NIWOT_LOG -- system startup times, shutdown times, and mode
dataHome	bin -- start and kill scripts for data service apps. DsServerMgr, Janitor, DataMapper params -- param files that will be copied to data directories (like the _Janitor files), plus static terrain files. data_lists -- lists of data directories on each host and param files that should reside in particular data directories.
utilsHome	bin contains checkout and build scripts of all apps and libs used in a particular installation of the autoNowcaster.

Following is a list of homes for applications. Each application home generally contains two subdirectories: bin and params. The bin subdir contains all start scripts and kill scripts(if there are any) for the applications. The params subdir contains all of the application parameter files.

$PROJ_DIR subdirectories: Homes for applications
Home	Description	Applications
adjointHome	Home to adjoint.	adjoint
advectHome	Home to all applications related to advection.	ctrec advectGrid stratiform_filter
ciddHome	Home to all applications related to CIDD the Configurable Interactive Data Display.	CIDD DrawFmq2Bdry GridPointSelect Sounding2Metar SoundingPlot SoundingText
cronusHome	Home to all applications which generate data for the forecast (with the exception of titanGrid which is in titanHome).	bdryCollision bdryGrid bdryStormCol cronus gandi radarTrigger
ingestHome	Home to all applications which ingest data into the AutoNowcast environment (except radar ingest which has its own home.)	LdmDynamic2Static Metar2Spdb NidsVad2Spdb derived_ruc_fields kavltg2spdb ruc_sounding surf_interp
radarHome	Home to all radar ingest, quality control, and reformatting applications.	nexrad2dsr Dsr2Vol MdvMerge BrightBand RadMon vadAnalysis
titanHome	Home to Titan and Titan display applications	Titan Rview TimeHist Tstorms2Spdb
sysviewHome	Home to applications, params files, and diagrams related to the system monitor called SysView.	SysView

Back to top

Environment Variables

The Auto-Nowcast Environment is run under the special user account nowcast The nowcast user account makes heavy use of UNIX environment variables for navigating around the Auto-Nowcast Environment. Environment variables provide an easy mechanism for installing a generic Auto-Nowcast Envrionment setup at different field sites. $PROJ_DIR subdirectories for process control, and host, data and software configuration.

Rap directory
$RAP_DIR /nfs/ncar/rap

Bins, Libs, and Includes Environment Variables
$RAP_SHARED_BIN_DIR /nfs/ncar/bin
$RAP_SHARED_INC_DIR /nfs/ncar/inc
$RAP_SHARED_LIB_DIR /nfs/ncar/lib
$RAP_BIN_DIR /nfs/ncar/rap/bin
$RAP_INC_DIR /nfs/ncar/rap/include
$RAP_LIB_DIR /nfs/ncar/rap/lib

The Project Directory Environment Variables
$PROJ_DIR $RAP_DIR/<project name>
$ADJOINT_HOME $PROJ_DIR/adjointHome
$ADVECT_HOME $PROJ_DIR/advectHome
$CONTROL_DIR $PROJ_DIR/control
$CIDD_HOME $PROJ_DIR/ciddHome
$COLIDE_HOME $PROJ_DIR/colideHome
$CRONUS_HOME $PROJ_DIR/cronusHome
$DATA_HOME $PROJ_DIR/dataHome
$CRONUS_HOME $PROJ_DIR/cronusHome
$COLIDE_HOME $PROJ_DIR/colideHome
$RADAR_HOME $PROJ_DIR/radarHome
$SATELLITE_HOME $PROJ_DIR/satelliteHome
$SYSVIEW_HOME $PROJ_DIR/sysviewHome
$TITAN_HOME $PROJ_DIR/titanHome
$UTILS_HOME $HOME/utilsHome

Data and Logs Environment Variables
$RAP_DATA_DIR $HOME/data
$LOG_DIR $HOME/logs

Back to top

Network Communications

Cross Mounted File Partition

All the hosts in the Auto-Nowcast Environment cross-mount (via NFS) one file partition called $RAP_SHARED_DIR. The partition is exported with read/write permissions to all Auto-Nowcast Environment hosts. $RAP_SHARED_DIR contains the NCAR executable applications as well as the project directory ($PROJ_DIR) in which the components of Auto-Nowcast Environment resides.

Local Area Network

The hosts within the Auto-Nowcast Environment communicate using TCP/IP protocol.

Radar Network

Each Auto-Nowcast Environment host which has radar data coming into it from a UDP broadcast has a second ethernet card for that purpose. The use of a second ethernet card for broadcasting radar data protects the load on the local area network in with the Auto-Nowcast Environment is installed. If radar data is being broadcast via TCP/IP there is no need for a second ethernet card.

Back to top

Data Display

Two types of graphical data displays are available in the Auto-Nowcast Environment:
Cidd
Rview

Cidd

Is the most general of the data display applications. It is highly user-configurable and is capable of displaying both MDV and SYMPROD (the graphical form of SPDB) data.

Cidd is a X-based, real-time data display application which supports PPI, CAPPI, and cross sectional views. Cidd is the main display tool within the Auto-Nowcast Environment. It is highly configurable and is used to display the majority of the MDV and SPDB formated data produced within the Environment.

cidd

Rview

Is part of the TITAN application and provides a detailed graphical view of the thunderstorm tracking algorithm.

The X-based display utility Rview is part of the TITAN analysis algorithm and is used to display thunderstorm tracks in real-time over top of CAPPI radar data as well as vertical sections through storms.

For more information on the TITAN application and using the Rview display, see TITAN: Thunderstorm identification, tracking, analysis and nowcasting - a radar-based methodology and The TITAN User's Manual.

rview

Analysis Algorithms

The Nowcast Engine
Thunderstorm Identification, Tracking, and Features
Advection
Variational Doppler Radar Analysis
Satellite Algorithms
Data Ingest and Quality Control

The Nowcast Engine

Cronus

Cronus gathers and combines the relevant user defined meterological data that have been generated by the various algorithms with user defined functions and weights to produce an initiation interest field and a growth and decay interest field. These These two interest fields as input into gandi.

Cronus is triggered to run by radarTrigger which watches the incoming radar beam data and sends a message or 'trigger' to cronus when a user specified tilt has begun. Cronus then sends a forecast trigger to algorithms producing data used to create the interest fields indicating that the algorithm should run and produce forecast data for specific time in future. Cronus then gathers output from the algorithms and produces its interest fields.

Gandi

Gandi takes as input the initiation and growth and decay interest fields which are ouput by cronus and the dbz advect field. User defined step functions are applied resulting in a final reflectivity or precip rate forecast. The initiation interest field is used to calculate and initiation reflectivity field (based on user defined step function.) The growth and decay interest field is used to modify the dbz advect field, either growing or decaying regions above a user specified reflectivity threshold. Finally, the initiation and growth and decay reflectivity fields are merged.

Scripts:	$CRONUS_HOME/bin
Parameters:	$CRONUS_HOME/params

Executables	Input Data		Output Data
Executables	Description	Format	Description	Format
radarTrigger	radar tilt indicator	fmq	tilt trigger	fmq
cronus	tilt trigger	fmq	forecast trigger	fmq
cronus	user selected interest fields	MDV	initiation or growth and decay interest fields	MDV
gandi	advected reflectivity field	MDV	2D reflectivity field	MDV
gandi	initiation or growth and decay interest fields	MDV	2D reflectivity field	MDV

Cronus operates in realtime mode by triggering at the start of the 0.5 degree radar tilt. When the nowcast domain includes several radars, the trigger can be set by MdvMerge , the application which merges the cartesian radar data from all radars.In addition to the realtime trigger, the user can manually trigger a nowcast at anytime using the graphical interface to the cronus application.

cronus

Back to top

Thunderstorm Identification, Tracking, and Features

TITAN is a collection of processes which identify and track thunderstorm above a user-specified reflectivity (dBZ) threshold.

Rview and TimeHist are both graphical data displays for Titan output.

titanGrid converts titan storms characteristics into gridded fields. It can also tag storms with point statistics such as lightning strike statistics and probability of hail. Optionally, storms can be extrapolated using titan motion information.

StormInitDetect detects new storms from Titan and outputs the latitude, longitude, and time of storm initiations.

StormInit2Field grids the output of StormInitDetect and produces regions of storm initiation defined by gaussians at each storm initiation location.

Scripts:	$TITAN_HOME/bin
Parameters:	$TITAN_HOME/params

Executables	Input Data		Output Data
Executables	Description	Format	Description	Format
Titan	cartesian radar data	MDV	thunderstorm information	Titan StormTrack
titanGrid	Titan thunderstorm information	Titan Storm Track	2D gridded storms tagged with storm characteristics or point data statistics	MDV
titanGrid	point data	SPDB		MDV
StormInitDetect	Titan thunderstorm information	Titan Storm Tracks	Points of thunderstorm initiation	SPDB
StormInit2Field	Points of thunderstorm initiation	SPDB	Gridded regions of thunderstorm initiation	MDV
Rview	Cartesion reflectivity data	MDV	Graphical data display
Rview	Storm information	Titan Storm Tracks	Graphical data display
TimeHist	Storm information	Titan Storm Tracks	Graphical data display
TimeHist

titan

For more information and detail on the TITAN application, see TITAN: Thunderstorm identification, tracking, analysis and nowcasting – A radar-based methodology and the TITAN User's Manual.

Back to top

Advection

ctrec employs cross-correlation analysis of echos to calculate motion vectors from radar data. ('trec' is an acronym for Tracking of Radar Echos by Correlation.)

advectGrid uses ctrec motion vectors or sounding data to advect gridded data. It is usually used to advect reflectivity and storm characteristics.

startiform_filter filters flat echoes from radar data at user specified elevation. Used to isolate radar cu.

Scripts:	$ADVECT_HOME/bin
Parameters:	$ADVECT_HOME/params

Executables	Input Data		Output Data
Executables	Description	Format	Description	Format
ctrec	cartestian radar data	MDV	motion vector field	MDV
advectGrid	motion vector field from ctrec	MDV	2D advected dataset image	MDV
	sounding data used as backup to motion vector field	SPDB
	dataset to be advected	MDV
stratiform_filter	cartesian radar data	2D filtered dataset	MDV

For more information on ptrec see Determination of the Boundary Layer Airflow from a Single Doppler Radar.

Back to top

Variational Doppler Radar Analysis (VDRAS)

VDRAS is a system that produces high-resolution three-dimensional wind in the boundary layer. The wind field is obtained through a retrieval process combining single-Doppler radar observations and a numerical model using a four-dimensional variational scheme. The output fields from VDRAS include three velocity components, divergence, and temperature.

Scripts:	$ADJOINT_HOME/bin
Parameters:	$ADJOINT_HOME/params

Executables	Input Data		Output Data
	Description	Format	Description	Format
adjoint	PPI radar volume	MDV	3D wind, divergence, and temperature fields	MDV
	surface observations	SPDB
	soundings	SPDB

For more information on the adjoint method see Forecasting Storm Growth and Decay using Low-level Radar Data and the Adjoint Method.

Back to top

Satellite Algorithms

satDerive calculates differences and standard deviations of satellite channel data. Calculates short wave, infrared reflectance, an icing index.

satThresh thresholds satellite data. Threshold values are applied to the visible and clear IR channels. Then these fields are used to mask all other fields, so that only pixels at locations that have passed the thresholding tests on IR and the visible are output - the rest are marked as bad or missing.

RateOfChange calculates the rate of change of MDV gridded data. Detects cloud growth by monitoring cloud top temperatures from infrared satellite data.

CloudClass classifies clouds based on a set of threshold based rules. Thresholds can vary sinusoidally according to the time of the day or year.

Scripts:	$SATELLITE_HOME/bin
Parameters:	$SATELLITE_HOME/params

Executables	Input Data		Output Data
Executables	Description	Format	Description	Format
satDerive	raw satellite data	MDV	derived satellite data fields	MDV
satThresh	satellite data	MDV	thresholded satellite data	MDV
RateOfChange	IR Satellite Data	MDV	2D rate of change data values	MDV
CloudClass	satellite data	MDV	bitwise map of cloud type	MDV

Back to top

Data Ingest and Quality Control

The algorithms in the Auto–Nowcast Environment operate on data RAL's internal data formats: FMQ, MDV, SPBD, or Titan Storm Tracks. Many data ingesters/converters have already been written and are readily available. For other datasets, special converters may need to be developed to bring the data stream into the Auto-Nowcast Environment. Following is a list of data ingest and quality control applications:

Application	Description	Input	Output
nexrad2dsr	Level-2 NEXRAD radar data ingester	Level-2 NEXRAD from LDM, TCP/IP port	DsRadar written to FMQ
JamesD	Dealiases radar data	DsRadar	DsRadar written to FMQ
Dsr2Vol	Collects radar beam data and writes radar volumes in various coordinates (polar, ppi, cartesian.)	DsRadar from FMQ	DsRadar written to FMQ
BrightBand	Removal of enhanced reflectivity in melting layer due to hydrometeor aggregation and water coating.	MDV (cartesion coordinates)	MDV (cartesion coordinates)
ApFilter	Anomalous propagation filter	MDV (cartesion coordinates)	MDV (cartesion coordinates)
NidsVad2Spdb	Ingests and reformats NEXRAD Level 3 Vad data	NIDS VAD data	Spdb
Terrascan2Mdv	Ingests satellite data	Terrascan	MDV
Metar2Spdb	Ingests and reformats surface observations	METAR	Spdb
sndgIngest or classIngest	ingests and reformats sounding data	class	Spdb
NWSsoundingIngest	ingests and reformats sounding data	WMO	Spdb
kavltg2spdb	ingest and reformats lightning data	Kavaouras	Spdb
rucIngest	ingest RUC model data	RUC GRIB	MDV
derived_ruc_fields	derive additional fields from primary RUC fields	MDV	MDV
ruc_sounding	Create a sounding from RUC model data	MDV	SPBD

Back to top

Installation

Using This Configuration Guide
Hardware Requirements & Purchasing
UNIX System Setup
The Nowcast User Account
Shared Project Space
Creating a new Auto-Nowcast Project in CVS
Checking out an Existing Auto-Nowcast Project from CVS
Modifying & Installing .dot Files
Establishing ssh authentication
Modifying Process Control Files
Defining the Data Hierarchy
Building Libraries & Applications
Modifying Parameter Files
Setting up a System Monitor
Ready, Set, Go...
Freezing the Source Code
Packaging up the Field Project
At the Field Site

Using This Configuration Guide

The Auto-Nowcast system has been installed more than a dozen times over the last 8 years using varying system layouts, configuration techniques, scripts, and algorithms. By and large the configuration has undergone significant changes with each installation, always striving for improvements over the previous installation. Keeping in mind that every approach has had its share of strengths and weaknesses, the following should be considered a guide not a cookbook recipe. To ease the burden of the installation process, explicit command are often included; however, these should be interpreted more as suggestions than instructions. This guide is neither complete nor comprehensive, but reflects a suggested configuration largely based on the Thor2003 project.

The UNIX commands in this installation guide are expressed in a combination of font style and the international standard EBNF notation for language syntax including:

|
meaning "or"
<>
angle brackets used to surround non-terminal symbols
[]
square brackets used to surround optional items
{}
curly brackets used to surround repetitve items of zero or more
()
parentheses used for grouping

The angle brackets are to distinguish non-specific syntax (also known as non-terminal symbols) from terminal symbols which are written exactly as they are to be executed. For example, the command:

mv projects/<prototypeProject> projects/<newProjectName>

indicates that the reader is to substitute exact content for the category names prototypeProject and newProjectName while all other parts of the command are to be types as is. The reader might therefore execute the command:

mv projects/Thor2003 projects/Mendoza

Back to top

Hardware Requirements & Purchasing

Before purchasing hardware for the Auto-Nowcast environment, you must determine which algorithms will be part of the field project and which machines will play which roles within the project. This determination will dictate how many machines should be purchased.

Typical Auto-Nowcast Machines
Role	Logical Name(s)
Process Control & Graphical Display	$CONTROL_HOST $DISPLAY_HOST
Storm Identification & Tracking	$TITAN_HOST
Data Ingest	$INGEST_HOST
Boundary Detection	$COLIDE_HOST
Satellite Applications	$SATELLITE_HOST
Ctrec Advection	$ADVECT_HOST
Winds Modelling	$ADJOINT_HOST
Final Nowcast	$CRONUS_HOST
Field Display	$FIELD_DISPLAY_HOST

The current hardware recommendations for each of the machines in the Auto-Nowcast environment include:

2.66 GHz dual processors
3 Gb RAM
One or two 146Gb HD

Back to top

UNIX System Setup

The first step in setting up the Auto-Nowcast machines is to decide upon a file partitioning. A reasonable recommendation is 2Gb for the root partition, 1Gb for the /home partition, with everything else in a data partition, say /d1.

Later, in the section Shared Project Space, when it is necessary to establish areas to be used for cross mounts, suggestions are made based on the existence of a /d1 partition.

The next step in the UNIX system setup for the Auto-Nowcast environment is to install the following required software:

Debian Linux version 3.0 or greater
gcc/g++/egcc (libc6 required)
g77
gdb
ddd
pthread library
ssh
NTP (network time protocol) service
The perl and python interpreters
The mysql libs (for Metar2Spdb)

Other support software which may need to be installed includes:

Xview on the machine(s) which will compile and/or run CIDD
ImageMagick on the $DISPLAY_HOST
Java if we are running SysView

Back to top

The Nowcast User Account

Create the following user account on all project machines:

User: nowcast
UID : 10140
GID : 678 (shared)

There are two types of passwords that are used for the nowcast user account.

The internal password is used for all machines in RAP which have a nowcast account. This password does not change.
The external password is used on Auto-Nowcast project machine which are out in the field. This password is project-specific, determined by the lead engineer on the project, and can be changed.

While project machines are inside of RAP being set up for a field project, they should use the internal password. Later, at the Field Site, you will change over to the external password.

Make sure that all other project members who get accounts on the field project machines are setup with the group "shared" (GID 678) as their default group. This is necessary for interactions with CVS.

Back to top

Shared Project Space

The Auto-Nowcast project space contains source code, libraries, application executables, and the field project configuration. At RAP these components are contained on the cross mount $RAP_SHARED_DIR (/rap) and $RAP_DIR (usually ~/rap). For Auto-Nowcast installation we do not use /rap in order to avoid confusion and in order to allow the Auto-Nowcast user nowcast to do a make_install_shared into a project-specific location rather than installing executables into /rap/bin. This approach allows us to take a snap-shot of the software libraries and applications that will be fielded. As is done at RAP, the $RAP_SHARED_DIR is installed on a cross mounted partition which we call the shared project space. To begin installing an Auto-Nowcast environment, you must select which machine you will use for serving the shared project space. Plan on at least 2 Gb to be used for the shared project space. It is recommended that the $RAP_SHARED_DIR is cross mounted in the following way:

On <Shared Project Space Host>, the machine where the disk is physically mounted:

Login as root
Create /d1/ncar writable by group shared, owned by user nowcast (Use unix commands chown and chmod)
Create /nfs
Create a link from /nfs/ncar to /d1/ncar
Add the following entry to /etc/exports: /d1/ncar <projectHostNames>(rw)
Execute the following commands:
/etc/init.d/nfs-kernel-server stop
/etc/init.d/nfs-kernel-server start

On all other project machines:

Login as root
Create /nfs
Add the following entry to /etc/auto.mnt: ncar <Shared Project Space host>: /d1/ncar
Execute the following commands:
/etc/init.d/autofs stop
/etc/init.d/autofs start
Verify that the cross mount exists using the command ls /var/autofs/mnt/ncar
Create a link from /nfs/ncar to /var/autofs/mnt/ncar
Verify that the link is okay using the command ls /nfs/ncar

Back to top

Creating a new Auto-Nowcast Project in CVS

The next step in creating and installing a new Auto-Nowcast project is to check out an existing Auto-Nowcast project from CVS to be used as a prototype. If, however, you are simply modifying an Auto-Nowcast project which already exists in CVS, do not perform this step! Instead you should skip to the step below Checking out an Existing Auto-Nowcast Project from CVS.

CAUTION! The find command in the sequence of steps below will remove all records of CVS for the prototype project. This is an important step in setting up a new Auto-Nowcast project from a prototype project. It is important, however, that this step not be taken on an existing Auto-Nowcast project which is already checked into CVS!

Login as user nowcast on the $CONTROL_HOST
cd /nfs/ncar
mkdir rapShared
cd rapShared
setenv CVSROOT :pserver:nowcast@cvs:/cvs
cvs login
cvs co utilities/niwot
cd /nfs/ncar
mkdir rap
cd rap
cvs co projects/<prototypeProject>
cd projects/<prototypeProject>
find . -name CVS -exec rm -r {} \;
cd /nfs/ncar/rap
mv projects/<prototypeProject> projects/<newProjectName>
cd projects/<newProjectName>
cvs import projects/<newProjectName> v1 r1

Verify that the import command has read the entire tree into CVS. There have been incidents in which not all of the new project has been entered into the CVS repository by the import command.

Back to top

Checking out an Existing Auto-Nowcast Project from CVS

If you are working on an already existing Auto-Nowcast project, you must first check the project out from CVS. To do this, perform the following steps:

cd /nfs/ncar/rapShared
setenv CVSROOT :pserver:nowcast@nut:/cvs
cvs login
cvs co utilities/niwot
cd /nfs/ncar/rap
cvs co projects/<projectName>

Back to top

Modifying & Installing .dot Files

Now that you have a prototype project, the next step is to modify project-specific files to reflect the correct settings for the new Auto-Nowcast project. The first set of files which need project-specific modifications are a subset of the .dot files which reside in $RAP_DIR/$PROJECT or $PROJ_DIR. The .dot files include:

.cshrc
.login
.nexrc
.xserverrc
.xtrc
.xearthmarker
.xinitrc
.xmodmaprc
.fvwm

After making necessary modifications to some of these files, we will set up links to these files in $HOME for user nowcast.

As user nowcast on the $CONTROL_HOST

cd
rm .cshrc -or- mv .cshrc .cshrc.bak
ln -s /nfs/ncar/rap/projects/<newProjectName>/.cshrc
Edit the file .cshrc and change the enviornment variable for $PROJECT to reflect the new project name, i.e., setenv PROJECT <newProjectName>
source .cshrc
cd $PROJ_DIR
Edit the files in the .fvwm.* directories. These files configure the window manager on various hosts based on the host function. A minimal configuration of the window manager might be designed like this: From the $CONTROL_HOST, the pull down menus include remote logins to all nowcast hosts, and system startup and shutdown. From the $DISPLAY_HOST, pull down menus are created to start and stop CIDD( the nowcast display) and login to hosts relevant from the display site. From all other hosts, menus include only remote logins.
Edit the file .xearthMarker to include the lat/lon position of the field project.

To install the new .dot files on each of the hosts, we will use the script setupNowcastUser.home. In addition to installing the dot files, this script will create directories $HOME/tmp, $HOME/controlLocal/runtime which is the directory which will contain the runtime process control files. The different options -c and -f for setupNowcastUser.home result in different window manager configuarations based on host function.

As user nowcast on the $CONTROL_HOST

setupNowcastUser.home -c -v -p $PROJ_DIR

As user nowcast on the $FIELD_DISPLAY_HOST

ln -s /nfs/ncar/rap/projects/<$PROJECT>/.cshrc
source ~/.cshrc
setupNowcastUser.home -f -v -p $PROJ_DIR

As user nowcast on all remaining hosts

ln -s /nfs/ncar/rap/projects/<$PROJECT>/.cshrc
source ~/.cshrc
setupNowcastUser.home -v -p $PROJ_DIR

Back to top

ssh Authentication

The next step is to establish ssh authentication between all of the project machines. Because many of the Auto-Nowcast process control scripts rely on ssh, it is necessary to establish ssh logins which do not involve password prompting for user nowcast.

To set up ssh authentication you need to create two files: known_hosts and authorized_hosts and distribute these files to each machine within the Auto-Nowcast system. Start by picking one host, we'll call this <STARTING_HOST>:

Login to the <STARTING_HOST> host and create/edit the file $HOME/authorized_keys.tmp We will generate host keys on each host and copy them to this temporary working file.
In another xterm, login to each host sequentially (including the <STARTING_HOST>) and do the following:

If asked, answer yes to the question:
Are you sure you want to continue connecting (yes/no)?
Once you are logged on, run the command:
ssh-keygen -t dsa
(Use default filename and no passphrase.) This command will create the file $HOME/.ssh/id_dsa.pub
cat $HOME/.ssh/id_dsa.pub
copy and paste the output to the temporary file authorized_keys.tmp
For example, after running ssh-keygen on machine gauss the cat results will look like:
ssh-dss AAAAB3NzaC1kc3<...a long string of numbers and letters>== cnc@gauss
Logout of the remote host so that you are back at the <STARTING_HOST>.
In the temporary file authorized_keys.tmp, make another copy of the host key, modifying the machine name to be a fully qualified domain name. For example,
ssh-dss AAAAB3NzaC1kc3<...a long string of numbers and letters>== cnc@gauss.rap.ucar.edu

After completing the above steps for each machine, including the <STARTING_HOST>, set the proper file mode on the temporary authorized_keys.tmp file and copy it to all machines, including the <STARTING_HOST>:

chmod 600 $HOME/authorized_keys.tmp
scp $HOME/authorized_keys.tmp <machineName>:$HOME/.ssh/authorized_keys

Now that you have logged onto each machine from the <STARTING_HOST>, the <STARTING_HOST> should have a file $HOME/.ssh/known_hosts which contains host ids for each machine in the autoNowcast system.
Edit this file, making a copy of each line, and modifying the machine name to be a fully qualified domain name.
Finally, copy the modified known_hosts file to all other machines:

scp $HOME/.ssh/known_hosts <machineName>:$HOME/.ssh/known_hosts

If password authentication is established correctly, you now should be able to ssh from machine to machine without password prompting. Be sure to verify that this is the case or you will have problems later with process control which will be very difficult to diagnose.

Now that each of the project machines has its $HOME directory and cross mounts setup, you should be able to access the shared project space from any machine to make modifications to the project configuration. Note that there are aliases in the installed .cshrc file to facilitate moving around within the Auto-Nowcast configuration.

Back to top

Modifying Process Control Files

The next set of files which need project-specific modifications are the process control files which will eventually be installed in $HOME/controlLocal/runtime by the host_startup script and used for starting up and shutting down the Auto-Nowcast environment. As user nowcast on any host in the system, perform the following steps:

cd $CONTROL_DIR/params
Edit the file host_env.realtime and change the machine names to reflect the roles that were determined in Hardware Requirements & Purchasing.
cd $CONTROL_DIR/proc_list
Edit all of the files in this directory to indicate which processes should be run on each of the logical hosts. Each entry in a proc list must contain: application name, instance name, start script, kill script, and host on which the application runs. In general the start scripts and kill scripts (other than the general utilities snuff or snuff_inst) reside in one of the $PROJ_DIR/*Home/bin directories.
cd $CONTROL_DIR/crontab
Edit all of the files in this directory to indicate which cron jobs should be run on each of the logical hosts.

Back to top

Defining the Data Hierarchy

Now you must establish the data directory tree on each host. The required data directories will depend on the algorithms which will be running operationally for the field project. As user nowcast on any host in the system, perform the following steps:

cd $DATA_HOME/data_lists
Modify the files in this directory as necessary to specify the output datasets on each host.

The format of each entry in the the data list file is as follows:

<dataDirectoryPath> local | (link <linkDestination>) { <dataDirectoryParameterFile> }

Lines starting with a '#' are comment lines. These lines as well as blank lines are ingnored.

The <dataDirectoryPath> is the directory to be made. It specified relative to $RAP_DATA_DIR if the directory path does not begin with a '/'. Directory paths beginning with '/' are treated as absolute rather than being relative to $RAP_DATA_DIR.

The second entry is either local, indicating that the directory is local and a mkdir -p $RAP_DATA_DIR/<dataDirectoryPath> is to be executed, or link indicating that the directory is a linked to another directory using the ln -s <linkDestination> <dataDirectoryPath> command. NOTE!!(bug needs a fix): The <linkDestination> needs to be an absolute path.

The optional series of <dataDirectoryParameterFile> specifies any run-time parameter files from $DATA_HOME/params to be installed into the specified data directory. This presently includes static files like terrain data files as well as param files from data managing applications. WORK TO BE DONE!: The static Mdv terrain data files should be moved to a different directory than $DATA_HOME/params since they arent paramter files and the script which installs them should look for them in a "static_files" directory.

NOTE: These specific parameter files are for applications that operate on or manage data and are specific to the data set therefore they reside with the data. The naming convention for for the parameter file is _<Data Managing Application>.<function or identifying info>+<optional identifying information> Note that the string following + will not appear as part of the filename when it is copied to the data directory. It merely allows the user to distinguish one param file from another in $DATA_HOME/params when the application is then same. Following are examples of applications that manage or operate on data and and corresponding param files:

Applications that Manage or Operate on Data and Example Parameter Files
Application	Example Parameter Files Which Reside With the Data	Role of Application
Bdry2Symprod	_Bdry2Symprod.bdryDetect	Converts Spdb boundary products into format which can be rendered by CIDD
Tstorms2Symprod	_Tstorms2Symprod.35dbzStormsDetect	Converts Spdb Titan products into format which can be rendered by CIDD
Ltg2Symprod	_Ltg2Symprod.THOR	Converts Spdb lightning products into format which can be rendered by CIDD
Metar2Symprod	_Metar2Symprod.surface	Converts Spdb surface data products into format which can be rendered by CIDD
Janitor	_Janitor+LogDir _Janitor+FastDelete _Janitor+KeepOut	Recurses through data directory tree and compresses and deletes data.
DsFileDist	_DsFileDist+ncar2Mit _DsFileDist+MitDisplay2ncarRelay	Data distribution
DsSpdbServer	_DsSpdbServer.distrib+ncar2Mit _DsSpdbServer.search+soundings	Spdb data distribution Spdb data server
DsMdvServer	_DsMdvServer.static _DsMdvServer.res1K	Mdv data server

Once you have defined the necessary data directories and any associated parameter files, you are ready to create the data directories and install the run-time parameter files.

Login as root on each host and perform the following steps:

mkdir /d1/fieldData
chown nowcast /d1/fieldData
chgrp shared /d1/fieldData

As user nowcast:

cd $RAP_DIR
cvs co make_include
cvs co make_bin
cd $RAP_SHARED_DIR
cvs co libs/perl5
cd /nfs/ncar/rapShared/libs/perl5/src
make install_shared
niwot_mkdata to create data dirs on every host or host_mkdata if you want to do each host separately.

Check each host to see that the data directories were made properly and that the param files for the data directories were installed.

Back to top

Building Libraries & Applications

Create & Check in Project-Specific Make Files

Login as user nowcast on any project machine
cd $RAP_SHARED_DIR
cvs co libs/Makefile
cvs co apps/Makefile
cd /nfs/ncar/rapShared/libs
cp Makefile Makefile.<$PROJECT>
Change the contents of the project-specific make file Makefile.<$PROJECT> to include those libraries which you want to build for the field project.
cd /nfs/ncar/rapShared/apps
cp Makefile Makefile.<$PROJECT>
Change the contents of the project specific make file Makefile.<$PROJECT> to include those application trees which you want to build for the field project.
Check all of your project specific make files into cvs.

Checkout Source Code

cd $UTILS_HOME/bin
Change the contents of the file niwot_checkout to reflect the libraries and applications that were just added to the project-specific make files.
niwot_checkout
Create project-specific make files within each of the application src subdirectories. Setting up these application-level make files reduces the number of libraries that you will need to check out and compile for the system and reduces the number of build errors that you are likely to encounter.
Check all of your project specific make files into cvs.

Now you are ready to build all of the software used by the Auto-Nowcast system. Because of library dependencies, building the necessary software is often an iterative process. Once you start building the code, you might identify missing libraries or apps and have to edit the Makefile.<$PROJECT> file for the libraries, apps or modify the niwot_checkout script to include the software you need.

There are several scripts in the directory $UTILS_HOME/bin for building the software in various combinations -- libraries only, applications only, with or without a cvs checkout, etc. Each of these scripts relies on the underlying build script $NIWOT_DIR/niwot_build with varying command line options.

If you want to use the niowt build script to check out, build, and install your libraries and applications AND you have already created your project specific makefiles, try niwot_build -m -bld -d. This will

install project specific makefiles
build and install the source code
print debug messages

If you want to do these steps by yourself and check the results of each step before moving on, you can do the following:

Install Project-Specific Make Files

cd $RAP_SHARED_DIR
installMakefiles --project <$PROJECT>

Build the Software

Install includes:

cd $RAP_SHARED_DIR/incs
make install_shared

Build and install libraries:

cd $RAP_SHARED_DIR/libs
make install_shared_include
make opt install_shared

Build and install applications:

cd $RAP_SHARED_DIR/apps
cd tdrp; make install_shared
cd ../
make opt install_shared

Back to top

Modifying Parameter Files

The next step in configuring the Auto-Nowcast environment is a fairly tedious one. It requires a large number of parameter files and scripts to start the applications to be modified. This is a very critical step. Although the applications may seem to run successfully, errors which are very difficult to diagnose later on will indeed occur if the parameter files for each application are not modified correctly.

Radar Information files and radar_env.lookup

The radar_env.lookup is a file whose entries contain radar id/ host (of radar data) pairs and a key word used to identify the pair. The file has has three different uses:

It is used by the host_startup script to generate host process lists. The lookup key word can be used in the pre-processed process lists in $CONTROL_DIR/proc_list with the syntax of preceding and following the key word by '?' symbol. At runtime, the key word will be replaced by the Radar Id. in the runtime process list which is placed in $HOME/controlLocal/runtime.
It is used by application start scripts find radar id, or radar Id/radar host pairs. This enables parameter files to be generic.
It is used by SysViewExpand to substitute radar id and radar data hosts into generic Sysview diagrams.

Modify relavant param files and start scripts:

cd $CONTROL_DIR/params.
Modify $CONTROL_DIR/params/radar_env.lookup. Enter all radar id/radar data host pairs and a key word for each pair.
cd $RADAR_HOME/bin.
Modify all start scripts that use radar_env.lookup. Generally these will have the suffix *. radarLookup. Make sure that the proper keys are being set and used in the niwot perl library lookup commands.
cd $CIDD_HOME/bin.
Similarly modify all start scripts that use radar_env.lookup.

Radar information param files are located in $RADAR_HOME/params. They set environment variables specific to the radar including radar id, latitude, longitude, altitude and other variables. These files are sourced by scripts which start applications that require radar specific information. This enables us to keep the parameter files for such applications generic and reduce the number of paramter files that must be maintained for a project using more than one radar data stream.

cd $RADAR_HOME/params
For each radar being used in your project, create a radar.<radar_id> param file. Note: The suffix of a radar info filename should match the <radar_id> of a corresponding entry in radar_env.lookup. The radar start scripts make the assumption that they will be the same. Also, There should be a one to one correspondence between each entry in radar_env.lookup and the radar information files created in $RADAR_HOME/params.

Configure Application Parameters

The application parameters which need to be modified vary depending on which applications are being used for a particualar project. Perhaps the best way to approach this task is to methodically go through each of the parameter files for each application in the autoNowcaster one by one, making sure to take a look at each parameter to determine its role. In many cases a parameter will need to be tuned by the scientists; in other cases a configuration change (like change in domain size, grid resolution, adding another algorithm or deleting one) will necessitate a change in the parameters. Note: In general the applications in the autoNowcaster contain a -print_params option. The command you would use to print a param file with default parameter settings is <application name> -print_params >& <param file name>

Build Project-Specific Map Database

Project maps are used by three different displays: CIDD, TITAN, and RDI. RAP's map database resides in http://www.rap.ucar.edu/maps. Any maps required by the Auto-Nowcast field project should be copied from the RAP database and installed in $PROJ_DIR/maps. Other maps specific to the new field project will need to be fetched from outside sources or digitized.

Build Project-Specific CIDD Colorscale Database

The colorscale database for CIDD is $CIDD_HOME/colorscales. RAP's colorscale database resides in http://www.rap.ucar.edu/colorscales. Any colorscales required for CIDD should be copied from the RAP database and installed in $CIDD_HOME/colorscales. Colorscales specific to the new field project might have to be created if not available in RAP's colorscale database.

Configure Data Distribution

Three applications which are useful for data distribution are DsSpdbServer for SPDB data, and the DsFileDist / DsFCopyServer pair for MDV data. The parameter files for these servers should ultimately reside in the directory containing the data to which they pertain. In configuring the distribution, create the files in $DATA_HOME/params, add them to the proper data list in $DATA_HOME/data_lists and they will be installed in the proper data directories with host_mkdata or niwot_mkdata commands

Configure Data Compression and Scrubbing

The Janitor is an application which continually recurses through the data tree and is typically used for compressing data files to save disk space and scrubbing those files determined to be old( a configurable parameter for each data set) and not in a time window specified for saving data by the $DATA_HOME/params/events_list. The run-time behavior of the Janitor is determined by the parameter files that reside in the data tree. Typically the top level _Janitor param file is very complete and specifies the entire set of Janitor application parameters. Lower level _Janitor files should contain only those few parameters which differ from and override the upper level parameter settings. In configuring _Janitor param files, create the files in $DATA_HOME/params, add them to the proper data list in $DATA_HOME/data_lists and they will be installed in the proper data directories with host_mkdata or niwot_mkdata commands.

Configure Data Servers To Run On Each Host

The four data service applications which typically run on all machines in the Auto-Nowcast system are:

DsServerMgr
DataMapper
Scout
Janitor

These applications typically appear in the process list file $CONTROL_DIR/proc_list/EVERY_HOST.always so you shouldn't have to make any modifications to enable these. You may, however, need to modify this approach if your system configuration includes an exposed data relay host for data distribution where running DsServerMgr could be considered a security risk.

If you are distributing data to/from an exposed host outside the firewall, you may want to run the server manager in secure mode using: DsServerMgr -secure or avoid running DsServerMgr altogether to minimize the risk of automatically starting up unexpected server processes. Instead of running the server manager you could use the exposed host strictly as a data relay machine and limit the use of that machine to the following processes maintained by auto_restart :

DsFileDist
DsFCopyServer
DsSpdbServer
Janitor
auto_restart

If you take this approach of explicitly running the Ds-applications via the auto restarter, then you might need to modify the process list for EVERY_HOST.always and take out DsServerMgr, DataMapper, Scout, and Janitor and add them to the process lists on a host by host basis.

Setup Nightly Status Email Lists

cd $CONTROL_DIR/params
Change the email addresses in the file email.list. These email addresses will receive two nightly emails:

24-hour summary of the process control status
machine status on uptime, disk space, and memory usage

cd $CONTROL_DIR/crontab. Edit the CONTROL_HOST crontab file and add the following:
#
# Run the reliability statistics once a day. This should be
# run on the CONTROL HOST (atNcar), just after midnight.
#
01 00 * * * csh -c "niwot_status >& /dev/null"
#
# Run the host machine status check once a day
#
59 23 * * * csh -c "query_hardware_status >&/dev/null"

Back to top

Setting up a System Monitor

At this point everything should be in place for developing a complete SysView diagram of the system configuration. If the SysView diagram has not already been developed for planning purposes, it is a very good idea to put one together for system documentation and realtime monitoring.

As user nowcast:

cd $SYSVIEW_HOME/bin/sysview2
java -jar sysview2.jar
Create diagrams for each $HOST and save these diagrams in $SYSVIEW_HOME/diagrams.

Note: You can use nowcast environment variables in your application hosts and data paths. You must write your environment variables with added parenthesis as displayed in the following example: $INGEST_HOST must be written $(INGEST_HOST). You can also substitute lookup key words for radar ids from the $CONTROL_DIR/params/radar_env.lookup. You must write these variables with added ? as displayed in the following example: key word radar1 must be written ?radar1?. All these variables will be resolved by a program called SysViewExpand which will be executed when you run SysView with the command run_Sysview.

Also note that SysView relies on the DataMapper application to get information in realtime about the datasets on various hosts. Therefore you must be running the DataMapper on hosts for which you want information about the local data.

Back to top

Ready, Set, Go...

By now the Auto-Nowcast system should be ready to begin testing the startup and shutdown procedures. This step will let you know if you have properly set up the start scripts, parameter files, process lists, data directories, etc.

A prudent approach to testing components of the system would be to manually start each application as you complete configuration of its parameter file. If you are feeling more daring, you can start up processes along with the auto-restart services.

To start up a single machine in the system:

login as user nowcast on the machine of interest
host_startup realtime

If you want to be brave and kick off the whole system at once:

login as user nowcast on the $CONTROL_HOST
niwot_startup realtime

When starting the whole system via niwot_startup, you should see each of the project hosts reporting back to the console window of the $CONTROL_HOST as they startup.

The corresponding commands for shutting down the Auto-Nowcast system are:

host_shutdown and niwot_shutdown

Some tools you can use to determine if your hosts started up properly are:

print_procmap -up -hb -c 5
This will tell you all of the processes which are registering with the process mapper called procmap . Make sure that all of the processes which should be running are showing up in this list.
Use tail -f $CONTROL_LOCAL_DIR/runtime/auto_restart.log
to see which process keep restarting on the host. This indicates that there is a problem with the start script, the param file for the application, perhaps the instance name in the process list doesnt match the one in the start script, or the application executable is missing.
Execute run_Sysview (assuming your SysView diagrams are set up properly).

Back to top

Freezing the Source Code

At some point once you are satisfied with the basic workings of the Auto-Nowcast applications, you will have to do a code freeze. This amounts to doing one final CVS checkout and build of all the software as described in Building Libraries & Applications. There's probably a lot that could be said here about both the importance and the potential pitfalls of a code freeze. Suffice it to say:

Don't wait until the day before you start Packaging up the Field Project before you do a code freeze.
Be circumspect when accepting any new code after the code freeze.

IMPORTANT: Once you are certain that no further updates or changes will be incorporated into the source code, mirror the libraries and applications (source and binaries) from the machine which is hosting the Shared Project Space to the $FIELD_SUPPORT_HOST. Maintaining a copy of the frozen source code which is operating in the field will allow you to debug and possibly correct software bugs which arise during operations without doing a full software upgrade during the field season.

Back to top

Packaging up the Field Project

Switch to the Field Host Names and IP Addresses

If you have been doing a setup at RAP on the actual field machines that are to be shipped out into the field, only the domain name and IP addresses of the machines will change. The hostnames should not change. In this case you will only need to update those project files in which domain names were specified:

Change the system domain name and IP address.
Reconfigure the second ethernet card of the $INGEST_HOST for the new IP address.
Change the file $PROJ_DIR/.shosts to the new domain name.

If instead of setting up the Auto-Nowcast system on the actual field machines, you have been doing a "mock-up" using various machines around RAP, all of the explicit hostnames in the project files will have to be changed in addition to the domain name and IP address changes made above.

In most cases, logical hostnames are used via UNIX environment variables. However, in some project files explicit hostnames must be specified. Refer to Modifying Parameter Files for the most complete explaination of the project files which should be examined. A quick list of the files in which explicit hostnames might be used include:

$HOME/.fvwmrc.*
$CONTROL_DIR/params/host_env.*

Do Final CVS Checkin of the Project

cd $PROJ_DIR
cvs -n update
cvs commit

Burn a CD-ROM of the Project

On at least one occasion, a PC has been shipped from RAP and has gone missing in the mail. This is particularly awkward when the machine in question has all the frozen source code and/or executable files on it. It is a good idea to write all the executables, the binary libraries, the applications source tree and the library source tree to a CD-ROM for backup purposes.

Make Final System Changes

BE SURE to remove any local cross mounted file systems!
Switch from DNS lookup to host table lookup. The system's administrators don't necessarily like to do this, but have 'em do it anyway!!!

Back to top

At the Field Site

Make sure ssh authentication still works with the new domain name. See Modifying & Installing .dot Files for a brief explaination of ssh authentication.
Make sure the cross mounts are set up properly for the Shared Project Space.
Make sure that the NTP (network time protocol) service is working properly for all machines.
Change the nowcast user password to a project-specific external password. Do NOT use the internal password for user nowcast on the project machines out in the field.
On the LDM servers, change the pqact.conf to point to the field project radars!
If you are maintaining previous season data, you may need to merge the previous data into the current data directory structure.

Back to top

Monitoring

Daily Checklist
Weekly Checklist

Daily Checklist

On a daily basis, the operator should perform the following tasks:

Check disk and memory usage
Check process status from the previous day
Check current process status
Check current data status
Monitor the system
Check email from cron jobs
Check data archiving mechanism

Check disk and memory usage

The various Auto-Nowcast Hosts must have sufficient disk and memory available to run the Auto-Nowcast Environment processes and store data files.

On a daily basis, a utility called niwot_status is run on $CONTROL_HOST to check the disk usage, uptime, and memory statistics on all the Auto-Nowcast Hosts. The results are emailed to addresses in $CONTROL_DIR/params/email.list. The status script is run nightly as a crontab job. If you want to receive daily mail regarding the status of the Auto-Nowcast Hosts, add your email address to the email list file.

For any Auto-Nowcast Host you can check the disk usage and memory usage at any time. The UNIX command df is used for checking disk usage. To check memory usage, either run top or cat /proc/meminfo.

Check process status from the previous day

On a daily basis, a utility called query_hardware_status is run to check process status for the day. The results are emailed to addresses in $CONTROL_DIR/params/stats_email.list. The status script is run nightly as a crontab job. If you want to receive daily mail regarding the daily process status, add your email address to the email list file.

The process status files are also date/time stamped and stored on the $CONTROL_HOST. To check the process status from a particular day, examine the corresponding file in $RAP_DATA_DIR/$PROJECT/other/reliability_stats/YYYYMMDD_procmap.stats. Keep in mind that the process status files get scrubbed by the Janitor along with other files and datasets in the Auto-Nowcast Environment.

Check current process status

SysView monitors the status of the Auto-Nowcast Environment datasets and processes. The User's Manual for SysView can be downloaded in *.pdf format. The procmap is also a useful utility to monitor the status of a given host in the Auto-Nowcast Environment.

If you suspect problems with any of the processes, see the Process Control FAQ for suggestions on diagnosing and correcting the problems.

Check current data status

SysView monitors the status of the Auto-Nowcast Environment datasets and processes. SysView can be downloaded in *.pdf format.

If you suspect problems with any of the datasets, see the Data Ingest & Management FAQ for suggestions on diagnosing and correcting the problems.

Check email from cron jobs

To check the results from any cron jobs that are running at your installation, read the email for the nowcast user on the $CONTROL_HOST. Most cron jobs are run near midnight, so checking email every morning is a good practice.

Check data archiving mechanism

Back to top

Weekly Checklist

On a weekly basis, the operator should perform the following tasks:

Update the events list
Backup data to tape

Update the events list

Update the meterologically interesting case dates in the file $DATA_HOME/params/events.list. The cases in this file will not be scrubbed off the disk by the Janitor off the disk.

Backup data to tape or utilize other archiving mechanism

Back to top

Data Architecture

RAL Internal Data Formats
RAL Data Directory Structure

Data Sources

The availability of various data sources at a particular Auto-Nowcast Environment installation will dictate the number and types of algorithms that can be run. The minimum data required for the Auto-Nowcast Environment is NEXRAD radar data. A typical configuration for the Auto-Nowcast Environment might include the folowing input data streams:

NEXRAD radar data
Satellite data
Soundings
Surface observations

Although the mechanism for acquiring the various external data sources may vary from one Auto-Nowcast Environment installation to another, the data must conform to data formats know to the applications in the Auto-Nowcast Environment.

RAL Internal Data Formats

MDV (or Meteorogical Data Volume): This is 3D or 2D gridded data. For example radar data volumes are stored in MDV format. Terrain data is stored in Mdv format.
SPDB (or Symbolic Product Data Base): This is a format for point data. For example, Lightning data and surface observations are stored in SPDB format. Titan storms can be converted to SPDB format.
FMQ (or File Message Queue): This is not so much of a data type but a fixed size file which which applications can use to exchange data of any type or messages. Some of our applications read and write data to and from these circular queues and therefore they are managed with our data.
titan storm track: This is the output format of storms generated by TITAN.
DsRadar: This is RALs format for radar beam data.

Back to top

RAL Data Directory Structure

The head of data heirarchy at RAP is $RAP_DATA_DIR. This is an environment variable which is set in the project .cshrc. The data servers which read and write data for the applications will assume all data paths that do not begin with a '/' are relative to $RAP_DATA_DIR and all data paths that begin with a '/' are absolute paths.

Under $RAP_DATA_DIR, data is divided by type. For example, typical subdirs of $RAP_DATA_DIR might include:

fmq
mdv
raw
spdb
titan

fmq directory and file structure:

The fmq directory contains the File Message Queues which have suffix .buf and corresponding .stat file. Typical FMQ files have the following paths relative to $RAP_DATA_DIR:

fmq/<fmq name&GT.buf
fmq/<fmq name&GT.stat

mdv directory and file structure:

mdv subdirectories are further divided by dataset name, and date. The dataset filename is the valid time of the data in hours, minutes and seconds. Following is the structure of an MDV dataset path relative to $RAP_DATA_DIR:

mdv/<data set name>/yyyymmdd/hhmmss.mdv

raw directory and file structure:

raw or sometimes called other subdirectories are further divided by dataset name but the rest of the directory structure depends on the raw data andits format.

spdb directory and file structure:

spdb subdirectories are further divided by dataset name and these directories contain the database files. The files are separated by day with basename yyyymmdd and for each day there is a *.data file and a *.index file. The database files for one day have the following paths relative to $RAP_DATA_DIR:

spdb/<data set name>/yyyymmdd.data and
spdb/<data set name>/yyyymmdd.index.

titan directory and file structure:

titan subdirectories are further divided by dataset name and these directories contain the database files. The files are separated by day with basename yyyymmdd and for each day there is a *.sd5 file, a *.sh5 file, *.td5 file, and a *.th5 file. The database files for one day have the following paths relative to $RAP_DATA_DIR:

titan/<data set name>/yyyymmdd.sd5
titan/<data set name>/yyyymmdd.sh5
titan/<data set name>/yyyymmdd.td5
titan/<data set name>/yyyymmdd.th5

All data files in the Auto-Nowcast Environment are timestamped based on UTC, a.k.a, GMT time.

Back to top

Operations Guide

Hardware

Software

Data Service

Process Control

Utilities

Environment Configuration

The Project Directory: Application Param Files and Start Scripts, Host Config, and Data Config.

Environment Variables

Network Communications

Data Display

Cidd

Rview

Analysis Algorithms

The Nowcast Engine

Thunderstorm Identification, Tracking, and Features

Advection

Variational Doppler Radar Analysis (VDRAS)

Satellite Algorithms

Data Ingest and Quality Control

Installation

Using This Configuration Guide

Hardware Requirements & Purchasing

UNIX System Setup

The Nowcast User Account

Shared Project Space

Creating a new Auto-Nowcast Project in CVS

Checking out an Existing Auto-Nowcast Project from CVS

Modifying & Installing .dot Files

ssh Authentication

Modifying Process Control Files

Defining the Data Hierarchy

Building Libraries & Applications

Modifying Parameter Files

Setting up a System Monitor

Ready, Set, Go...

Freezing the Source Code

Packaging up the Field Project

At the Field Site

Monitoring

Daily Checklist

Weekly Checklist

Data Architecture

Data Sources

RAL Internal Data Formats

RAL Data Directory Structure