This document describes the THOR-North network setup at Lincoln Labs as I understand it. It will be updated after the networking is installed to reflect the final configuration. Please excuse any mis-uses of networking terminology. I believe I understand the ideas, but I definitely don't know the networking details.
Table of Contents:
The following diagram shows the preliminary network configuration as drawn by Bob Boldi at Lincoln Labs:
The portion of the system labelled "Collaborators' Network" is the RAP controlled network. This network consists of all of the machines we are sending to Lincoln. This network will have a firewall around it that matches the firewall we have in RAP. All of the machines will be behind this firewall except for zenith, which will be partially exposed to allow data to flow from RAP to the system at Lincoln.
The "Filters/Firewall" object is a Solaris box provided by Lincoln to bridge the gap between their network and ours. I will call this machine the "firewall Solaris box" in the rest of this document. This machine has very limited functionality, but provides us access to the data provided by Lincoln and to the "slow lane" internet communication path for non-time critical datasets.
The firewall Solaris box will have a disk which is cross-mounted on appropriate Lincoln machines inside of their firewall and on specified RAP machine(s) on our side of the firewall. This cross-mounted disk will be used to provide the promised Lincoln MDV datasets to the RAP system. Lincoln will write the datasets to this disk in specified RAP-style directories and RAP will read the datasets from this disk and distribute them as appropriate. There is currently no limit on the number of RAP machines which would be able to cross-mount this data disk, but one idea would be to cross-mount the disk on one RAP NCWF machine and on one RAP AutoNowcast machine and distribute the data within each subsystem as appropriate. For this first go-round, the VIL mosaics and raw satellite data will be provided using this mechanism.
The raw RUC datasets (in GRIB format as received from NCEP) will also be provided to RAP using the firewall Solaris box cross-mounted disk. Again, these will be put in a specified directory for RAP to pick up and distribute as needed. To begin with, we will just get the 40km hybrid 3-hour forecast datasets through this mechanism since these are the ones used by the NCWF growth forecast algorithm. We can add the dataset needed by the AutoNowcaster (the 40km pressure 0-hour, I believe) once the NCWF data is running smoothly. We need to get the RUC data through Lincoln rather than retrieving it directly from NCEP ourselves to reduce the data bandwidth going into Lincoln. They are already retrieving most of these datasets for their own use anyway.
The lighting data will be retrieved directly from the Lincoln lightning data server as has been discussed in the past. Lincoln will open up their firewall on this server to allow us to make a TCP/IP connection and retrieve the data. This should work because the lightning server machine is outside of their main Lincoln firewall and the RAP network will be set up like the main RAP network in Boulder which allows machines within the firewall to initiate TCP/IP connections with machines outside of the firewall.
We did not discuss how we will get the LDM data because I don't know enough about the LDM and about the AutoNowcaster data needs to address this issue effectively myself. When these other datasets are flowing successfully it should be easier to address this need and any other unaddressed needs of the AutoNowcaster.
Datasets sent from RAP to the RAP system in Lincoln will use the path previously discussed. These datasets will be pushed from RAP to zenith over the T1 line. zenith will then push the datasets to the appropriate machine within the RAP network.
Datasets will be distributed to RAP via two different connections, known as the "fast-lane" and "slow-lane" connections. The fast-lane connection is provided by the purchased T1 line and is used for time-critical datasets. The slow-lane connection is provided Lincoln's regular connection to the internet and is used for datasets that are only needed in near-realtime.
The fast-lane connection uses the purchased T1 line. Since we are using the firewall configuration used in RAP's main network, all of the RAP machines at Lincoln will be able to push data directly to the exposed host at RAP (moonbow) via this line. So, data distribution to RAP using the fast-lane connection will not have to go through zenith in this configuration.
The slow-lane connection goes through the Lincoln main network to get the their regular internet connection. Because of this, we will need to make an effort to send as much data as possible over the fast-lane connection, without overloading that line and slowing down the time-critical datasets.
In order to push data using this mechanism, Bob Boldi will run a copy of DsFileDist and a copy of DsSpdbServer on the firewall Solaris box. These processes will look at a directory that is cross-mounted on one or more RAP machines and will distribute the datasets based on the parameter files we put in those directories. Since this box has access to the Lincoln main network, data can be pushed by processes on this machine "directly" to moonbow. When I say that the data is pushed "directly" to moonbow, I mean that the DsFileDist and DsSpdbServer parameter files specify moonbow as the destination, but the data itself will travel through the Lincoln main network, out the firewall and onto the internet to get to moonbow.
So, for MDV datasets, we just write the data to the cross-mounted disk in the appropriate place and install DsFileDist parameter files as needed in these directories.
I just now realized that we'll have to double-check on whether we'll be able to distribute SPDB datasets through the slow-lane connection. The problem is that the SPDB data distribution requires us to make a TCP/IP connection to the DsSpdbServer process on the distribution machine and send the data that way. This may not be allowed. But it's probably not a big problem to always send the SPDB datasets through the fast-lane connection because most of them are considerably smaller than the MDV datasets.
Author: Nancy Rehak