Obtaining Jason-2 IGDRs from NOAA/CLASS
John L. Lillibridge - NOAA/OSTM Project Scientist
2009-02-13

 

When a new altimetry user needs to get started with Jason-2 datasets, they typically want all previous data as well as access to new data going forward. In this tutorial we’ll assume that the user is interested in the Level-2 Interim Geophysical Data Records (IGDRs) - past, present, and future. The following steps lead you through obtaining all existing data, setting up a subscription for incoming data, and how to manage the download and filing of the IGDRs on the local workstation.

Step 1 - Obtaining previous data

After obtaining a CLASS user ID, login to the CLASS website http://www.class.noaa.gov and click on the ‘Login’ link in the upper left-hand corner of the main CLASS web page:

CLASS web site with Login area hightlighted

Enter the Username and Password:

CLASS web site with the UserID entry page indicating where to enter UserID and Password

Note that the ‘Login’ link has changed to ‘Logout’, indicating a successful login.
Select ‘J2-XGDR’ from the top-center pull-down menu, and click on the ‘Go’ button:

CLASS web site showing the change in link from login to logout and search selections for data highlighted.

This takes you to the main J2-XGDR search page.

Select ‘Jason-2 Interim GDR in NetCDF Format’ for the IGDRs:

CLASS web site Serch selection page for Jason-2 XGDR

To change the search criteria from a Date range to a Cycle range, click on the ‘Cycle’ radio button.
Note that there are 254 IGDRs per 10-day cycle, and the maximum number of items per order is 500.
To simplify matters, place a single order for each cycle as shown below, using Cycle-15 as an example:

CLASS web site showing the change in selection criteria

Click on the ‘Order Now’ button, to go immediately to the Shopping Cart screen:

CLASS Shopping Cart screen

Click on the ‘Place Order’ button and you will receive an Order Confirmation. Note the number shown highlighted below, as you can use it later to download this cycle’s worth of IGDRs:

CLASS web site with the Order confirmation number highlighted.

When you receive e-mail notification that the order is complete, you can ftp all the IGDRs with a client like lftp, using the CLASS ftp site and the order number shown above:

lftp mirror ftp.class.ngdc.noaa.gov/10058624 (for this example using Cycle-15 IGDRs).

Continue to place orders for each of the cycle’s of data that are desired. There will be a separate order number/download directory for each order. Managing the ftp sessions from CLASS to your local workstation, including filing into cyclic subdirectories, is discussed further in Step 3 below. Once all orders have been completed, click on the Logout link in the upper left and/or close the browser window to the CLASS web site.

Step 2 - Setting up subscriptions for new data

The next step is to set up a subscription so that all new IGDRs received by CLASS are automatically queued for download by the user. A new user must first request subscription services for their account, by sending an e-mail to the CLASS help desk: info@class.noaa.gov.

Once this request has been processed, as ‘Subscriptions’ menu item will appear on the left hand side of the main CLASS window, after the user has logged in (see main screen pictures above).

Begin managing subscriptions by clicking on the ‘Subscription’ menu item. Again select ‘J2-XGDR’ from the pull-down menu, and click on the ‘Add New’ button:

CLASS web site screen to manage subscriptions.

Select the ‘Interim GDR in NetCDF Format’ product, and click on the ‘Delivery Options >>’ button:

CLASS web site with selections available for the subscription.

If you prefer not to receive e-mail notifications for your subscription, select the two ‘No’ radio buttons. Click on the ‘Save’ button to save your preferences:

CLASS web site witht he option to receive email.

Your subscription for IGDRs, in this example, is now enabled. Typically a group of ~13 IGDRs are received daily around 1900 UTC. Note that setting up a subscription will NOT download previous files, only those coming in after the subscription is enabled. You must follow Step 1 above in order to obtain older datasets. If you return to modify your subscription later, you will see a screen like the following. Click on the link in the subscription name to make modifications to an existing subscription:

CLASS web site screen showing current subscriptions.

Step 3 - Managing FTP downloads and filing of datasets

Once online (shopping cart) orders have been placed, and/or subscription orders have been filled, the requested datasets will be made available from CLASS ftp servers. Online orders are available from the CLASS server at NGDC, in a subdirectory with the name of the order number. For example:

ftp://ftp.class.ngdc.noaa.gov/########

Subscription orders are available from the CLASS server at NSOF, in the subscription directory, in a subdirectory with the user’s Username. For example:

ftp://ftp.class.noaa.gov/sub/User.Name

Presently files provided by subscription have a CLASS order number prefix which most users will want to strip off. A future version of the CLASS system will likely eliminate this annoyance, but it isn’t difficult to remove the unwanted prefix. Additionally, all Level-2 Jason-2 datasets (OGDR/IGDR/GDR) are accompanied by XML-metadata files. In the next major CLASS release it will be possible to order data alone, XML-metadata alone, or both. But for now the orders will routinely supply both the file and it’s accompanying metadata. The use of smart ftp clients like ‘lftp’ and a few simple Unix shell scripts can easily deal with both the prefix and metadata issues, as well as routinely filing IGDRs into cyclic directories.

The following lftp command file, kindly provided by Remko Scharroo, downloads subscription based files from the CLASS NSOF ftp server, avoids downloading the XML-metadata, and places all of the files in a local directory ‘class’:

open ftp.class.noaa.gov/sub
mirror --parallel=5 -x xml -e -L User.Name class

 

The following Unix bash-shell script sorts these downloaded files into cyclic directories, and removes the CLASS order number prefix:

#!/bin/bash
#############################################
# The purpose of the script is to sort all newly arrived
# IGDR/GDR files into separate directories per cycle.
# The sorted files are hard links to the unsorted files.
#
# The source directory is given as argument to the script
# The cycle directories are created as {i,}gdr/c???
#
# 27-Aug-2008 - Created by Remko Scharroo
#############################################

alldirs=

# Repeat this process for different file types

for type in IPN GPN ; do
    case $type in
    IPN) dir=igdr ;;
    GPN) dir=gdr ;;
    *)   dir=$type ;;
    esac

# List the new files

    newdir=`ls -dt $dir/c??? | head -1`
    if [[ ${#newdir} > 0 ]] ; then
        files=(`find $1 -name "*JA2_${type}_*" -newer $newdir -print`)
    else
        files=(`find $1 -name "*JA2_${type}_*" -print`)
    fi
    echo "$dir: ${#files[@]} file(s) to sort ..."

# Store all listed files in the appropriate directories
# Example file name:
# class/L0254093.JA2_IPN_2PTP005_137_20080826_003922_20080826_013535

    n=0
    for file in ${files[@]} ; do
        destfile=${file/#*JA2/JA2}
        destdir=$dir/c${destfile:12:3}
        mkdir -p $destdir
        ln -f $file $destdir/$destfile
        alldirs=`ls -d $alldirs $destdir | sort -u`
    done
done