U.S. Geological Survey

USGS participation in NSDI Formal metadata
Metadata and the National Hydrography Dataset (NHD)

How to convert NHD metadata into indented text compatible with mp

Problem:

Metadata provided with data from the U.S. National Hydrography Dataset contain many elements appropriate for use in the National Spatial Data Infrastructure (NSDI). However the format of these files is not directly compatible with software created for the NSDI.

The NHD high-resolution data packages are distributed from http://edcw2ks56.cr.usgs.gov/fod/hftp.asp Each package includes a directory labeled metadata containing many files whose names end in .met. Files with the same root name (8-digit hydrologic unit code) can be merged to form a reasonable metadata record.

Example:

Data for the Truckee hydrologic unit (code 16050102) are at

    ftp://edc.usgs.gov/pub/data/nhd/fod_cache/high_resolution/arc/16050102.tgz
When unpacked, a directory 16050102 is created and within it, a directory metadata containing, at this writing,
  Size Date created File name
-----------------------------------
 12855 Jan 21 10:25 16050101001.met
   477 Jan 21 10:25 16050101002.met
   166 Jan 21 10:25 16050101003.met
   126 Jan 21 10:25 16050101004.met
   400 Jan 21 10:25 16050101005.met
 12849 Jan 21 10:25 16050102001.met
   126 Jan 21 10:25 16050102002.met
   704 Jan 21 10:25 16050102003.met
   400 Jan 21 10:25 16050102004.met
 12859 Jan 21 10:25 16050201001.met
   714 Jan 21 10:25 16050201002.met
   400 Jan 21 10:25 16050201003.met
  4763 Jan 21 10:25 nhd.met
Although the data in this package pertain only to the hydrologic unit 16050102, metadata are included for units 16050101 and 16050201 as well. Those can be ignored.

The task at hand is to re-format and merge the information provided in the files pertaining to 16050102 so that they can be understood by software that handles metadata. Here the files that need to be merged and reformatted are


    16050102001.met
    16050102002.met
    16050102003.met
    16050102004.met
    nhd.met

Method with high-resolution data example

Each specific file is merged, in turn, with a "boilerplate" file using a script merge.tcl which is written in Tcl and interpreted by the Tcl shell ("tclsh"). This script utilizes the Tcl extension mq to read and edit the metadata.

  1. Install Tcl/Tk on the system by following directions at
        http://aspn.activestate.com/ASPN/Downloads/ActiveTcl
    
    (follow the link labeled "Download" under Product Information on the left side)

    These instructions assume you installed Tcl/Tk into the folder C:\Program Files\Tcl.

  2. Install mq in the Tcl/Tk directory

    1. Download and run the self-extracting package file
          http://geology.usgs.gov/tools/metadata/all_win.exe
      
      These instructions assume you installed in C:\USGS
    2. Copy \USGS\tools\bin\mq26.dll and paste it into C:\Program Files\Tcl\lib
    3. Make directory C:\Program Files\Tcl\lib\mq
    4. Copy C:\USGS\tools\bin\pkgIndex.tcl and paste it into C:\Program Files\Tcl\lib\mq
      or download pkgIndex.tcl.win, save it there but rename it as pkgIndex.tcl
    5. Verify installation by running wish (Start->Programs->Tcl->Wish) in the window containing %, type package require mq. The response should be a version number like 2.6.0.
  3. You'll need to have a number of specific files from me. You can follow their links throughout this page, or you can download all of the NHD-specific files as tools.zip. Extract them into the directory that contains the NHD metadata for this HUC.
  4. Replace the boilerplate metadata nhd.met with one that has been edited to conform to the structure and format of FGDC metadata. This edited boilerplate file is supplied with the name nhd-hr.met (for high-resolution data); you need to make a copy of this file and name it nhd0.met.
  5. Place copies of the NHD metadata configuration file nhd.cfg and the NHD metadata extension file nhd.ext in the same directory as contains the NHD metadata.
  6. Place a copy of merge.tcl in the directory that contains the NHD metadata as well. This script could be placed anywhere, but if you put it in a different directory, you'll need to specify its full path name when you run the Tcl interpreter.
  7. Open a command prompt window.
  8. Make sure the command processor can find Tcl.
    1. First, just type tclsh at the command prompt.
    2. If you get a % prompt, then the command procesor can find Tcl, so you don't need to do the next step.
    3. If you get a message like this:
        'tclsh' is not recognized as an internal or external command,
        operable program or batch file.
      
      then you need to set the execution path to include the Tcl interpreter. Type
          PATH %PATH%;C:\Program Files\Tcl\bin
      at the prompt. Then try typing tclsh again.
  9. Set the current directory to the one that contains the NHD metadata. Do this by typing something like cd \gis\NHD\16050102\metadata at the prompt. But instead of \gis\NHD\16050102\metadata (which I've made up as an example), use the real directory on your system where the NHD metadata for this HUC are.
  10. For each of the HUC-specific metadata files, run the Tcl interpreter, providing it with the name of the merge script (merge.tcl) and the name of one of the HUC-specific metadata files.
    In this example, you would run this script four times, once for each of the files
    
        16050102001.met
        16050102002.met
        16050102003.met
        16050102004.met
    
    Each time you run the script, you'll want to use the results as the template for the next run. So rename the output to match the template name:
    
        copy  nhd-hr.met  nhd0.met
    
        tclsh  merge.tcl  16050102001.met
        move  merged-16050102001.met  nhd0.met
    
        tclsh  merge.tcl  16050102002.met
        move  merged-16050102002.met  nhd0.met
    
        tclsh  merge.tcl  16050102003.met
        move  merged-16050102003.met  nhd0.met
    
        tclsh  merge.tcl  16050102004.met
        move  merged-16050102004.met  nhd0.met
    
    Finally, rename the file nhd0.met, which now contains all of the merged metadata, so that its name is the number of your HUC, with .met as the extension:
    
        move nhd0.met 16050102.met
    
    This looks like a pain, so I've written a Tcl script mkbat.tcl that will create a batch file containing all of the commands you need for this step. From the command prompt, run tclsh mkbat.tcl. Then inspect the file doit.sh, which should have been created by this script. If all went well, all you have to type now is doit.
  11. The resulting file 16050102.met contains the information applicable to all NHD high-resolution metadata (from nhd-hr.met) as well as the specific information for the hydrologic unit 16050102 and all of the revisions to those data. This file can be brought into ArcCatalog or other metadata-processing software such as mp or Tkme.

What you need

Technical contact:
  Peter N. Schweitzer
  Mail Stop 954 National Center
  U.S. Geological Survey
  12201 Sunrise Valley Drive
  Reston VA 20192

  703-648-6533
  703-648-6252 FAX
  pschweitzer@usgs.gov


Privacy statement -- General disclaimer
This page is <http://geology.usgs.gov/tools/metadata/nhd/>
Maintained by Peter Schweitzer
Last updated Wednesday, 14-Oct-2009 14:49:00 EDT