NOAA/National Weather Service's Warning Decision Training BranchUnited States Department of Commerce
Commerce Learning Center

WES-2 Bridge News

The current WES-2 Bridge release is 17.3.1 for RedHat 7. A 19.x build to support Hazard Services is under development.
Training Events Planning Calendar
Residence/Virtual Courses
Web-Based Training Release Dates

Warning Decision Training Division

Office of Chief Learning Officer

AWIPS-2 Archiver

Weather.gov > Warning Decision Training Division > Tools > WES-2 Bridge > AWIPS-2 Archiver

Enter Your Main Content Here


Every site needs to perform some local configuration for the AWIPS-2 archiver to include their local datasets which could not be archived otherwise. This page includes actual ("tested") configurations for situations encountered at NWS field offices for both LDM pqact.conf and the archiver configuration files RAW_DATA.xml and PROCESSED_DATA.xml. These configurations ensure the archives are in a format that WES-2 Bridge can use.

Below are examples of configurations for Raw Data, Processed Data, and the LDM pqact.conf file. Click on any of the blue bars to open or close the item.

These examples were updated and tested on June 6, 2016.

Raw Data

 

Note that the RAW_DATA.xml entries below are very sensitive to the punctuation and other characters that immediately precede or follow the digits that represent the date. In many cases, in the actual filename, an underscore character precedes or follows the ten digits that represent YYYYMMDDHH. While this underscore could be captured by a .* regular expression, specifying it explictly ensures that those ten digits are used for the date (as opposed to the second ten digits that sometimes appear at the end of the filename which represent the Epoch seconds for the LDAD file write time).


Canadian Model (HRDPS)

This data is found in /data_store/manual/grib/YYYYMMDD/HH, where YYYYMMDD/HH is the time the data were received and processed by AWIPS-2. The processed versions are typically found in /awips2/edex/data/hdf5/grid/Canadian-HRDPS or /archive/grid/Canadian-HRDPS or /archive/grid/Canadian-HiRes or something similar.

Sample Filenames: CMC_hrdps_east_VGRD_ISBL_0850_ps2.5km_2016053112_P042-00.Grb2.1464720741

RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>.*(CMC_hrdps).*_(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-Canadian Model (HRDPS)</displayLabel>
    <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

Canadian Model (Northern Hemisphere)

This data is found in /data_store/manual/grib/YYYYMMDD/HH, where YYYYMMDD/HH is the time the data were received and processed by AWIPS-2. The processed versions are typically found in /awips2/edex/data/hdf5/grid/Canadian-NH or /archive/grid/Canadian-NH.

Sample Filenames:

  • CMC2_nh_ACPCP_SFC_0_latlon.6x.6_2015082500_P144.Grb2.1440478052
  • LDAD-GRIB-CMC2_nh_APCP_SFC_0_latlon.6x.6_2015082100_P126.Grb2_2015-08-21_043450

This sample entry works for both of these filenames.

RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>.*(CMC2_nh).*_(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-Canadian Model (Northern Hemisphere)</displayLabel>
    <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>


Sample Filename: LDAD-GRIB.GEMWRSubsector_2015111012_P156.grib2.1447175704

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(GEMWR).*_(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Canadian</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

 

Sample Filename: LDAD-GRIB.GEMWRSubsector_.1446524277

This file was in /data_store/manual/grib/20151103/04. The epoch seconds (1446524277) represents Nov 3, 2015 at 041757 UTC, but the file itself was from the 00Z run of the Canadian model (determined by ingesting this particular file on a standalone system).

Can the LDAD configuration be changed to include the model cycle time in the filename? The other alternative, which is used below, is to use the directory name to get the time (or the Epoch seconds from the filename), but it is not representative of the actual data, because it is the time the file was processed by the system, and not the time of the data.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <filePattern>.*(GEMWRSubsector)_.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Canadian Model Western Region Sector</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

 

Canadian Model (Regional)

This data is found in /data_store/manual/grib/YYYYMMDD/HH, where YYYYMMDD/HH is the time the data were received and processed by AWIPS-2. The processed versions are typically found in /awips2/edex/data/hdf5/grid/Canadian-Reg or /archive/grid/Canadian-Reg.

Sample Filenames:

  • CMC2_reg_DEPR_ISBL_700_ps15km_2015082500_P009.Grb2.1440471459
  • LDAD-GRIB-CMC2_reg_UGRD_ISBL_850_ps15km_2015082400_P030.Grb2_2015-08-24_030615

This sample entry works for both of these filenames.

RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>.*(CMC2_reg).*_(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-Canadian Model (Regional)</displayLabel>
    <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

Climo PoP

This data may be found in /data_store/manual/grib/YYYYMMDD/HH, where YYYYMMDD/HH is the time the data were received and processed by AWIPS-2. The processed versions are probably found in /awips2/edex/data/hdf5/grid and /archive/grid with names like MAVClimoPoP, METClimoPoP, MEXClimoPop, WPCClimoPop.


Sample Filenames:

  • LDAD-GRIB-MAVClimoPoP_20151008_0600__20151009_0000.grb2_2015-10-08_104822
  • LDAD-GRIB-METClimoPoP_20151002_0000__20151002_1200.grb2_2015-10-02_034711
  • LDAD-GRIB-MEXClimoPoP_20151008_1200__20151013_1200.grb2_2015-10-08_172719
  • LDAD-GRIB-WPCClimoPoP_20151002_0000__20151009_1200.grb2_2015-10-02_052738
Since there are four different types of ClimoPop grids, they can be archived either separately or together; both strategies are shown below:

Separate Entries:


RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{4}\d{4}/\d{2}</dirPattern>
     <filePattern>.*(MAVClimoPoP)_(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-MAVClimoPoP</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

<dataSet>
     <dirPattern>(manual)/grib/\d{4}\d{4}/\d{2}</dirPattern>
     <filePattern>.*(METClimoPoP)_(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-METClimoPoP</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

<dataSet>
     <dirPattern>(manual)/grib/\d{4}\d{4}/\d{2}</dirPattern>
     <filePattern>.*(MEXClimoPoP)_(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-MEXClimoPoP</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

<dataSet>
     <dirPattern>(manual)/grib/\d{4}\d{4}/\d{2}</dirPattern>
     <filePattern>.*(WPCClimoPoP)_(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-WPCClimoPoP</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

 

Combined Entry:


RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{4}\d{4}/\d{2}</dirPattern>
     <filePattern>.*(ClimoPoP)_(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-ClimoPoP</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

ECMWF MOS Text

This data is found in /data_store/ecmwf-mos-text.

Sample Filename: ecmwf_mos_decrypted_20160523/23/FOAK57_PAJK_232300_117520249.2016052323.5gIGN4

RAW_DATA.xml

<dataSet>
    <dirPattern>(ecmwf-mos-text)/.*_(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
    <displayLabel>{1}</displayLabel>
    <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

Extended ETA or NAM (66-84 hours)

This data consists of ETA grids beyond those that are normally distributed over the SBN. In AWIPS-2, the raw data will be found in /data_store/manual/grib/YYYYMMDD/HH. The processed data is stored with the normal ETA model (e.g., /awips2/edex/data/hdf5/grid/ETA218 or /archive/grid/ETA218). For raw data, several different filename conventions have been observed, depending on local LDAD configurations.

Sample Filename: 2015082418.ExtEtaF18.1440447412. The distinguishing characteristic in the filename is ExtEtaF##.


RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>(\d{4})(\d{2})(\d{2})(\d{2}).ExtEtaF(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-Extended NAM</displayLabel>
    <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

 

Sample Filename: LDAD-GRIB-2015082206.GribF66_2015-08-22_084555. The distinguishing characteristic in the filename is GribF##.


RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>.*(\d{4})(\d{2})(\d{2})(\d{2}).GribF(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-Extended NAM</displayLabel>
    <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

 

Sample Filename: namA280.2016053112.F66.Grb2.1464706237.


RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>namA280.*(\d{4})(\d{2})(\d{2})(\d{2}).F(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-Extended NAM 66-84 hr</displayLabel>
    <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

GFS Simulated Satellite

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH.

 

Sample Filename: /data_store/manual/grib/20160531/17/gfsA2.t12z.goes215f45.Grb2.1464714353

The model cycle time for this is May 31, 2016 at 1200 UTC. The date is taken from the directory name and the model cycle time (e.g. 12) is from the filename.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <filePattern>.*gfsA2.t(\d{2})z.goes.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-GFS Simulated Satellite</displayLabel>
     <dateGroupIndices>2,3,4,6</dateGroupIndices>
</dataSet>

GOES-16 Raw Data

The raw data is found in /data_store/goes-r/YYYYMMDD/HH.

 

Sample Filenames:
/data_store/manual/grib/20170614/17/TIRC01_KNES_141727_PAA.2017061417 (ABI)
/data_store/manual/grib/20170614/17/IXTG99_KNES_141727.2017061417 (Derived Product)

This entry grabs all GOES-16 data for an hour.

RAW_DATA.xml

<dataSet>
     <dirPattern>(goes-r)/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-GOES-16 Satellite</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

hpcqpf
The raw data is found in /data_store/manual/grib/YYYYMMDD/HH. The processed data is stored as HPCqpfNDFD (e.g., /awips2/edex/data/hdf5/grid/HPCqpfNDFD or /archive/grid/HPCqpfNDFD).

Sample Filenames:

  • hpcqpf67_p06m_2015082500f126.Grb2.1440448847
  • LDAD-GRIB-hpcqpf67_p06m_2015070612f126.Grb2_2015-07-06_081041
This entry will work for both filenames.

RAW_DATA.xml

<dataSet>
   <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
   <filePattern>.*(hpcqpf67).*_(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
   <timeType>Date</timeType>
   <displayLabel>{1}-HPC QPF for NDFD</displayLabel>
   <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

HPCWWD (HPC Winter Weather Desk)

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH, where YYYYMMDD/HH is the time the data were received and processed by AWIPS-2. The processed versions are typically found in /awips2/edex/data/hdf5/grid/HPCWWD or /archive/grid/HPCWWD.

Sample Filename: LDAD-GRIB-2014-2015_HPCWWD_2.5km_gfs_snoden_2015082200f006.Grb2_2015-08-21_162036

RAW_DATA.xml

<dataSet>
   <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
   <filePattern>.*(HPCWWD).*_(\d{4})(\d{2})(\d{2})(\d{2}).</filePattern>
   <timeType>Date</timeType>
   <displayLabel>{1}-HPCWWD</displayLabel>
   <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

HREF

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH.

 

Sample Filename: /data_store/manual/grib/20160524/11/href.t06z.mean.f18.grib2.1464089568

The model cycle time for this is May 24, 2016 at 0600 UTC. The date is taken from the directory name and the model cycle time (e.g. 06) is from the filename.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <filePattern>.*href.t(\d{2})z.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-HREF Ensembles</displayLabel>
     <dateGroupIndices>2,3,4,6</dateGroupIndices>
</dataSet>

HRRR (CR/GSD version)

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH, where YYYYMMDD/HH is the time the data were received and processed by AWIPS-2. The processed versions are typically found in /awips2/edex/data/hdf5/grid/HRRR-CR or /archive/grid/HRRR-CR.

Sample Filename: LDAD-GRIB-HRRR.001Hour.DPT.2m_FHAG.201508210400.grb2_2015-08-21_052748


RAW_DATA.xml

<dataSet>
   <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
   <filePattern>.*(HRRR).*\.(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
   <timeType>Date</timeType>
   <displayLabel>{1}-GSD/CR HRRR</displayLabel>
   <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

Sample Filename: LDAD-GRIB-CR.HRRR.000Hour.TMP.2m_FHAG.201508240100.grb2_2015-08-24_025340

The above example actually works with this filename pattern as well. However, regular expression could also be changed to (CR.HHRR) rather than (HRRR), if desired.

LDAD.mesonet and LDAD.hydro (/data_store/ldad)
Local Mesonet, Local Hydro and some SHEF data from LDAD are stored in /data_store/ldad. Almost every WFO ingests data using this method, but there is no baseline archiver configuration for this data. Sample Filenames:
  • LDAD.hydro.1440448352.CoCoRaHS.decoded.xml
  • LDAD.mesonet.1440449193.okmesnet.decoded.xml
  • LDAD.mesonet.okmesnet.1440453600.36.msas_qc.decoded.xml
  • SXUS44 KWOHshef.1440795065

The local mesonet name ("okmesnet" above) varies, but all these filenames contain Epoch seconds. We can make a single RAW_DATA.xml entry, or have a separate entries for hydro and mesonet data.

Single Entry

 

RAW_DATA.xml

<dataSet>
    <dirPattern>(ldad)</dirPattern>
    <timeType>EpochSec</timeType>
    <filePattern>.*\.(\d{10}).*</filePattern>
    <displayLabel>{1} Mesonet/Hydro</displayLabel>
    <dateGroupIndices>2</dateGroupIndices>
</dataSet>

Separate Entries

 

RAW_DATA.xml

<dataSet>
    <dirPattern>(ldad)</dirPattern>
    <timeType>EpochSec</timeType>
    <filePattern>LDAD.mesonet.*\.(\d{10}).*</filePattern>
    <displayLabel>{1} Mesonet</displayLabel>
    <dateGroupIndices>2</dateGroupIndices>
</dataSet>

<dataSet>
    <dirPattern>(ldad)</dirPattern>
    <timeType>EpochSec</timeType>
    <filePattern>(LDAD.hydro|SXUS44).*\.(\d{10}).*</filePattern>
    <displayLabel>{1} Hydro</displayLabel>
    <dateGroupIndices>3</dateGroupIndices>
</dataSet>

Lightning Mapper Array (LMA)

This raw data was found in /data_store/manual/sportlma/YYYYMMDD/HH. Though the data were processed by the sportlma plugin, the processed data were stored in /awips2/edex/data/hdf5/grid/NALMA and /archive/grid/NALMA.

Sample Filename: sportlma_nalma_fed_20151002_1606

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/(sportlma)/(\d{4})(\d{2})(\d{2})/\d{2}</dirPattern>
     <timeType>Date</timeType>
     <filePattern>.*(nalma).*_(\d{4})(\d{2})(\d{2})_(\d{2})(\d{2}).*</filePattern>
     <displayLabel>{1}-{2} for Alabama</displayLabel>
     <dateGroupIndices>7,8,9,10</dateGroupIndices>
</dataSet>

When there is a new plugin like this, you ordinarily would be concerned where the processed data would be located (for example, named after the same plugin, such as nucaps). But this plugin actually contributes to the grid data type (/archive/grid/NALMA), which means a new PROCESSED_DATA.xml entry is not necessary.

Local METARs

This raw data is found in /data_store/manual/obs/YYYYMMDD/HH/, where YYYYMMDD/HH is the time the data were received and processed by AWIPS-2. The processed versions are found with all other METAR data in /awips2/edex/data/hdf5/obs or /archive/grid/obs.

Sample Filename: /data_store/manual/obs/YYYYMMDD/HH/SAUS80KWBC.022055.GTFMTR1S3

Because the filename does not contain a full date/time specification, it is necessary to use the date/time from the directory (which represents the receipt time and not the valid time of the data).

RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/obs/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-Local Obs</displayLabel>
    <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

 

Sample Filename:SAUS70_KGID_240211_440382479.2015082402_2015-08-24_021503

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/obs/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(SAUS70).*(\d{4})(\d{2})(\d{2})(\d{2})_.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Local METARs</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

Sample Filename:SAUS44KWBC_KDWU.dat.1462072462

In this situation, we need to use the last ten digits as the Epoch seconds.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/obs/\d{8}/\d{2}</dirPattern>
     <filePattern>(SAUS).*(\d{10})</filePattern>
     <timeType>EpochSec</timeType>
     <displayLabel>{1}-Local METARs</displayLabel>
     <dateGroupIndices>3</dateGroupIndices>
</dataSet>

Sample Filename:/data_store/manual/obs/20160531/00/SWRALBMTRABG.dat.1464656042.310054

Here the date/time from the directory apparently matches the date/time from the filename (310004).

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/obs/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <filePattern>SWR.*dat.*.(\d{10}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Various Local Surface Obs</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

Local WRF

Local WRF models typically have their raw data located in /data_store/manual/grib/YYYYMMDD/HH. Exact filenames differ depending on local configurations. The processed data also are in /awips2/edex/data/hdf5/grid and /archive/grid, but the exact names typically differ at each WFO. A number of examples follow.

All of these examples do not have the full year specified in the filename, so we have to get the year from the directory (YYYYMMDD/HH).

Sample Filename: LDAD-GRIB-1509301200_arw_d03.grb2f530000_2015-90-30_165747

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})\d{2}\d{2}/\d{2}</dirPattern>
     <filePattern>LDAD-GRIB-(\d{2})(\d{2})(\d{2})(\d{2}).*(arw_d03).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-ARW8KM</displayLabel>
     <dateGroupIndices>2,4,5,6</dateGroupIndices>
</dataSet>

 


Sample Filename: 1509071200_arw_d01.Grb2LDAD-GRIBf010000.1441655188

RAW_DATA.xml


<dataSet>
     <dirPattern>(manual)/grib/(\d{4})\d{4}/\d{2}</dirPattern>
     <filePattern>\d{2}(\d{2})(\d{2})(\d{2})(\d{2})_arw.*.Grb.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Local WRF Model</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

 

Sometimes a WFO runs the WRF at different resolutions. In this case, the arw_d01 and arw_d02 files are different resolutions. They can be archived either separately or together as shown below:

Sample Filenames:

  • LDAD-GRIB-MTRWRF-15101512_arw_d01.grbF0410000.1444934189
  • LDAD-GRIB-MTRWRF-15101512_arw_d02.grbF0410000.1444933869

Combined Entry:


RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})\d{4}/\d{2}</dirPattern>
     <filePattern>.*\d{2}(\d{2})(\d{2})(\d{2})_arw.*.grb.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Local WRF Model</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>


Separate Entries:


RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})\d{4}/\d{2}</dirPattern>
     <filePattern>.*\d{2}(\d{2})(\d{2})(\d{2})_arw_d01.*.grb.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Local WRF Model</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>


<dataSet>
     <dirPattern>(manual)/grib/(\d{4})\d{4}/\d{2}</dirPattern>
     <filePattern>.*\d{2}(\d{2})(\d{2})(\d{2})_arw_d02.*.grb.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Local WRF Model (1km)</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>


mPING (Meteorological Phenomena Identification Near the Ground)

The raw data is found in /data_store/local/mping/YYYYMMDD/HH. The processed data is stored as mping (e.g., /awips2/edex/data/hdf5/mping or /archive/mping).

 

Sample Filename: mping_2016050812089.xml.1462709322

The data are well-organized into date/hour directories.

RAW_DATA.xml

<dataSet>
     <dirPattern>local/(mping)/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <timeType>Date</timeType>
     <displayLabel>{1}</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

NAM CONUS Nest

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH. The processed data is stored as NAMNestt or CR-NAMNest (e.g., /awips2/edex/data/hdf5/grid/CR-NAMNest or /archive/grid/CR-NAMNest).

 

Sample Filename: LDAD-GRIB-CR-nam.t00z.conusnest.hiresf01.CAPE_SFC.grib2_2015-11-20_023149

The model cycle time for this is November 20, 2015 at 0000 UTC. The date is taken from the LDAD time.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(nam).t(\d{2})z.(conusnest).*_(\d{4})-(\d{2})-(\d{2})_(\d{2})(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-CR NAM CONUS Nest</displayLabel>
     <dateGroupIndices>5,6,7,3</dateGroupIndices>
</dataSet>

 

Sample Filename: /data_store/manual/grib/20160531/22/nam.t18z.conusnest.hiresf27.tm00.grib2.1464732054.

The model cycle time for this is May 31, 2016 at 1800 UTC. The date is taken from the directory name and the model cycle time (e.g. 18) is from the filename.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <filePattern>.*(nam).t(\d{2})z.(conusnest).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-NAM CONUS Nest</displayLabel>
     <dateGroupIndices>2,3,4,6</dateGroupIndices>
</dataSet>

NAM Simulated Satellite

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH.

 

Sample Filename: /data_store/manual/grib/20160531/02/nam.t00z.goes21822.tm00.grib2.1464662771

The model cycle time for this is May 31, 2016 at 0000 UTC. The date is taken from the directory name and the model cycle time (e.g. 00) is from the filename.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
     <filePattern>.*nam.t(\d{2})z.goes.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-NAM Simulated Satellite</displayLabel>
     <dateGroupIndices>2,3,4,6</dateGroupIndices>
</dataSet>

NARRE (North America Rapid Refresh Ensemble)

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH. The processed data is stored as NARRE-TL-CR or just NARRE (e.g., /awips2/edex/data/hdf5/grid/NARRE-TL-CR or /archive/grid/NARRE-TL-CR or /archive/grid/NARRE).

 

Sample Filename: LDAD-GRIB-cr.narre.t19z.mean.grd130.f03.grib2_2015-11-20_204102

The model cycle time for this is November 20, 2015 at 1900 UTC. The date is taken from the LDAD time.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(narre).t(\d{2})z.*_(\d{4})-(\d{2})-(\d{2})_(\d{2})(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-NARRE</displayLabel>
     <dateGroupIndices>4,5,6,3</dateGroupIndices>
</dataSet>

NBM (National Blend of Models)

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH. The processed data is stored as NBM (e.g., /awips2/edex/data/hdf5/grid/NBM or /archive/grid/NBM).

Sample Filename: LDAD-GRIB-WR-NBM-mdlblendcogb215082412sky.1440458294

This filename is problematic because the model cycle time only contains a two-digit year (15 in the above). In the above, the model cycle time is August 24, 2015 at 1200 UTC. This entry below obtains the year from the parent directory structure.

<dataSet>
   <dirPattern>(manual)/grib/(\d{4})\d{4}/\d{2}</dirPattern>
   <filePattern>.*(NBM-).*(\d{2})(\d{2})(\d{2}).*\.\d{10}</filePattern>
   <timeType>Date</timeType>
   <displayLabel>{1}-NBM</displayLabel>
   <dateGroupIndices>2,4,5,6</dateGroupIndices>
</dataSet>

 

Sample Filename: LDAD-GRIB-CR-NBM-mdlblendcogb215100112apt_2015-10-01_232324

I am assuming the model run time for this is October 1, 2015 at 1200 UTC. Again the four-digit year for the model cycle time is not in the filename, so it must be obtained from the LDAD time (2015-10-01_232324; less accurate):

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{4}\d{4}/\d{2}</dirPattern>
     <filePattern>.*(NBM-).*\d{1}(\d{2})(\d{2})(\d{2})(\d{2}).*\_(\d{4})-.*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-NBM</displayLabel>
     <dateGroupIndices>7,4,5,6</dateGroupIndices>
</dataSet>

NCAR 3km ensembles

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH. The processed data may be stored as NCAR-WRF1 through NCAR-WRF10 (e.g., /awips2/edex/data/hdf5/grid/NCAR-WRF1 or /archive/grid/NCAR-WRF10).

Sample Filename: LDAD-GRIB-UNR-ncar_3km_2015082800_mem9_f046.grb2f_2015-08-28_075143

(Note there are ten members, labeled mem1, mem2, mem3, mem4, mem5, mem6, mem7, mem8, mem9, and mem10.)

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>.*(UNR-ncar_3km).*(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}-NCAR 3km members</displayLabel>
    <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

NWPS (Nearshore Wave Prediction System)
The NWPS raw grib data is found in /data_store/manual/grib/YYYYMMDD/HH. Filenames typically include the WFO name (for this example, it's "mtr").

Sample Filenames:
  • mtr_nwps_CG0_Trkng_20151015_0600.grib2.1444977368
  • mtr_nwps_CG1_20151009_1800.grib2.1444411943

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
     <filePattern>(mtr_nwps).*(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Nearshore Wave Prediction System (NWPS)</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

NSSL WRF

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH. The processed data is probably stored as NSSL-WRF (e.g., /awips2/edex/data/hdf5/grid/NSSL-WRF or /archive/grid/NSSL-WRF).

Sample Filename: LDAD-GRIB-UNR-wrf4nssl_2015082812.fgrib2f.f08_2015-08-28_160539

RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
    <filePattern>.*(-wrf4nssl).*(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
    <timeType>Date</timeType>
    <displayLabel>{1}- NSSL WRF</displayLabel>
    <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

nucaps

This data consists of soundings from NOAA Unique CrIS/ATMS Processing System from NPP (National Polar-Orbiting Partnership) Satellites. This data is not included in the baseline RAW_DATA.xml archiver configuration file so it needs to be added. The raw data is typically located in /data_store/nucaps.

RAW_DATA.xml

<category>
    <name>Profiles</name>
    <selectedRetentionHours>168</selectedRetentionHours>
    <dataSet>
       <dirPattern>(bufrua|goessounding|nucaps|poessounding|profiler|raobs|upperair)/(\d{4})(\d{2})(\d{2})/(\d{2})</dirPattern>
       <displayLabel>{1}</displayLabel>
       <dateGroupIndices>2,3,4,5</dateGroupIndices>
   </dataSet>
</category>


See also: nucaps under Processed Data.

ProbSevere (convectprob)

This data is produced by CIMSS and is distributed via regional LDMs and ingested into AWIPS through LDAD. The raw data is typically located in /data_store/manual/convectprob.

Sample Filename: SSEC_AWIPS_CONVECTPROB_20160501_201039.ascii.2016-05-01_201113

RAW_DATA.xml

   <dataSet>
       <dirPattern>(manual)/(convectprob)/\d{8}/\d{2}</dirPattern>
       <filePattern>.*_CONVECTPROB_(\d{4})(\d{2})(\d{2})_(\d{2})(\d{2}).*ascii.*</filePattern>
       <displayLabel>{1}-ProbSevere</displayLabel>
       <dateGroupIndices>3,4,5,6</dateGroupIndices>
   </dataSet>


See also: ProbSevere under Processed Data.

RegionalSat

This raw data is stored under /data_store/manual/regionalsat/YYYYMMDD/HH. The processed data is stored alongside the regular satellite data in /awips2/edex/data/hdf5/satellite/ in directories named like centralRegionEast, centralRegionWest, goesEast, goesWest, conus, eastConus, goesEast, goesWest, westConus, etc.

Depending on local LDAD configurations and the source of the data, slightly different filenames have been observed.

Sample Filename: LDAD-REGIONALSAT-20150824_1815_sport_econus_uahci.nc_2015-08-24_182943

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/regionalsat/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(\d{4})(\d{2})(\d{2})_(\d{2}).*(sport).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Regional Sat (SPoRT)</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

 

Sample Filename: 20151002_1625_sport_econus_uahci.nc

(Note: The above RAW_DATA.xml also works for this sample file as well as the one below. The above example works because the leading .* means to capture zero or more characters before before the specified digits (the eight digits, underscore, and two digits).

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/regionalsat/\d{8}/\d{2}</dirPattern>
     <filePattern>(\d{4})(\d{2})(\d{2})_(\d{2}).*(sport).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Regional Sat (SPoRT)</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

 

Sample Filename:
GOESR-PG_SSEC_AWIPS_GEOCAT-GEOW_CONUS_4KM_FOGDEPTH_20151009_2230.55779861.1444430872


RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/regionalsat/\d{8}/\d{2}</dirPattern>
     <filePattern>(GOESR-PG_SSEC).*(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Regional Sat (GOESR-PG SSEC)</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

 

Sample Filename: 20151009_1645_sportA2_nesdis_conusa_qpe000hr

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/regionalsat/\d{8}/\d{2}</dirPattern>
     <filePattern>(\d{4})(\d{2})(\d{2})_(\d{2}).*(sport).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Regional Sat (SPoRT)</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>

Note: this entry is identical to the one used for the uahci.nc file above.

 

WFOs typically ingest multiple satellite products, and it is possible to archive them separately, as shown here:


Sample Filenames:

  • 20151002_1519_sport_modis_sre_12.0um
  • 20151002_1601_sport_goesSndr_conus_airmass

RAW_DATA.xml

<dataSet>
<dirPattern>(manual)/regionalsat/\d{8}/\d{2}</dirPattern>
     <filePattern>(\d{4})(\d{2})(\d{2})_(\d{2}).*(sport_modis).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Regional Sat (SPoRT) MODIS</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>
<dataSet>
     <dirPattern>(manual)/regionalsat/\d{8}/\d{2}</dirPattern>
     <filePattern>(\d{4})(\d{2})(\d{2})_(\d{2}).*(sport_goesSndr).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Regional Sat (SPoRT) GOES SOUNDER</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>


 

Sample Filename: satxgoesvsfg_20150824_2030.nc


RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/regionalsat/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(satx).*_(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <timeType>Date</timeType>
     <displayLabel>{1}-Regional Sat</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

 

satellite.mcidas

This raw data is stored in /data_store/manual/satellite.mcidas/YYYYMMDD/HH, where the date and time represent the time the data is received and processed. The processed data is stored alongside the regular satellite data in /awips2/edex/data/hdf5/satellite in AREA#### directories (e.g., AREA1881).

Sample Filenames:

  • LDAD-MCIDAS.CIRA_AWIPS2_RGB_AIRMASS_20150824_1901_2015-08-24_204547
  • LDAD-MCIDAS-CIRA_AWIPS2_WC_LCFOG_20150826_2230_2015-08-26_224851

RAW_DATA.xml

<dataSet>
     <dirPattern>manual/(satellite.mcidas)/.*/.*</dirPattern>
     <filePattern>.*CIRA_AWIPS2.*_(\d{4})(\d{2})(\d{2})_(\d{2})(\d{2})_.*</filePattern>
     <displayLabel>McIDAS Experimental GOES-R Satellite</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>


Sample Filename: CIRA_AWIPS2_SIMWRF_VIS_20151009_1200.gz.1444403021

RAW_DATA.xml

<dataSet>
     <dirPattern>manual/(satellite.mcidas)/.*/.*</dirPattern>
     <filePattern>.*_(\d{4})(\d{2})(\d{2})_(\d{2})(\d{2}).*</filePattern>
     <displayLabel>{1}</displayLabel>
     <dateGroupIndices>2,3,4,5</dateGroupIndices>
</dataSet>


Note: the above entry actually captures all three sample files for satellite.mcidas.

sportlis (SPoRT Land Information System)

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH. The processed data is probably found in /awips2/edex/data/hdf5/grid/LIS-CONUS or /archive/grid/LIS-CONUS.

Sample Filename: sportlis_conus3km_awips_20151002_1800.grb2.1443830086

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(sportlis).*(\d{4})(\d{2})(\d{2})_(\d{2})(\d{2}).*</filePattern>
     <displayLabel>{1}-SPoRT Land Info System</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

SPoRT SST

The raw data is found in /data_store/manual/grib/YYYYMMDD/HH.

Sample Filenames:

  • ERA2_sport_sstlate_20160530_1800.Grb2.1464665179
  • ERA2_sport_sstlate_20160531_0600.Grb2.1464708371
  • ERA2_sport_sstcomp_20160531_0600.Grb2.1464708376

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/grib/\d{8}/\d{2}</dirPattern>
     <filePattern>.*(sport_sst).*_(\d{4})(\d{2})(\d{2})_(\d{2}).*</filePattern>
     <displayLabel>{1}-SPoRT SST</displayLabel>
     <dateGroupIndices>3,4,5,6</dateGroupIndices>
</dataSet>

Various Products in /data_store/manual/text

The raw data is found in /data_store/manual/text. Most often, products that are stored here are outgoing products issued by a WFO. Examples include: RWSRAP.wan1434881882 and KUNRAFDUNR-0-1438788073. Operational outgoing products are identified by a trailing set of ten digits which are the epoch seconds of when product was created or transmitted. The baseline RAW_DATA.xml file handles these normal outgoing products.

Some sites have additional data files in /data_store/manual/text that are not outoging products and do not end with the epoch seconds. An example is ALBRR7ZKC. This particular file is a shef-encoded file that probably should have been processed by the shef plugin and therefore stored under /data_store/manual/shef. But there could be regular text products that could be stored in /data_store/manual/text that don't end with epoch seconds. The baseline RAW_DATA.xml pattern will not correctly handle files in /data_store/manual/text that do not end with ten digits.

Sample Filename: /data_store/text/YYYYMMDD/HH/ALBRR7ZKC

Add this entry to the Products category of RAW_DATA.xml. Since there is no date in the filename, this entry associates the file write-time as the time the archiver uses to determine if the file should be saved in a particular case. This filePattern finds files that do not end with any numeric digits.

RAW_DATA.xml

<dataSet>
     <dirPattern>(manual)/text/\d{8}/\d{2}</dirPattern>
     <filePattern>(.*[^0-9]$)</filePattern>
     <displayLabel>{1}-Various Local Text Products</displayLabel>
     <timeType>File</timeType>
     <dateGroupIndices>2</dateGroupIndices>
</dataSet>

text_decrypted (Watch County List or WCL)

The raw data is found in /data_store/text/text_decrypted/

Sample Filename: /data_store/text/text_decrypted/wcl_decrypted_NWUS64_KWNS_270246.128922525.2016052702.Ldlepg

Add this entry to the Products category of RAW_DATA.xml:

RAW_DATA.xml

<dataSet>
     <dirPattern>text/text_decrypted</dirPattern>
     <filePattern>.*(\d{4})(\d{2})(\d{2})(\d{2}).*</filePattern>
     <displayLabel>text - Watch County List (WCL)</displayLabel>
     <dateGroupIndices>1,2,3,4</dateGroupIndices>
</dataSet>

unknown (/data_store/manual/unknown)

Most of the time, when data appears in /data_store/manual/unknown, it is a symptom of an AWIPS misconfiguration where the system did not know how to process a certain kind of data. However, at a few WFOs, SHEF data from the U.S. Bureau of Reclamation was actually processed by the system and still appeared in this directory. This situation is covered here. In this case, there are no dependable times in the filenames for the archiver to use other than the Epoch seconds added by the LDAD configuration.

Sample filenames:
  • fm_hydmt_usbr.1435976647.1435976670
  • fm_hydmt.0704.072601.1435994788

RAW_DATA.xml

<dataSet>
    <dirPattern>(manual)/unknown/\d{8}/\d{2}</dirPattern>    
    <filePattern>fm_hydmt.*.(\d{10})</filePattern>
    <displayLabel>{1} - Various Local SHEF Products (fm_hydmt)</displayLabel>
    <timeType>EpochSec</timeType>
    <dateGroupIndices>2</dateGroupIndices>   
</dataSet>

Processed Data


FFMP (Flash Flood Monitoring and Prediction)

In AWIPS Build 16.2.1, the directory hierarchy for FFMP data changed but corresponding changes were not made to PROCESSED_DATA.xml. The processed data is located in /archive/ffmp. In order to archive FFMP data, the change below must be made to PROCESSED_DATA.xml. The change removes ffmp from the dirPattern that contains the other decision assistance tools (like cwat, fssobs, and scan) and makes a new dataSet specifically for FFMP.

 

PROCESSED_DATA.xml

<category>
    <name>Decision Assistance</name>
    <selectedRetentionHours>168</selectedRetentionHours>
    <dataSet>
        <dirPattern>(cwat|fog|fssobs|preciprate|qpf|scan|vil)/.*(\d{4})-(\d{2})-(\d{2})-(\d{2}).*</dirPattern>
        <displayLabel>{1}</displayLabel>
        <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>
    <dataSet>
        <dirPattern>(ffmp)/.*/.*(\d{4})-(\d{2})-(\d{2})-(\d{2}).*</dirPattern>
        <displayLabel>{1}</displayLabel>
        <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>
</category>

mPING (Meteorological Phenomena Identification Near the Ground)

The processed data is located in /archive/mping.

 

PROCESSED_DATA.xml

<category>
    <name>Observation</name>
    <selectedRetentionHours>168</selectedRetentionHours>
    <dataSet>
        <dirPattern>(acars|airep|binlightning|bufrascat|bufrmthdw|bufrssmi|idft|lsr|mping|obs|pirep|recco|svrwx)/.*-(\d{4})-(\d{2})-(\d{2})-(\d{2})</dirPattern>
    <dataSet>
        <dirPattern>(sfcobs)/.*/.*-(\d{4})-(\d{2})-(\d{2})-(\d{2})</dirPattern>
        <displayLabel>{1}</displayLabel>
        <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>
</category>

nucaps

This data consists of soundings from NOAA Unique CrIS/ATMS Processing System from NPP (National Polar-Orbiting Partnership) Satellites. This data is not included in the baseline RAW_DATA.xml archiver configuration file so it needs to be added. The processed data is located in /archive/nucaps.

 

PROCESSED_DATA.xml

<category>
    <name>Profiles</name>
    <selectedRetentionHours>168</selectedRetentionHours>
    <dataSet>
        <dirPattern>(acarssounding|bufrua|goessounding|nucaps|poessounding|profiler)/.*-(\d{4})-(\d{2})-(\d{2})-(\d{2})</dirPattern>
        <displayLabel>{1}</displayLabel>
        <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>
</category>

ProbSevere (convectprob)

This data is produced by CIMSS and distributed via regional LDMs through LDAD. In addition, this data is not included in the baseline RAW_DATA.xml archiver configuration file so it needs to be added. The processed data is located in /archive/convectprob.

 

PROCESSED_DATA.xml

    <dataSet>
        <dirPattern>(convectprob)/.*-(\d{4})-(\d{2})-(\d{2})-(\d{2}).*</dirPattern>
        <displayLabel>{1}-ProbSevere</displayLabel>
        <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>

Spot Forecast Requests (STQ)

A new STQ plugin processes fire weather spot forecast requests and makes a plot accessible with the NCEP/Hydro menu. The raw data for these requests are already archived under the Products category as fire_wx_spot_fcst_reports. The baseline PROCESSED_DATA.xml does not contain the ability to archive STQ data.

 

PROCESSED_DATA.xml

<category>
    <name>Products</name>
    <selectedRetentionHours>168</selectedRetentionHours>
    <dataSet>
        <dirPattern>(airmet|atcf|aww|bufrncwf|ccfp|convsigmet|cwa|ffg|intlsigmet|nonconvsigmet|stormtrack|stq|taf|tcg|tcm|tcs|text|vaa|warning|wcp)/.*-(\d{4})-(\d{2})-(\d{2})-(\d{2})</dirPattern>
        <dirPattern>(bufrsigwx|redbook)/.*/.*-(\d{4})-(\d{2})-(\d{2})-(\d{2})</dirPattern>
        <displayLabel>{1}</displayLabel>
        <dateGroupIndices>2,3,4,5</dateGroupIndices>
    </dataSet>
</category>

LDM pqact.conf configurations


CADAS and/or HADS SHEF data
Many WFOs ingest DCP data from the CADS and HADS systems which are transmitted over the SBN. The baseline pqact.conf file stores all products from the IDS|DDPLUS feed that have WMO headers that begin with SR in /data_store/shef/YYYYMMDD/HH. HADS data for central and western regions have WMO headers that begin with SX, so a local entry is required because the baseline pqact.conf file does not handle WMO headers starting with SX. In addition, CADAS data comes over the HDS feed rather than the IDS|DDPLUS feed, so a local pqact.conf entry may be needed for that as well, for any region.

 

Here is one WFO's local entry for CADAS and HADS data. Their pqact.conf entry is correct, except that it does not store the data in /data_store/shef/YYYYMMDD/HH. Therefore, the data cannot be archived. As an aside, some WFOs have incorrectly used the IDS|DDPLUS feedtype in pqact.conf entries intended for CADAS, which resulted in a duplication of the HADS gauges with no additional CADAS gauge data.

incorrect pqact.conf.xxx

IDS|DDPLUS     ^(SXUS3[123]) (KWOH) (..)(..)(..)
     FILE      -overwrite -log -close -edex /data_store/shef/hads/\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d
HDS            ^(SXUS3[123]) (KWOH) (..)(..)(..)
     FILE      -overwrite -log -close -edex /data_store/shef/cadas/\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d

 

Here is the correct entry, and notice the highlighted directory.

correct pqact.conf.xxx

IDS|DDPLUS     ^(SXUS3[123]) (KWOH) (..)(..)(..)
     FILE      -overwrite -log -close -edex /data_store/shef/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d
HDS            ^(SXUS3[123]) (KWOH) (..)(..)(..)
     FILE      -overwrite -log -close -edex /data_store/shef/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d

 

/data_store/firewx directory

Several WFOs have the local portion of their pqact conf file to save fire weather observations in a /data_store/firewx directory. One such configuration is shown below:

pqact.conf.XXX

IDS|DDPLUS ^(SHUS..) (.{4}) (..)(..)(..)
      FILE -overwrite -log -close -edex /data_store/firewx/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H

 


The baseline pqact.conf also has the same WMO header but saves the data in /data_store/misc_sfc_obs. This makes the locally-added /data_store/firewx directory redundant.

pqact.conf

# DR 15716 - Add in SHUS products for fire weather obs (and others)
IDS|DDPLUS ^(SHUS..) (.{4}) (..)(..)(..)
     FILE   -overwrite -log -close -edex /data_store/misc_sfc_obs/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H

 

If a local pqact.conf entry matches this baseline entry, the local entry should be removed.

Local addition for GLERL

Some WFOs near the Great Lakes have added a Great Lakes wave model (GLERL) to their pqact.conf.XXX file. This particular WFO stored its data like this: /data_store/grib/GLERL/GLERL_120000_23241343.grib, which does not fit the expected /data_store/grib/YYYYMMDD/HH paradigm.

pqact.conf.XXX (bad entry)

#GLERL Wave Model
ANY      ^(O[LMN]N.88) (KWNB) (..)(..)(..)
     FILE -overwrite -log -close -edex /data_store/grib/GLERL/GLERL_\3\4\5_(seq).grib

pqact.conf.XXX (corrected version)

ANY      ^(O[LMN]N.88) (KWNB) (..)(..)(..)[^!]*!(grib|grib2)/[^/]*/([^/]*)/#([^/]*)/([0-9]{8})([0-9]{4})/(F[0-9]{3})
     FILE -overwrite -log -close -edex /data_store/\6/(\3:yyyy)(\3:mm)\3/\4/GLERL/GRID\8/\(10)Z_\(11)-\1_\2_\3\4\5_(seq).\6.%Y%m%d%H


This works because running the notifyme command for this data reveals there is sufficient metadata (the !grib/161/161/#255/201508210000/f001/WVHGT/sfc/) to populate all the regular expressions:

notifyme output

Aug 21 01:22:44 notifyme[29417] INFO: 2114050 20150821012242.638
HDS 501312036 OLNC88 KWNB 210100 /m99 !grib/161/161/#255/201508210000/F001/WVHGT/sfc/

Two-digit directories in /data_store/shef

The symptom of this problem is two-digit directories (typically representing HH) in /data_store/shef that are alongside proper 8-digit directories that represent YYYYMMDD, as shown below:

dx2-xxx$ cd /data_store/shef
dx2-xxx$ ls
00 02 04 06 08 10 12 14 16 18 20       20151103 20151105 20151107 20151109 22
01 03 05 07 09 11 13 15 17 19 20151102 20151104 20151106 20151108 21       23


The two-digit directories result from entries in LDM's pqact.conf file that do not match the storage paradigm used throughout AWIPS-2 and expected by the archiver.

Example 1:

Here are some sample incorrect pqact.conf entries followed by corrected versions.

IDS|DDPLUS      ^(SXUS53) (KWOH) (..)(..)(..)
     FILE     -overwrite -log -close -edex     /data_store/shef/\4/\1_\2_\3\4\5_(seq).txt
IDS|DDPLUS     ^(SXUS58) (KWOH) (..)(..)(..)
     FILE     -overwrite -log -close -edex     /data_store/shef/\4/\1_\2_\3\4\5_(seq).txt
IDS|DDPLUS     ^(SXUS59) (KWOH) (..)(..)(..)
     FILE     -overwrite -log -close -edex     /data_store/shef/\4/\1_\2_\3\4\5_(seq).txt
IDS|DDPLUS     ^(SXUS60) (KWOH) (..)(..)(..)
     FILE     -overwrite -log -close -edex     /data_store/shef/\4/\1_\2_\3\4\5_(seq).txt
IDS|DDPLUS     ^(SXUS56) (KSCS) (..)(..)(..)
     FILE     -overwrite -log -close -edex     /data_store/shef/\4/\1_\2_\3\4\5_(seq).txt
IDS|DDPLUS     ^(CXUS86) (KSCS) (..)(..)(..)
     FILE     -overwrite -log -close -edex     /data_store/shef/\4/\1_\2_\3\4\5_(seq).txt

In the above sample, the highlighted \4 expressions are responsible for creating the directory that is locatedd just under /data_store/shef. The \4 means to take the fourth regular expression group that was defined using the WMO header. So if a sample WMO header was SXUS53 KWOH 091200 then the 4th regular expression group in this sample would be 12. The solution is to insert the YYYYMMDD expression in the FILE entries just before this \4 as shown here:

 IDS|DDPLUS     ^(SXUS53) (KWOH) (..)(..)(..)
      FILE     -overwrite -log -close -edex     /data_store/text/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
 IDS|DDPLUS     ^(SXUS58) (KWOH) (..)(..)(..)
      FILE     -overwrite -log -close -edex     /data_store/text/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
 IDS|DDPLUS     ^(SXUS59) (KWOH) (..)(..)(..)
      FILE     -overwrite -log -close -edex     /data_store/text/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
 IDS|DDPLUS     ^(SXUS60) (KWOH) (..)(..)(..)
      FILE     -overwrite -log -close -edex     /data_store/text/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
 IDS|DDPLUS     ^(SXUS56) (KSCS) (..)(..)(..)
      FILE     -overwrite -log -close -edex     /data_store/text/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
 IDS|DDPLUS     ^(CXUS86) (KSCS) (..)(..)(..)
      FILE     -overwrite -log -close -edex     /data_store/text/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H

Example 2:

This WFO has two-digit directories as well (the \2 expression below creates the two-digit directory).

pqact.conf.XXX

IDS|DDPLUS      ^SXUS52 KWBC (..)(..)(..)
      FILE    -overwrite -close -edex /data_store/shef/\2/SXUS52_KWBC_\1\2\3_(seq).txt
IDS|DDPLUS     ^SXUS54 KORN (..)(..)(..)
      FILE    -overwrite -close -edex /data_store/shef/\2/SXUS54_KORN_\1\2\3_(seq).txt
IDS|DDPLUS     ^SEXX60 PAAQ (..)(..)(..)
      FILE    -overwrite -close -edex /data_store/text/\2/SEXX60_PAAQ_\1\2\3_(seq).txt

replacement lines for pqact.conf.XXX

IDS|DDPLUS      ^(SXUS52) (KWBC) (..)(..)(..)
      FILE     -overwrite -close -edex /data_store/shef/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
IDS|DDPLUS     ^(SXUS54) (KORN) (..)(..)(..)
      FILE     -overwrite -close -edex /data_store/shef/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H
IDS|DDPLUS     ^(SEXX60) (PAAQ) (..)(..)(..)
      FILE     -overwrite -close -edex /data_store/shef/(\3:yyyy)(\3:mm)\3/\4/\1_\2_\3\4\5_(seq).%Y%m%d%H

 

Follow WDTD on Facebook  Follow WDTD on Facebook
Follow NWSTC on YouTube  Follow NWSTC on YouTube
NWS RSS Feed  NWS RSS Feed