What's New with the I/O API

Contents

Go to the I/O API Availability/Download page


Bugfix for corrected handling of circular-buffer/RESTART files in READ3(), WRITE3(), INTERP3(), and DDTVAR3().: Aug. 29, 2002

There was an error in the handling of record numbers for I/O API circular-buffer (or "restart") files. This bug was discovered by Dr. Prakash Karamchandani at AER, who suggested the correct bug-fix as well.

(Circular-buffer files keep two time steps, the "even" and the "odd" time step; this allows the maintenance of what would otherwise be huge model-restart files without consuming excessive disk space. Negative values for time-step attribute TSTEP3D are used to indicate to OPEN3() that the requested file to be created is a circular-buffer file...)

Back to Contents


Bugfix in function ISDSTIME() and programs GREGDATE and JULDATE: July 9, 2002

There was an error in the Julian date for October 31, that caused ISDSTIME(), .GREGDATE and JULDATE to think that Daylight Savings Time ended on the last Sunday in September instead of the last Sunday in October. This error is now fixed.

Back to Contents


Bugfix in SETLAM(): June 7, 2002

There was an initialization problem in SETLAM() that caused irregular errors, particularly when called from C. This error is now fixed.

Back to Contents


Bugfix in SETENVVAR(): May 10, 2002

There was an off-by-one problem in SETENVVAR() that caused irregular errors in reading the resulting environment variables. This error is now fixed.

Back to Contents


Version 2.2: Due May 2002

The following set of enhancements does not break backward compatibility -- either at the source, link, or data levels. There is an additional round of enhancements planned for I/O API Version 3 to support the Weather Research and Forecasting Meteorology and Atmospheric Chemistry (WRF; WRF-CHEM) Models; this latter version (to appear 3rd quarter, 2002) will break source and link compatibility to some degree.
CMAQ Enhancements
Integration into the main development path of work by Jeff Young and David Wong at EPA ORD:
INTERPX
time-interpolation for a window into the grid defined by column, row, and layer subranges.

SETENV functionality
Allows the setting of environment variables from within model code. This will also wind up supporting NAMELISTs in the follow-on WRFIO implementation using I/O API Version 3

Support for T3E-style parallel I/O
Support for message-passing distributed-memory CMAQ on T3E/Fortran-90 (and similar platforms).

Minor Restructuring
There has been minor restructuring to improve the decoupling between the interface layer and the data storage/retrieval layer, in order more easily to support additional lower layer implementations (such as an MPI coupling mode).

tbd
...

Enhancements for the SMOKE emissions model
Support for input file-lists for READ3() and INTERP3():
setenv foo "LIST:name1,...,namen
setenv name1 <path>
...
setenv namen <path>
indicates that foo will be treated as a concatenation of files name1, ... namen for READ3() and INTERP3(). The files must have the same header structure, except for the time step structure, which must be compatible (identical time step, and all starting dates and times must be valid time steps for the time step sequence of the chronologically-earliest file). READ3() and INTERP3() take their data from the first listed file that has the relevant data (in case the files overlap).

This enhancement will also make it possible to get around the 2GB limits on input file sizes on various 32-bit platforms.

WRF Enhancements
Routines INQATT3(), RDATT3(), and WRATT3() support the specification of additional per-file and per-variable metadata attributes beyond the standard list, found in the FDESC COMMONs and accessed by the DESC3() routine.
Affects INCLUDE-files IODECL3.EXT and iodecl3.h.

The I/O routines has been re-coded so that it is only required that trimmed lengths (i.e., length exclusive of trailing blanks) of logical names and variable-names must not exceed NAMELEN3=16 (e.g, CHARACTER(LEN=24) FOO = 'v23456789012345   ' will be legal since its trimmed length is 15, it number of nonblank characters (which are then followed by 9 blanks to make out its total declared length of 24).

WRF eventually needs a number of additional enhancements that will break source, link, and backwards data compatibility with the current I/O API. These are being considered for an eventual I/O API Version 3, particularly:

F9x Enhancements

Global Climate I/O API Integration
The Global Climate I/O API is a variant of the I/O lAPI intended for use in global clmate modeling, where some modelers work in terms of a 360-day year having twelve months of exactly 30 days each. The library produced "lives" in a directory of its own, that has a _360 suffix (so is called io_360 e.g., Linux2_alpha_360 For Version 2.2 of the I/O API, we will unify the code of this version with the code for the main branch, so that one may obtain the io_360 by #defineing the preprocessor symbol IO_360 as in the Makefile.io_360 and using the usual build system.

LIST-variable Enhancements
Routines INTLIST(), REALIST(), and STRLIST() have been modified so that they accept an optional LIST: prefix for the value-list, so that one may set the list-value as in the following script-example for an INTLIST() list:
setenv MYLIST "LIST:1,24,9876,5"

Bug-fixes
LAMBERT() entries SET*() had a bug introduced when the extra map projections were added in Version 2.1.

Data type PROFIL3 had implementation errors which were corrected. (We finally had our first user of this data type, which was designed to store and retrieve rawinsonde profile data efficiently.)

I/O API Version 3 will support NAMELEN3=32 character variable and logical names, "staggered" variables, and various other things yet to be determined, in support of WRF. It will not be data- nor link-compatible (and may not be fully source-code compatible) with previous versions of the I/O API. We will maintain forwards data compatibility (so that Version 3 will read Version 1 and Version 2 generated files, but possibly not vice versa.

STATUS: under development. Version 2.2 is due out by the end of 2Q 2002; Version 3 should appear during 2Q 2003.

Back to Contents


bugfix in NEXTIME(): Jan. 15, 2002

It turns out that there was an algorithm error in subroutine NEXTIME() for the case of a negative time step that backed up the JDATE:JTIME argunments across a year boundary (e.g., stepping backwards by an hour from 2002001:003000 by -10000 should yield 2001366:233000 but did not get the year correct.

STATUS: fixed.

Back to Contents


I/O API Version 2.1: December 5, 2001

There were bugs in certain warning-output formats in routines CHKFIL3(), KFOPEN(), KFINDX(), OPNLOG3(), SHUT3(). These bugs arose during the effort to make all of the I/O API OpenMP thread-safe. LBLANK() also had an algorithm fix to make it safe for (F90) zero-length strings.

Intel Fortran and C, and Lahey-Fujitsu Fortran compilers for Linux are now supported by the Makeinclude system.

New routines INTLIST(), REALLIST(), and STRLIST() that read in comma-delimited-list-valued environment variables.

New programs for analysis, model-coupling, grid-to-grid transformation, etc.:

Back to Contents


Bug-fix in  CHECK3(), Makefile

There was a bug in certain warning-output formats in routine CHECK3(). This bug arose during the effort to make all of the I/O API OpenMP thread-safe.

There was also a bug in the way the Makefile was building GCTP(). The problem is that this is a very old Fortran routine from the US Geological Survey, which requires that compile-flags be set so as to SAVE all its local variables (and there turned out to be significant differences how various platforms handled the matter).

Back to Contents


More I/O API map projection support

Parameters have been added to INCLUDE file FDESC3.EXT for the following horizontal map projection types:
POLGRD3 = 6
Polar (secant) stereographic
EQMGRD3: = 7
Equatorial Mercator
TRMGRD3 = 8
Transverse Mercator
The implementation of map-transformations for these is provided directly by USGS routine GCTP. Support for these map projection types has been put into the user-friendly wrapper-module LAMBERT and into interactive map-transformation utility program UTMTOOL.

Back to Contents

New I/O API routine FILCHK3

New routines FILCHK3 (for Fortran) and filchk3c (for C) provide an easy check of whether an input-file's file-type and dimensions match the caller's expectations, with a report to the program log of any mis-matches that it finds. Typical Fortran use might be:
    ...
    IF ( .NOT.OPEN3( 'MYFILE', FSREAD3, 'Myprogram' ) ) THEN
C              MYFILE could not be opened for read-only
        ...
    ELSE IF ( .NOT. FILCHK3( 'MYFILE', 
 &                       GRDDED3, NCOLS, NROWS, NLAYS, NTHIK ) THEN
C            Either the file type for 'MYFILE' was not GRIDDED, or 
C            it had unexpected dimensions (see log for more details).
        ...
    END IF
    ...
    

Back to Contents

"Tools" Programs

In conjunction with various real-time numerical air quality prediction projects such as NAQP, a variety of new I/O API-related "Tools" programs have been developed:

Back to Contents

New Release Version

In cooperation with EPA, we will be releasing the new and updated public version 2.0 of the I/O API in late Fall, 1999. This version incorporates both the task-parallel and the coupling-mode extensions, as well as the time series operations.
The new version is now 2.0.0c, available here.

With this version, the source-code-level build-process uses Makefiles rather than the previous Makeit shell-scripts. This allows much faster re-building of the library for small bug-fixes.

With this version, we will be supporting only netCDF 3.x, because of the support it provides for synchronization during model coupling. Please note that the NCAR folks accidentally incorporated a link-incompatibility between versions 2.x and 3.x of netCDF in some of the "magic numbers" used to describe modes of opening files. Note also that versions 3.4 and later use different error numbers than version 3.3.1 and earlier. The I/O API Troubleshooting Page will be updated to reflect both sets of error numbers. (As of 8/20/1999, the current released version of netCDF is 3.4, although we have been testing the new version 3.5b2 in-house recently.)

Also with this version, the default build of the library will require linking with libpvm3.a for coupling mode support. PVM 3.4 or later is required, because of the requirement for PVM mailboxes.

For some vendors, linking of serial programs will also require the explicit specification of an OpenMP run-time library as well -- on SGI's, for example, this requires an additional -lmp in the linking-step command line for serial programs.

Back to Contents

OpenMP Task-Parallel Extensions

OpenMP is a recent standard for directive-based shared-memory parallel Fortran and C. The Fortran and C standards were released in October, 1997 and October, 1998, respectively; a revision of the Fortran standard is expected in October, 1999. Compilers supporting the OpenMP standard are available for a variety of platforms, including Compaq (DEC), HP, IBM, Linux (both x86 and Alpha), SGI, Sun, and Win32 (95, 98, and NT (both x86 and Alpha)). OpenMP supports not only the kind of loop-level parallelism that has long been available from a variety of vendors, but also task-parallelism as well. In addition to much better portability, it offers much better support (critical sections, barriers, nested parallelism) for large projects than did the proprietary vendor specific directives. We are using OpenMP Fortran in-house with various air quality modeling projects; with its MM5 meteorology model, NCAR is moving to OpenMP directives as of its MM5 Version 3.

We are making the I/O API thread-safe for OpenMP, and have been testing the result in various real-time air quality forecasting and other meteorological projects. The basic principle is that I/O API calls that read or write distinct variables and/or distinct files may be done concurrently from OpenMP programs. The biggest gains for model-users are with INTERP3 and DDTVAR3, where the call involves linear algebra computations (linear interpolation or differencing) in addition to I/O (which is performed sometimes--but not always, because of the internal double-buffering mechanisms. Note that I/O to/from disk is itself sequential in any case.)

Building a parallel program that uses an OpenMP-enabled libioapi.a generally requires compiler-specific parallelization directives in both the compile-lines and the link-lines that make your executable. For examples, ook at your machine/compiler's Makeinclude.${BIN} for variables OMPFLAGS andOMPLIBS used with this build of the library

Back to Contents

Data-Parallel Extensions

For message-passing data parallel programming, our colleagues at EPA's Atmospheric Research and Exposure Assessment Laboratory (AREAL) have been constructing an extension to the I/O API which distributes gridded data for domain-decomposition data parallel message passing models. Inquiries should be addressed to Dr. Jeffrey Young, yoj@hpcc.epa.gov.

As part of the Practical Parallel Project, MCNC has developed an HPF2 implementation of an HPF I/O API module which calls the I/O API and then re-destributes gridded or boundary data according to the data distribution of its caller. Unfortunately, however, the IBM XL HPF and Portland Group HPF compilers we use support only their own extensions to HPF 1.1 at this time, so we have had to develop revised HPF I/O API modules for each which are not fully general but which do fulfill the requirements of our HPF air quality and hydrology models.

Back to Contents

Parallel Extensions for Coupled Models

As part of the Practical Parallel Project, MCNC has developed an extended Model-Coupling Mode for the I/O API. This mode, based upon PVM 3.4 mailboxes, allows the user to specify in the run-script whether "file" means a physical file on disk or a PVM mailbox-based communications channel. In the latter case, the I/O API schedules the various processes in a coupled model on the basis of data availability: requests for data which has not yet been written put the reading process to sleep until the data arrives, at which point the reader is awakened and given the data it requested. The one requirement is that the programmers structure reads and writes so as to avoid deadlocks (two or more models, each asleep while waiting for input from the other).

This has several advantages from the model-engineering point of view:

Back to Contents

Time Series READ/WRITE

In order to support the needs of surface-water, lake and estuary, and bay modeling, at the request of EPA-AREAL we have added two routines to the I/O API which will access an entire time step sequence of data in a single operation ("read or write N time steps of data starting at date and time D:T with step DT for variable V to/from file F"). The new routines are READ4D() and WRITE4D(), and very much resemble READ3() and WRITE3(), except for the specification of an entire time step sequence as arguments, the restriction to single-variable operations (ALLVAR3 is not supported for these) and file types CUSTOM3, GRDDED3, BNDARY3, and TSRIES3.

Back to Contents

Additional Supported Platforms

Conditional compilation directives in the I/O API were recently modified to make it easier to port the I/O API to other platforms. Three issues are prominent: Preprocessor definitions and conditional compilation directives recognizing them have been provided in the code: defining FLDMN=0,1 specifies the first; REAL8=0,1 specifies the second, and AUTO_ARRAYS=0,1 the last of these. Work is in progress to provide the I/O API on at least the following additional platforms:

Back to Contents


MCPL: an I/O API output module for MM5

MCPL is an I/O API output module for MM5 developed under the PPAR project. It is MCPL is designed to fit into MM5 with a minimum of effort, to offer great flexibility of run-time configuration, and to be written in "boring" code that is easy to understand, extend, and maintain. It generates output files with a variety of kinds of variables:

Back to Contents


Go to the I/O API Availability/Download page

Go to the I/O API Home Page

Send comments to

Carlie J. Coats, Jr.
coats@emc.mcnc.org