What's New with the I/O API

Contents

Go to the I/O API Availability/Download page


ug fix for GCTP;
Format fix for envrealc() and envdblec()

January 18, 2005:

Catherine Seppannen of UNC-Carolina Environmental Program diagnosed an obscure bug that occurs in the GCTP map-projection package from USGS:

I've traced the problem to the subroutine SPHDZ0 in gctp.f. When GTPZ0 is first called, it calls SPHDZ0 with dummy values that cause the spheroid to be set to Clarke 1866. A later call to SPHDZ0 is made that looks like it should reset the spheroid. However, at line 5475, the subroutine checks if the spheroid needs to be set and then returns. This means that the spheriod is never changed from Clarke 1866. When processing UTM coordinates, the spheroid is changed outside of SPHDZ0 (at lines 1273 and 1274) so the correct spheroid is used in later calculations. For now, simply removing line 5475 and moving the label to the next line seems to solve the problem.

Dr. Michael Bane of the University of Manchester (UK) notes that there are problemswith underflow with the existing %f format used in envrealc() and envdblec(). This has now been changed to a more-robust %G,that should not be subject to underflow.

Back to Contents


Range checks for NVARS in OPEN3

December 9, 2004:

To facilitate user changes to parameter MXVARS3 in INCLUDE-file PARMS3,EXT, additional checks were inserted into OPEN3 and its worker routines to ensure that one may not inadvertently open files with NVARS too large for the current libioapi.a build.

Note that environments in which nonstandard values of parameters from PARMS3,EXT are used must be managed with great care: inconsistent values of these parameters for different objects or files within a build- or usage-scenario can lead to very hard to diagnose, obscure failures, and requesting help for such difficulties will still invoke the triple-time support penalties in the I/O API license.

Back to Contents


Fixes to M3TPROC

December 1, 2004:

Dr. Michael Bane of the University of Manchester, Manchester, UK, submits changes to program M3TPROC to allow clean compilation on the new Intel Version 8.1 compilers: removal of a duplicate declaration for ASTEPS, and reversion to F77 style declarations (instead of F90 style) for MENUITMS and OPNAMES.

Back to Contents


Duplicate declaration removed from STATE3.EXT; new experimental Fortran-90 MODULE M3UTILIO

November 30, 2004:

Dr. Michael Bane of the University of Manchester, Manchester, UK, noticed a duplicate declaration of variable VERSN3 in INCLUDE-file STATE3.EXT. This was clearly an error, and may have been responsible for cryptic error messages on various platforms. He also points out that Intel "ifort" Version 8 generates a "Command line warning: overriding '-auto' with '-save'" message during the compilation of GCTP; this latter message is not a symptom of an underlying problem (the build should still be correct.

MODULE M3UTILIO: Fortran-90 INTERFACE Blocks

There is a new experimental Fortran-90 module MODULE M3UTILIO, available, in file m3utilio.f, coming as a consequence of development work to enable the I/O API and SMOKE to couple with WRF-Chem. This module is not built by a default build (partly because of the lack of standards for naming module-files in the various Fortran 90/95 compilers), but may be built and used by the interested modeler. In addition to some cleaned-up utility routines used by SMOKE, it contains Fortran-90 INTERFACEs for most of the public I/O API utility routines (excepting those for which normal usage depends upon using an argument as a void pointer, i.e., in a non-typesafe manner (as in the sparse matrix multiplication routine SMATVEC,when used with an ALLOCATEd matrix-buffer filled from READ3). It also INCLUDEs the PARMS3.EXT, FDESC3.EXT, and IODECL3.EXT files, so that a single
USE M3UTILIO
replaces declarations of all the functions which have the INTERFACE blocks, as well as the inclusions:
INCLUDE 'PARMS3.EXT'
INCLUDE 'FDESC3.EXT'
INCLUDE 'IODECL3.EXT'
This greatly simplifies many I/O API applications.

I am thinking of splitting this module into two pieces, however: an INTERFACE block/declaration module (M3IO?), and a module for the SMOKE-related utility routines (MUTIL?); I welcome feedback on that matter (and in fact on the whole idea). -- CJC

SunOS SPARC 64-Bit? Opteron 32- and 64-bit? Others?

I would be interested in adding 64-bit compile-support for Sun SPARC and both 32-bit and 64-bit support for Sun Opteron systems. I no longer have access to the Sun compiler suite even for 32-bit SPARC, and would be glad to give credit for someone who would contribute the compile-flags to support these systems -- as well as whatever others you know how to build.-- CJC

Back to Contents


Enhancement for CHECK3

November 22, 2004:

During the transition to I/O API Version 3, the responsibility for "timestep not available" status and for messages about that status was transferred from routines CHECK3 DDTVAR3 INTERP3 READ3 READ4D XTRACT3 to worker routine RDTFLAG. However, such messages should be generated for all of these routines except CHECK3, as was discovered during the implementation of the new WRF-enabled / sub-gridscale terrain version of the SMOKE TMPMOBIL program. Therefore, a new "everbose-output" flag was added to the argument list for RDTFLAG, and the other routines were modified accordingly.

Back to Contents


Bug fix, version in sample program VERTOT

November 19, 2004:

Bill Hutzell of EOA's Atmospheric Modeling Division, National Exposure Research Laboratory, pointed out a bug in sample program VERTOT, and provided the fix: when used to process all variables, the previous version of the program did not handle the list of output variable names (etc.) correctly.

Back to Contents


Bug fix, version in CRTFIL3()

November 1, 2004:

There was a coding bug in routine GCD() and bugs in the error messages for worker-routine RDTFLAG(). These are now corrected.

Program M3TPROC was enhanced so that it will now do time aggregation of minima, in addition to the previously supported arithmetic means and maxima (so that one can now compute daily-minimum layer 1 temperature, for example). The time-stepping logic was enhanced to support partial-input aggregation, and the program control logic was completely overhauled to get rid of the absurdly complex hierarchy of environment variables and command-line prompts used in the prior version. Consequently, scripts used with the previous version will no longer work: the new version requires re-scripting.

Back to Contents


Bug fix, version in CRTFIL3()

October 15, 2004:

As pointed out by Dr. Michael Bane of the University of Manchester, Manchester, UK, worker routines CRTFIL3() and CRTKF3() were not correctly picking up the I/O API version to insert into file headers. This is now corrected.

Back to Contents


Bug fix, virtual mode INTEPR3()

September 7, 2004:

The virtual-mode part of the interp3() system still had a bug that occurred for PVM-based "virtual" files (netCDF file behavior is unaffected). This has now (finally!) been fixed and the fix validated.

Back to Contents


Bug fixes; program m3interp

September 3, 2004:

ungridb() had a fencepost error in dealing with the last row of the input grid, leading to errors when the output grid extended beyond that edge. Bug was reported by Catharine Sepannen, UNC CEP.

open3() now checks for repeated occurrences of variabl names when opening files as "'new"' or "unknown", leading to more meaningful error messages than those which would otherwise come from netCDF.

The virtual-mode part of the interp3() system has been re-written to eliminate a bug which occurs forPVM-based "virtual" files (netCDF file behavior is unaffected).

New program m3interp is similar in capabilities to m3cple but in addition does time-interpolation to a potentially higher-resolution time step sequence. (This program has been moved from iotests directory to m3tools.)

Back to Contents


Bug fix in currstepc()

August 4, 2004:

currstepc() was wrong -- I found the bug during attempt to rationalize/re-engineer the structure of PAVE. The previous version would almost certainly have caused a seg-fault.

Back to Contents


Bug fix in M3COMBO

July 30, 2004:

M. Talat Odman of Georgia Tech points out a bug in M3COMBO: the first term in the linear combinations was wrong for multi-variable M3COMBO runs.

Back to Contents

<


Bug fixes in M3MSG2 and UNGRIDB

July 26, 2004:

Catherine Seppanen of UNC-CEP points out a bug in UNGRIDB: several clauses of the form

    ELSE IF ( C .GE. NCOLS - 1 ) THEN !  c right of grid
    
should instead be
    ELSE IF ( C .GT. NCOLS - 1 ) THEN !  c right of grid
    
The result of the bug was an inappropriate extend-by-constant computation for the last (N-S) column of the horizontal grid.

M3MESG and M3MSG2 had an argument-mismatch problem if they had to call worker routine INITLOG3 (which would occur if they were called prior to callingINIT3; this has been fixed.

Back to Contents

HR>

Message fix in MTXCALC

May 25, 2004:

Luca Delle Monache of the Atmospheric Science Programme at the University of British Columbia points out that the warning message for unsupported coordinate system types was incorrect in program MTXCALC. This is now fixed in the I/O API Version 3 download.

Back to Contents


I/O API Version 3.0 Pre-Release 2

May 17, 2004:

David Wong of SAIC characterized a bug in the BINFIL3 version of XTRACT3(), which is now fixed. There was also an over-dimensioning of the malloc()ed input buffer in that routine, now corrected.

Back to Contents


I/O API Version 3.0 Pre-Release 1

May 11, 2004:

I/O API Version 3.0 is now available for final user testing. Among its features are the following:

  • New/improved build-system with a new ew top-level Makefile which allows one to build various configurations of the entire (I/O API + M3TOOLS) system with one make command.

  • More complete WRF support
    • File and variable name argumentss are now declared CHARACTER*(*) with a maximum length of 16, instead of being fixed-length CHARACTER*16). (This is still not fully compatible with WRF, which uses a maximum length of 32...)
    • Fortran INCLUDE-files are now in the ISO-suggested "hybrid format" (see below).

  • INCLUDE files are now safe for both fixed-form (F77-style) and free-form (F90-style) source code format: line continuations have two ampersands, one in column 73 (where it is not visible to F77-style compilation, and one in column 6, where it is the first non-blank character in the line.

  • New script fixed_src.csh and build target make fixed_src to deal with INCLUDE files for nonstandard CMAQ fixed-132 source code form.

  • A new "native-binary I/O" (BINFIL3) lower layer, in addition to the current netCDF, BUFFERED, VIRTUAL, and FILE-LIST lower layers. This is the least-tested part of the code, and is primarily intended for use with ETA/CMAQ at NOAA/NCEP, where the departmental politics prevents the existence of libnetcdf.a on any NCEP computer system.

  • A more thorough-going treatment of Fortran/C interface type-matching, to deal with portability to platforms where one or more of the following do not occur:
    • Fortran REAL is not C float
    • Fortran INTEGER is not C int
    • Fortran CHARACTER string length type is not C int

  • Support for additional compiler/OS combinations, including Intel Fortran and C for both Linux2_x86 and Linux2_ia64, as well as the new g95 GNU Fortran-95 compiler for Linux2_x86 (see URL http://www.g95.org/)

  • Various new routines, including M3PARAG, M3FLUSH, IOPARMS3 GET_ENDIAN with corresponding parameters LITTLE_ENDIAN, BIG_ENDIAN, and PDP_ENDIAN in PARMS3.EXT and parms3.h

  • Messaging routines M3MESG and M3MSG2, M3WARN, and M3EXIT, have been modified to get around problems with SGI compilers that were causing duplicate copies of log output on SGIs.

  • Improved OpenMP thread-safety.

  • The tar-file naming scheme now matches GNU standards.

Documentation for I/O API Version 3 continues to be in preparation.

Back to Contents


New Makefiles for I/O API version 2.2; beta availability for I/O API version 3.

May 4, 2004:

In the retrofitting of M3PARAG, M3FLUSH, etc., into I/O API Version 2.2, I failed to update the Makefiles to match. This error has now been corrected for the new download tar-file.

The beta release of I/O API Version 3, as described below is now available for download.

Further documentation for I/O API Version 3 is in preparation.

Back to Contents


New subroutines M3PARAG, M3FLUSH

May 1, 2004:

New subroutine M3PARAG ("M3-paragraph") allows the caller to write multi-line messages (in a thread-safe manner) to the log.

New subroutine M3FLUSH allows the caller to flush a specified Fortran file to disk. This routine is useful for dealing with portability problems due to the fact that various operating system/Fortran compiler combinations define FLUSH inconsistently.

Subroutines M3MESG, M3WARN, M3EXIT, and M3ERR have been re-worked to enhance OpenMP thread-safety. (Actually, all of these except M3WARN, M3EXIT,and M3ERR are ENTRYs to the nified message-treatment now implemented in m3msg2.F.)

Back to Contents


New subroutine IOPARMS3

April 16, 2004:

New subroutine IOPARMS3 allows for run-time checks in model-level code, to check the I/O API dimensioning parameters from INCLUDE-file PARMS3.EXT being used to build the model match those used to build the libioapi.a being linked against. This is made the more necessary by the tendency of some among the SMOKE emissions modeling community to change dimensioning parameters in PARMS3.EXT and not simultaneously update the INCLUDE-files used for building SMOKE itself.

Back to Contents


Bug-fix to subroutines LAMBERT; enhancements to CKFILE3, and CKDESC3

March 30, 2004:

Eric Giroux of the National Research Council of Canada contributes a bug-fix and code enhancements made by his colleague Dazhong Yin:

  • bug-fix to routine LAMBERT, dealing with conversions from polar sterographic to Lat-Lon coordinates.
  • enhancements to CKFILE3, and CKDESC3 to provide appropriate "sanity checking" to file descriptions, when the underlying coordinate system is polar sterographic.

Back to Contents


Bug-fix to M3STAT

December 2, 2003:

There were two problems with M3STAT: one serious, the other less so:

  • For single-variable boundary files (FTYPE3D=BNDARY3), the program did not set local variable SIZE correctly, leading to segmentation faults when the worker routine tried to use SIZE as a dimension for the input buffer.
    Fixed.

  • When run without command-line arguments, the program did not accept NONE as the logical name for the REPORT file, so that the program would generate output to the log file or screen.
    Fixed.

Back to Contents


GDTYP=STEGRD3 checking; Type-portability, I/O API Version 3

October 17, 2003:

Éric Giroux points out that the treatment in CKDESC() for map projection parameter PROJ_GAMMA for polar sterographic grids (GDTYP=STEGRD3) is incorrect: this parameter should have been ignored, instead of being checked against the range [-180,180]. Fixed in the new 2.2 release for Oct. 17.

David Wong points out that ENVREAL does not work correctly on the Cray T3E. A correct fix for this problem involves the use throughout of typedef FREAL and of type-casts back and forth between FREAL and float. Stopgap-fix in the new 2.2 release for Oct. 17; this will be treated more carefully in the upcoming I/O API Version 3 (below)

I/O API Version 3 is now under development. Among its features will be the following:

  • More complete WRF support.
  • INCLUDE files are now safe for both fixed-form (F77-style) and free-form (F90-style) source code format.
  • New script fixed_src.csh and build target make fixed_src to deal with INCLUDE files for nonstandard CMAQ fixed-132 source code form.
  • A new "native-binary I/O" (BINFIL3) lower layer, in addition to the current netCDF, BUFFERED, VIRTUAL, and FILE-LIST lower layers.
  • A more thorough-going treatment of Fortran/C interface type-matching, to deal with platforms where one or more of the following do not occur:
    • Fortran REAL is not C float
    • Fortran INTEGER is not C int
    • Fortran CHARACTER string length type is not C int
For more information, contact the author,
Carlie J. Coats, Jr.
carlie.coats@baronams.com

Back to Contents


Makeinclude.OSF1 enhancements

August 20, 2003:

Jinyou Liang, of the California Air Resources Board, suggests the following additions to Makeinclude.OSF1, for making the parallel version of the I/O API on Alpha OSF1 systems:

Add to FOPTFLAGS: -arch host -tune host
Add to OMPFLAGS: -omp
Add to OMPLIBS: -omp

Back to Contents


Compiler-Bug Report: IRIX F90 Version 7.4

July 9, 2003:

Conditional-compilation for the I/O API now takes care of the fact that netCDF open-flag NF_SHARE is not implemented in netCDF for CRAYs. This affects only users who have more than one program simultaneously accessing the files. These users can still share files safely if both reader and writer processes declare the shared file to be "volatile".

In the process of doing the May 19 and June 5 bugfixes and enhancements, an OpenMP critical-section deadlock related to INIT3() was created in various routines. The critical sections related to INIT3() have now been re-organized in a cleaner, more consistent fashion, avoiding this deadlock. Affected routines are:

xtract3.f
write4d.F
write3.F
wrdict3.f
wratt3.f
sync3.f
snoop3.F
read3.F
read4d.F
read3.1.F
rdatt3.f
promptmfile.f
open3.F
m3warn.f
kfwrite.F
kfread.F
kfindx.f
interp3.F
inqatt3.f
getffile.F
getefile.F
getdfile.F
flush3.f
desc3.F
ddtvar3.F
ckgeom.f
check3.F
ll2utm.f
init3.F
m3abort.F

SGI IRIX F90 Version 7.4 does not follow industry-standard practice for ensuring that BLOCK DATA INITBLK3 is correctly linked in, to initialize I/O API internal data structures. Among other things, this causes problems with a user-setLOGFILE; for more details, look here. I have submitted this problem to SGI in an error report.
--CJC Note, Spring 2004: SGI contacted me about this, and have fixed it for the 7.4.1 and later compilers.

Back to Contents


Enhancements for IRIX F90

June 5, 2003:

When standard output is re-directed to a log file, SGI IRIX6 F90 Version 7.4 has anomalies in the way standard output is flushed to a file. Routines GETDBLE, GETMENU, GETNUM, GETREAL, GETSTR, M3MESG, M3MSG2, PROMPTDFILE, PROMPTFFILE, PROMTMFILE, PROMPTGRID have been enhanced to attempt to deal with these anomalies; additionally, new entry point M3FLUSH( INTEGER IUNIT ) for M3MSG2()has been added, for user-code to use for flushing its standard output, in order to similarly avoid these anomalies.

Back to Contents


Minor BugFixes/Enhancements for BUFFERED Files.

May 19, 2003:

There were errors in OPEN3(), XTRACT3(), and RDBUF3() dealing with the return data-types of worker routines when these routines were called on virtual files of type

BUFFERED. There was also a format-mismatch error in a warning message in OPNKF3() These were discovered, and fixes provided by David Wong, US EPA.

Back to Contents


Minor BugFixes/Enhancements

May 15, 2003:

The May 8, 2003 source-distribution tar-file was accidentally constructed using full path-names to specify the directories, rather than relative path names. This was corrected; moreover, make-target gtar was added to the Makefile so that the tar-file building process would be both correct and automated.

Function CRLF(), that had originated in SMOKE and that is used (afaik) only in that model, was modified so that it generates correct C-style line terminators for both UNIX and Windows (instead of only for Windows, as its predecessor did...) [UNIX traditionally uses the ASCII line-feed character as a line terminator; Windows uses a line-feed/carriage-return pair; Mac OS (not yet supported ;-( ) uses a carriage-return/line-feed pair.] Note that the insertion of C-style line terminators into the middle of Fortran strings for carriage control with Fortran I/O is not standard-conforming and in fact will not work on some systems (e.g., VMS, Crays).

Function XTRACT3() was modified to use time-step-check routine RDTFLAG() to bring it into conformity with the rest of the Version 2.2 I/O API input routines (which no longer have in-lined code for time step availability-checking, but use that routine instead).

Back to Contents


Enhancement to allow for OpenMP-parallel INTERP3() and/or KFOPEN() on files not yet opened
New make-target nametest checks linker-compatibility for libioapi.a and libnetcdf.a
New script nm_test.csh for testing linker-name compatibility.

May 8, 2003:

Enhancement allows for more relaxed/less careful model implementation, and removes the possibilities of deadlocks on opening multiple virtual files in parallel.

make nametest assumes that libioapi.a is built, and that libnetcdf.a has been correctly placed in ${OBJDIR}, and then checks that the compilers used to generate these two libraries use compatible name-mangling systems (e.g., the same number of trailing underscores, if any).

make all will run make nametest and report the results to the screen.

nm_test.csh <file1> <file2> <name> may be used to test that the compiler systems used to build file1 and file2 generate the same linker symbol name. Note that file1 and file2 should be object, executable, or archive (".a") files. For testing against libnetcdf.a, a good symbol is nf_open; for testing against libioapi.a, a good symbol is open3 (or possibly the all-caps versions of these names).

Back to Contents


Bugfix for SETTRM(); and new spheroid-selection routines SETSPHERE(), INITSPHERES(), and SPHEREDAT(). Apr. 14, 2003

These routines allow the user to select a non-default sphere or spheroid for use with the I/O  wrappers (LAM2LL, LL2UTM, POL2TRM, etc.) for GCTP, either by means of environment variables or explicit subroutine calls.

Code for SETTERM had an incorrect initializations for XCENT and YCENT; this would silently generate corrupt results.

Back to Contents


Bugfix for I/O API Version 2.2 rdatt3c(), wratt3c(), BUFINTX, and BUFINTXD: Feb. 18, 2003

These bugs were discovered by David Wong, US EPA.

Code for Cray versions of rdatt3c() and wratt3c() was incorrect (backwards test of whether the variable was of type CHARACTER or not, then incorrect Cray-specific treatment of file, variable, and attribute-names.

Code for BUFINTX and BUFINTXD had incorrect pointer-initializations; this would silently generate corrupt results.

Code for the Cray version of public routine SETENVVAR() and private routine lentrim() had syntax bugs, that are now fixed.

Back to Contents


Bugfix for I/O API Version 2.2 program M3TPROC: Feb. 7, 2003

This bug was found by Éric Giroux of the Institute for Chemical Process and Environmental Technology, National Research Council, Canada. He also aided in testing the solution as implemented.

Run-time memory allocation of internal work arrays was handled incorrectly when there were mixed output-lists of M3REAL and M3INT variables with M3DBLE variables. New code uses F90 "auto" work arrays, and completely avoids the problem.

Back to Contents


Bugfix for I/O API Version 2.2 program M3TPROC: Nov. 14, 2002

The previous edition would silently generate garbage for variables of type INTEGER or DOUBLE, or for files not of the types listed above. The new version handles all three types of variables correctly, and generates meaniangful error messages for the inappropriate file types. The new version is much cleaner in its coding, but for Fortran-90 only.

Back to Contents


Corrected handling of circular-buffer/RESTART files in READ3(), WRITE3(), INTERP3(), and DDTVAR3(): Aug. 29, 2002

There was an error in the handling of record numbers for I/O API circular-buffer (or "restart") files. This bug was discovered by Dr. Prakash Karamchandani at AER, who suggested the correct bug-fix as well.

(Circular-buffer files keep two time steps, the "even" and the "odd" time step; this allows the maintenance of what would otherwise be huge model-restart files without consuming excessive disk space. Negative values for time-step attribute TSTEP3D are used to indicate to OPEN3() that the requested file to be created is a circular-buffer file...)

Back to Contents


Bugfix in function ISDSTIME() and programs GREGDATE and JULDATE: July 9, 2002

There was an error in the Julian date for October 31, that caused ISDSTIME(), .GREGDATE and JULDATE to think that Daylight Savings Time ended on the last Sunday in September instead of the last Sunday in October. This error is now fixed.

Back to Contents


Bugfix in SETLAM(): June 7, 2002

There was an initialization problem in SETLAM() that caused irregular errors, particularly when called from C. This error is now fixed.

Back to Contents


Bugfix in SETENVVAR(): May 10, 2002

There was an off-by-one problem in SETENVVAR() that caused irregular errors in reading the resulting environment variables. This error is now fixed.

Back to Contents


Version 2.2: Due May 2002

The following set of enhancements does not break backward compatibility -- either at the source, link, or data levels. There is an additional round of enhancements planned for I/O API Version 3 to support the Weather Research and Forecasting Meteorology and Atmospheric Chemistry (WRF; WRF-CHEM) Models; this latter version (to appear 3rd quarter, 2002) will break source and link compatibility to some degree.
CMAQ Enhancements
Integration into the main development path of work by Jeff Young and David Wong at EPA ORD:
INTERPX
time-interpolation for a window into the grid defined by column, row, and layer subranges.

SETENV functionality
Allows the setting of environment variables from within model code. This will also wind up supporting NAMELISTs in the follow-on WRFIO implementation using I/O API Version 3

Support for T3E-style parallel I/O
Support for message-passing distributed-memory CMAQ on T3E/Fortran-90 (and similar platforms).

Minor Restructuring
There has been minor restructuring to improve the decoupling between the interface layer and the data storage/retrieval layer, in order more easily to support additional lower layer implementations (such as an MPI coupling mode).

tbd
...

Enhancements for the SMOKE emissions model
Support for input file-lists for READ3() and INTERP3():
setenv foo "LIST:name1,...,namen
setenv name1 <path>
...
setenv namen <path>
indicates that foo will be treated as a concatenation of files name1, ... namen for READ3() and INTERP3(). The files must have the same header structure, except for the time step structure, which must be compatible (identical time step, and all starting dates and times must be valid time steps for the time step sequence of the chronologically-earliest file). READ3() and INTERP3() take their data from the first listed file that has the relevant data (in case the files overlap).

This enhancement will also make it possible to get around the 2GB limits on input file sizes on various 32-bit platforms.

WRF Enhancements
Routines INQATT3(), RDATT3(), and WRATT3() support the specification of additional per-file and per-variable metadata attributes beyond the standard list, found in the FDESC COMMONs and accessed by the DESC3() routine.
Affects INCLUDE-files IODECL3.EXT and iodecl3.h.

The I/O routines has been re-coded so that it is only required that trimmed lengths (i.e., length exclusive of trailing blanks) of logical names and variable-names must not exceed NAMELEN3=16 (e.g, CHARACTER(LEN=24) FOO = 'v23456789012345   ' will be legal since its trimmed length is 15, it number of nonblank characters (which are then followed by 9 blanks to make out its total declared length of 24).

WRF eventually needs a number of additional enhancements that will break source, link, and backwards data compatibility with the current I/O API. These are being considered for an eventual I/O API Version 3, particularly:

  • 32-character names
  • millisecond-accurate date&time functionality (with accompanying interface changes to READ3(), WRITE3(), etc.
  • New WRF-specific vertical coordinate types
  • Built-in support for WRF "staggered" variables.
  • Support in INTERP3() (and DDTVAR3() for double precision variables.

F9x Enhancements
  • INCLUDE-files have been modified so that they are compatible with both the free and the fixed F9x source forms. Among other things, this supports new development for CMAQ, SMOKE, and WRF. Script fix_src.csh is available for those users (e.g., CMAQ) who demand nonstandard 132-column fixed F90 source format; Makefile goal fixed_src constructs nonstandard 132-column INCLUDE-files in a user-configurable directory.

  • I/O API Data type PROFIL3 now uses REAL position-variables in each time step record, instead of REAL*8.

Global Climate I/O API Integration
The Global Climate I/O API is a variant of the I/O lAPI intended for use in global clmate modeling, where some modelers work in terms of a 360-day year having twelve months of exactly 30 days each. The library produced "lives" in a directory of its own, that has a _360 suffix (so is called io_360 e.g., Linux2_alpha_360 For Version 2.2 of the I/O API, we will unify the code of this version with the code for the main branch, so that one may obtain the io_360 by #defineing the preprocessor symbol IO_360 as in the Makefile.io_360 and using the usual build system.

LIST-variable Enhancements
Routines INTLIST(), REALIST(), and STRLIST() have been modified so that they accept an optional LIST: prefix for the value-list, so that one may set the list-value as in the following script-example for an INTLIST() list:
setenv MYLIST "LIST:1,24,9876,5"

Bug-fixes
LAMBERT() entries SET*() had a bug introduced when the extra map projections were added in Version 2.1.

Data type PROFIL3 had implementation errors which were corrected. (We finally had our first user of this data type, which was designed to store and retrieve rawinsonde profile data efficiently.)

I/O API Version 3 will support NAMELEN3=32 character variable and logical names, "staggered" variables, and various other things yet to be determined, in support of WRF. It will not be data- nor link-compatible (and may not be fully source-code compatible) with previous versions of the I/O API. We will maintain forwards data compatibility (so that Version 3 will read Version 1 and Version 2 generated files, but possibly not vice versa.

STATUS: under development. Version 2.2 is due out by the end of 2Q 2002; Version 3 should appear during 2Q 2003.

Back to Contents


bugfix in NEXTIME(): Jan. 15, 2002

It turns out that there was an algorithm error in subroutine NEXTIME() for the case of a negative time step that backed up the JDATE:JTIME argunments across a year boundary (e.g., stepping backwards by an hour from 2002001:003000 by -10000 should yield 2001366:233000 but did not get the year correct.

STATUS: fixed.

Back to Contents


I/O API Version 2.1: December 5, 2001

There were bugs in certain warning-output formats in routines CHKFIL3(), KFOPEN(), KFINDX(), OPNLOG3(), SHUT3(). These bugs arose during the effort to make all of the I/O API OpenMP thread-safe. LBLANK() also had an algorithm fix to make it safe for (F90) zero-length strings.

Intel Fortran and C, and Lahey-Fujitsu Fortran compilers for Linux are now supported by the Makeinclude system.

New routines INTLIST(), REALLIST(), and STRLIST() that read in comma-delimited-list-valued environment variables.

New programs for analysis, model-coupling, grid-to-grid transformation, etc.:

  • M3TPROC: compute time period aggregates (e.g., 08:00-16:00 gridded daily maxima) and write them to an output file

  • M3CPLE: copy to the same grid, or interpolate to another grid a time sequence of all variables from a source file

  • MTXBLEND: Use a sparse-matrix file to interpolate/transform data from an input file to the grid of a "base" file and to merge it with data from the "base" file.

  • MTXBUILD: Build a sparse-matrix transform file from user-supplied ASCII coefficient inputs.

  • MTXCALC: Build a grid-to-grid sparse-matrix transform file using a sub-sampling algorithm.

  • MTXCPLE: Use a sparse-matrix file to interpolate a time sequence of all variables from a source file to a target file.

  • PRESTERP: Interpolate from a 3-D sigma-coordinate file to a new 3-D pressure-coordinate file, using coefficients from PRES_CRO_3D from the MCPL I/O API output module for MM5.

  • SELMRG2D: select multiple 2D layer/variable combinations from multiple gridded input files, and write result to merged 2D gridded output file

Back to Contents


Bug-fix in  CHECK3(), Makefile

There was a bug in certain warning-output formats in routine CHECK3(). This bug arose during the effort to make all of the I/O API OpenMP thread-safe.

There was also a bug in the way the Makefile was building GCTP(). The problem is that this is a very old Fortran routine from the US Geological Survey, which requires that compile-flags be set so as to SAVE all its local variables (and there turned out to be significant differences how various platforms handled the matter).

Back to Contents


More I/O API map projection support

Parameters have been added to INCLUDE file FDESC3.EXT for the following horizontal map projection types:
POLGRD3 = 6
Polar (secant) stereographic
EQMGRD3: = 7
Equatorial Mercator
TRMGRD3 = 8
Transverse Mercator
The implementation of map-transformations for these is provided directly by USGS routine GCTP. Support for these map projection types has been put into the user-friendly wrapper-module LAMBERT and into interactive map-transformation utility program UTMTOOL.

Back to Contents

New I/O API routine FILCHK3

New routines FILCHK3 (for Fortran) and filchk3c (for C) provide an easy check of whether an input-file's file-type and dimensions match the caller's expectations, with a report to the program log of any mis-matches that it finds. Typical Fortran use might be:
    ...
    IF ( .NOT.OPEN3( 'MYFILE', FSREAD3, 'Myprogram' ) ) THEN
C              MYFILE could not be opened for read-only
        ...
    ELSE IF ( .NOT. FILCHK3( 'MYFILE', 
 &                       GRDDED3, NCOLS, NROWS, NLAYS, NTHIK ) THEN
C            Either the file type for 'MYFILE' was not GRIDDED, or 
C            it had unexpected dimensions (see log for more details).
        ...
    END IF
    ...
    

Back to Contents

"Tools" Programs

In conjunction with various real-time numerical air quality prediction projects such as NAQP, a variety of new I/O API-related "Tools" programs have been developed:
  • mtxblend: Use a sparse-matrix linear-operator transform-file (as from mtxbuild, below) to perform a linear grid-to-grid transform data from an input file to a new output grid (general purpose linear-operator-transform tool), and then perform a masked merge of the result with data from another "base" file on the same grid as the output from the sparse-matrix transform. Supported output file type: GRIDDED.

  • mtxbuild: Construct a sparse-matrix linear-operator transform-file (as for mtxcple, below) from a user-supplied ASCII file of coefficients.

  • mtxcple: Use a sparse-matrix linear-operator transform-file (as from mtxbuild, above) to perform a linear grid-to-grid transform from an input file to a new output file (general purpose linear-operator-transform tool). Supported output file types are GRIDDED, BOUNDARY, or CUSTOM.

  • m3cple: copy to the same grid, or interpolate to another grid a time sequence of all variables from a source file to a target file, under the optional control of an I/O API coupling-mode "synch file".

  • selmrg2d: select multiple 2D layer/variable combinations from multiple gridded input files, and write result to merged 2D gridded output file, e.g., for use preparing inputs to various "canned" 2-D graphics programs in automated web-generation, e.g., for NCSC's Numerical Air Quality Prediction project.

Back to Contents

New Release Version

In cooperation with EPA, we will be releasing the new and updated public version 2.0 of the I/O API in late Fall, 1999. This version incorporates both the task-parallel and the coupling-mode extensions, as well as the time series operations.
The new version is now 2.0.0c, available here.

With this version, the source-code-level build-process uses Makefiles rather than the previous Makeit shell-scripts. This allows much faster re-building of the library for small bug-fixes.

With this version, we will be supporting only netCDF 3.x, because of the support it provides for synchronization during model coupling. Please note that the NCAR folks accidentally incorporated a link-incompatibility between versions 2.x and 3.x of netCDF in some of the "magic numbers" used to describe modes of opening files. Note also that versions 3.4 and later use different error numbers than version 3.3.1 and earlier. The I/O API Troubleshooting Page will be updated to reflect both sets of error numbers. (As of 8/20/1999, the current released version of netCDF is 3.4, although we have been testing the new version 3.5b2 in-house recently.)

Also with this version, the default build of the library will require linking with libpvm3.a for coupling mode support. PVM 3.4 or later is required, because of the requirement for PVM mailboxes.

For some vendors, linking of serial programs will also require the explicit specification of an OpenMP run-time library as well -- on SGI's, for example, this requires an additional -lmp in the linking-step command line for serial programs.

Back to Contents

OpenMP Task-Parallel Extensions

OpenMP is a recent standard for directive-based shared-memory parallel Fortran and C. The Fortran and C standards were released in October, 1997 and October, 1998, respectively; a revision of the Fortran standard is expected in October, 1999. Compilers supporting the OpenMP standard are available for a variety of platforms, including Compaq (DEC), HP, IBM, Linux (both x86 and Alpha), SGI, Sun, and Win32 (95, 98, and NT (both x86 and Alpha)). OpenMP supports not only the kind of loop-level parallelism that has long been available from a variety of vendors, but also task-parallelism as well. In addition to much better portability, it offers much better support (critical sections, barriers, nested parallelism) for large projects than did the proprietary vendor specific directives. We are using OpenMP Fortran in-house with various air quality modeling projects; with its MM5 meteorology model, NCAR is moving to OpenMP directives as of its MM5 Version 3.

We are making the I/O API thread-safe for OpenMP, and have been testing the result in various real-time air quality forecasting and other meteorological projects. The basic principle is that I/O API calls that read or write distinct variables and/or distinct files may be done concurrently from OpenMP programs. The biggest gains for model-users are with INTERP3 and DDTVAR3, where the call involves linear algebra computations (linear interpolation or differencing) in addition to I/O (which is performed sometimes--but not always, because of the internal double-buffering mechanisms. Note that I/O to/from disk is itself sequential in any case.)

Building a parallel program that uses an OpenMP-enabled libioapi.a generally requires compiler-specific parallelization directives in both the compile-lines and the link-lines that make your executable. For examples, ook at your machine/compiler's Makeinclude.${BIN} for variables OMPFLAGS andOMPLIBS used with this build of the library

Back to Contents

Data-Parallel Extensions

For message-passing data parallel programming, our colleagues at EPA's Atmospheric Research and Exposure Assessment Laboratory (AREAL) have been constructing an extension to the I/O API which distributes gridded data for domain-decomposition data parallel message passing models. Inquiries should be addressed to Dr. Jeffrey Young, yoj@hpcc.epa.gov.

As part of the Practical Parallel Project, MCNC has developed an HPF2 implementation of an HPF I/O API module which calls the I/O API and then re-destributes gridded or boundary data according to the data distribution of its caller. Unfortunately, however, the IBM XL HPF and Portland Group HPF compilers we use support only their own extensions to HPF 1.1 at this time, so we have had to develop revised HPF I/O API modules for each which are not fully general but which do fulfill the requirements of our HPF air quality and hydrology models.

Back to Contents

Parallel Extensions for Coupled Models

As part of the Practical Parallel Project, MCNC has developed an extended Model-Coupling Mode for the I/O API. This mode, based upon PVM 3.4 mailboxes, allows the user to specify in the run-script whether "file" means a physical file on disk or a PVM mailbox-based communications channel. In the latter case, the I/O API schedules the various processes in a coupled model on the basis of data availability: requests for data which has not yet been written put the reading process to sleep until the data arrives, at which point the reader is awakened and given the data it requested. The one requirement is that the programmers structure reads and writes so as to avoid deadlocks (two or more models, each asleep while waiting for input from the other).

This has several advantages from the model-engineering point of view:

  • The same programs work unchanged both in standalone mode (reading input from files and writing output to files) and in coupled-model mode (reading and writing selected inputs or outputs to/from PVM mailboxes).
  • Readers and writers do not need to know about each other in detail. In particular, any reader only needs to know that some writer will put the variables it needs into the mailbox. Writers don't care whether readers even exist or not.
  • One writer can supply multiple readers without special programming (and without needing to know who they are). For example, in a coupled system with the MM5/MCIP meteorology model, the SMOKE emissions model, and the MAQSIP air quality model, MM5 produces 5 time-stepped output "virtual files", some variables of two of which are read by SMOKE and all of which are read by MAQSIP; and SMOKE produces one output "virtual files" read by MAQSIP. MAQSIP produces a "synchronization file" read by MM5/MCIP and used to keep MM5/MCIP from running ahead and exhausting available mailbox-buffer space.
  • Since data is tagged by variable-name, simulation date, and time, the system is not subject to data scrambling because of implicit programming assumptions about the data ordering in the way that stream-like communications channels are.

Back to Contents

Time Series READ/WRITE

In order to support the needs of surface-water, lake and estuary, and bay modeling, at the request of EPA-AREAL we have added two routines to the I/O API which will access an entire time step sequence of data in a single operation ("read or write N time steps of data starting at date and time D:T with step DT for variable V to/from file F"). The new routines are READ4D() and WRITE4D(), and very much resemble READ3() and WRITE3(), except for the specification of an entire time step sequence as arguments, the restriction to single-variable operations (ALLVAR3 is not supported for these) and file types CUSTOM3, GRDDED3, BNDARY3, and TSRIES3.

Back to Contents

Additional Supported Platforms

Conditional compilation directives in the I/O API were recently modified to make it easier to port the I/O API to other platforms. Three issues are prominent:
  • Feldman-style name mangling (trailing underscores) and CHARACTER string-passing conventions or not; and
  • Native length for REAL, INTEGER, and DOUBLE
  • Dynamic memory allocation mechanism, whether via Cray-pointers and MALLOC or by automatic arrays
Preprocessor definitions and conditional compilation directives recognizing them have been provided in the code: defining FLDMN=0,1 specifies the first; REAL8=0,1 specifies the second, and AUTO_ARRAYS=0,1 the last of these. Work is in progress to provide the I/O API on at least the following additional platforms:
  • Compaq fort and gcc on Linux Alpha.
  • gcc/g77 (FLDMN=1, REAL8=0, and AUTO_ARRAYS=1)
  • Win32 (Windows 3.1/win32s, Windows 95, and Windows NT/x86) using either the Digital or the Absoft Fortran-90 compilers and the Microsoft C compiler (this requires that the Fortran pre-processing be done in advance on a UNIX system; FLDMN=1, REAL8=0, and AUTO_ARRAYS=1);
  • Windows NT/Alpha using the Digital compilers (also requires Fortran pre-processing n advance; FLDMN=1, REAL8=0, and AUTO_ARRAYS=1);
  • <suggestions? Do you want to donate us a machine (or machine-time) for it?>

Back to Contents


MCPL: an I/O API output module for MM5

MCPL is an I/O API output module for MM5 developed under the PPAR project. It is MCPL is designed to fit into MM5 with a minimum of effort, to offer great flexibility of run-time configuration, and to be written in "boring" code that is easy to understand, extend, and maintain. It generates output files with a variety of kinds of variables:
  • MM5 internal variables (including all those written by OUTTAP);
  • all the variables needed for input to the MAQSIP air quality model or the SMOKE emissions model;
  • variables needed by the TOPLATS land surface/hydrology model;
  • variables interpolated to pressure-surfaces; and
  • a variety of other diagnostic or analysis variables

Back to Contents


Go to the I/O API Availability/Download page

Go to the I/O API Home Page

Go to the I/O API User Manual

Send comments to

Carlie J. Coats, Jr.
carlie.coats@baronams.com