REAL FILES AND "VIRTUAL FILES"
Introduction
The I/O API provides both real, disk-based files (which may be
"volatile" or not, and are implemented
on top of netCDF) and "virtual files" that may be used to
provide safe, structured exchange of data -- the latter of
"gridded,"
"boundary," or
"custom" types
only -- between different modules in the same program. You may even
safely interchange between using real files and virtual files in
different executions of the same program merely by changing the
values of the files logical names (this would allow you to
look at the data being shared between modules whenever you want to,
for example, at high temporal resolution). There are two types of
virtual files: memory-resident BUFFERED virtual
files that can be used to share data between modules of a single
program; and PVM-mailbox based COUPLING-MODE
virtual files that can be used to share data and coordinate
scheduling between different programs, even if they are executing
on different machines half a continent apart across the Internet.
Real (disk-based) I/O API files may optionally be declared
"volatile" by the addition of a trailing
" -v"
to the value of the file's
logical name in order to tell the
I/O API to perform disk-synch operations before every input
and after every output operation on that file:
...
setenv QUX "/tmp/mydir/volatiledata.mymodel -v"
NetCDF attempts the I/O optimization of not writing a file's header
-- needed in order to interpret the file's contents -- out to disk
until either a "synch" operation is performed, or until
the file is closed. This has the effect of making non-volatile
output files unreadable until the program that writes them does a
SHUT3() or M3EXIT()
(or if it crashes unexpectedly). This extra "synch"
operation does cause some (usually small) performance penalty, but
it allows other programs to read I/O API files while they are
still being written, and prevents data loss upon program crashes.
NOTE: There is a bug in the interaction of
netCDF 3.4 and the SGI IRIX6 operating system NFS implementation
that sometimes causes NFS-based data exchange between
concurrently-running programs using volatile files to fail. If you
plan to do this, either go back to netCDF 3.3.1 or before, or go to
netCDF 3.5-beta2 or later.
For memory-resident BUFFERED files, one restriction at present is
that the basic data type of
all variables in the virtual file be either integer or real. The
other restriction is that only two time steps of data are kept in
the buffered file -- the "even step" and the "odd
step" (which in normal usage are the last two time steps
written). Otherwise, you write code to open, read, write, or
interpolate data just as you would with a "real" file.
This provides for structured name based identity-tagged
exchange of data between different modules in the same program --
since data are stored and accessed by file-name, variable-name,
date, and time, the system will detect at run-time the attempt to
request data not yet initialized (unlike the situation where data
is exchanged via Fortran COMMON
s - we've detected
some obscure use-before-calculate bugs by replacing
COMMON
s with BUFFERED virtual files.)
To set up a buffered virtual file, setenv the value of
the file's logical name to the
value BUFFERED
(instead of to the pathname of a real
physical file), as given below:
...
#
# myprogram uses "qux" for internal data sharing:
#
setenv qux BUFFERED
...
/user/mydir/myprogram
...
Restrictions:
- For all-variable READ3()
and WRITE3() calls, all of the variables in
the file must be of
type
M3REAL
.
- Prior to the I/O API V 2.2-beta-May-3-2002
release, all variables in buffered virtual files must
be of type
M3REAL
.
As part of the
Practical
Parallel Project, MCNC has developed an extended
Model Coupling Mode for the I/O API.
This mode, implemented using
PVM 3.4
mailboxes, allows the user to specify in
the run-script whether "file" means a physical file on
disk or a PVM mailbox-based communications channel (a
virtual file), on the basis of the value of the
file's logical name:
setenv FOO "virtual BAR"
setenv IOAPI_KEEP_NSTEPS 3
declares that FOO
is the logical name of a virtual
file whose physical name (in terms of PVM mailbox names) is
BAR
. The additional environment variable
IOAPI_KEEP_NSTEPS
determines the number of time
steps to keep in PVM mailbox buffers -- if it is 3 (as here), and
there are already 3 timesteps of variable QUX
in the
mailboxes for virtual file FOO
, then writing a fourth
time step of QUX
to FOO
causes the earliest
time step of QUX
to be erased, leaving only timesteps 2,
3, and 4. This is necessary, so that the coupled modeling system
does not require an infinite amount of memory for its sustained
operation. If not set, IOAPI_KEEP_NSTEPS
defaults
to 2 (the minimum needed to support INTERP3()
's
double-buffering).
The (UNIX) environments in which the modeler launches multiple models
each of which reads or writes from a virtual file must all agree
on its physical name (usually achieved by sourcing some script that
contains the relevant setenv commands).
For models exchanging data via virtual files of the I/O API's
coupling mode, the I/O API schedules the various processes on
the basis of data availability:
- The modeler must start up a PVM session that will
"contain" all the virtual files and enroll in
it all those machines which will be running the various
modeling programs before starting up the various models
in a coupled modeling system on those respective machines.
-
OPEN3()
calls for read-access to virtual
files that haven't yet been opened for write access by
some other process put the caller to sleep until the
file is opened; and
-
READ3()
, INTERP3()
, or
DDTVAR3()
calls for virtual-file data which
has not yet been written put the reading process to sleep
until the data arrives, at which point the reader is
awakened and given the data it requested.
There are two requirements on the modeler:
- structuring reads and writes so as to avoid deadlocks
(two or more models, each asleep while waiting for input
from the other); and
- providing enough feedbacks to prevent one process from
"racing ahead" of the others. In a one-way
coupled system, this may mean the introduction of
artificial synchronization files which exist
solely to provide these feedbacks.
Using coupling mode to construct complex modeling systems has several
advantages from the model-engineering point of view:
- Since data is tagged by variable-name, simulation date,
and time, the system is not subject to data scrambling
because of implicit programming assumptions about the
data ordering in the way that stream-like communications
channels are.
- The same programs work unchanged both in standalone mode
(reading input from files and writing output to files)
and in coupled-model mode (reading and writing selected
inputs or outputs to/from PVM mailboxes).
- Readers and writers do not need to know about each other
in detail. In particular, any reader only needs to know
that some writer will put the variables it needs
into the mailbox. Writers don't care whether readers
even exist or not. It is easy to change system configuration
by just adding additional processes or by deleting processes
and replacing them by appropriate disk-based files containing
the data that would have been produced. In MCNC's Real-Time
Ozone Forecast System, for example, the set of programs that
runs to compute each day's ozone forecast varies from day to
day, on the basis of such things as whether particular data
ingest feeds have succeeded or failed over the past two days.
- One writer can supply multiple readers without special
programming (and without needing to know who they are).
For example, in a coupled system with the MM5/MCIP
meteorology model, the SMOKE emissions model, and the
MAQSIP air quality model, MM5 produces 5 time-stepped
output "virtual files", some variables of two
of which are read by SMOKE and all of which are read by
MAQSIP; and SMOKE produces one output "virtual
files" read by MAQSIP. SMOKE is itself a system
of five programs coupled together by virtual files and
fed by a number of additional disk-files produced
off-line. MAQSIP produces a "synchronization file"
read by MM5/MCIP and used to keep MM5/MCIP from running ahead
and exhausting all memory available for mailbox-buffer space.
Previous Section: Variables and Layers and Time Steps
Up: Conventions
To: Models-3/EDSS I/O API: The Help Pages