README - 3 Sep 2003 This README file outlines the steps necessary to build and run CMAQ models. The code has been tested on a variety of platforms, but the build and run scripts that are included in the tar files are set up to compile and run on Linux (we tested on Redhat Linux 7.3 with the Portland Group F90 compiler (pgf90 version 4.0-2). We did not include scripts nor test for use with the Intel F90 compiler as in the previous "interim" release. In case you want to run on another platform, the C-shell scripts are easy to modify for any Unix implementation. The Stand-Alone CMAQ package contains data and default setups to execute a series of tutorial runs to demonstrate the usage of the scripts and data. The tutorial produces concentrations for a 32 km horizontal resolution grid (coarse domain) and a nested 8 km grid (fine domain). The package will have datasets in the $M3DATA input directories only; the user must work through the system of CMAQ models to produce the inputs required for the downstream model(s). These inputs (preprocessor model outputs) must reside in the specified $M3DATA subdirectories (see below in item 13). We have provided model-generated datasets for the purpose of comparison with your outputs in a separate tar file, M3DATA_REF.tar. The run scripts and comparison output data reflect the scenario period for the tutorial runs: two 24 hour periods starting 0Z:1999183 (2 July 1999) and 0Z:1999184 (3 July 1999). The distribution package consists of the following files: o README - this readme text file o CVS_NETCDF - text file explaining the CVS configuration management system and the netCDF data system and how to set them up o IOAPI - text file explaining how to get the I/O API system and set up the libraries o SMOKE - text file describing the SMOKE emissions system and how to set it up (not required for the tutorial) o PARALLEL_NOTES - text file containing comments related to running the CMAQ CCTM in Linux MPICH clusters o TUTORIAL_PROCEDURE - text file describing how to work through the tutorial o RELEASE_NOTES - text file containing a list of the major changes since the last release, along with the previous release notes o MODELS.tar.gz - gzipped tar file (~2.7 Mb) containing model, tools and libraries source code CVS archives o M3DATA.tar.gz - gzipped tar file (~106 Mb) containing the required datasets not produced by this package o SCRIPTS.tar.gz - gzipped tar file (~12 Kb) containing C-Shell scripts to build and execute the CMAQ models o M3DATA_REF.tar.gz - gzipped tar file (~315 Mb) containing reference data to compare with datasets produced by the tutorial on a Linux workstation ** NOTE: You must have CVS, IOAPI and netCDF (see the CVS_NETCDF and IOAPI readme's for help). The following outlines a sequence of steps you can follow to build and run the codes: 1) Set environment variable (path) for M3HOME, e.g. user "yoj" could set: setenv M3HOME /project/cmaq/yoj 2) Set environment variables (paths) for M3MODEL, M3LIB and M3DATA as: setenv M3MODEL $M3HOME/models setenv M3LIB $M3HOME/lib (you may need to create this subdirectory) setenv M3DATA $M3HOME/data 3) cd to $M3HOME and gunzip and untar the data tar file, M3DATA.tar. This will produce the following subdirectories: data/ cctm/ <<<<<<< empty, to be filled by the user bcon/ <<<<<<< empty, to be filled by the user icon/ <<<<<<< empty, to be filled by the user jproc/ <<<<<<< empty, to be filled by the user mcip2/ M_32_99NASH/ M_08_99NASH/ emis/ tut02/ raw/ phot/ icon/ bcon/ 4) Create (mkdir) the subdirectory $M3LIB and the following subdirectories under $M3LIB: build/ netCDF/ ioapi_22/ stenex/ pario/ dynmem/ Concerning netCDF: The scripts assume that netCDF resides in the $M3LIB path as $M3LIB/netCDF. You need to install your own netCDF libraries built for Linux (SunOS5) in this directory (see CVS_NETCDF) or symbolically link to the existing netcdf on your system. Example for Linux cluster: mkdir -p $M3LIB/netCDF/Linux cd $M3LIB/netCDF/Linux ln -s /home/showard/netcdf-3.4_linux/lib/libnetcdf.a libnetcdf.a Concerning ioapi_22: We no longer support the I/O API. We recommend that you download IOAPI version 2.2 from the CMAS/EMC web site and compile the libraries that you need. This is done by editing the appropriate Makeinclude file(s) for the compiler flags, if necessary, and setting the "BIN" environment variable appropriately - see the IOAPI readme included with this release. 5) In $M3HOME untar the models archive tar file, MODELS.tar. This will produce the following subdirectories: models/ CCTM/ PARIO/ include/ BUILD/ DYNMEM/ STENEX/ PROCAN/ JPROC/ ICON/ BCON/ 6) Make a working directory (NOT in either the $M3MODEL, $M3LIB or $M3DATA trees), cd there and untar the SCRIPTS.tar. This will produce the following subdirectories, which contain "bldit" and "run" C-shell scripts and a GRIDDESC file (see item c. under "other details" below): bcon/ icon/ cctm/ build/ stenex/ jproc/ pario/ dynmem/ GRIDDESC1 Not necessary, but for the sake of further discussion create an environment variable for the top level of your working directory, $WORK. 7) First, create the IOAPI library required for the models. See the IOAPI readme file included with this release. mkdir $M3LIB/ioapi_22 <<<<<<< install here 8) Next create the stencil exchange library required for parallel processing and the no-op version for serial processing: cd $WORK/stenex Execute (type) bldit.se.pgf Execute (type) bldit.se_noop.pgf 9) For parallel CCTM operation create the parallel I/O and associated dynamic memory libraries: cd $WORK/pario Execute (type) bldit.pario.pgf cd $WORK/dynmem Execute (type) bldit.dynmem.pgf 10) Create m3bld, the tool required to build all the other executables. cd $WORK/build execute (type) bldit.m3bld Note: Although m3bld is really a tool, we put it in with the "libraries." 11) Now create the model executables: JPROC is created and run only once for the tutorial; ICON and BCON need to be compiled and run separately for profile data (coarse grid) and for nest data (fine grid); CCTM is compiled only once. See the TUTORIAL_PROCEDURE readme file for details. Generally, you will need to get the MCIP2 code and run it to create met data from MM5 for CCTM. MCIP2 can be downloaded from the same site as this distribution package. And of course, you will need "model-ready" emissions data - presumably from SMOKE. See the SMOKE readme file included with this package. For this tutorial release we have provided the model-ready emissions and met data. Start with JPROC (cd to $WORK/jproc). Invoke "bldit.jproc.pgf". There will be a lot of text displayed to standard out (which you can capture of course, by redirecting to a file). The process should end with a JPROC executable, which is invoked in the second script, "run.jproc", producing output data files. These data files will be inserted into the path predefined in the run script, $M3DATA/jproc. Note: The "run.jproc" script is set up to produce daily J-value tables for the cb4_ae3_aq mechanism starting from 30 June to 14 July 1999. This works as long as you're not using TOMS data, in which case you would need to run one day at a time. Note: It's always a good idea to capture in a log file the text written to standard out when running these models. In each "run" script, near the top, is a suggested method. e.g. for JPROC, run.jproc >&! jproc.log & 12) Check the JPROC log file to ensure complete and correct execution. Then cd to $WORK/icon and follow the same procedure; invoke "bldit.icon.pgf", followed by "run.icon >&! icon.log &". This will produce the first (profile) dataset for the first run of CCTM on the coarse domain. After CCTM finishes, you will need to generate a nest dataset for the fine domain. See the TUTORIAL_PROCEDURE readme file for details. 13) Follow this procedure for each of the model subdirectories after icon/ (the order is not mandatory). If you are running through the tutorial, see the TUTORIAL_PROCEDURE readme file. 14) Finishing with CCTM, you should have a complete collection of datasets, which you can compare with the distribution datasets in $M3DATA_REF.tar. Unless you modify the run scripts, the output data from all the models will reside in the following (automatically generated) paths: $M3DATA/ jproc/ icon/ bcon/ cctm/ 15) Concerning parallel CCTM operation: We have tested the "bldit" script for both serial and parallel compilation. The source code is the same for both. Only some libraries are different as well as the run scripts. The "stenex" library for parallel is different than for serial; "pario" and "dynmem" are needed only for parallel. We ran successfully on a Scyld Beowulf Linux cluster, but this release was set up and tested for a "standard" MPICH linux cluster, requiring the addition of a C code that distributes the run time environment from the node that launches the run to the other participating nodes. Thanks to Bo Wang and Zion Wang of CERT-UC-Riverside, who developed and tested this code. Also, see the PARALLEL_NOTES readme. (Note: The initial concentrations pre-processor, ICON can also be executed in parallel, but we have not yet tested this for Linux clusters.) 16) Concerning non-parallel CCTM operation (to run the model in serial): In bldit.cctm.linux, deselect ParOpt (comment it out) Modify the run.cctm script as: 9c9 < # method: run.cctm >&! cctm_e2a.log & --- > # method: run.cctm >&! cctm_e1a.log & 19,20c19,20 < set APPL = e2a < set CFG = e2a --- > set APPL = e1a > set CFG = e1a 24,27c24,27 < #setenv NPCOL_NPROW "1 1" < #set NPROCS = 1 < setenv NPCOL_NPROW "5 2" < set NPROCS = 10 --- > setenv NPCOL_NPROW "1 1" > set NPROCS = 1 > #setenv NPCOL_NPROW "5 2" > #set NPROCS = 10 206c206 < #time $BASE/$EXEC --- > time $BASE/$EXEC 208,211c208,211 < set MPIRUN = /share/linux/bin/mpich-ch_p4/bin/mpirun < set TASKMAP = $BASE/machines10 < cat $TASKMAP < time $MPIRUN -v -machinefile $TASKMAP -np $NPROCS $BASE/$EXEC --- > #set MPIRUN = /share/linux/bin/mpich-ch_p4/bin/mpirun > #set TASKMAP = $BASE/machines10 > #cat $TASKMAP > #time $MPIRUN -v -machinefile $TASKMAP -np $NPROCS $BASE/$EXEC Note: You can change the default script by using the Unix "patch" utility. Cut the indented section listed above into a file, say "mod." Then type "patch run.cctm mod." 17) Other details: a. You can check output ioapi file headers (and data) using ncdump. This utility will be located in the same place as netcdf, mentioned in (4) above. b. You can run a model using the required inputs from the reference datasets, instead of your preprocessor outputs. You might want to do this to run just the CCTM, compiled with a different set of module options, for example. c. To run CCTM, ICON and BCON requires a GRIDDESC file that contains horizontal projection and grid domain definitions. The run scripts contain environment variables that point to this file that contains the user's horizontal grid definitions. The horizontal grid definition can be set to window from the met and emissions input files. However, the window must be a proper subset - not lie along any of the boundaries of the domain represented by the input files. Note: The domains represented by the met and emissions data must be the same. Of course, you don't have to window, and the domain represented by the input files is a valid CCTM domain. d. Running CCTM for a windowed domain or a higher resolution nested domain from larger or coarser met and emissions datasets still requires creating initial and boundary data for the target domain using ICON and BCON.