1. Overview
    1.1 Overall Objective and Basic Structure
    1.2 Concept of an AMET “Project”
    1.3 Organization of This User’s Guide
2. Directory Structure
3. Configuration
    3.1 R Configuration File (amet-config.R)
4. Datasets
    4.1 Model Data
    4.2 Observational Data
5. Database Setup
    5.1 AMET Setup
    5.2 Basic MySQL Commands
6. Project Creation and Database Population
    6.1 The metExample_wrf and metExample_mpas Projects
    6.2 The aqExample Project
    6.3 Creating a New MET Project
    6.4 Creating a New AQ Project
7. Analysis
    7.1 metExample (WRF, MPAS or MCIP)
    7.2 aqExample
    7.3 Summary of the AQ Analysis Scripts
    7.4 Creating a New Analysis Project
8. Adding a New AQ Network to AMET
9. CMAS Support for AMET
References
Appendix A: Overview Flow Diagram
Appendix B: Configuration and Input Files
Appendix C: Statistics/Metrics Formula Reference Document

Table 2‑1. Directories under $AMETBASE
Table 3‑1. Most common variables that need to be changed in amet-config.R
Table 6‑3. aqProject.csh script options
Table 7‑2. AQ analysis script options
Table B‑1. amet-config.R
Table B‑2. MET analysis input variables
Table B‑3. AQ analysis input variables

The Atmospheric Model Evaluation Tool (AMET) (Appel et al., 2011) is a suite of software designed to facilitate the analysis and evaluation of predictions from meteorological and air quality models. AMET matches the model output for particular locations to the corresponding observed values from one or more networks of monitors. These pairings of values (model and observation) are then used to statistically and graphically analyze the model’s performance.

More specifically, AMET is currently designed to analyze outputs from the Weather Research and Forecasting (WRF) model, and the Community Multiscale Air Quality (CMAQ) model, as well as the Model for Prediction Across Scales (MPAS) - a new global model developed by NCAR. Output from other regional and global air quality models, such as the Comprehensive Air Quality Model with Extensions (CAMx), can also be formatted for analysis with AMET.

The basic structure of AMET consists of two fields and two processes.

-   The two fields (scientific topics) are MET and AQ, corresponding to meteorology and air quality data.

-   The two processes (actions) are database population and analysis. Database population refers to the underlying structure of AMET; after the observations and model data are paired in space and time, the pairs are inserted into a database (MySQL). Analysis refers to the statistical evaluation of these pairings and their subsequent plotting.

Practically, a user may be interested in using only one of the fields (either MET or AQ), or may be interested in using both fields. That decision is based on the scope of the study. The two main software components of AMETv1.4+ are MySQL (an open-source database software system) and R (a free software environment for statistical computing and graphics). The previous versions of AMET also utilized Perl (an open-source cross-platform programming language), but the Perl requirement was removed from AMETv1.3 and beyond in an effort to streamline the tool.

A central organizing structure for AMET applications is a project. A project organizes a particular model simulation (specific model, physics, spatial domain, grid scale, etc.) with AMET database tables that correspond to that simulation, the scripts necessary to populate that database, the scripts required to analyze that project and statistical analysis outputs. For example, you might have one project for a 2016 12-km continental U.S. simulation, and another project for a 2016 4-km Eastern U.S. simulation. A project can be for either MET or AQ, not for both. It is essential that you both uniquely and concisely name each project. It is recommended that you follow the directory structure when creating new projects, by copying one of the example directories (aqExample, metExample_wrf/mpas/mcip) provided with the installation and then renaming it to the new project’s name.

The U.S. EPA and the Community Modeling and Analysis System (CMAS) Center created this guide to assist you in applying the AMET system in your work.

The contents of the remaining sections of this User's Guide are listed below.

• Section 2 describes the overall directory structure of the AMET installation.

• Section 3 gives instructions on how to configure the R configuration files.

• Section 4 is an overview of the various model outputs and observed data provided with the AMET release.

• Section 5 provides instructions on how to create the AMET MySQL database, with specific instructions for each of the MET and AQ models. Sample MySQL commands are also shown for illustrative SQL queries.

• Section 6 gives instructions on how to populate the AMET MySQL database, with specific instructions for WRF and CMAQ models, and also on how to create new MET and AQ projects for subsequent analyses.

• Section 7 includes instructions on how to perform model evaluation for WRF and CMAQ models, and includes an overview of the functionality of all availiable MET and AQ evaluation scripts.

Important note: The set of analyses/evaluation scripts provided in this release are strictly for illustration purposes on the functionality/design of AMET, and are not to be construed as a recommended suite of analyses scripts for model evaluation. We encourage the user community to use the scripts we have provided as examples as well as a basis to start developing other analyses scripts and contribute them to the modeling community to increase AMET functionality.

• Section 8 discusses how to obtain support for AMET from the Community Modeling and Analysis System (CMAS) Center (http://www.cmascenter.org).

• Appendix A is an overall flow diagram for AMET and its various components.

• Appendix B provides information on the various input files used in AMET. For each input file, a table lists brief descriptions of all user-defined variables that can be set by the user for a given evaluation.

Before using AMET and this user’s guide, you must first install the AMET package on your system. For information on the installation process, please see the Atmospheric Model Evaluation Tool (AMET) Installation Guide.

In this guide, the top level of the AMET directory structure is referred to as “AMETBASE”. This environment variable is actually set in many of the scripts discussed below. For example, if you were to run the AMET (example for version 1.5) installation Git command in user's home directory /home/user:

git clone -b 1.5 https://github.com/USEPA/AMET.git AMET_v15

The setting of AMETBASE would be /home/user/AMET_v15

Table 2-1 shows the directories contained in the $AMETBASE directory.

Table 2‑1. Directories under $AMETBASE.

After installing the AMET code and data, the next step is to configure the AMET system. In the $AMETBASE/configure directory, you will find the R MySQL configuration file amet-config.R.

The AMET R configuration file is used by the underlying R programs to perform AMET setup and statistical analysis on pairs of model and observational data. Most users will need to modify only a few specific lines of this configuration file. The most common variables to change are shown in Table 3-1.

Table 3‑1. Most common variables that need to be changed in amet-config.R.

The amet_login and amet_pass variables are MySQL database credentials. The MySQL credentials specified here are always used in the analysis scripts that come with AMET, which require only database read access to function. Therefore, the MySQL user specified here can be limited to read access only. However, these credentials can also be setup to be used by the database loading scripts. For those scripts to work properly, the MySQL user specified must have permission to create databases and tables, in addition to read access. If the setting in the database loading scripts is to read the amet_login and amet_pass variables from the amet-config.R file, those credentials must be for a user with full MySQL permissions.

For simplicity, it is suggested that the MySQL credentials specified in the amet-config.R file be for a user with full database permissions. For database security purposes, it is recommended that the amet-config.R file be made read-only by the user.

The AMET release includes example datasets of both model and observational data.

For the model data, we have included both meteorological and air quality data. We have organized the data into several example projects: "metExample_wrf", "metExample_mpas", "metExample_mcip" and "aqExample". On the MET side, there is a 1-month WRF simulation, 1-month MCIP output, and 1-month MPAS simulation provided for July 2016. We included the same period in case users wanted to compare the two models. These are model output subsets with only the variables needed for the evaluation scripts. Model outputs are availiable for download via the https://www.cmascenter.org/ site. Users need to register and then look under AMET options for the link. Note, these are rather large, so for basic testing only a few files may be enough for testing. The full month can allow users to fully exercise AMET before their own model analysis is done.

The WRF data consist of 31 WRF output files in netCDF format:

$AMETBASE/model_data/MET/metExample_wrf/

wrfout_subset_2016-07-01_00:00:00 ... wrfout_subset_2016-07-31_00:00:00

The MCIP data consist of 1 GRIDCRO2D, 31 METCRO2D and 31 METCRO3D output files in netCDF format:

$AMETBASE/model_data/MET/metExample_mcip/

GRIDCRO2D

METCRO2D_160701.nc ... METCRO2D_160731.nc

METCRO3D_160701.nc ... METCRO3D_160731.nc

The MPAS data consist of 31 MPAS output files in netCDF format:

$AMETBASE/model_data/MET/metExample_mpas/

history.subset.2016-07-01.nc ... history.subset.2016-07-31.nc

Note that we have bolded “metExample_*” in the directory names above to highlight the fact that we are using the project name to organize the model output files into directories.

On the AQ side, we have included two CMAQ output files for the period July 01 2011 0:00 UTC to July 31 2011 23:00 UTC. The two files:

$AMETBASE/model_data/AQ/aqExample/

AMET_CMAQ_July_2016_Test_Data.aconc

AMET_CMAQ_July_2016_Test_Data.dep

correspond to the concentration and wet deposition output files from CMAQ, after they have been postprocessed with the combine utility.

All of the spatial domains cover the continental U.S. and have a 12-km grid resolution.

As with the model data, the observations directory structure is divided between MET and AQ fields. On the MET side, the bulk of the observations come from the Meteorological Assimilation Data Ingest System (MADIS), provided by the National Oceanic and Atmospheric Administration (NOAA). Contact MADIS to obtain a MADIS account for downloading these data (see http://www-sdd.fsl.noaa.gov/MADIS for details). Two other dataset are compatible for the evaluation of shortwave radiation. Users can either select a global Baseline Surface Radiation Network (BSRN; https://bsrn.awi.de/) or a US-centric NOAA-based SURFace RADiation (SURFRAD; https://gml.noaa.gov/grad/surfrad/) network data. There is also an example of a simple text file format for surface meteorology if users have non-standard text data.

The new autoFTP option in AMET uses the MADIS anonymous FTP account; if using that option, please acknowledge the MADIS group in applications that use AMET. Alternatively, users may also manually download all MADIS observations before the model-observation matching step. The autoFTP option is also fully compatible for both surface shortwave radiation datasets. This option ignores data that was already acquired as long as it is located in the AMET directory structure.

In the AMET directory structure, all of the MADIS data are stored under $AMETBASE/obs/MET. The MADIS observation directory structure is provided in this directory in the release. The AMET example data distribution from CMAS includes standard MET surface observations from MADIS for the model output periods (July 2011 and July 2013). To list the contents of the example met observational data directory:

ls $AMETBASE/obs/MET
point/metar
point/raob
point/maritime
point/sao
LDAD/mesonet
bsrn
surface_text

Each of the MADIS observation files is a netCDF file representing one hour’s worth of surface and/or upper-air meteorological data from all available observation sites. The netCDF files can be stored in a gzip compressed format to save space, but should be unzipped before use. These files are now directly read during the model-obs matching step instead of using an external utility (e.g.; sfcdump.exe) to extract into text and parsed by Perl. BSRN observation files are monthly and include all global sites. SURFRAD data are daily for individual sites. Surface text example file is hourly for all sites like MADIS. It is advised that users utilize the autoFTP routine that handles the download, decompress and organization of all observational dataset.

On the AQ side, example observational data are available for a number of networks including:

• Air Quality System (AQS) network
• Clean Air Status and Trends Network (CASTNET)
• Interagency Monitoring of PROtected Visual Environments (IMPROVE)
• Mercury Deposition Network (MDN)
• National Atmospheric Deposition Program (NADP)
• SouthEastern Aerosol Research and Characterization Study (SEARCH)
• Chemical Speciation Network (CSN; formerly STN)
• National Air Pollution Surveillance (NAPS) Program (Canada AQ network)
• NOAA Earth Systems Research Laboratory (ESRL)
• Tropospheric Ozone Assessment Report (TOAR)

The observational datasets have been preprocessed and reformatted (in some instances from their original sources) for use with AMET. The temporal range is network dependent. The monitoring station locations are provided by a series of .csv files under the subdirectory $AMETBASE/obs/AQ/site_meta_files. A brief synopsis of each network, along with the steps taken to create these data for AMET, is given below.

Note that in the species lists, each line is of the format “observed species name; model species name (units)”.

Weekly CASTNet data are obtained through the CASTNet web site: http://www.epa.gov/castnet/. Weekly CASTNet data can be obtained by downloading the “drychem” file under the prepackaged datasets on the CASTNet web site. No postprocessing of the downloaded data is necessary in order for them to be compatible with the Site Compare (sitecmp) software packaged with the AMET system. Note that the species MG, CA, K, NA, and CL are available when using the CMAQ AERO6 module.

Weekly CASTNet Species used with AMET:

tso4; ASO4T (µg/m³)
tno3; ANO3T (µg/m³)
tnh4; ANH4T (µg/m³)
tno3+nhno3; ANO3T+HNO3_UGM3 (TNO3; µg/m³)
nhno3; HNO3_UGM3 (µg/m³)
wso2; SO2_UGM3 (µg/m³)
MG; AMGJ (µg/m³)
CA; ACAJ (µg/m³)
K; AKJ (µg/m³)
NA; ANAIJ (µg/m³)
CL; ACLIJ (µg/m³)

Hourly CASTNet data are obtained through the CASTNet web site: http://www.epa.gov/castnet/. Hourly CASTNet ozone data can be obtained by downloading the files labeled “ozone_yyyy” under the prepackaged datasets on the CASTNet web site. Additionally, a “metdata_yyyy” file is also available on the CASTNet web site, which contains several meteorological variables in addition to ozone. No postprocessing of the downloaded data is necessary in order for them to be compatible with AMET’s sitecmp.

Hourly CASTNet Species used with AMET:

Ozone; ozone (ppb)

Additional species that could be used with AMET:

Surface Temperature Precipitation
Relative Humidity 10m Wind Speed
Solar Radiation 10m Wind Direction

IMPROVE data are obtained through the IMPROVE web site: http://vista.cira.colostate.edu/improve/. The IMPROVE web site links to the Visibility Information Exchange Web System (VIEWS) web site, which is an interactive system for downloading various air-quality-related data. IMPROVE data obtained through the VIEWS system do not require any additional processing to work with AMET’s sitecmp.

IMPROVE Species used with AMET:

SO4f_val; ASO4T (µg/m³)
NO3f_val; ANO3T (µg/m³)
NH4f_val; ANH4T (µg/m³)
MF_val; PM25 (µg/m³)
OCf_val; PM_OC (µg/m³)
ECf_val; AECT (µg/m³)
OCf_val+ECf_val; PM_OC+AECT (TC; µg/m³)
CHLf_val; ACLIJ (µg/m³)
MT_val; PM10 (µg/m³)
CM_calculated_val; PMC_TOT (µg/m³)
NAf_val; ANAIJ (µg/m³)
NAf_val+CHLf_val; ANAIJ+ACLIJ (NaCl; µg/m³)
FEf_val; AFEJ (µg/m³)
ALf_val; AALJ (µg/m³)
SIf_val; ASIJ (µg/m³)
TIf_val; ATIJ (µg/m³)
CAf_val; ACAJ (µg/m³)
MGf_val; AMGJ (µg/m³)
Kf_val; AKJ (µg/m³)
MNf_val; AMNJ (µg/m³)
2.20*ALf_val+2.49*SIf_val+1.63*CAf_val+2.42*FEf_val+1.94*TIf_val; ASOILJ (µg/m³)
MF_val-SO4f_val-NO3f_val-0.2903*NO3f_val-0.375*SO4f_val-OCf_val-ECf_val-NAf_val-CHLf_val-2.2*\ALf_val-2.49*SIf_val-1.63*CAf_val-2.42*FEf_val-1.94*TIf_val; AUNSPEC1IJ (OTHER; µg/m³)
; ANCOMIJ (NCOM; µg/m³)
; AUNSPECIJ2 (OTHER_REM; µg/m³)

MDN data are obtained through the NADP/MDN network web site: http://nadp.sws.uiuc.edu/mdn/. Data are available for download as a comma-delimited file for all sites. No postprocessing of the downloaded data is necessary in order for them to be used with AMET’s sitecmp.

MDN Species used with AMET (from CMAQ deposition file):

HGconc; TWDEP_HG (ng/L)
HGdep; TWDEP_HG (µg/m²)

NADP data are obtained through the NADP/NTN web site: http://nadp.sws.uiuc.edu/. Weekly wet concentration data are downloaded in comma-delimited format directly from the NADP web site. No postprocessing of the downloaded data is necessary in order for them to be used with AMET’s sitecmp.

NADP Species used with AMET (from CMAQ deposition file):

Valcode
Invalcode
NH4; WDEP_NHX (mg/L or kg/ha)
NO3; WDEP_TNO3 (mg/L or kg/ha)
SO4; WDEP_ASO4T (mg/L or kg/ha)
Cl; WDEP_TCL (mg/L or kg/ha)
Na; WDEP_ANAJK (mg/L or kg/ha)
Ca; WDEP_CAJK (mg/L or kg/ha)
Mg; WDEP_MGJK (mg/L or kg/ha)
K; WDEP_KJK (mg/L or kg/ha)
precip; RT (mm)

SEARCH data are obtained through the SEARCH web site: http://www.atmospheric-research.com/public/index.html. The SEARCH data can be downloaded as comma-delimited files for each SEARCH site. In order to be used with sitecmp and AMET, the individual site files must first be merged together into a single file. The example SEARCH data file provided with AMET should serve as an example of how the raw SEARCH data need to be combined and formatted in order to work with sitecmp and AMET. The list of SEARCH species listed here is just an example of the species available from SEARCH, as the exact species available varies depending on year and whether the data are hourly or daily. AMET formatted SEARCH data files are available for download from the CMAS website.

SEARCH Species used with AMET:

o3; O3 (ppb)
co; CO (ppb)
so2; SO2 (ppb)
no; NO (ppb)
hno3; HNO3 (ppb)
teom; PM25 (µg/m³)
no3; ANO3T (µg/m³)
so4; ASO4T (µg/m³)
nh4; ANH4T (µg/m³)
noy; NOY (ppb)

CSN data are obtained through the EPA’s Air Quality System (AQS), located at http://www.epa.gov/ttn/airs/airsaqs/. The data provided with AMET are a sample of the CSN data that can be obtained through the AQS. Some postprocessing of the downloaded CSN data is necessary in order for them to work with sitecmp and AMET. However, AMET compatible CSN data files are available from the CMAS website.

CSN Species used with AMET:

m_so4; ASO4T (µg/m³)
m_no3; ANO3T (µg/m³)
m_nh4; ANH4T (µg/m³)
FRM PM_2.5 Mass; PM_2.5 (µg/m³)
oc_adj; PM_OC (µg/m³)
ec_niosh; AECT (µg/m³)
oc_adj+ec_niosh; PM_OC+AECT (TC; µg/m³)

AQS data are obtained through the EPA’s Air Quality System (AQS), located at http://aqsdr1.epa.gov/aqsweb/aqstmp/airdata/download_files.html. Various species of atmospheric gases are available for download through the AQS. The pre-generated files on this site need to be combined into a single data file in order to work best with AMET. AMET compatible AQS data files are available for download from the CMAS website.

Hourly AQS species used with AMET:

O3; O3 (ppb)
NO; NO (ppb)
NOY; NOY (ppb)
NO2; NO2 (ppb)
NOX; NO+NO2 (NOX, ppb)
CO; CO (ppb)
SO2; SO2 (ppb)
PM25; PMIJ (ug/m3)
PM10; PM10 (ug/m3)
Isoprene; ISOP (ppb)
Ethylene; ETH (ppb)
Ethane; ETHA (ppb)
Toluene; TOL (ppb)
Temperature: SFC_TMP (C)
RH; RH (%)
Wind_Speed; WSPD10 (m/s)
; PBLH (m)
; SOL_RAD (watts/m2)
; 10*precip (mm/hr)

Daily AQS species used with AMET:

PM25; ATOTIJ (ug/m3)
PM25_88101; ATOTIJ (ug/m3)
PM25_88502; ATOTIJ (ug/m3)
PM10; ATOTIJK (ug/m3)
Isoprene; ISOP (ppb)
Ethylene;ETH (ppb)
Ethane; ETHA (ppb)
Toluene; TOL (ppb)
Acetaldehyde; ALD2 (ppb)
Formaldehyde; FORM (ppb)
OC+OC_Blank; AOCIJ (ug/m3)
EC,ug/m3; AECIJ (ug/m3)
OC+OC_Blank+EC; AOCIJ+AECIJ (ug/m3)
Na; ANAIJ (ug/m3)
Cl; ACLIJ (ug/m3)
Na+Cl; ACLIJ+ANAIJ (ug/m3)
SO4; ASO4IJ (ug/m3)
NO3; ANO3IJ (ug/m3)
NH4; ANH4IJ (ug/m3)
Fe; AFEJ (ug/m3)
Al; AALJ(ug/m3)
Si; SIJ (ug/m3)
Ti; ATIJ (ug/m3)
Ca; ACAJ (ug/m3)
Mg; AMGJ (ug/m3)
K; AKJ (ug/m3)
Mg; AMNJ (ug/m3)
2.2*Al+2.49*Si+1.63*Ca+2.42*Fe+1.94*Ti; ASOILJ (soil, ug/m3)
PM25-SO4-NO3-NH4-OC-EC-[Na]-[Cl]-2.2*Al-2.49*Si-1.63*Ca-2.42*Fe-1.94*Ti; AUNSPEC1IJ (OTHER, ug/m3)
0.8*OC; ANCOMIJ (NCOM, ug/m3)
PM25-SO4-NO3-NH4-OC-EC-[Na]-[Cl]-2.2*Al-2.49*Si-1.63*Ca-2.42*Fe-1.94*Ti-0.8*OC; UNSPEC2IJ (OTHER_REM, ug/m3)

NOAA ESRL species used with AMET:

O3; O3(ppb)

TOAR species used with AMET:

O3; O3(ppb)

This section describes how to set up the MySQL database. Please refer to the flow diagram in Appendix A to understand the overall flow of data among the various modules within AMET. This section must be completed before you populate the database with your project-specific data (Section 6). This setup process is required only once for a given AMET installation. There are separate setup procedures for the two fields, MET and AQ. If you are using AMET for only one of those fields, you need to run only the corresponding setup. If you are running AMET for both fields, you will need to run both setups. In the following discussion, we assume the default name of the AMET MySQL user is “ametsecure”. If you decide to change the AMET database user or password, then you will need to update the appropriate variables in the R configuration files in the directory $AMETBASE/configure (see Section 3). Before you run the setup scripts, you will need to know the “root” password for the MySQL administrator. Note that this is not necessarily the same as the “ametsecure” password that will be created using the scripts discussed below.

Note that the database is required for the meteorological side of AMET. However, as of AMETv1.4, the database is no longer a requirement to process and analyze air quality data. An option has been added to read the csv output files from site compare directly, bypassing the need to use the database. This has both advantages and disadvantages. The primary advantage is that an AMET user would not be required to install and setup the MySQL database for AMET, eliminating and streamlining the AMET installation process. The primary disadvantages are that all output files must be retained and organized, as they are needed to do any analysis, and analyses that require the use of site metadata may not work without the database. Also, the use of the database provides the ability to fully query and subset the AQ data in the database using the metadata provided. This functionality would be highly limited without the use of the database. It is up to the user to choose whether or not to install and utilize the database. However, keep in mind that the database is still required to process and analyze meteorological data.

The instructions provided below assume the use of the MySQL database. If only processing AQ data and not employing the use of the database, those portions of the instructions that deal with setup and interfacing with the database can be ignored.

Go to the AMET database setup directory

cd $AMETBASE/scripts_db/dbSetup

To create an AMET user, you will need to edit and run the create_amet_user.csh script. Confirm that the value of AMETBASE in this script corresponds to the AMET installation on your system. Run this script to set up the AMET database:

./create_amet_user.csh

After executing the script you will be prompted for MySQL’s “root” password. The MySQL root user must have permission to create and modify user. It will then set up the AMET user, here assumed to be “ametsecure”.

The dbSetup directory also contains a script for removing the amet user and database. To delete a specific AMET database, edit and run the delete_db.csh script, setting the $AMETBASE and $AMET_DATABASE variables to specify the database to be removed.

./delete_db.csh

After executing the script you will be prompted for MySQL’s “root” password. Use this script with EXTREME CAUTION because this will delete all of the data in the database corresponding to all of the projects (both MET and AQ).

As you proceed through the amet database setup and the project-specific database populate process, you may want to query the database directly using command line SQL commands. Here are a few commands to help you interact directly with the MySQL amet database. For more specifics, see one of the many MySQL books available, or look at the documentation under http://dev.mysql.com/doc.

To log onto the MySQL server from the command line:

$ mysql -u ametsecure -D amet –p

This will give you a MySQL prompt (“mysql>“). Note that all MySQL commands are case insensitive, and they must end with a semicolon (“;”).

To get a list of all the tables in your database

mysql> show tables;

After you have populated all of the example projects (end of Section 6), that command will yield a table like this:

+---------------------+

| Tables\_in\_amet |

+---------------------+

| aqExample |

| aq\_project\_log |

| project\_log |

| project\_units |

| site\_metadata |

| stations |

| metExample_wrf_surface |

| metExample_wrf_raob |

+---------------------+

To select every column and row in your project_log table:

mysql> select \* from project\_log;

To select the latitude, longitude, and common name columns from the stations metadata table and limit the results to the first 20 rows:

mysql> select lat,lon,common\_name from stations limit 20;

To select all station metadata where the monitor is from the CASTNET network:

mysql> select \* from site\_metadata where network=‘castnet’;

To determine which networks are included in the aqExample project:

mysql> select distinct network from aqExample;

Note: It is wise for users to understand simple queries like this or to download an interactive MySQL database tool as a means to look at the database, table structure and even sample data therein in the case of problems.

The database population phase of AMET must be performed for each new project. As discussed in Section 1.2, the project is the organizing structure used to group a particular model simulation with the scripts and data used to populate the AMET tables. Navigate to the database populate directory by typing

cd $AMETBASE/scripts_db

This directory contains two project directories and one input files directory, in addition to the dbSetup directory described earlier. The projects are

• metExample_wrf, metExample_mpas and metExample_mcip: MET examples for the WRF, MPAS models plus MCIP support
• aqExample: an AQ example for the CMAQ model

In the following subsections, we describe how to run each project.

Go to the metExample_wrf project directory. Note these are the same steps for metExample_mpas and metExample_mcip.

cd $AMETBASE/scripts_db/metExample_wrf

The C-shell file matching_surface.csh is a wrapper script for calling the R MET_matching_surface.R that actually populates the AMET database with the project data. Verify that the variable AMETBASE is set to the correct AMET project. The example script has detailed instructions on all variables that are passed to R. Modify according to your setup and run the script by typing

./matching_surface.csh |& tee log.populate.sfc

After executing the script you will be prompted for MySQL’s “root” password. The script can be configured to not prompt for a password by adding the variable password to the script and setting it to the MySQL "root" pass. This non-interactive option is useful for batch processing or for enabling the script to run in the background.

This C-shell script for surface meteorology will create an empty project tables in the AMET database: wrfExample_wrf_surface. It is important to understand that if users specify a database via AMET_DATABASE that is not present on the MySQL server, a new database will automatically be created. The wrfExample_wrf_surface table contains the matched pairs of model outputs and surface observations. After creating the table, the script then excutes the matching process. This process consists of retreiving data from the MADIS web site for the model's temporal period, unzipping the downloaded data, finding the geographic location of each observation site on the model grid and interpolating to those locations, populating the appropriate table with the model-obs pairs for each variable. Finally, the script updates the project_log with summary information for the wrfExample project like project creation date, last matching execution, period of record and description of project.

./matching_radiation.csh |& tee log.populate.radiation

This C-shell script is executed the same as the surface script above. It is used to compare the model with Baseline Surface Radiation Network (BSRN) or SURFRAD shortwave radiation measurements. The setting RADIATION_DSET should be set to bsrn or surfrad. BSRN has a lag period of months to a year for data curation, but SURFRAD is real-time. BSRN is global and SURFRAD is US only. When executed, the model-observation pairs are inserted in the wrfExample_wrf_surface table. The script will automatically download the BSRN or SURFRAD observations from the FTP site (RAD_SERVER) in the matching_radiation.csh script. Users should contact the BSRN organization and request access, which will follow with a login and password that should be specified (RAD_LOGIN and RAD_PASS) https://bsrn.awi.de/data/data-retrieval-via-ftp/ . SURFRAD uses anonymous FTP + a users email for their internal tracking. BSRN files are monthly text files with 1 minute data. It takes a few minutes of processing to read these file, but after, the script runs very fast. This is a new option in AMETv1.4. AMETv1.5 adds the SURFRAD capability. These data are hourly like MADIS and seperate files for each site, but autoFTP coordinates this more complex retrieval so this option is advised.

./matching_raob.csh |& tee log.populate.raob

This C-shell is executed like the ones above. It will create a new table wrfExample_wrf_raob specifically for profile observations. Like the surface meteorology, this script will download MADIS rawinsonde observations and match with the model profiles. This was a new option added in AMETv1.4 and allows users to evaluate the entire troposphere.

For the metExample_mcip, the matching_raob.csh needs to increment over the number of days. Recommend using the loop_over_days.csh script.

./loop_over_days.csh |& tee log.populate.raob

Go to the AQ example project directory:

$ cd $AMETBASE/scripts_db/aqExample

The C-shell file aqProject_pre_and_post.csh is a wrapper script for calling the R programs that actually create an AMET AQ project (and AMET database if necessary) and populate the AMET database with the project data.

Start by opening the aqProject_pre_and_post.csh and verify that the variable AMETBASE is set to the correct AMET project. Run the script by typing

./aqProject_pre_and_post.csh |& tee log.populate.aqExample

After executing the script you will be prompted for MySQL’s “root” password. The script can be configured to not prompt for a password by adding the variable password to the script and setting it to the MySQL "root" pass. This non-interactive option is useful for batch processing or for enabling the script to run in the background. By default, the script is setup to prompt you for the MySQL login/password.

This C-shell script will create the AMET database (if it does not already exist), three required AQ database tables (i.e. project_units, site_metadata and aq_project_log), and one empty project table in the AMET database: aqExample. After creating this table, the script then begins the matching process. This consists of calling a series of Fortran helper programs. The two Fortran helper programs are $AMETBASE/bin/sitex_daily.exe and $AMETBASE/bin/sitex_daily_O3.exe; the first one matches the AQS network’s data to the nearest grid cell in the CMAQ model, and the second one does the same for the other networks. These programs need to be downloaded and built (and the path to the executable specified in the amet-config.R file before running the aqProject.csh script. After each network has been matched to the model, the aqExample table is populated with the model-obs pairs. In addition to creating and populating the aqExample table, the script updates the project_units table with each network for that project. This table defines the physical units of the species variables for this network (e.g., ppb vs. µg/m³). Finally, the script updates the aq_project_log with summary information for the aqExample project.

When you create your own projects, we recommend that you utilize the AMET convention of naming your directories after your projects. If you choose not to do this, you will have to modify the provided run scripts to suite your own needs.

To create a new project, follow these basic steps:

1. Copy the appropriate example project (metExample_wrf) to a new directory name (see below).

2. Rename these directories after your new project (use the exact project name, as the utility scripts use the project name to navigate directories and organize analyses).

3. Create a new project directory under $AMETBASE/model_data/MET for the model output files. In most cases you will want to link in model outputs from an archive or work directory. This also serves to keep model outputs organized.

4. Configure the C-shell script matching_surface.csh for the new project.

5. Run the matching_surface.csh (or matching_radiation.csh, or matching_raob.csh) script to populate the AMET database.

TIP: Name the directory of each new project the same name as the AMET_PROJECT variable in the database and analysis scripts.

For example, to create a new WRF project called “wrfNC2007”, use the following commands:

cd $AMETBASE/scripts_db

cp -r metExample_wrf wrfNC2007

cd wrfNC2007

cd $AMETBASE/scripts_db
cp -r metExample_wrf wrfNC2007

Create a new model data directory and move or link model data into it, as follows:

cd $AMETBASE/model_data/MET
mkdir wrfNC2007
cd wrfNC2007
ln -s /model/data/directory/wrfout* $AMETBASE/model_data/MET/wrfNC2007/.

Here, you would replace "/model/data/directory/" with the path to your model data file(s). The matching_surface.csh, matching_radiation.csh and matching_raob.csh will perform the model-obs matching of all model outputs in this new project directory. Users can use wildcards like the example above to link all or just specific WRF/MPAS/MCIP outputs into the MET output directory.

Next, edit the $AMETBASE/script_db/wrfNC2007/matching_surface.csh variables AMET_PROJECT ("wrfNC2007") and RUN_DESCRIPTION (your description of the project).

Finally, run the surface model-obs matching script (or others):

cd $AMETBASE/scripts\_db/wrfNC2007
./matching_surface.csh |& tee matching.wrfNC2007.log

The matching_surface.csh script will create a new MET project in the AMET database if it does not exist (a new database will also be created if it does not already exist). Specifically, it will create a new row in the AMET project_log table and wrfNC2007_surface. The matching_radiation.csh script will put radiation data into the same wrfNC2007_surface table. The matching_raob.csh script will put upper-air meteorology data in a wrfNC2007_raob table. Once this script completes, the AMET database will be ready to produce meteorology model performance analysis plots and statistics.

Before illustrating the creation of a new AQ project, the relationship between model species and monitor species will be described. In order for AQ database population to work, there must be a mapping between the model species and the various observational network species. This mapping is accomplished through a combination of post-processing the CMAQ model data and species formulas in the AQ_species_list.input file, which is located in the $AMETBASE/scripts_db/input_files directory. The model data used in the aqExample project (Section 6.3) have already been post-processed. For a new project, the CMAQ data will need to be post-processed before they are ingested into the AMET database. This post-processing is accomplished by using the Fortran program Combine.

The directions below assume you are running combine separately to create the combine files for AMET to use, and therefore will only need to run the AMET script designed for post analysis only using the aqProject_post_only.csh script found in $AMETBASE/scripts_db/aqExample. There is also a more comprehensive script that performs both the pre analysis functions (e.g. running combine) and the post analysis functions (e.g. running site compare and AMET). That script is named aqExample_pre_and_post.csh and can also found in the $AMETBASE/scripts_db/aqExample directory. Instructions for using that script can be found in a separate guide here: aqProject Pre and Post Analysis Script Guide.

TIP: Name the directory of each new project the same name as the AMET_PROJECT variable in the database and analysis scripts.

To create a new AQ project, follow these basic steps:

1. Copy $AMETBASE/scripts_db/aqExample to a new directory with the name of the new project.

2. Post-process the model data using the Fortran program Combine.

3. Create a new project directory under $AMETBASE/model_data/AQ for the input model data and copy or link post-processed model data to this directory.

4. Configure the C-shell script aqProject_post_only.csh for the new project.

5. Run the aqProject_post_only.csh script to populate the AMET database.

For example, to create a new AQ project called “aqNC2007”, use the following commands:

cd $AMETBASE/scripts_db
cp -r aqExample aqNC2007

Next, create a new model data directory and move or link post-processed model data into it, as follows:

cd $AMETBASE/model_data/AQ
mkdir aqNC2007
cd aqNC2007
ln -s <model data directory> .

Replace “<model data directory>” in this example with the path to post-processed model data file(s) output from Combine.

The program Combine is used to post-process CMAQ (and CAMx) I/O API-netCDF formatted files for pairing with observational data. The source code and scripts for Combine are available in the CMAQ GitHub repository.

For detailed instructions on using Combine, see https://github.com/USEPA/CMAQ/tree/5.2/POST/combine.

After installing the model data in the AMET directories, configure the $AMETBASE/scripts_db/aqProject.csh script. The aqProject.csh script does several things:

• Creates a project table in the AMET database (if requested). It will also create the database if it does not already exist.
• Creates, writes and runs site compare run scripts for each requested network. This step is required regardless of whether or not you plan to you the MySQL database.
• Populates that database with the model and observational data from the site compare scripts. This step can also be skipped if you do not plan on using the database.

The configuration options for the aqProject.csh script are documented in the script and briefly described below. Upon execution, the script calls several R scripts to run the Fortran program Site Compare and then populates the AMET database (assumed AMET_DB = T) with paired model-observation data. As this script will be used for setting up different AMET-AQ projects, it will likely only need to be fully configured once and then reused with little modification for future projects.

Set the following variables to configure the aqProject.csh script for a new project.

• Set AMETBASE to the root AMET installation directory for the project.
• Set AMET_DATABASE to the name of the database to use (by default this is set to "amet"). This does not need to be set if not using the database.
• Set MYSQL_CONFIG to the AMET R configuration file. This does not to be set regardless of whether or not you are using the database since it contains the paths to the site compare executables. However, you do not need to specify the database information if you do not plan to use the database.
• If desired, you can specify the MySQL login information using the mysql_login and mysql_password variables. If these variables are set to "config_file" the login information will be taken from the amet-config.R file. If you comment out these variables, the script will prompt you for the MySQL login and password.
• Set AMET_PROJECT to the name of the AMET project; this should be the same name as the project directory, it needs to be unique and contain no spaces. This is required.
• Set the AQ MODEL_TYPE (e.g. "CMAQ" or "CAMx"). Not required when not using the database.
• Set RUN_DESCRIPTION to a short description of the AMET project. Not required when not using the database.
• The variable USER_NAME will default to the system user ID and can be changed as desired. The USER_NAME is only used to identify you in the AMET database and is not used to as a login to the database. Not required when not using the database.
• Set EMAIL_ADDR to associate and email address with the project. This setting is not currenlty used for anything in AMET and is simply stored along with the project information. Not required when not using the database.

The Table 6-3 below describes the other options and file locations that need to be specified in the aqProject_post_only.csh and aqProject_pre_and_post.csh scripts.

After configuring the aqProject.csh script, it can be run to load the air quality model and observation data to the AMET database:

cd $AMETBASE/scripts_db/aqNC2007
./aqProject.csh |& tee populate.aqNC2007.log

This will create a new AQ project in the AMET database. Specifically, it will create a new row in the aq_project_log table, a series of new rows (one for each network) in the project_units table, and a new project table called aqNC2007.

The analysis capabilities of AMET consists of performing statistical analyses on the model-obs pairs. The basic process is to query the project’s database table(s) using SQL criteria; to perform statistical analyses on the returned data; and to create plots, tables, and text/R data file outputs. The AMET installation contains a series of preprogrammed statistical analysis and plotting routines, based on the R language. These scripts are provided strictly as a starting point and as illustrative examples. Because all the model-obs pairs are stored in a MySQL database, an advanced user can decide to access those data in any desired manner, including other software packages. All that is required is a MySQL interface and some exploration of the table structure. We encourage advanced users to extend these R scripts to create more specific or advanced plotting capabilities, to use other languages to expand AMET analysis capabilities, and to contribute these updates to the CMAS community.

As with the database population routines, a project is the organizing structure used to group a particular model run (specific model, physics or chemistry, spatial domain, scale, etc.) with the scripts used to analyze the AMET tables and with the output from the analysis (plots and data).

Example air quality and met analysis scripts are located in $AMETBASE/scripts_analysis/aqExample and $AMETBASE/scripts_analysis/metExample_wrf, respectively. Within each of these directories are C-shell wrapper scripts and a subdirectory called input_files containing a R configuration input file with similar names as the analysis scripts (e.g., run_timeseries.csh and input_files/timeseries.input). These two files set up everything that is necessary to configure and run the underlying AMET analysis R script (located in $AMETBASE/R_analysis_code). In AMETv1.5+ the R configuration input file was split into an input and a static.input file (e.g., timeseries.input and timeseries.static.input). This was mostly done to simplify the configuration for a new AMET GUI, but also helps distinquish settings a user can modify with static settings fundamental to the main R script. Also new to AMETv1.5+ is a universal confguration input file (input_files/run_info_MET.R) that can be used for all scripts and the GUI. These changes are all backward compatible with older analysis scripts. The use of the C-shell interface allows users who are not very familiar with R to perform a set of predefined analyses with AMET.

Use the following command to navigate to the met analysis example project directory:

cd $AMETBASE/scripts_analysis/metExample_wrf
or
cd $AMETBASE/scripts_analysis/metExample_mpas
or
cd $AMETBASE/scripts_analysis/metExample_mcip

This directory contains a series of C-shell scripts and their accompanying input files (./input_files/ directory). There are detailed comments in the analysis scripts describing the main configuration options in the script. Other configuration options are set in the files under the ./input_files directory; the settings in these scripts are for fine tuning the AMET plots and typically do not need modification, especially the static files. Some of these other settings include color scales, text output options, and Quality Control limits. The settings for the met analysis scripts are all detailed in Appendix B Table B-2.

The run_spatial_surface.csh script is used here as an example of how to run the analysis scripts. This particular script creates a series of maps comparing the surface observations to models for a specificed period (date start/end). Each plot provides color-coded model performance metrics (RMSE, MAE, BIAS and Anomaly Correlation) at each of the monitor locations.

Edit the run_spatial_surface.csh file and change the AMETBASE variable to the root AMET installation directory on your system. Set the location of the amet-config.R file; the default location is in the $AMETBASE/configure directory. As noted previously in this guide, the settings in this file could be secured by limiting the read access to only the user. Alternatively this file can be "hidden" by saving it to $HOME/.amet-config.R. The corresponding input file for this example is input_files/spatial_surface.input and will likely not need to be changed unless extra query specifications are needed or plot symbol size needs adjustment. Most of the primary configuration settings for the AMET analysis scripts are in the .csh run script. Other AMET installation and database settings to check in the run_spatial_surface.csh script are the AMET_DATABASE, MYSQL_SERVER, and AMET_PROJECT variables. Analysis configuration settings in the script include:

• AMET_OUT: directory where the plots and text output will be written
• AMET_PTYPE: output plot format (pdf is recommended, but png is an option)
• AMET_DATES and AMET_DATEE: start and end dates for the AMET analysis (format: YYYYDDMM HH)
• THRESHOLD: number of valid observations required for statistics to be computed
• AMET_BOUNDS_LAT and AMET_BOUNDS_LON: latitude and longitude bounds of the plot

After configuring the example met analysis script, save and run the script:

./run_spatial_surface.csh |& tee spatial_surface.log

The plots from this script will be written to the $AMETBASE/output/metExample_wrf directory. A subdirectory is created in this output directory for each analysis (e.g.; spatial_surface, summary, timeseries and daily_barplot). After the script has completed, go to the output directory to view the plots:

cd $AMETBASE/output/metExample_wrf/spatial_surface

or

cd $AMETBASE/output/metExample_mcip/spatial_surface

or

cd $AMETBASE/output/metExample_mpas/spatial_surface

You should see a whole series of plots of the form:

metExample_wrf.<stats>.<variable>.20160701-20160801.pdf

A brief summary of each of the C-shell scripts that drive the main R analyses, with example plots from each script, is given below.

run_spatial_surface.csh (Example Spatial Plot)

• spatial_surface.input
• spatial_surface.static.input
• Creates maps of statistics at each observation site
• Creates a csv file of the site specific statistics (Example csv)

run_timeseries.csh (Example Timeseries Plot)

• timeseries.input
• timeseries.static.input
• Creates a 4 panel timeseries of model and observed temperature, mixing ratio, wind speed and direction.
• Creates a text file and R data file of the time series (Example of text output)

run_timeseries_rh.csh (Example Timeseries RH Plot)

• timeseries_rh.input
• timeseries.static.input
• Creates a 4 panel timeseries of model and observed temperature, mixing ratio, relative humidity and surface pressure.
• Creates a text file and R data file of the time series (Example of text output)

run_summary.csh (AMET Plot Diurnal Plot)

• summary.input
• Creates two plots for each met variable. A diurnal statistics plot and summary plot with panels that include scatter plot, stats table, statistics as a function of the observation range.
• Creates a csv file of both dirunal and overall statistics (Example csv)

run_daily_barplot.csh (Example Daily Plot)

• daily_barplot.input
• Creates a barplot of daily statistics values over the range of dates specified by user. One plot for each met variable and statistic.
• Creates a csv file of daily statistics (Example csv)

run_plot_radiation.csh (Example plots: Diurnal, Spatial, Timeseries, Histogram)

• plot_radiation.input (note that pre-AMETv1.5 named this script "plot_srad" rather than "plot_radiation")
• plot_radiation.static.input
• Creates several shortwave radiation evaluations plots. Spatial, diurnal, histogram and timeseries.
• Creates a csv file for (Example csv)

run_raob.csh (Example plots: Spatial, Profile, Daily)

• raob.input
• raob.static.input
• Creates a number of plots. Not all are shown above. See script for full details.
• Creates a csv file for daily and spatial statistics (Example csv)

run_prism_comp.csh (NetCDF example, Text Stats, Leaflets HTML Plot)

• Creates a NetCDF file in the model output format of PRISM and model total precip + a text file with grid average statistics.
• Uses R prism package to automatically download PRISM for daily, monthly or annual periods.
• Looping capability in run script allows loop over days, months or years for much easier control for long model runs.
• Leaflet HTML plotting if new automated download of PRISM raster format (BIL) is used as recommended.
• Users have the flexibility to use Verdi, Ncview, IDV or other software to plot as desired or read via NetCDF modules and do external analysis on the data.

wrapper.csh

• A new script in AMETv1.5 that acts as a wrapper for most of the scripts listed above (spatial, daily barplot, summary, raob and radiation).
• Uses an wrapper run code (e.g., DB.RM) that drives an analysis script and analysis mode. DB.MN for example runs the daily barplot (DB) statistics for each US Climate Region & Month of a defined year (RM). Example script only does July 2016 for each region. SM.RM is another option activated in example script for summary statistics (SM) for each region and month (RM). Notes in script provide all options.
• The script lists all availiable options that include monthly, seasonal, regional-monthly and regional-seasonal statistics for the various analysis scripts.
• The run_info_MET.R configuration allows further refinement so users can build a complete model evaluation protocol and run from a single script execution.
• The new AMET GUI will also allow have capability from a user interface.

Use the following command to navigate to the air quality analysis example project directory:

cd $AMETBASE/scripts_analysis/aqExample

The directory includes a set of C-shell scripts and their accompanying input files in the subdirectory input_files. The example below provides details on running one of the scripts in the example project.

The run_scatterplot.csh script creates a scatterplot for one species from one or more monitoring networks. It compares the observed values to the corresponding model values. As not all of the AQ monitoring networks monitor all species, users need to know which network(s) to select for particular model species. See Section 4.2 and Appendix B for more details on the various species that are monitored (or available) from each AQ network.

Edit the run_scatterplot.csh file to run an example AQ analysis. Below is a table describing the option available in the run_scatterplot.csh script. Note that for this example the default script has SO4 as the selected species and the IMPROVE and CASTNET networks as the observational data to use for the SO4 evaluation. The corresponding input file, scatterplot.input, will likely not need to be changed for this example.

Each script requires an input file, located in the subdirectory input_files. The input file contains all the options available for each script, and allows the user to customize scripts to their liking. Unlike previous version of AMET where each script had its own individual input file, for AMETv1.4 and beyond, all scripts by default will use the all_scripts.input file, which contains all the options available for all the analysis scripts. This eliminates the need to edit each individual input file.

A brief summary of each of the typical options for a AQ analysis script is given below in table 7-2.

Also note that all AQ analysis scripts make use of the Network input file. This file contains information about each observational network available to the project that is needed by the R scripts. More information about this file can be found in Appendix B.

Edit the AMETBASE and AMET_DATABASE variables to be consistent with the AMET installation on your system. Save and run the script:

./run_scatterplot.csh |& tee scatterplot.log

The output plots, a CSV file of the plotted data, and a detailed log file will be written to the directory $AMETBASE/scripts_analysis/aqExample. After the script has completed, go to the output directory and view the plots:

cd $AMETBASE/output/aqExample/scatterplot

You should see files of the form:

aqExample_SO4_scatterplot.pdf

A brief summary of each of the C-shell scripts, with example plots from each script, is given below. To view the html files, you'll need to follow the link, click on "view raw", then save the page as an html file. You'll then be able to view the interactive html plot.

run_boxplot.csh (Example Plot)

• Creates a box plot of model-obs quartiles
• single network; single species; multi simulation

run_boxplot_DofW.csh (Example Plot)

• Creates a box plot of model-obs quartiles parsed by the day of the week
• single network; single species; single simulation

run_boxplot_ggplot.csh (Example Plot)

• Creates a box plot of model-obs quartiles using the ggplot2 R package
• single network; single species; multi simulation

run_boxplot_hourly.csh (Example Plot)

• Creates side-by-side boxplots to create a diurnal average curve. Hourly data only
• single network; hourly species only; multi simulation

run_boxplot_plotly.csh (Example HTML)((Example Plot - static image)

• Creates a interactive box plot of model-obs quartiles using the plotly R package
• single network; single species; multi simulation

run_boxplot_roselle.csh (Example Plot)

• Creates a box plot of model-obs quartiles, with select statistics provided underneath the box plot
• single network; single species; multi simulation

run_bugleplot.csh (Example Plot)

• Model performance criteria are adjusted as a function of the average concentration of the observed value for that species. As the average concentration of the species decreases, the acceptable performance criteria increase. Creates a bias and error plot
• multiple networks; single species; single simulation

run_histogram.csh (Example Plot)

• Creates a histogram of model-obs quartiles
• single network; single species; multi simulation

run_kellyplot.csh (Example Plot)

• Creates a kellyplot of model performance color coded by season and NOAA climate region
• single network; single species; single simulation; full year data required

run_kellyplot_region.csh (Example Plot)

• Creates a kellyplot of model performance color coded by simulation and NOAA climate region
• single network; single species; multiple simulations allowed;

run_kellyplot_season.csh (Example Plot)

• Creates a kellyplot of model performance color coded by simulation and season (winter, spring, summer, and fall)
• single network; single species; multiple simulations allowed;

run_monthly_stat_plot.csh ([Example Plot])

• Creates a monthly average time series of obs/mod concentration, bias, RMSE, NMB, NME and correlation.
• single network; single species; multi simulation; full year data required

run_plot_spatial.csh (Example Plot)

• Plots the observed value, model value, and difference between the model and obs for each site. Multiple values for a site are averaged to a single value for plotting purposes
• multiple networks; single species; single simulation

run_plot_spatial_leaflet.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Plots the observed value, model value, and difference between the model and obs for each site. Multiple values for a site are averaged to a single value for plotting purposes. Uses R leaflet package to allow map zooming
• multiple networks; single species; single simulation

run_plot_spatial_diff.csh (Example Plot)

• Plots the difference in bias and error between two model simulations each site. Multiple values for a site are averaged to a single value for plotting purposes
• multiple networks; single species; multi simulations required

run_plot_spatial_diff_leaflet.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Plots the difference in bias and error between two model simulations each site. Multiple values for a site are averaged to a single value for plotting purposes. Uses R leaflet package to allow map zooming
• multiple networks; single species; multi simulations required

run_plot_spatial_mtom.csh (Example Plot)

• Plots the absolute difference between two model simulations at observation sites, regardless if valid observations exist or not. Multiple values for a site are averaged to a single value for plotting purposes.
• multiple networks; single species; multiple simulations required

run_plot_spatial_mtom_species.csh (Example Plot)

• Plots the min, max, average, and ratio between two model species for a single simulation at observation sites, regardless if valid observations exist or not. Multiple values for a site are averaged to a single value for plotting purposes. Most useful to plot the ratio difference between two species. This script does not directly utilize any observation data.
• multiple networks; two species required; single simulation

run_plot_spatial_ratio.csh (Example Plot)

• Plots the model/obs ratio for each site. Multiple values for a site are averaged to a single value for plotting purposes
• multiple networks; single species; single simulation

run_raw_data.csh

• Used to extract raw data from the database. Output is a csv file containing the data requested
• single network; single species; multi simulation

run_scatterplot.csh (Example Plot)

• Creates a single model vs. obs scatterplot. This script will plot a single species from up to three networks on a single plot. Summary statistics are also included on the plot
• multiple networks; single species; multiple simulations

run_scatterplot_ggplot.csh (Example Plot)

• Creates a single model vs. obs scatterplot. This script will plot a single species from up to three networks on a single plot. Summary statistics are also included on the plot. Uses the ggplot2 R package
• multiple networks; single species; multiple simulations

run_scatterplot_plotly.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Creates a single interactive model vs. obs scatterplot. This script will plot a single species from up to three networks on a single plot. Summary statistics are also included on the plot. Uses the plotly R package
• multiple networks; single species; multiple simulations

run_scatterplot_bins.csh (Example Plot)

• Creates a multi-panel scatterplot of bias and RMSE, where the values are binned by the observed or modeled concentration. This script will plot a single species for a single network
• single networks; single species; multiple simulations

run_scatterplot_bins_plotly.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Creates an interactive multi-panel scatterplot of bias and RMSE, where the values are binned by the observed or modeled concentration. This script will plot a single species for a single network. Uses the R plotly package
• single networks; single species; multiple simulations

run_scatterplot_density.csh (Example Plot)

• Creates a single model vs. obs scatterplot with shading to represent the density of points
• multiple networks; single species; single simulation

run_scatterplot_density_ggplot.csh (Example Plot)

• Creates a single model vs. obs scatterplot with shading to represent the density of points. Uses the ggplot2 R package
• multiple networks; single species; single simulation

run_scatterplot_mtom.csh (Example Plot)

• Creates a single model-to-model scatterplot. Note: The model points correspond to network’s site locations only
• multiple networks; single species; multiple simulations

run_scatterplot_mtom_denisty.csh (Example Plot)

• Creates a single model-to-model scatterplot with shanding to represent the density of points. Note: The model points correspond to network’s site locations only
• multiple networks; single species; multiple simulations required

run_scatterplot_multi.csh (Example Plot)

• Creates a single model vs. obs scatterplot, designed specifically for plotting many simulations on a single plot. This script will plot a single species from a single network for up to six different simulations. Summary statistics are also included on the plot
• single networks; single species; multiple simulations

run_scatterplot_multi_plotly.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Creates a single interactive model vs. obs scatterplot, designed specifically for plotting many simulations on a single plot. This script will plot a single species from a single network for up to six different simulations. Summary statistics are also included on the plot. Uses the plotly R package
• single networks; single species; multiple simulations

run_scatterplot_percentiles.csh (Example Plot)

• Creates a single model vs. obs scatterplot, color coding the 5th, 25th, 50th, 75th and 95th percentiles
• single networks; single species; single simulation

run_scatterplot_single.csh (Example Plot)

• Creates a scatter plot for a single network that includes more specific statistics than run_scatterplot.csh
• single network;single species;multiple simulations

run_scatterplot_skill.csh (Example Plot)

• Creates a forecast skill scatter plot. The script is designed to work specifically with O₃
• all AQS networks; O₃; single simulation

run_scatterplot_soil.csh (Example Plot)

• Creates a single model vs. obs scatterplot designed specifically for plotting soil species (e.g. Si, Fe, Al, etc.). This script will plot the soil species from a single network on a single plot
• single network; multiple soil species; single simulation

run_soccerplot.csh (Example Plot)

• Creates a soccerplot for one or more species over one or more networks. Criteria and goal lines are plotted in such a way as to form a “soccer goal” on the plot area. Two statistics are then plotted: Bias [NMB (normalized mean), FB (fractional), or NMdnB (normalized median)] on the x-axis and Error [NME (normalized mean), FE(fractional), or NMdnE(normalized median)] on the y-axis. The better the performance of the model, the closer the plotted points will fall within the “goal” lines
• multiple network; multiple species; multiple simulations

run_spectral_analysis.csh (Example Plot)

run_stacked_barplot.csh (Example Plot)

• Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC, and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot, along with the percent of the total PM_2.5 that each species constitutes. Does not include soil, so is compatible with simulations using CMAQ aerosol modules prior to AERO6.
• CSN, IMPROVE or SEARCH; species predefined; multiple simulations

run_stacked_barplot_AE6.csh (Example Plot)

• Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC, soil, NCOM and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot, along with the percent of the total PM_2.5 that each species constitutes
• CSN, IMPROVE or SEARCH; species predefined; multiple simulations

run_stacked_barplot_AE6_ggplot.csh (Example Plot)

• Creates a single stacked bar plot. Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC, individual trace elements (e.g. Si, Fe), NCOM and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot. Uses the ggplot2 R package
• CSN or IMPROVE; species predefined; multiple simulations

run_stacked_barplot_AE6_plotly.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Creates a single interactive stacked bar plot. Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC, individual trace elements (e.g. Si, Fe), NCOM and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot. Uses the plotly R package
• CSN or IMPROVE; species predefined; multiple simulations

run_stacked_barplot_AE6_ts.csh (Example Plot)

• Creates static and interactive stacked bar time series plots. Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC, individual trace elements (e.g. Si, Fe), NCOM and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot. Uses the ggplot and plotly R packages
• CSN or IMPROVE; species predefined; multiple simulations

run_stacked_barplot_panel_AE6.csh

• Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC, soil, NCOM and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot, along with the percent of the total PM_2.5 that each species constitutes. Specifically designed to plot data for an entire year (separated by season) for four different geographic regions.
• CSN, IMPROVE or SEARCH; species predefined; single simulation; full year data required

run_stacked_barplot_panel_AE6_multi.csh

• Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC, soil, NCOM and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot. Specifically designed to plot data for an entire year (separated by season) for four different geographic regions for multiple simulations.
• CSN, IMPROVE or SEARCH; species predefined; multiple simulations; full year data required

run_stacked_barplot_panel_AE6.csh

• Data are averaged (mean or median) for SO₄, NO₃, NH₄, EC, OC and PM_2.5 other for the model and observed values. Averages are then plotted on a stacked bar plot, along with the percent of the total PM_2.5 that each species constitutes. Specifically designed to plot data for an entire year (separated by season) for four different geographic regions.
• CSN, IMPROVE or SEARCH; species predefined; single simulation; full year data required

run_stacked_barplot_soil.csh (Example Plot)

• Data are averaged (mean or median) for the soil species (e.g. Si, Fe, Ti, Mg, etc.) for the model and observed values. Averages are then plotted on a stacked bar plot, along with the percent of the total soil concentration that each species constitutes
• CSN and IMPROVE networks; species predefined; single simulation

run_stacked_barplot_soil_multi.csh (Example Plot)

• Data are averaged (mean or median) for the soil species (e.g. Si, Fe, Ti, Mg, etc.) for the model and observed values. Averages are then plotted on a stacked bar plot, along with the percent of the total soil concentration that each species constitutes
• CSN and IMPROVE networks; species predefined; multiple simulations

run_stats.csh (Example File)

• Generates CSV files with domain- and site-specific statistics. Also provides a raw data query CSV file. No images are created with this script.
• multiple networks; single species; single simulation

run_stats_plots.csh (Example Plot)

• Generates a series of spatial plots of NMB, NME, FB, FE, and Correlation. CSV files with additional domain- and site-specific statistics are also included
• multiple networks; single species; single simulation

run_stats_plots_leaflet.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Generates a series of interactive spatial plots of NMB, NME, FB, FE, and Correlation. CSV files with additional domain- and site-specific statistics are also included. uses the R leaflet package to create interactive html files with zoom capability
• multiple networks; single species; single simulation

run_timeseries.csh (Example Plot)

• Creates a time series plot. With multiple sites; the sites are time averaged to create a single plot. Also plots the bias and error between the obs and model
• single network;single species; multiple simulations

run_timeseries_dygraph.csh (Example Leaflet HTML)(Example Leaflet Plot - Screenshot)

• Creates an interactive time series plot. With multiple sites; the sites are time averaged to create a single plot. Also plots the bias and error between the obs and model. Uses R dygraph package to allow time-series zooming
• single network;single species; multiple simulations

run_timeseries_mtom.csh (Example Plot)

• Creates a model to model time series plot. With multiple sites; the sites are time averaged to create a single plot. Also plots the bias between the and model
• single network;single species; multiple simulations

run_timeseries_multi_species.csh (Example Plot)

• Creates a time series plot for multiple species. With multiple sites; the sites are time averaged to create a single plot. Also plots the bias between the obs and model
• single network;multiple species; multiple simulations

run_timeseries_multi_species_plotly.csh (Example Plot)

• Creates a interactive (html) time series plot for multiple species. With multiple sites; the sites are time averaged to create a single plot. Also plots the bias between the obs and model
• single network;multiple species; multiple simulations

run_timeseries_multi_networks.csh (Example Plot)

• Creates a time series plot for up to two networks. With multiple sites; the sites are time averaged to create a single plot. Also plots the bias between the obs and model
• multiple networks;single species; multiple simulations

run_timeseries_plotly.csh (Example Leaflet HTML) (Example Leaflet Plot - Screenshot)

• Creates an interactive time series plot. With multiple sites; the sites are time averaged to create a single plot. Also plots the bias and error between the obs and model. Uses R plotly package to allow time-series zooming
• single network; single species; multiple simulations

run_temporal.csh (Example Plot)

• Creates four plots: a CDF plot; a Q-Q plot; a Taylor diagram; and a periodogram
• single network; single species; multiple simulations

Creating a new analysis project in AMET requires the same basic steps for both MET and AQ data. It is recommended that for new analysis projects, consistency is enforced in the naming of directories and projects. In other words, name the directory of the analysis scripts for a project the same as the project name in the analysis scripts (PROJECT_NAME).

Before running an of the AMET analysis scripts, ensure that the database loading scripts for project completed successfully. See Section 6.3 and Section 6.4 for additional details on creating new MET and AQ projects in AMET, respectively.

To create a new analysis project, follow these basic steps:

1. Copy the appropriate example project to a new directory named the same as the PROJECT_NAME in the script. Use the exact project name, as many scripts use the project name to navigate directories.

2. Configure the variables in the analysis C-shell scripts for the new project.

3. Run the new analysis scripts.

For example, to create a new WRF analysis project called “wrfNC2007”:

cd $AMETBASE/scripts_analysis
cp -r metExample_wrf wrfNC2007
cd wrfNC2007

Edit each of the C-shell analysis scripts to set the variable AMET_PROJECT to wrfNC2007. The scripts will likely require other changes, like the analysis dates, output options, plot format, etc.

Adding support for a new AQ network to AMET is relatively simple, but does require several steps.

1. Create a properly formatted observation data file for site compare.

This is generally the first step to setting up a new network for AMET. Format your new network observation data in a format that site compare can read. Use one of the existing network observation data files as a template for creating your new data file. The most versatile data format is that for the SEARCH hourly data, as it contains both a start date/time and end date/time, which allows for maximum flexibility in pairing the observation data with the model data. In addition to the data file, you will need to create site list for your new network. Site files are available for download from the CMAS website along with the network data for the existing networks. These two files, the data file and site list file, will be referred to in Step 3. The data file is assumed to have the name "NewAQNet_data_$year.csv" and the site file should either be in the tab delimited text format with the name "NewAQNet_sites.txt" or a comma separated metadata file with the name "NewAQNet_full_site_list.csv". Note that the txt format of the site file is being phased out in favor of the comma separated metadata file. You will also need this site metadata file to populate the site_metadata_table in the database.

2. Modify the AQ_species_list.input file

The AQ_species_list.input file, located in the input_files subdirectory in the scripts_db directory off the AMET base code directory, is used to setup the observation species to model species mapping. This is a R formatted input script. Again, the best method for setting up a new network is to follow the formatting of an existing network. Start by choosing a name for your new AQ network that does already exist in AMET. The name should be short, but descriptive of your network. In the example below the new network is called NewAQNet. There are three "categories" of species that can be setup with AMET. There is a standard set of species which are always computed, and then there are two optional sets of species call "cutoff" and "AE6". For most users, the "cutoff" and "AE" species can simply be left empty, as in the example below. The example below creates a new network with three species, SO4, NO3 and PM_TOT. Any number of addition species could be added using the same formatting and sequential numbering as below.

species_NewAQNet <- paste("
setenv AERO_1 "SO4f_val,ug/m3,ASO4IJ,,SO4" # sulfate
setenv AERO_2 "NO3f_val,ug/m3,ANO3IJ,,NO3" # nitrate
setenv AERO_3 "MF_val,ug/m3,",PM_MOD_SPEC,",ug/m3,PM_TOT" # Total PM2.5 mass
",sep="")
species_cutoff_NewAQNet <- ""
species_AE6_NewAQNet <- ""

Once you have setup the species for a new network as above, move to the bottom of the AQ_species_list.input file. In that file, you will find three lists containing the species definition names for all the exising networks in AMET. You will need to add your new network to each of those lists, using the names above (i.e. species_NewAQNet, species_cutoff_NewAQNet, and species_AE6_NewAQNet). Follow the formatting of the existing networks. Once you've done that, you can save your modified AQ_species_list.input file and move on to step 3.

3. Add your new network to the AQ_matching.R code

The third step to adding new network support to AMET is to modify the AQ_matching.R code in the $AMETBASE/R_db_code directory. Again, the best method for adding a new network is to follow the formatting of an existing network. In the AQ_matching.R code you will see a section near the top of the file called "Network Flags". Here you will add your new network, following the format of an existing network as per the example below.

NewAQNet_flag <- Sys.getenv('NEWAQNET') # Flag to include NewAQNet data in the analysis

After you've done that, move down to the section titled "Create and Execute Site Compare Scripts". In this section you will again add support for your new network following the format of an existing network as per the example below.

if ((NewAQNet_flag == "y") || (NewAQNet_flag == "Y") || (NewAQNet_flag == "t") || (NewAQNet_flag == "T")) {
table_type <- "SEARCH"
network <- "NewAQNet"
site_file <- paste(obs_data_dir,"/site_file_directory/NewAQNet_site_file_name",sep="")
ob_file <- paste(obs_data_dir,"/",year,"/NewAQNet_data_",year,".csv",sep="")
EXEC <- EXEC_sitex
run_sitex(network)
}

Once you've modified the AQ_matching.R code as above, you can save your modified version and move on the step 4.

4. Modify the aqProject.csh script

As mentioned in step 1, you will need to add the site_metadata table to the database in order for database queries for your new network to function properly. Follow the format of an existing network metadata file to create your new file. Once you have created your new file, place it in the $AMETBASE/obs/AQ/site_metadata_files directory. Next move to the $AMETBASE/scripts_db/input_files directory. Here you will modify the sites_meta.input file and add your new metadata file to the list of existing files. It does not matter where in the list you place your new file, just be sure to update the file numbering according. Once you have done that, you will next need to modify and re-run the project creation script (default name is aqProject.csh in the $AMETBASE/script_db/aqExample directory).

If you are adding a new network to an existing project (i.e. database is already created and setup), then all you need to do is re-run the loading of the site_metadata (no new tables need to be created). This can be accomplished using the flag RELOAD_METADATA in the aqProject.csh script. Setting that flag to T will reload the site metadata for all networks, regardless of whether or not the site_metadata table already exists and is populated (existing data are simply overwritten with the same data). To add your new network site metadata, set the flag to T. The next time you execute the run script, the site metadata table will be repopulated, this time including your new network. You only need to do this once. So, after you have run the script and added your new network metadata, you should set the RELOAD_METADATA flag to F.

The next step is to further modify the aqProject.csh script. Open the aqProject.csh script and move to the section containing the flags for the networks to include in the analysis. Here you will add your network to the list of network to process using the same formatting as an existing network per the example below.

setenv NEWAQNET T

By setting this flag to true, it will tell AMET you want to process your new network data for analysis. This is all you need to modify in this script. You can now move on step 5.

5. Modify the analysis script files

The final step to adding your new network to AMET is to modify the analysis scripts to include your new network. This is accomplished by modifying the run scripts in $AMETBASE/scripts_analysis/aqExample and the input files in $AMETBASE/scripts_analysis/aqExample/input_files/. Begin by opening one of the run scripts, for example the run_boxplot.csh script. In the run script, under the section titled with "Observation Network to plot", you will need to add a new environment variable for your new network. This will be used to set whether or not your new network is used in the analysis. You will need to modify the other run scripts with the same environment variable.

setenv AMET_NEWAQNET y

Once you have done that, the last step is to modify the Network.input file in the input_files subdirectory. Move to the input_files subdirectory and open the Network.input file. In there you will see a section called "Network selection flags from run script". Here, you will need to add your new network as per the example below.

inc_newaqnet <- Sys.getenv("AMET_NEWAQNET")

Next under the "Setup Network Arrays", you will need to add your new network as per the example below.

if (inc_newaqnet == "y") {
network_names <- c(network_names,"NewAQNet")
network_label <- c(network_label,"NewAQNet")
}

Once you have done that, you can save your modified version of the Network.input file.

After you have completed all the steps above, you should be ready to process your new network with AMET. The modifications above will allow you to run site compare to create paired model/ob data files and add those data to the database, and also allow you to run analysis scripts using your new network.

AMET is supported by the Community Modeling and Analysis System (CMAS) Center. See the CMAS Center Help Desk for instructions on how to get technical support for using AMET.

Appel, K.W., Gilliam, R.C., Davis, N., Zubrow, A., and Howard, S.C.: Overview of the Atmospheric Model Evaluation Tool (AMET) v1.1 for evaluating meteorological and air quality models, Environ. Modell. Softw.,26, 4, 434-443, 2011.

Overview Flow Diagram

Configuration and Input Files

Statisical metric reference document

This is the configuration file for all R scripts used in database population—for example, $AMETBASE/configure/amet-config.R.

**Table B-1. amet-config.R variables **

The analysis input files are found in $AMETBASE/scripts_analysis/metExample_wrf. The following is a partial list of variables. Not all of these variables are available in every input file. And, the analysis scripts and configuration files have comments describing these variables and others that may be missed below.

Table B‑2. MET analysis input variables

The analysis input files are found in $AMETBASE/scripts_analysis/aqExample/input_files. The following is a list of the variables found in the AQ analysis input files all_script.input and AMET_batch.input.

Table B‑3. AQ analysis input variables

In addition to the analysis input files which are script specific, AMET makes use of the input file $AMETBASE/scripts_analysis/aqExample/input_files/Network.input. This file allows all of the network-specific processing to be handled in one location, and allows for easier addition of new networks into the analysis scripts. This file does not need to be modified unless adding new AQ networks to AMET.

The input file $AMETBASE/scripts_db/input_files/AQ_species_list.input describes the mapping of observation species from each network to CMAQ model species (post-processed using combine) for use with the program site compare. This file is read as an input file to the AQ_matching.R script and used to create the site compare scripts for each network. This file is necessary since most air quality networks use different names to represent species. Site compare requires the observed and modeled species names to work correctly. The basic structure to map observation and species names for site compare is:

ob_species_name, ob_species_unit, model_species_name, model_species_unit, output_species_name

The output_species_name above is the name used by AMET to identify the species in the database. For example, for the IMPROVE network, sulfate is calculated in site compare by the following line:

SO4f_val, ug/m3, ASO4IJ, ug/m3, SO4

where SO4f_val is the name given to SO4 in the IMPROVE observation file, ASO4IJ is the name given to fine SO4 in the CMAQ combine file, and SO4 is the column name used to identify SO4 in the site compare output .csv file. Note that the units only need to be specified for either the observation or model species if they are the same (typically they are), but can be specified for both species.

The AQ_species_list.input is setup to work with the default AQ networks in AMET. However, it can be modified to work with additional networks, or the species for existing networks can be modified if desired. Just follow the existing entries in the file to add additional species. Note that the species are numbered sequentially, and therefore when adding or removing species be careful to correctly number the remaining species.