The SNODAS Tools process original SNODAS data files, extracted from a tar file, into zonal snowpack statistics for watershed basins. Zonal statistics are statistics calculated by zone where the zones are defined by an input zone dataset and the values are defined by a raster grid. For the SNODAS Tools, the input zone dataset is the watershed basin boundary shapefile and the raster grid is the clipped and projected SNODAS daily SWE grid. Originally, these tools were developed for processing snowpack statistics for the state of Colorado. Therefore, the examples throughout this documentation reference the Colorado study area.

After the data is processed, final snowpack statistics can easily be viewed in a choropleth map or exported in tabular form. Below is an image of the results from the SNODAS Tools displayed as a choropleth map. Right-click on the image and click Open image in new tab to see a larger view.

CDSS SNODAS Tools Chloropleth Map

As explained in the Processing Workflow section, the tool iterates through various data-manipulation processes (ex: downloading the data, clipping the national grid to the extent of the basins, calculating the zonal statistics). Each data-manipulation process can generate one or more output files. The majority of intermediate data files are currently saved to allow for process verification, troubleshooting, and avoiding re-downloads should the full analysis period need to be rerun due to changes in the software (each daily download of the SNODAS national files takes approximately 7 seconds).

The following illustrates the overall folder structure for the SNODAS Tools for Windows. The software is configured u sing the config\SNODAS-Tools-Config.ini file, which specifies locations of folders and files on the operational system.

Nested folders are represented by: '---'.
Files are represented by: '--->'.

File Structure of SNODAS Tools

C:\CDSS\SNODAS\

--- SNODAS_Tools
- - - - - aws\
- - - - - - - - > copyAllToOWFAmazons3.bat
- - - - - config\
- - - - - - - - > SNODAS-Tools-Config.ini
- - - - - processedData\
- - - - - - - - > SNODASTools.log
- - - - - - - - - - 1_DownloadSNODAS\
- - - - - - - - - - - - > SNODAS_YYYYMMDD.tar
- - - - - - - - - - 2_SetFormat\
- - - - - - - - - - - - > us_ssmv11034tS__T0001TTNATSYYYYMMDD05HP001.tif
- - - - - - - - - -3_ClipToExtent\
- - - - - - - - - - - - > SNODAS_SWE_ClipAndProjYYYYMMDD.tif
- - - - - - - - - - 4_CreateSnowCover\
- - - - - - - - - - - - > SNODAS_SnowCover_ClipAndProjYYYYMMDD.tif
- - - - - - - - - - 5_CalculateStatistics\
- - - - - - - - - - - - - SnowpackStatisticsbyBasin\
- - - - - - - - - - - - - - - - > SnowpackStatisticsByBasin_LOCALID.csv
- - - - - - - - - - - - - SnowpackStatisticsbyDate\
- - - - - - - - - - - - - - - - > SnowpackStatisticsByDate_YYYYMMDD.csv
- - - - - - - - - - - - - - - - > SnowpackStatisticsByDate_YYYYMMDD.geojson
- - - - - - - - - - - - - - - - > SnowpackStatisticsByDate_YYYYMMDD.zip
- - - - - - - - - - - - - - - - > SnowpackStatisticsByDate_LatestDate.csv
- - - - - - - - - - - - - - - - > ListOfDates.txt
- - - - - - - - - - 6_CreateTimeSeriesProducts
- - - - - - - - - - - - - SnowpackGraphsByBasin\
- - - - - - - - - - - - - - - - > LOCALID_SNODAS_SWE.png
- - - - - - - - - - - - - - - - > LOCALID_SNODAS_SnowCover.png
- - - - - - - - - - - - - - - - > LOCALID_SNODAS_SWE_Volume.png
- - - - - - - - - - - - - - - - > LOCALID_SNODAS_SWE_Volume_1WeekChange.png
- - - - - - - - - - - - - - - - > LOCALID-SNODAS-SWE-Volume-Gain-Cumulative.png
- - - - - scripts\ - - - - - - - - > SNODASDaily_Automated.py - - - - - - - - > SNODASDaily_Interactive.py - - - - - - - - > SNODAS_utilities.py - - - - - - - - > SNODASDaily_Automated_forTaskScheduler.bat - - - - - staticData\
- - - - - - - - > studyAreaExtent_prj.shp (created and exported by the SNODAS Tools)
- - - - - - - - > watershedBasinBoundary.shp (must be added before running the SNODAS Tools)
- - - - - - - - - WatershedConnectivity\
- - - - - - - - - - - - > Watershed_Connectivity_v3.xlsx (must be added before running the SNODAS Tools)
- - - - - TsTool\
- - - - - - - - - 6_CreateTimeSeriesProducts\
- - - - - - - - - - - - > ~$create-snodas-swe-graphs.TSTool.log
- - - - - - - - - - - - > create-snodas-swe-graphs-tstool-control.txt
- - - - - - - - - - - - > create-snodas-swe-graphs.TSTool
- - - - - - - - - - - - > snodas-localid-swe-graph-template.tsp
- - - - - - - - - - - - > snodas-localid-snowcover-graph-template.tsp
- - - - - - - - - - - - > snodas-localid-swe-volume-graph-template.tsp
- - - - - - - - - - - - > snodas-localid-swe-volume-1weekchange-graph-template.tsp
- - - - - - - - - - - - > snodas-localid-swe-volume-gain-cumulative-graph-template.tsp
- - - - - - - - - - - - - SnowpackGraphsByBasin-tsp\
- - - - - - - - - - - - - - - - > LOCALID-SNODAS-SWE.tsp
- - - - - - - - - - - - - - - - > LOCALID-SNODAS-SnowCover.tsp
- - - - - - - - - - - - - - - - > LOCALID-SNODAS-SWE-Volume.tsp
- - - - - - - - - - - - - - - - > LOCALID-SNODAS-SWE-Volume-1WeekChange.tsp
- - - - - - - - - - - - - - - - > LOCALID-SNODAS-SWE-Volume-Cumulative.tsp

CDSS\SNODAS\

All SNDOAS scripts, input data and output data are stored within the CDSS\SNODAS\ folder.

The SNODAS Python and batch scripts are contained within the SNODAS\pycharm-project\ folder.
The SNODAS\SNODAS_Tools\ folder contains the data (both input and output), the configuration file, and the batch files designed to create the time series graphs and upload the data to amazon web services.

SNODAS\SNODAS_Tools\

The SNODAS scripts (Python and .bat) are contained within the SNODAS_Tools\scripts\ folder.
The SNODAS input data files are contained within the SNODAS_Tools\staticData\ folder.
The SNODAS output data files are contained within the SNODAS_Tools\processedData\ folder.
The SNODAS cofiguration file is contained within the SNODAS_Tools\config\ folder.
The input TsTool files associated with creating the output time series graphs are contained withing the SNODAS_Tools\TsTool\ folder.
The batch file designed to upload the SNODAS output data to Amazon Web Services is contained within the SNODAS_Tools\aws\ folder.

SNODAS_Tools\scripts\

The C:\CDSS\SNODAS\SNODAS_Tools\scripts\ folder holds all SNODAS Tools' scripts. In total there are four scripts:

1. SNODASDaily_Automated.py 
2. SNODASDaily_Interactive.py 
3. SNODAS_utilities.py 
4. SNODASDaily_Automated_forTaskScheduler.bat

1. SNODASDaily_Automated.py

The SNODASDaily_Automated.py Python script downloads the current date's SNODAS data from the SNODAS FTP site and exports daily snowpack zonal statistics for each basin of the watershed basin shapefile input.

This script only processes the current date's SNODAS data. For information on the data processing steps of SNODASDaily_Automated.py, refer to the Processing Workflow section. For information on the SNODAS FTP site, refer to the SNODAS Data of the user guide. For information on the output snowpack products, refer to the SNODAS Tools Products section of the user guide.

The SNODASDaily_Automated.py script is designed to automatically run using a task scheduler program. Refer to the Task Scheduler section for more information. Once the task is set up, the script downloads the daily SNODAS data on a daily timer and exports the daily zonal statistics to the processedData\ folder.

2. SNODASDaily_Interactive.py

The SNODASDaily_Interactive.py script downloads historical SNODAS data from the SNODAS FTP site and exports daily snowpack zonal statistics.

The SNODASDaily_Interactive.py script is designed to be interactive. Users specify historical dates of interest and the script exports the zonal statistics corresponding to those dates. The exported statistics of the SNODASDaily_Interactive.py are saved in the processedData\ folder alongside the exported statistics of the SNODAS_DailyAutomated.py script.

For information on the data processing steps of SNODASDaily_Interactive.py, refer to the Processing Workflow section. For information on the SNODAS FTP site, refer to the SNODAS Data section of the user guide. For information on the output snowpack products, refer to the SNODAS Tools Products section of the user guide.

The SNODASDaily_Interactive.py script is to be utilized in the following scenarios:

The historical SNODAS repository has not yet been processed.
- The temporal coverage of the SNODAS data is September 28th, 2003 to the current date. The SNODASDaily_Automated.py script automatically creates an ongoing export of SNODAS zonal statistics, once the SNODASDaily_Automated_forTaskScheduler.bat has been set up with a task scheduler program. The SNODAS historical repository, however, must be created with the SNODASDaily_Interactive.py script.
The SNODASDaily_Automated_forTaskScheduler.bat failed to run.
- The SNODASDaily_Automated_forTaskScheduler.bat is designed to automatically run SNODASDaily_Automated.py every day. There are instances, however, when the task could fail to run for a single or range of days. This could occur, for example, if the task properties are set to run only when the user is signed in and the user was signed off for one or a range of days. The missed days must then be manually processed with the SNODASDaily_Interactive.py script.
The SNODAS grid displayed incorrect information.
- The SNODAS products for a specific day could require an overwrite if the National Snow & Ice Data Center uploads incorrect SNODAS data, only to reload the correct dataset later. The reprocessing of SNODAS data for that date would be accomplished with the SNODASDaily_Interactive.py script.

3. SNODAS_utilities.py

The SNODAS_utilities.py script contains all of the functions utilized in the SNODASDaily_Automated.py and the SNODASDaily_Interactive.py scripts. For descriptions of the individual SNODAS_utilities.py functions refer to the Tool Utilities and Functions section.

4. SNODASDaily_Automated_forTaskScheduler.bat

The SNODASDaily_Automated_forTaskScheduler.bat is a batch file to be called by a task scheduler program. It automatically runs the SNODASDaily_Automated.py script every day with the correct environment settings. Refer to the Task Scheduler section for a tutorial on how to initially set up the SNODASDaily_Automated_forTaskScheduler.bat within a task scheduler program.

Components of the SNODASDaily_Automated_forTaskScheduler.bat File

Code	Description	Defaulted to:
SET OSGEO4W_ROOT=	Set the base install folder for OSGEO4W.	C:\OSGeo4W64
SET QGISNAME=	Set the install folder name for QGIS.	qgis
SET QGIS=	Set the full pathname to the install folder for QGIS.	%OSGEO4W_ROOT%\apps\%QGISNAME%
SET QGIS_PREFIX_PATH=	Set the QGIS prefix path to the full pathname to the install folder for QGIS.	%QGIS%
SET GDAL_DATA=	Set the install folder for GDAL.	%OSGEO4W_ROOT%\share\gdal\|
SET PATH=	Set paths to OSGEO4W bin and QGIS bin.	%OSGEO4W_ROOT%\bin;%QGIS%\bin;%PATH%
SET PYTHONHOME=	Set the install folder for Python.	%OSGEO4W_ROOT%\apps\Python27
SET PYTHONPATH=	Set up the default search path for Python module files.	%QGIS%\python;%PYTHONPATH%
SET PYCHARM=	Set full pathname to location of PyCharm program.	"C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.2.3\bin\pycharm.exe"
SET PYTHON_JOB=	Set the full pathname to the location of the Python script.	D:\SNODAS\bin\SNODASDaily_Automated.py
python %PYTHON_JOB%	Launch the Python program.	N/A

SNODAS_Tools\staticData\

There are three files that are contained within the SNODASTools\staticData\ folder. The first file, the watershed basin shapefile input, must be provided by the user of the SNODAS Tools before any scripts are run. The second file, the watershed basin extent shapefile, is produced by the SNODAS Tools after the first script of the SNODAS Tools is run. The user of the tool does not provide this shapefile - it is created automatically by the SNODAS Tools.

Watershed Basin Shapefile Input

The Watershed Basin Shapefile Input (watershedBasinBoundary.shp) is a shapefile of basin features for the study area of interest. The SNODAS Tools calculate daily snowpack statistics for each basin of the watershed basin shapefile input. Originally the SNODAS Tools were developed to perform snowpack analysis for the state of Colorado. Below is an image of the watershed basin shapefile input (displayed in green) used to run the SNODAS Tools for the Colorado project. The black, boxed outline is the Colorado state boundary.

What is the purpose of the watershed basin shapefile input?
- Zonal statistics are statistics calculated by zone where the zones are defined by an input zone dataset and the values are defined by a raster grid. This shapefile is the input zone dataset.
How is the watershed basin shapefile input projected?
- The projection of the watershed basin shapefile input is defaulted to USA Contiguous Albers Equal Area Conic (EPSG code: 102003). If the watershed basin shapefile input has a projection other than USA Contiguous Albers Equal Area Conic, the default EPSG code in the configuration file under section Projections option calcstats_proj_epsg must be altered.
- The clipped SNODAS daily rasters are projected into the projection of the watershed basin shapefile input before the zonal statistics are calculated to ensure that the basin boundaries "match up" with the correct SNODAS raster data.

colorado Basins Shapefile

The Watershed Basin Extent Shapefile

The Watershed Basin Extent Shapefile (studyAreaExtent_prj.shp), referred to as extent shapefile from this point onward, is a single-feature bounding-box shapefile that extends to the extent of the watershed basin shapefile input. The extent shapefile is used within the SNODAS Tools to clip the daily SNODAS data to the study area. Below is an image of the extent shapefile (displayed in green) used for the Colorado project. The Colorado watershed basin shapefile input is overlaid with a transparent fill.

What is the purpose of the extent shapefile?
- The extent shapefile decreases the processing time of the scripts by clipping the national SNODAS grid to a manageable size of the study area.
How is the extent shapefile projected?
- The extent shapefile is defaulted to have the same datum and projection as the national SNODAS grid. The national SNODAS grid does not have a projection however the "SNODAS fields are grids of point estimates of snow cover in latitude/longitude coordinates with the horizontal datum WGS84". Therefore, the extent shapefile is defaulted to have a WGS84 datum and no projection. Although highly unrecommended, the projection of the extent shapefile can be changed from the defaulted WGS84 projection in the configuration file under section Projections option datum_epsg.
How is the extent shapefile created?
- The SNODAS Tools create the extent shapefile by drawing a bounding box extent around the watershed basin shapefile input. The projection of the watershed basin shapefile input is set by the user in the configuration file. The extent shapefile, therefore, is originally projected in the same projection as the watershed basin shapefile input. However, the SNODAS Tools are designed to project the extent shapefile to the projection of the daily SNODAS data grids (defaulted to WGS84).

colorado Basins Extent

Watershed Connectivity File

The watershed connectivity file is an excel file listing all basins of the watershed basin input shapefile. It is read by the create-snodas-swe-graphs TsTool command file to create the time series graphs.

SNODAS_Tools\processedData\

All output products of SNODASDaily_Automated.py and SNODASDaily_Interactive.py are saved within the processedData\ folder. For each processed day of data, 13 output products are created. To see a larger view of the images below, right-click on the image and click Open image in new tab.

The originally downloaded national SNODAS .tar file
- SNODAS_YYYYMMDD.tar
The reformatted national SNODAS SWE data in .tif format
- us_ssmv11034tS__T0001TTNATSYYYYMMDD05HP001.tif
The clipped and projected SNODAS SWE .tif file
- SNODAS_SWE_ClipAndProjYYYYMMDD.tif
The clipped and projected snow cover binary .tif file
- SNODAS_SnowCover_ClipAndProjYYYYMMDD.tif
The snowpack statistics organized by basin ID
- SnowpackStatisticsByBasin_LOCALID.csv (a .csv file)
The snowpack statistics organized by date
- SnowpackStatisticsByDate_YYYYMMDD.csv (a .csv file)
- SnowpackStatisticsByDate_YYYYMMDD.GeoJSON (a .GeoJSON file)
- SnowpackStatisticsByDate_YYYYMMDD.shp (a shapefile, with required extensions)
The snowpack time series graphs organized by basin
- LOCALID_SNODAS_SWE.png
- LOCALID_SNODAS_SnowCover.png
- LOCALID_SNODAS_SWE_Volume.png
- LOCALID_SNODAS_SWE_Volume_1WeekChange.png
- LOCALID-SNODAS-SWE-Volume-Cumulative.png

The 13 output products are saved within subfolders of the processedData folder. Each subfolder is explained in further detail below. The name of each subfolder is described by the default name. However, the following folder names can be edited in the configuration file under section Folders.

processedData\1_DownloadSNODAS\

The SNODAS Tools access the SNODAS FTP site and download the daily SNODAS .tar file. If the SNODASDaily_Automated.py script is utilized, the daily SNODAS .tar file is the current date. If the SNODASDaily_Interactive.py script is utilized, the daily SNODAS .tar file is the date of interest defined by user input.

The file is downloaded to the 1_DownloadSNODAS folder and is named SNODAS_YYYYMMDD.tar where YYYYMMDD represents the date of data. Note that the date does not represent the download date but rather the date when the SNODAS data is collected.

Example: 
Downloaded SNODAS file for January 9th, 2013 -> SNODAS_20130109.tar

Refer to the Processing Workflow section for a general description of the SNODAS Tools' downloading step. Refer to Tool Utilities and Functions for detailed information on the Python functions called to download the SNODAS data.

processedData\2_SetFormat\

The SNODAS Tools manipulate the SNODASYYYYMMDD.tar file to produce a SNODAS Snow Water Equivalent (SWE) national grid in .tif format, shown below.

nationaltif SNODAS Snow Water Equivalent Masked Grid for January 16th, 2017

Refer to the Processing Workflow section for a general description of the SNODAS Tools' set format step. Refer to Tool Utilities and Functions for detailed information on the Python functions called to set the format of the SNODAS data.

The manipulated SNODAS SWE .tif file is saved to the 2_SetFormat folder and is named us_ssmv11034tS__T0001TTNATSYYYYMMDD05HP001.tif where YYYYMMDD represents the date of data. Note that the date does not represent the download date but rather the date when the SNODAS data is collected.

Example: 
SNODAS SWE .tif file for January 9th, 2013 -> us_ssmv11034tS__T0001TTNATS2013010905HP001.tif

The long and cryptic name of this file can be explained with the NSIDC SNODAS user guide.
The naming convention variables are described below:

us: region of the file
'us': United States

ssm: model used to generate the estimates
'ssm': simple snow model

v1: signifies if the file contains snow model driving data or model output
'v1': operational snow model output

1034: product code representing the snow parameter represented by the data
'1034': snow water equivalent

tS__: a vertical integration code that denotes what type of snow pack data are being collected
'tS__': integral through all the layers of the snow pack

T0001: time integration code
'T0001': a one-hour snapshot

TTNA: detail of snow modeling operations
'TTNA': will always be TTNA

TS: time step code
TS is followed by the year, month, day, and hour of the start of the last time step of the integration period for which the data applies. For example, the time integration code, T0024, and time step code, TS2003102305, are for the time interval 2003-10-22 06 to 2003-10-23 05.

YYYY: 4-digit year
'YYYY': dependent on date of data

MM: 2-digit month
'MM': dependent on date of data

DD: 2-digit day of month
'DD': dependent on date of data

05: 2-digit hour of day
'05': 5th hour of the day

H: time interval
'H': hourly

P001: offset code referring to where the data applies during a snow model time step in the snow model's differencing scheme
'P001': field represents a total flux for the entire time step such as precipitation or that a field represents data at the end of a time step

2_SetFormat\OtherParameters folder
There are multiple SNODAS parameters that are included in the downloaded SNODAS .tar file. Those parameters are:

Snow Water Equivalent (SWE)
Snow Depth
Snow Melt Runoff at Base of Snow Pack
Sublimation from the Snow Pack
Sublimation of Blowing Snow
Solid Precipitation
Liquid Precipitation
Snow Pack Average Temperature

The SNODAS Tools are defaulted to delete all SNODAS parameters except for the Snow Water Equivalent data. However, section SaveALLSNODASparameters of the configuration file allows for users to save all the data from the seven other SNODAS parameters. If configured, the SNODAS Tools create a new folder called OtherParameters under the 2_SetFormat folder. All extracted data regarding the SNODAS parameters, other than SWE, is saved within the 2_SetFormat\OtherParameters folder.

The SNODAS data files saved in the 2_SetFormat\OtherParameters folder follow the file naming convention described by the NSIDC SNODAS user guide.

processedData\3_CliptoExtent\

The SNODAS Tools clip the us_ssmv11034tS__T0001TTNATSYYYYMMDD05HP001.tif file to the Watershed Basin Extent Shapefile (studyAreaExtent_prj.shp). The clipped daily SNODAS grid, shown below for the Colorado dataset, is then projected into the desired projection configured in the configuration file under section Projections option calcstats_proj_epsg (this is the same projection as the Watershed Basin Shapefile Input, defaulted to USA Contiguous Albers Equal Area).

withExtent
Above: The SNODAS Snow Water Equivalent Masked Grid for January 16th, 2017 with the Colorado basin extent outline overlaid in black. clippedtif Above: The SNODAS Snow Water Equivalent Grid for January 16th, 2017 clipped to the Colorado basin extent. The skewed image is due to projections built into the SNODAS Tools.

The clipped and projected SNODAS .tif file is saved to the 3_ClipToExtent folder and is named SNODAS_SWE_ClipAndProjYYYYMMDD.tif where YYYYMMDD represents the date of data. Note that the date does not represent the download date but rather the date when the SNODAS data is collected.

Example: 
Clipped and projected SNODAS file for January 9th, 2013 -> SNODAS_SWE_ClipAndProj20130109.tif

Refer to the Processing Workflow section for a general description of the SNODAS Tools' clip and projecting steps. Refer to Tool Utilities and Functions for detailed information on the Python functions called to clip and project the SNODAS data.

processedData\4_CreateSnowCover\

The SNODAS Tools create a daily binary raster grid displaying presence and absence of snow cover. The created snow cover .tif file is saved to the 4_CreateSnowCover folder and is named SNODAS_SnowCover_ClipAndProjYYYYMMDD.tif where YYYYMMDD represents the date of data. Note that the date does not represent the download date but rather the date when the SNODAS data is collected.

Example: 
Clipped and projected snow cover file for January 9th, 2013 -> 
SNODAS_SnowCover_ClipAndProj20130109.tif

SNODAS_SnowCover_ClipAndProjYYYYMMDD.tif, shown below for the Colorado dataset, is created by iterating through the cells of the SNODAS_SWE_ClipAndProjYYYYMMDD.tif file and assigning cell values dependent on the following guidelines:

SNODAS_`SWE`_ClipAndProjYYYYMMDD	SNODAS_`SnowCover`_ClipAndProjYYYYMMDD
If a cell has a value greater than 0 (there is snow on the ground)	then the corresponding cell is assigned a value of '1' (presence of snow displayed in blue)
If a cell has a value equal to 0 (there is no snow on the ground)	then the corresponding cell is assigned a value of '0' (absence of snow displayed in brown)
If a cell has a value equal to -9999 (a null value)	then the corresponding cell is assigned a value of '-9999' (a null value displayed in white)

snowCover Above: The binary Colorado snow cover grid for January 16th, 2017. Blue = presence of snow. Brown = absence of snow.

Refer to the Processing Workflow section for a general description of the SNODAS Tools' Create Snow Cover step. Refer to Tool Utilities and Functions for detailed information on the Python functions called to create the daily snow cover .tif file.

processedData\5_CalculateStatistics\

The SNODAS Tools perform zonal statistics on the SNODAS_SWE_ClipAndProjYYYYMMDD.tif file where zones are defined by the individual features of the Watershed Basin Shapefile Input (watershedBasinBoundary.shp). The statistics are exported into two types of data - snowpack statistics organized by basin and snowpack statistics organized by date.

Types of Snowpack Statistics

There are default snowpack statistics that are non-configurable and required inclusions in the output results. There are also additional snowpack statistics that can be added to the output results if configured in the configuration file under section OptionalZonalStatistics.

Default Snowpack Statistics	Additional Snowpack Statistics
Mean Snow Water Equivalent	Minimum Snow Water Equivalent
Effective Area	Maximum Snow Water Equivalent
Percent Area of Snow Cover	Standard Deviation of Snow Water Equivalent
Total Snow Volume	-
One Week Change in Total Snow Volume	-

Snowpack Statistics organized By Basin

There are multiple basins in the watershed basin shapefile input. For every day of processedSNODAS data, new daily statistics are produced for each individual basin. The output snowpack statistics organized by basin are contained within .csv files. There is one .csv file for each basin of the watershed basin shapefile input. Each row of the byBasin .csv file represents the daily snowpack statistics. As a new date of SNODAS data is processed by the SNODAS Tools, a new row is appended to each byBasin .csv file with the newly calculated statistics.

Refer to the processedData\5_CalculateStatistics\StatisticsbyBasin\ section for more information.

Snowpack Statistics organized By Date

For each date of processed SNODAS data, daily statistics are calculated for every basin of the watershed basin shapefile input. The snowpack statistics organized by date files contain all daily snowpack statistics for a specific date.

Three different formats of byDate data are exported each day.

File Format	Description
CSV file	Tabular data.
GeoJSON file	Spatial data with statistics stored in the attribute table.
Shapefile	Spatial data with statistics stored in the attribute table.

A new file of each byDate format is created for every day of processed SNODAS data completed by the SNODAS Tools.

Refer to the processedData\5_CalculateStatistics\StatisticsbyDate\ section for more information.

Refer to the Processing Workflow section for a general description of the SNODAS Tools' calculate statistics step. Refer to Tool Utilities and Functions for detailed information on the Python functions called to create the .csv files and calculate and export the zonal statistics.

processedData\5_CalculateStatistics\StatisticsbyBasin\

As previously explained, the snowpack zonal statistics are organized and exported into two different groupings, by basin and by date.

There is one SnowpackStatisticsByBasin_XXXX.csv file for each feature of the Watershed Basin Shapefile Input (watershedBasinBoundary.shp). The statistics organized by basin provide change analysis capabilities - the data displays the change in snowpack statistics for a specific basin throughout time.

'XXXX' is the unique ID identifying each basin feature. This ID is located as a field within the attribute table of the Watershed Basin Shapefile Input (watershedBasinBoundary.shp). Before running the SNODAS Tools, it is important to assign the name of the attribute field holding the basin ID information in section BasinBoundaryShapefile option basin_id_fieldname of the configuration file.

Example: 
Zonal statistics by basin .csv file for basin 'ALAC2' on January 9th, 2013 -> 
SnowpackStatisticsByBasin_ALAC2.csv

Each SnowpackStatisticsByBasin_XXXX.csv contains rows of snowpack statistics organized by processed date. Every time a new date of SNODAS data is run, each SnowpackStatisticsByBasin_XXXX.csv file is updated with a new row of statistics.

An example of a SnowpackStatisticsByBasin_XXXX.csv file is shown below. You can see that the dates, January 1st through January 5th, 2017, have been processed by the SNODAS Tools. The red circle shows that all values under the Local_ID column (the watershed basin ID) are equivalent. Right-click on the image and click Open image in new tab to see a larger view.

statsByBasin

processedData\5_CalculateStatistics\StatisticsbyDate\

As previously explained, the snowpack zonal statistics are organized into two different groupings, by basin and by date. The by basin data is exported solely as a csv file. However, the by date data is exported into three different formats - a csv file, a GeoJSON file and a shapefile.

Along with the by date data, a single text file is also stored within the 5_CalculateStatistics\SnowpackStatisticsbyDate folder. The ListofDate.txt section below will explain further.

Statistics By Date CSV Files

There is one SnowpackStatisticsByDate_YYYYMMDD.csv file for each date of processed SNODAS data where YYYYMMDD represents the date of data. Note that the date does not represent the download date but rather the date when the SNODAS data is collected. The statistics organized by date provide landscape comparison capabilities - the data displays the varying daily snowpack statistics for each basin across the entire study area.

Example: 
Zonal statistics by date .csv file for January 9th, 2013 -> 
SnowpackStatisticsByDate_20130109.csv

Each SnowpackStatisticsByDate_YYYYMMDD.csv contains rows of snowpack statistics organized by basin ID. Each basins' statistics are saved within individual rows of the .csv file.

An example of a SnowpackStatisticsByDate_YYYYMMDD.csv file is shown below. The daily statistics (October 15th, 2014) for all basins of the Watershed Basin Shapefile Input (watershedBasinBoundary.shp) are represented. The red circle shows that all values under the Date_YYYYMMDD column are equivalent. Right-click on the image and click Open image in new tab to see a larger view.

statsByBasin

The GeoJSON file and the Shapefile

The by date snowpack statistics are also exported in formats that can be spatially rendered. The GeoJSON and Shapefile exports obtain the geometry from the Watershed Basin Shapefile Input (watershedBasinBoundary.shp) and append the daily snowpack statistics to the attribute table.

A GeoJSON is an open standard format designed for representing simple geographical features, along with their non-spatial attributes, based on JavaScript Object Notation. A Shapefile is a popular geospatial vector data format for GIS software developed and regulated by ESRI.

One GeoJSON file and one shapefile are produced for each date of processed SNODAS data. The exports have the same name as the by date csv file, SnowpackSatisticsByDate_YYYYMMDD, but different extensions (GeoJSON: .geojson, Shapefile: .cpg, .dbf, .prj, .qpj, .shp, and .sgx).

A shapefile is made up of many separate files with different extensions. The SNODAS Tools, by default, will zip up all of the shapefile component files for each day into one zipped file. Under section OutputLayers option shp_zip of the configuration file, the user can choose to keep the shapefile component files unzipped. If the default is true and the daily shapefile is zipped, the SNODAS Tools, by default, will delete the unzipped shapefile component files. Under section OutputLayers option shp_delete_originals of the configuration file, the user can choose to keep both the zipped folder AND the unzipped shapefile component files.

SnowpackStatisticsByDate_LatestDate.csv

The SnowpackStatisticsByDate_LatestDate.csv is a copy of the most recent date's SnowpackStatisticsByDate_YYYYMMDD.csv file. The contents of the file are overwritten when a later date of SNODAS data is processed.

ListOfDates.txt

The ListOfDates.txt file is a text file that contains a list of all processed dates of SNODAS data. All dates in the list correspond to a SnowpackStatisticsByDate_YYYYMMDD.csv file in the StatisticsbyDate folder. The dates in the ListOfDates.txt file are in the YYYYMMDD format. This text file is used in the development of the CDSS SNODAS Tools Map Application.

processedData\6_CreateTimeSeriesProducts\

The SNODAS Tools utilize TsTool to create time series graphs of the following snowpack statistics for each basin of watershed basin shapefile input:

Mean Snow Water Equivalent LOCALID-SNODAS-SWE.png
Percent Snow Cover LOCALID-SNODAS-SnowCover.png
SWE Volume LOCALID-SNODAS-SWE-Volume.png
1-Week Change in SWE Volume LOCALID-SNODAS-SWE-Volume-1WeekChange.png
cumulative SWE Volume LOCALID-SNODAS-SWE-Volume-Gain-Cumulative.png

The time series graphs are stored in the processedData\6_CreateTimeSeriesProducts\SnowpackGraphsByBasin\ folder. The graphs can be configured to update daily OR weekly (default). The configurations specific to the time series graphs are option tsgraph_weekly_update and option tsgraph_weekly_update_date of section OutputLayers. Refer to the configuration file section for more details about these settings.

Refer to the Processing Workflow section for a general description on how the SNODAS Tools generate the snowpack time series products. Refer to Tool Utilities and Functions for detailed information on the Python function called to create the time series product.

processedData\SNODASTools.log

The SNODAS Tools are set to export logging messages to aid in troubleshooting. The logging setting for the SNODAS Tools are configured with the configuration file format.

Levels of Logging Messages
The SNODAS Tools are set to export logging messages to both the console and the SNODASTools.log file. Warning and error messages export to both the console and the SNODASTools.log file. Info messages are defaulted to export solely to the SNODASTools.log file. The logging level of messages exported to the SNODASTools.log file can be changed from the defaulted DEBUG level in the configuration file under section logger_log02 option level.

Formatting of Logging Messages
All logging messages are formatted to the default simpleFormatter. The simpleFormatter outputs the date and local time of the created log record in the following format" YYYY-MM-DD HH:MM:SS,MSS where:

Format	Description	Example for a log message created at 5:26.21,00AM on 2/4/2017.
YYYY	The year when the logging message is created with century as a decimal number.	2017
MM	The month, as a zero-padded decimal number, when the logging message is created.	02
DD	The day, as a zero-padded decimal number, when the logging message is created.	04
HH	The hour (24-hour clock), as a zero-padded decimal number, when the logging message is created.	13
MM	The minute, as a zero-padded decimal number, when the logging message is created.	26
SS	The second, as a zero-padded decimal number, when the logging message is created.	21
MSS	The millisecond, as a three-digit zero-padded decimal number, when the logging message is created.	000

The logging message follows the format: SSSS: EEEE: MMMM where:

Format	Description	Example
SSSS	The name of the .py script or function for which the log message is regarding.	SNODASDaily_Interactive.py:
EEEE	The logging level of the log message. Only present if the logging level is a warning or an error.	WARNING:
MMMM	The logging message.	SNODAS_20110217.tar has been untarred.

The format of the logging messages can be changed from the defaulted %(asctime)s %(message)s in the configuration file under section formatter_simpleFormatter option format.

Timed Rotating File Handler
If the SNODASTools.log logging level is set to default DEBUG, all logging messages will be written to the SNODASTools.log file. For each processed day of SNODAS data, the size of the SNODASTools.log file will increase by approximately 12KB. The SNODAS Tools are designed to run every day. This high frequency of processing would quickly cause the SNODASTools.log file to become incredibly large. To address this issue, the SNODASTools.log file is configured to run on a Timed Rotating File Handler.

The timed rotating file handler creates and updates multiple versions of the SNODASTools.log file based upon a configured temporal schedule. After an allotted amount of time (defaulted to 5 weeks), the oldest version of the SNODASTools.log is deleted and only the most recent log files are available. By default, a new SNODASTools.log file is created every Monday, local time. The SNODASTools.log file from the previous week is assigned a suffix of .YYYY-MM-DD and saved in the CDSS\SNODASTools\ folder. The .YYYY-MM-DD refers to the day that the log file was originally created.

Note:
The timed rotating file handler defaults to adding the dated suffix as an extension. For example SNODASTools.log will become SNODASTools.log.YYYY-MM-DD. This locks up the previous log files to be opened and viewed in Notepad. However, the previous log files can still be opened and viewed using Notepad++.

The default setting saves 5 versions of the SNODASTools.log file. This means that any processing from the past 5 weeks can be accessed. SNODASTools.log files older than 5 weeks are deleted. The previous 5 versions of the SNODASTools.log are saved under the CDSS\SNODASTools\ folder.

As previously mentioned, each processed SNODAS date increases the individual SNODASTools.log file by approximately 12KB. Given the default setting of the timed rotating file handler, each log file will be approximately 84KB (daily size of 12KB multiplied by the 7 days of the week). The total size of file space used for SNODAS Tools' logging will be approximately 420KB (weekly log file size of 84KB multiplied by 5 weeks of backup files).

Note:
The size increase of the SNODASTools.log file will be larger than 12KB for each processed date of SNODAS data if the processed date of SNODAS data is being rerun and the original files are being overwritten. The file size estimates in the previous paragraph assume that the only SNODAS Tools' script that is running is the SNODASDaily_Automated.py script where only one date of SNODAS data is being processed each day.

The settings of the timed rotating file handler can be changed in the configuration file under section handler_fileHandler option args. There are 4 arguments (filename, type of time interval, interval, and backupConut) that can be altered within the option args. These 4 editable features are explained below.

Argument	Description	Defaulted to:
Filename	The full pathname to the location of the log file.	'..\SNODASTools.log'
Type of Time Interval	Time interval type when a new log file is to be created. Options: seconds, days, weekdays, etc.	'W0' Monday
Interval	Time interval. Example (if type of time interval = 'days'): 1 - every day 2 - every other day 5 - every five days, etc.	1
backupCount	The number of previous SNODASTools.log files to be saved.	5

Refer to the Python tutorial documentation on the TimedRotatingFileHandler class
for further information regarding the argument options.

SNODAS_Tools\config\

The SNODAS-Tools-Config.ini is located in the SNODAS_Tools\config\ folder and contains Python input variables and logging settings for the SNODAS Tools.

Design of the Configuration File

The configuration file is divided into sections. The sections are the broad categories of SNODAS configurable variables and are recognizable by the brackets - [] - that surround them.

Example:

['BasinBoundaryShapefile'] is the section header for all configurable variables related to the  
Watershed Basin Shapefile Input.

Under each section, there are corresponding options that relate to the section.

Example:

'pathname' and 'basin_id_fieldname' are options under the ['BasinBoundaryShapefile'] section.

'pathname' refers to the file location of the Watershed Basin Shapefile Input. 
'basin_id_fieldname' refers to the field name of the Watershed Basin Shapefile Input that
    uniquely defines each basin.

The Sections and Options of the Configuration File

To explain the components of the configuration file, each section is represented by a table below. The options of each section are represented by a row in the table.

Software Installation
Configuration File Section: [ProgramInstall]

Configurable Parameter	Description	Defaulted to:
qgis_pathname	The full location to the QGIS installation on the local desktop.	C:/OSGeo4W/apps/qgis
tstool_pathname	The full location of the TsTool program (TsTool.exe) on the local desktop.	C:/CDSS/TSTool-12.00.00beta/bin/TSTool.exe
tstool_create-snodas-graphs_pathname	The full location of the create-snodas-swe-graphs.TSTool command file.	D:/SNODAS/bin/create-snodas-swe-graphs.TSTool

NSIDC FTP Site
Configuration File Section: [SNODAS_FTPSite]

Configurable Parameter	Description	Defaulted to:
host	The FTP site hosting the SNODAS data.	sidads.colorado.edu
username	Sidads.colorado.edu is a public site so the username can remain 'anonymous'.	anonymous
password	Sidads.colorado.edu is a public site so the password can remain 'None'.	None
folder_path	The pathname to the SNODAS data (defaulted to 'masked' data).	/DATASETS/NOAA/G02158/masked/
null_value	The no data value of the SNODAS data. This information can be found in this PDF.	-9999

The Watershed Basin Shapefile Input
Configuration File Section: [BasinBoundaryShapefile]

Configurable Parameter	Description	Defaulted to:
pathname	Location and name of the Watershed Basin Shapefile Input. The shapefile should be stored in the Static Data Folder.	N/A
basin_id_fieldname	The name of the field in the shapefile attribute table that uniquely identifies each basin. The values of this field will be exported to the output statistics csv files.	LOCAL_ID

Projections
Configuration File Section: [Projections]

Configurable Parameter	Description	Defaulted to:
datum_epsg	The EPSG code of the datum used to define the national SNODAS daily grid. WGS84 (EPSG 4326) is recommended.	4326
calcstats_proj_epsg	The EPSG code of the projection used to calculate the zonal statistics, an equal-area projection is recommended. This should be the same projection as that of the Watershed Basin Shapefile Input.	102003
calculate_cellsize_x	The desired cell size (x axis) to resample the daily SNODAS grid before calculating the zonal statistics. Remember to apply units used in the calcstats_proj_epsg projection.	463.1475
calculate_cellsize_y	The desired cell size (y axis) to resample the daily SNODAS grid before calculating the zonal statistics. Remember to apply units used in the calcstats_proj_epsg projection.	463.1475
output_proj_epsg	The EPSG code of the desired projection for the output layers, the daily GeoJSON and the daily shapefile.	26913

Output Folders
Configuration File Section: [Folders]

Configurable Parameter	Description	Defaulted to:
root_pathname	Location and name of root folder. Contains the following 2 folders.	D:/SNODAS/
static_data_folder	Name of folder containing the static data, including the Watershed Basin Shapefile Input.	staticData/
processed_data_folder	Name of folder containing all of the processed data. Contains the following 5 folders.	processedData/
download_snodas_tar_folder	Name of folder containing all daily SNODAS .tar files downloaded from FTP site.	1_DownloadSNODAS
untar_snodas_tif_folder	Name of folder containing national daily SNODAS .tif files.	2_SetFormat
clip_proj_snodas_tif_folder	Name of folder containing clipped and projected daily SNODAS .tif files.	3_ClipToExtent
create_snowvover_tif_folder	Name of folder containing clipped binary snow cover .tif files.	4_CreateSnowCover
calculate_stats_folder	Name of folder containing the output products. Contains the following 2 folders.	5_CalculateStatistics/
output_stats_by_date_folder	Name of folder containing the output zonal snowpack statistics organized by date. Also contains the daily output GeoJSONs & shapefiles.	SnowpackStatisticsByDate
output_stats_by_basin_folder	Name of folder containing the output zonal snowpack statistics organized by basin.	SnowpackStatisticsByBasin

Output Layers
Configuration File Section: [OutputLayers]

Configurable Parameter	Description	Defaulted to:
shp_zip	Boolean logic to determine if the output shapefile files should be zipped. True: Shapefile files are zipped. False: Shapefile files are left independent.	True
shp_delete_orginals	Boolean logic to determine if unzipped shapefile files should be deleted. Only applied if shp_zip = True. True: Independent shapefile files are deleted. False: Independent shapefile files are saved along with the zipped file.	True
geojson_precision	The number of decimal places included in the GeoJSON output geometry. The more decimal places, the more accurate the geometry and the larger the file size.	5
tsgraph_weekly_update	Boolean logic to determine whether or not to update the snowpack time series graphs daily or weekly. True: Time series graphs are updated weekly, based on tsgraph_weekly_update_date setting. False: Time series graphs are updated daily.	False
tsgraph_weekly_update_date	The day of the week that the snowpack time series graphs are set to update (Monday: 0, Tuesday: 1 ...). Only applied if tsgraph_weekly_update = True. Note that the SNODAS Tools must be run on this set day in order for the graphs to update. The graphs will not update automatically if one of the SNODAS Tools' scripts is not run.	0
upload_results_to_amazon_s3	Boolean logic to determine whether or not to upload the SNODAS_Tools results to amazon web services given the specifics of the batch file input in function 'push_to_AWS' in 'SNODAS_utilities.py'.	False

Daily SNODAS Parameters
Configuration File Section: [SNODASparameters]

Configurable Parameter	Description	Defaulted to:
save_all_parameters	The downloaded daily SNODAS .tar file contains 8 snowpack parameters. The SNODAS Tools only compute statistics from the SWE parameter. Boolean logic to determine whether or not to delete the other 7 national grids of SNODAS parameters. True: The daily 7 national grids of SNODAS parameters (other than SWE) are saved in a folder called download_snodas_tar_folder/OtherParameters. False: The daily 7 national grids of SNODAS parameters are deleted.	False

Optional Statistics
Configuration File Section: [OptionalZonalStatistics]

Configurable Parameter	Description	Defaulted to:
calculate_swe_minimum	Boolean logic to enable calculation of the daily minimum SWE zonal statistic (mm and in). True: Enable. False: Disable.	False
calculate_swe_maximum	Boolean logic to enable calculation of the daily maximum SWE zonal statistic (mm and in). True: Enable. False: Disable.	False
calculate_swe_standard_deviation	Boolean logic to enable calculation of the SWE standard deviation zonal statistic (mm and in). True: Enable. False: Disable.	False

The Logging Files
The configuration of the logging files is slightly more complicated than the other sections of the configuration file. The configuration of the logging files is set up with multiple sections, following the design provided by the Python Software Foundation - Logging HOWTO.

The following table differs than the previous tables in that there are multiple sections represented in the table (instead of just one). In the first column, the section AND the option are listed (instead of just the option). All of the sections and options in the following table are in reference to configuring the logging files.

Section Option	Description	Defaulted to:
loggers keys	The available SNODASTools logs. This should not be changed unless a new log configuration is to be created.	root, log02
handlers keys	The available SNODASTools handlers. This should not be changed unless a new handler is to be created.	fileHandler, consoleHandler
formatters keys	The available SNODASTools formatters. This should not be changed unless a new formatter is to be created.	simpleFormatter
logger_root level	The log level of the 'root' logger.	WARNING
logger_root handlers	The handler used for the 'root' logger.	consoleHandler
logger_log02 level	The log level of the 'log02' logger.	DEBUG
logger_log02 handlers	The handler used for the 'log02' logger.	fileHandler
logger_log02 qualname	The name used to call the log in the SNODASTools' applications.	log02
logger_log02 propogate	Propogation setting. 1 to indicate that messages must propagate to handlers higher up the logger hierarchy from this logger, or 0 to indicate that messages are not propagated to handlers up the hierarchy.	0
handler_consoleHandler class	The class type of the consoleHandler.	StreamHandler
handler_consoleHandler level	The log level of the consoleHandler.	NOTSET
handler_consoleHandler formatter	The formatter of the consoleHandler.	simpleFormatter
handler_consoleHandler args	The location of output log messages for the consoleHandler.	(sys.stdout,)
handler_fileHandler class	The class type of the fileHandler.	handlers.TimedRotatingFileHandler
handler_fileHandler level	The log level of the fileHandler.	NOTSET
handler_fileHandler formatter	The formatter of the fileHandler.	simpleFormatter
handler_fileHandler args	The options for the TimedRotatingFileHandler (filename, when to rotate, rotation interval, backupCount)	('../processedData/SNODASTools.log', 'W0', 1, 5)
formatter_simpleFormatter foramt	The format of log messages.	%(asctime)s %(message)s

SNODAS_Tools\TsTool\6_CreateTimeSeriesProducts\

All files inside of this folder are required inputs and outputs for the TsTool program to create the time series graphs.

create-snodas-swe-graphs.TSTool

The create-snodas-swe-graphs.TSTool command file is read by TSTool to produce the snowpack time series graphs. The create-snodas-swe-graphs.TSTool command file is prompted in the SNODASDaily_Interactive.py script and the SNODASDaily_Automated.py script, under function create_SNODAS_SWE_graphs(), after all dates of data have been processed.

Refer to the Generate Time Series Products section of the Software Design - Overview page for more information regarding the creation of the snowpack time series graphs

Other .tsp and .txt Files

The other .tsp and .txt files included in this folder are as follows:
create-snodas-swe-graphs-tstool-control.txt
snodas-localid-snowcover-graph-template.tsp
snodas-localid-swe-graph-template.tsp
snodas-localid-swe-volume-1weekchange-graph-template.tsp
snodas-localid-swe-volume-gain-cumulative-graph-template.tsp
snodas-localid-swe-volume-graph-template.tsp

These files are designed to aid TsTool in the drawing settings of each graph. To learn more about these files, read the TsTool software documentation.

~$create-snodas-swe-graphs.TSTool.log

The log file, ~$create-snodas-swe-graphs.TSTool.log, contains all logging messages developed from TsTool while creating the above time-series graphs.

SNODAS_Tools\aws\

The copyAllToOWFAmazons3.bat file is contained within the SNODAS_Tools\aws\ folder. The copyAllToOWFAmazons3.bat file is designed to push the output SNODAS Tools' products to Open Water Foundation's Amazon Web Service to update the CDSS SNODAS Tools Map Viewer.

Table of Contents

Overview