noaa/etl/noaa_hdta_etl.ipynb

{"nbformat_minor": 0, "cells": [{"source": "# Historical Daily Temperature Analysis \n\n## Objective\n\n>If there is scientific evidence of extreme fluctuations in our weather patterns due to human impact to the environment then we should be able to identify factual examples of increase in the frequency in extreme temperatures.\n\nThere has been a great deal of discussion around climate change and global warming. Since NOAA has made their data public, let us explore the data ourselves and see what insights we can discover. \n\n1. How many weather stations in US?\n2. For US weather stations, what is the average years of record keeping?\n3. For each US weather station, on each day of the year, identify the frequency at which daily High and Low temperature records are broken.\n4. Does the historical frequency of daily temperature records (High or Low) in the US provide statistical evidence of dramatic climate change?\n5. What is the average life-span of a daily temperature record (High or Low) in the US?\n\nThis analytical notebook is a component of a package of notebooks. The package is intended to serve as an exercise in the applicability of IPython/Juypter Notebooks to public weather data for DIY Analytics.", "cell_type": "markdown", "metadata": {}}, {"source": "<div class=\"alert alert-warning\">\n    <strong>WARNING:</strong> This notebook requires a minimum of <font color=\"red\"><strong>12GB</strong></font> of free disk space to <strong>Run All</strong> cells for all processing phases. The time required to run all phases is **1 hr 50 mins**. The notebook supports two approaches for data generation (CSV and HDF). It is recommended that you choose to run either <emphasis>Approach 1</emphasis> or <emphasis>Approach 2</emphasis> and <font color=\"red\"><strong>avoid</strong></font> running all cells.\n</div>", "cell_type": "markdown", "metadata": {}}, {"source": "## Assumptions\n\n1. We will observe and report everything in Fahrenheit.\n2. The data we extract from NOAA may be something to republish as it may require extensive data munging.\n\n### Data\n\nThe Global Historical Climatology Network (GHCN) - [Daily dataset](http://gis.ncdc.noaa.gov/all-records/catalog/search/resource/details.page?id=gov.noaa.ncdc:C00861) integrates daily climate observations from approximately 30 different data sources. Over 25,000 worldwide weather stations are regularly updated with observations from within roughly the last month. The dataset is also routinely reconstructed (usually every week) from its roughly 30 data sources to ensure that GHCN-Daily is generally in sync with its growing list of constituent sources. During this process, quality assurance checks are applied to the full dataset. Where possible, GHCN-Daily station data are also updated daily from a variety of data streams. Station values for each daily update also undergo a suite of quality checks.", "cell_type": "markdown", "metadata": {}}, {"source": "<div class=\"alert alert-info\">\nThis notebook was developed using a <strong>March 16, 2015</strong> snapshot of USA-Only daily temperature readings from the Global Historical Climatology Network. The emphasis herein is on the generation of human readable data. Data exploration and analytical exercises are deferred to separate but related notebooks.\n</div>\n\n#### Ideal datasets \n\nNOAA's [National Climatic Data Center](http://www.ncdc.noaa.gov/about-ncdc)(NCDC) is responsible for preserving, monitoring, assessing, and providing public access to the USA's climate and historical weather data and information. Since weather is something that can be observed at varying intervals, the process used by NCDC is the best that we have yet it is far from ideal. Optimally, weather metrics should be observed, streamed, stored and analyzed in real-time. Such an approach could offer the data as a service associated with a data lake.\n\n> [Data lakes](http://www.pwc.com/us/en/technology-forecast/2014/cloud-computing/features/data-lakes.jhtml) that can scale at the pace of the cloud remove integration barriers and clear a path for more timely and informed business decisions.\n\nAccess to cloud-based data services that front-end a data lake would help to reduce the possibility of human error and divorce us from downstream processing that alters the data from it's native state.   \n\n#### Available datasets\n\nNOAA NCDC provides public FTP access to the **GHCN-Daily dataset**, which contains a file for each US weather station. Each file contains historical daily data for the given station since that station began to observe and record. Here are some details about the available data:\n\n* The data is delivered in a **text file based, fixed record machine-readable format**. \n* Version 3 of the GHCN-Daily dataset was released in September 2012. \n* Changes to the processing system associated with the Version 3 release also allowed for updates to occur 7 days a week rather than only on most weekdays. \n* Version 3 contains station-based measurements from well over 90,000 land-based stations worldwide, about two thirds of which are for precipitation measurement only. \n* Other meteorological elements include, but are not limited to, daily maximum and minimum temperature, temperature at the time of observation, snowfall and snow depth. \n\n##### Citation Information\n\n* [GHCN-Daily journal article](doi:10.1175/JTECH-D-11-00103.1): Menne, M.J., I. Durre, R.S. Vose, B.E. Gleason, and T.G. Houston, 2012:  An overview of the Global Historical Climatology Network-Daily Database.  Journal of Atmospheric and Oceanic Technology, 29, 897-910.\n* Menne, M.J., I. Durre, B. Korzeniewski, S. McNeal, K. Thomas, X. Yin, S. Anthony, R. Ray, R.S. Vose, B.E.Gleason, and T.G. Houston, 2012: [Global Historical Climatology Network - Daily (GHCN-Daily)](http://doi.org/10.7289/V5D21VHZ), [Version 3.20-upd-2015031605], NOAA National Climatic Data Center [March 16, 2015].", "cell_type": "markdown", "metadata": {}}, {"source": "## Getting Started\n\n### Project Setup\n\nThis project is comprised of several notebooks that address various stages of the analytical process. A common theme for the project is the enablement of reproducible research. This notebook will focus on the creation of new datasets that will be used for downstream analytics.\n\n#### Analytical Workbench\n\nThis notebook is compatible with [Project Jupyter](http://jupyter.org/).  ", "cell_type": "markdown", "metadata": {}}, {"source": "<div class=\"alert\" style=\"border: 1px solid #aaa; background: radial-gradient(ellipse at center, #ffffff 50%, #eee 100%);\">\n<div class=\"row\">\n    <div class=\"col-sm-1\"><img src=\"https://knowledgeanyhow.org/static/images/favicon_32x32.png\" style=\"margin-top: -6px\"/></div>\n    <div class=\"col-sm-11\">Execution of this notebook depends on one or more features found in [IBM Knowledge Anyhow Workbench (KAWB)](https://marketplace.ibmcloud.com/apps/2149#!overview). To request a free trial account, visit us at https://knowledgeanyhow.org. You can, however, load it and view it on nbviewer or in IPython / Jupyter.</div>\n</div>\n</div>", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "# Import special helper functions for the IBM Knowledge Anyhow Workbench.\nimport kawb", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "##### Workarea Folder Structure\n\nThe project will be comprised of several file artifacts. This project's file subfolder structure is: \n\n```\n noaa_hdta_*.ipynb                         - Notebooks\n noaa-hdta/data/ghcnd_hcn.tar.gz           - Obtained from NCDC FTP site.\n noaa-hdta/data/usa-daily/15mar2015/*.dly  - Daily weather station files\n noaa-hdta/data/hdf5/15mar2015/*.h5        - Hierarchical Data Format files \n noaa-hdta/data/derived/15mar2015/*.h5     - Comma delimited files \n```\n**Notes**:\n\n1. The initial project research used the 15-March-2015 snapshot of the GHCN Dataset. The folder structure allows for other snapshots to be used.\n2. This notebook can be used to generate both Hierarchical Data Format and comma delimited files. It is recommended to pick one or the other as disk space requirements can be as large as:\n\n    * HDF5: <font color=\"red\">8GB</font>\n    * CSV: <font color=\"red\">4GB</font>", "cell_type": "markdown", "metadata": {}}, {"source": "<div class=\"alert\" style=\"border: 1px solid #aaa; background: radial-gradient(ellipse at center, #ffffff 50%, #eee 100%);\">\n<div class=\"row\">\n    <div class=\"col-sm-1\"><img src=\"https://knowledgeanyhow.org/static/images/favicon_32x32.png\" style=\"margin-top: -6px\"/></div>\n    <div class=\"col-sm-11\">In IBM Knowledge Anyhow Workbench, the  user workarea is located under the \"resources\" folder.</div>\n    </div>\n</div>", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "%%bash\n# Create folder structure\nmkdir -p noaa-hdta/data noaa-hdta/data/usa-daily/15mar2015\nmkdir -p noaa-hdta/data/hdf5 \nmkdir -p noaa-hdta/data/derived/15mar2015/missing\nmkdir -p noaa-hdta/data/derived/15mar2015/summaries\nmkdir -p noaa-hdta/data/derived/15mar2015/raw\nmkdir -p noaa-hdta/data/derived/15mar2015/station_details\n# List all project realted files and folders\n!ls -laR noaa-*", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "##### Tags\nKAWB allows all files to be tagged for project organization and search. This project will use the following tags.\n\n* ```noaa_data```: Used to tag data files (.dly, .h5, .csv)\n* ```noaa_hdta```: Used to tag project notebooks (.ipynb)\n\nThe following inline code can be used throughout the project to tag project files.", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "import glob\n\ndata_tagdetail = ['noaa_data',\n                  ['/resources/noaa-hdta/data/',\n                   '/resources/noaa-hdta/data/usa-daily/15mar2015/',\n                   '/resources/noaa-hdta/data/hdf5/15mar2015/',\n                   '/resources/noaa-hdta/data/derived/15mar2015/'\n                  ]]\nnb_tagdetail = ['noaa_hdta',['/resources/noaa_hdta_*.ipynb']]\n\ndef tag_files(tagdetail):\n    pathnames = tagdetail[1]\n    for path in pathnames:\n        for fname in glob.glob(path):\n            kawb.set_tag(fname, tagdetail[0])\n\n# Tag Project files\ntag_files(data_tagdetail)\ntag_files(nb_tagdetail)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "#### Technical Awareness\n\nIf the intended use of this notebook is to generate *Hierarchical Data Formatted* files, then the user of this notebook should be familiar with the following software concepts:\n\n* [HDF5 Files](http://docs.h5py.org/en/latest/)\n", "cell_type": "markdown", "metadata": {}}, {"source": "### Obtain the data\n\n1. Copy URL below to a new browser tab and then download the ```noaa-hdta/data/ghcnd_hcn.tar.gz``` file.\n```\n    ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily\n``` \n2. Upload the same file to your workbench. \n3. Move and unpack the tarball into the designated project folders\n```\n    %%bash\n    mv /resources/ghcnd_hcn.tar.gz /resources/noaa-hdta/data/ghcnd_hcn.tar.gz\n    cd /resources/noaa-hdta/data\n    tar -xzf ghcnd_hcn.tar.gz\n    mv ./ghcnd_hcn/*.dly ./usa-daily/15mar2015/\n    rm -R ghcnd_hcn\n    ls -la\n```", "cell_type": "markdown", "metadata": {}}, {"source": "<div class=\"alert\" style=\"border: 1px solid #aaa; background: radial-gradient(ellipse at center, #ffffff 50%, #eee 100%);\">\n<div class=\"row\">\n    <div class=\"col-sm-1\"><img src=\"https://knowledgeanyhow.org/static/images/favicon_32x32.png\" style=\"margin-top: -6px\"/></div>\n    <div class=\"col-sm-11\">In IBM Knowledge Anyhow Workbench, you can drag/drop the file on your workbench browser tab to simplify the uploading process.</div>\n</div></div>", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "%%bash\n# Provide the inline code for obtaining the data. See step 3 above.", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "### Dependencies\n\nIf the intended use of this notebook is to generate *Hierarchical Data Formatted* files, then this notebook requires the installation of the following software dependencies:\n\n```\n    $ pip install h5py\n\n```", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "# Provide the inline code necessary for loading any required libraries", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"source": "## Tidy Data Check\n\nUpon review of the NOAA NCDC GHCN Dataset, the data **fails** to meet the requirements of [Tidy Data](http://www.jstatsoft.org/v59/i10/paper). A dataset is tidy if rows, columns and tables are matched up with observations, variables and types. In tidy data:\n1. Each variable forms a column.\n2. Each observation forms a row.\n3. Each type of observational unit forms a table.\nMessy data is any other arrangement of the data.\n\nIn the case of the GHCN Dataset, we are presented with datasets that contain observations for each day in a month for a given year. Each \".dly\" file contains data for one station.  The name of the file corresponds to a station's identification code. Each record in a file contains one month of daily data. Each row contains observations for more than 20 different element types. The variables on each line include the following:\n\n```\n------------------------------\nVariable   Columns   Type\n------------------------------\nID            1-11   Character\nYEAR         12-15   Integer\nMONTH        16-17   Integer\nELEMENT      18-21   Character\nVALUE1       22-26   Integer\nMFLAG1       27-27   Character\nQFLAG1       28-28   Character\nSFLAG1       29-29   Character\nVALUE2       30-34   Integer\nMFLAG2       35-35   Character\nQFLAG2       36-36   Character\nSFLAG2       37-37   Character\n  .           .          .\n  .           .          .\n  .           .          .\nVALUE31    262-266   Integer\nMFLAG31    267-267   Character\nQFLAG31    268-268   Character\nSFLAG31    269-269   Character\n------------------------------\n```\nA more detailed interpretation of the format of the data is outlined in ```readme.txt``` here ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily.\n\n### Sample Preview\n\nThe variables of interest to this project have the following definitions:\n\n* **ID** is the station identification code.  See \"ghcnd-stations.txt\" for a complete list of stations and their metadata.\n* **YEAR** is the year of the record.\n* **MONTH** is the month of the record.\n* **ELEMENT** is the element type. There are more than 20 possible elements. The five core elements are:\n    * *PRCP* = Precipitation (tenths of mm)\n    * *SNOW* = Snowfall (mm)\n\t* *SNWD* = Snow depth (mm)\n    * *TMAX* = Maximum temperature (tenths of degrees C)\n    * *TMIN* = Minimum temperature (tenths of degrees C)\n* **VALUE(n)** is element value on the n<sup>th</sup> day of the month (missing = -9999).\n* **NAPAD(n)** contains non-applicable fields of interest.\n\nwhere **n** denotes the day of the month (1-31) If the month has less than 31 days, then the remaining variables are set to missing (e.g., for April, VALUE31 = -9999, NAPAD31 = {MFLAG31 = blank, QFLAG31 = blank, SFLAG31 = blank}). \n\nHere is a snippet depicting how the data is represented:\n```\nUSC00011084201409TMAX  350  H  350  H  344  H  339  H  306  H  333  H  328  H  339  H  339  H  322  H  339  H  339  H  339  H  333  H  339  H  333  H  339  H  328  H  322  H  328  H  283  H  317  H  317  H  272  H  283  H  272  H  272  H  272  H-9999   -9999   -9999   \nUSC00011084201409TMIN  217  H  217  H  228  H  222  H  217  H  217  H  222  H  233  H  233  H  228  H  222  H  222  H  217  H  211  H  217  H  217  H  211  H  206  H  200  H  189  H  172  H  178  H  122  H  139  H  144  H  139  H  161  H  206  H-9999   -9999   -9999   \nUSC00011084201409TOBS  217  H  256  H  233  H  222  H  217  H  233  H  239  H  239  H  233  H  278  H  294  H  256  H  250  H  228  H  222  H  222  H  211  H  206  H  211  H  194  H  217  H  194  H  139  H  161  H  144  H  194  H  217  H  228  H-9999   -9999   -9999   \nUSC00011084201409PRCP    0  H    0  H    0  H   13  H   25  H    8  H    0  H    0  H    0  H    0  H    0  H    0  H    0  H    0  H    0  H   25  H  178  H    0  H    0  H   56  H    0  H    0  H    0  H    0  H    0  H    0  H    0  H    0  H-9999   -9999   -9999\n```\n\n\n### Remove noise\n\nThe NOAA NCDC GHCN Dataset includes ```-9999``` as an indicator of missing observation data. We will take this into consideration as we transform the data into a usable format for the project.", "cell_type": "markdown", "metadata": {}}, {"source": "## Data Processing Decision\n\nBefore we can attempt to answer the questions driving this project, we must first map the data into a more reasonable format. As a result, the focus of this notebook will be the creation of new datasets that can be consumed by other notebooks in the project. \n\n> [Data munging](http://en.wikipedia.org/wiki/Data_wrangling) or data wrangling is loosely the process of manually converting or mapping data from one \"raw\" form into another format that allows for more convenient consumption of the data.\"\n\nThis data munging endeavor is an undertaking unto itself since our goal here is to extract from the machine-readable content new information and store it in a human-readable format. Essentially, we seek to explode (decompress) the data for general consumption and [normalize](http://en.wikipedia.org/wiki/Database_normalization#Normal_forms) it into a relational model that may be informative to users.\n\n### Desired Data Schema\n\n#### Historical Daily Summary\nThe goal here is to capture summary information about a given day throughout history at a specific weather station in the US. This dataset contains 365 rows where each row depicts the aggregated low and high temperature data for a specific day throughout the history of the weather station.  \n\nColumn | Description \n--- | --- \nStation ID | Name of the US Weather Station\nMonth |Month of the observations\nDay | Day of the observations\nFirstYearOfRecord | First year that this weather station started collecting data for this day in in history.\nTMin | Current record low temperature (F) for this day in history.\nTMinRecordYear | Year in which current record low temperature (F) occurred.  \nTMax | Current record high temperature (F) for this day in history. \nTMaxRecordYear | Year in which current record high temperature occurred. \nCurTMinMaxDelta | Difference in degrees F between  record high and low records for this day in history.\nCurTMinRecordDur | LifeSpan of curent low record temperature.\nCurTMaxRecordDur | LifeSpan of current high record temperature.\nMaxDurTMinRecord | Maximum years a low temperature record was held for this day in history.\nMinDurTMinRecord | Minimum years a low temperature record was held for this day in history.\nMaxDurTMaxRecord | Maximum years a high temperature record was held for this day in history.\nMinDurTMaxRecord | Minimum years a high temperature record was held for this day in history.\nTMinRecordCount | Total number of TMin records set on this day (does not include first since that may not be a record).\nTMaxRecordCount | Total number of TMax records set on this day (does not include first since that may not be a record).\n\n#### Historical Daily Detail\nThe goal here is to capture details for each year that a record has changed for a specific weather station in the US. During the processing of the Historical Daily Summary dataset, we will log each occurrence of a new temperature record. Information in this file can be used to drill-down into and/or validate the summary file. \n\nColumn | Description \n--- | --- \nStation ID | Name of the US Weather Station\nYear | Year of the observation\nMonth |Month of the observation\nDay | Day of the observation\nType | Type of temperature record (Low = *TMin*, High = *TMax*)\nOldTemp | Temperature (F) prior to change.\nNewTemp | New temperature (F) record for this day.\nTDelta | Delta between old and new temperatures.\n\n#### Historical Daily Missing Record Detail\nThe goal here is to capture details pertaining to missing data. Each record in this dataset represents a day in history that a specific weather station in the USA failed to observe a temperature reading.\n\nColumn | Description \n--- | --- \nStation ID | Name of the US Weather Station\nYear | Year of the missing observation\nMonth |Month of the missing observation\nDay | Day of the missing observation\nType | Type of temperature missing (Low = *TMin*, High = *TMax*)\n\n#### Historical Raw Detail\n\nThe goal here is to capture raw daily details. Each record in this dataset represents a specific temperature observation for a day in history for a specific that a specific weather station.\n\nColumn | Description \n--- | --- \nStation ID | Name of the US Weather Station\nYear | Year of the observation\nMonth |Month of the observation\nDay | Day of the observation\nType | Type of temperature reading (Low = *TMin*, High = *TMax*)\nFahrenheitTemp | Fahrenheit Temperature\n\n### Derived Datasets\n\nWhile this notebook is focused on daily temperature data, we could imagine  future work associated with other observation types like snow accumulations and precipitation. Therefore, the format we choose to capture and store our desired data should also allow us to organize and append future datasets.\n\nThe [HDF5 Python Library](http://docs.h5py.org/en/latest/) provides support for the standard Hierarchical Data Format. This library will allow us to:\n\n1. Create, save and publish derived datasets for reuse.\n2. Organize our datasets into groups (folders). \n3. Create datasets that can be easily converted to/from dataframes. \n\nHowever, HDF5 files can be very large which could be a problem if we want to share the data. Alternatively, we could store the information in new collections of CSV files where each ```.csv``` contained weather station specific content for one of our target schemas.\n", "cell_type": "markdown", "metadata": {}}, {"source": "# Extract, Transform and Load (ETL)\n\nThe focus of this notebook is to perform the to compute intensive activities that will **explode** the **text file based, fixed record machine-readable format** into the derived data schemas we have specified.", "cell_type": "markdown", "metadata": {}}, {"source": "### Gather Phase 1\n\nThe purpose of this phase will be to do the following:\n\n* For each daily data file (*.dly) provided by NOAA:\n     * For each record where ELEMENT == TMAX  or TMIN\n         * Identify missing daily temperature readings, write them to the missing dataset.\n         * Convert each daily temperature reading from celcius to fahrenheit, write each daily reading to the raw dataset.\n        \n#### Approach 1: Comma delimited files\n\nUse the code below to layout your target project environment (if it should differ from what is described herein) and then run the process for *Gather Phase 1*. This will take about 20 minutes to process the **1218** or more weather station files. You should expect to see output like this:\n\n```\n>> Processing file 0: USC00207812\nExtracting content from file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00207812.dly.\n>> Processing Complete: 7024 lines of file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00207812.dly.\n>>   Elapsed file execution time 0:00:00\n>> Processing file 1: USC00164700\nExtracting content from file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00164700.dly.\n>> Processing Complete: 9715 lines of file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00164700.dly.\n.\n.\n.\n>> Processing file 1217: USC00200230\nExtracting content from file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00200230.dly.\n>> Processing Complete: 10112 lines of file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00200230.dly.\n>>   Elapsed file execution time 0:00:00\n>>      Processing Complete.\n>>   Elapsed corpus execution time 0:19:37\n```", "cell_type": "markdown", "metadata": {}}, {"source": "<div class=\"alert\" style=\"border: 1px solid #aaa; background: radial-gradient(ellipse at center, #ffffff 50%, #eee 100%);\">\n<div class=\"row\">\n    <div class=\"col-sm-1\"><img src=\"https://knowledgeanyhow.org/static/images/favicon_32x32.png\" style=\"margin-top: -6px\"/></div>\n    <div class=\"col-sm-11\">[IBM Knowledge Anyhow Workbench (KAWB)](https://marketplace.ibmcloud.com/apps/2149#!overview) provides support for importable notebooks that address the challenges of code reuse. It also supports the concept of code injection from reusable cookbooks. For more details, please refer to the Share and Reuse tutorials that come with your instance of KAWB.</div>\n</div>\n</div>", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "import mywb.noaa_hdta_etl_csv_tools as csvtools", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "csvtools.help()", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "%inject csvtools.noaa_run_phase1_approach1", "cell_type": "markdown", "metadata": {"collapsed": false}}, {"execution_count": null, "cell_type": "code", "source": "# Approach 1 Content Layout for Gather Phases 1 and 2\nhtda_approach1_content_layout = {\n    'Content_Version': '15mar2015',\n    'Daily_Input_Files': '/resources/noaa-hdta/data/usa-daily/15mar2015/*.dly',\n    'Raw_Details': '/resources/noaa-hdta/data/derived/15mar2015/raw',\n    'Missing_Details': '/resources/noaa-hdta/data/derived/15mar2015/missing',\n    'Station_Summary': '/resources/noaa-hdta/data/derived/15mar2015/summaries', \n    'Station_Details': '/resources/noaa-hdta/data/derived/15mar2015/station_details',\n    }", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "# Run Gather Phase 1 for all 1218 files using approach 1 (CSV)\ncsvtools.noaa_run_phase1_approach1(htda_approach1_content_layout)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "##### Observe Output \n\nYou can compute the disk capacity of your *Gather Phase 1* results.\n```\n96M\t/resources/noaa-hdta/data/derived/15mar2015/missing\n24M\t/resources/noaa-hdta/data/derived/15mar2015/summaries\n3.2G\t/resources/noaa-hdta/data/derived/15mar2015/raw\n```", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "%%bash\n# Compute size of output folders\ndu -h --max-depth=1 /resources/noaa-hdta/data/derived/15mar2015/", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "#### Approach 2: HDF5 files\n\nUse the code below to layout your target project environment (if it should differ from what is described herein) and then run the process for *Gather Phase 1*. It will take less than 20 minutes to process the **1218** or more weather station files. **Note**: You will need to have room for about <font color=\"red\">**6.5GB**</font>. You should expect to see output like this:\n\n```\n>> Processing file 0: USC00207812\nExtracting content from file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00207812.dly.\n>> Processing Complete: 7024 lines of file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00207812.dly.\n>>   Elapsed file execution time 0:00:00\n>> Processing file 1: USC00164700\nExtracting content from file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00164700.dly.\n>> Processing Complete: 9715 lines of file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00164700.dly.\n.\n.\n.\n>> Processing file 1217: USC00200230\nExtracting content from file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00200230.dly.\n>> Processing Complete: 10112 lines of file /resources/noaa-hdta/data/usa-daily/15mar2015/USC00200230.dly.\n>>   Elapsed file execution time 0:00:00\n>>      Processing Complete.\n>>   Elapsed corpus execution time 0:17:43\n```", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "import mywb.noaa_hdta_etl_hdf_tools as hdftools", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "hdftools.help()", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "%inject hdftools.noaa_run_phase1_approach2", "cell_type": "markdown", "metadata": {"collapsed": false}}, {"execution_count": null, "cell_type": "code", "source": "# Approach 2 Content Layout for Gather Phases 1 and 2\nhtda_approach2_content_layout = {\n    'Content_Version': '15mar2015',\n    'Daily_Input_Files': '/resources/noaa-hdta/data/usa-daily/15mar2015/*.dly',\n    'Raw_Details': '/resources/noaa-hdta/data/hdf5/15mar2015/noaa_ncdc_raw_details.h5',\n    'Missing_Details': '/resources/noaa-hdta/data/hdf5/15mar2015/noaa_ncdc_missing_details.h5',\n    'Station_Summary': '/resources/noaa-hdta/data/hdf5/15mar2015/noaa_ncdc_station_summaries.h5', \n    'Station_Details': '/resources/noaa-hdta/data/hdf5/15mar2015/noaa_ncdc_station_details.h5',\n    }", "outputs": [], "metadata": {"collapsed": true, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "# Run Gather Phase 1 for all 1218 files using approach 2 (HDF5)\nhdftools.noaa_run_phase1_approach2(htda_approach2_content_layout)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "%%bash\n# Compute size of output folders\ndu -h --max-depth=1 /resources/noaa-hdta/data/hdf5/15mar2015/", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "### Gather Phase 2\n\nThe purpose of this phase will be to do the following:\n\n* For each daily data file (*.dly) provided by NOAA:\n     * For each record where ELEMENT == TMAX  or TMIN\n         * Identify missing daily temperature readings, write them to the missing dataset.\n         * Convert each daily temperature reading from celcius to fahrenheit, write each daily reading to the raw dataset.\n         \n* For each raw dataset per weather station that was generated in *Gather Phase 1*:\n     * For each tuple in a dataset, identify when a new temperature record has occurred (TMIN or TMAX):\n         * Create a Station Detail tuple and add to list of tuples\n         * Update the Summary Detail list of tuples for this day in history for this station\n     * Store the lists to disk\n        \n#### Approach 1: Comma delimited files\n\nUse the code below to layout your target project environment (if it should differ from what is described herein) and then run the process for *Gather Phase 2*. This will take about 40 minutes to process the **1218** or more raw weather station files. You should expect to see output like this:\n\n```\nProcessing dataset 0 - 1218: USC00011084\nProcessing dataset 1 - 1218: USC00012813\n.\n.\n.\nProcessing dataset 1216: USC00130133\nProcessing dataset 1217: USC00204090\n>> Processing Complete.\n>>   Elapsed corpus execution time 0:38:47\n```", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "# Decide if we need to generate station detail files.\ncsvtools.noaa_run_phase2_approach1.help()", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"execution_count": null, "cell_type": "code", "source": "# Run Gather Phase 2 for all 1218 raw files using approach 1 (CSV)\ncsvtools.noaa_run_phase2_approach1(htda_approach1_content_layout, create_details=True)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "##### Observe Output \n\nYou can compute the disk capacity of your *Gather Phase 2* results.\n```\n96M\t/resources/noaa-hdta/data/derived/15mar2015/missing\n24M\t/resources/noaa-hdta/data/derived/15mar2015/summaries\n3.2G\t/resources/noaa-hdta/data/derived/15mar2015/raw\n129M\t/resources/noaa-hdta/data/derived/15mar2015/station_details\n3.4G\t/resources/noaa-hdta/data/derived/15mar2015/\n```", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "%%bash\n# Compute size of output folders\ndu -h --max-depth=1 /resources/noaa-hdta/data/derived/15mar2015/", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "#### Approach 2: HDF5 files\n\nUse the code below to layout your target project environment (if it should differ from what is described herein) and then run the process for *Gather Phase 2*. This will take about 30 minutes to process the **1218** or more raw weather station files. **Note**: You will need to have room for about <font color=\"red\">**6.5GB**</font>. You should expect to see output like this:\n\n```\nFetching keys for type = raw_detail\n>> Fetch Complete.\n>>   Elapsed key-fetch execution time 0:00:09\nProcessing dataset 0 - 1218: USC00011084\nProcessing dataset 1 - 1218: USC00012813\n.\n.\n.\nProcessing dataset 1216 - 1218: USW00094794\nProcessing dataset 1217 - 1218: USW00094967\n>> Processing Complete.\n>>   Elapsed corpus execution time 0:28:48\n```", "cell_type": "markdown", "metadata": {}}, {"source": "%inject hdftools.noaa_run_phase2_approach2", "cell_type": "markdown", "metadata": {}}, {"source": "### noaa_run_phase2_approach2\nTakes a dictionary of project folder details to drive the processing of *Gather Phase 2 Approach 2* using **HDF files**.\n\n#### Disk Storage Requirements\n\n* This function creates a **Station Summaries** dataset that requires ~2GB of free space. \n* This function can also create a **Station Details** dataset. If you require this dataset to be generated, modify the call to ```noaa_run_phase2_approach2()``` with ```create_details=True```. You will need additional free space to support this feature. Estimated requirement: <font color=\"red\">**5GB**</font>", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "# Run Gather Phase 2 for all 1218 files using approach 2 (HDF)\nhdftools.noaa_run_phase2_approach2(htda_approach2_content_layout)", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "##### Observe Output  TODO\n\nYou can compute the disk capacity of your *Gather Phase 2* results.\n```\nHDF File Usage (Phases 1 & 2) - Per File and Total\n4.9G\t/resources/noaa-hdta/data/hdf5/15mar2015/noaa_ncdc_raw_details.h5\n1.4G\t/resources/noaa-hdta/data/hdf5/15mar2015/noaa_ncdc_missing_details.h5\n1.3G\t/resources/noaa-hdta/data/hdf5/15mar2015/noaa_ncdc_station_summaries.h5\n7.4G\t/resources/noaa-hdta/data/hdf5/15mar2015/\n```", "cell_type": "markdown", "metadata": {}}, {"execution_count": null, "cell_type": "code", "source": "%%bash\n# Compute size of output folders\necho \"HDF File Usage (Phases 1 & 2) - Per File and Total\"\ndu -ah /resources/noaa-hdta/data/hdf5/15mar2015/", "outputs": [], "metadata": {"collapsed": false, "trusted": true}}, {"source": "# References\n\n* [Data Analysis Workflow Navigation repository](https://github.com/vinomaster/dawn): This notebook outline was derived from the **Research Analysis Navigation Template**.\n* [NOAA National Climatic Data Center](http://www.ncdc.noaa.gov/)\n* [NOAA Data Fraud News](https://stevengoddard.wordpress.com/2013/01/11/noaa-temperature-fraud-expands-part-1/)\n* [HDF5](http://www.h5py.org/)\n* [HDF5 v. Database](http://nbviewer.ipython.org/github/dolaameng/tutorials/blob/master/ml-tutorials/BASIC_pandas_io%28specially%20hdf5%29.ipynb)\n\n# Summary\n\nThis notebook provides two approaches in the creation of human readable datasets for historical daily temperature analytics. This analytical notebook is a component of a package of notebooks. The tasks addressed herein focused on data munging activities to produce a desired set of datasets for several predefined schemas. These datasets can now be used in other package notebooks for data exploration activities. \n\nThis notebook has embraced the concepts of reproducible research and can be shared with others so that they can recreate the data locally.", "cell_type": "markdown", "metadata": {}}, {"source": "# Future Investigations \n\n1. Fix the ordering of the record lifespan calculations. See code and comments for clarity. \n2. Fix FTP link so that it can be embedded into notebook.\n2. Explore multi-level importable notebooks to allow the tools files to share common code. ", "cell_type": "markdown", "metadata": {}}, {"source": "<div class=\"alert\" style=\"border: 1px solid #aaa; background: radial-gradient(ellipse at center, #ffffff 50%, #eee 100%);\">\n<div class=\"row\">\n    <div class=\"col-sm-1\"><img src=\"https://knowledgeanyhow.org/static/images/favicon_32x32.png\" style=\"margin-top: -6px\"/></div>\n    <div class=\"col-sm-11\">This notebook was created using [IBM Knowledge Anyhow Workbench](https://knowledgeanyhow.org).  To learn more, visit us at https://knowledgeanyhow.org.</div>\n    </div>\n</div>", "cell_type": "markdown", "metadata": {"collapsed": true}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "Python 2", "name": "python2", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.6", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}}