Chapter 4 - Data Download
The Global Fluxnet Synthesis Data Set is large and a download of all sites years and variables is about 15 Gb of data. A single zip file of this size would be difficult for users to download. To improve reliability and speed, we have enabled download of data via several reliable web transfer mechanisms. Before you download data, you may also want to tune your machine's TCP to improve performance.
The first step in downloading the data is to choose which data you would like to download.
- To select the sites, choose individual sites from the list. To select a range of sites, select the first one and then hold down shift when selecting the last site in the range (For example: if you want to download all sites, select the first site in the list and then scroll down to the last site, hold down shift, and then select the last site). If you hold down "control", you will be able to select multiple individual sites. Once all of your selected sites are highlighted, proceed to the next step.
- Now choose what format you want to download the data in:
Select the time granularity:
- Binary - This format is a binary representation where each value is stored as a Int32. The data files have a .bin extension. The header files have a .hdr extension. The header files provide the name and number of each data column in the associated binary file. In the binary format, all available variables will be downloaded for each site/year.
- Core Vars - This format is comma separated value (csv) ASCII file which will contain a first row that is only the core variables (the first row of the file contains variable names). The core variables are defined on the "Dataset Information" page listing the variables under "Data Explanations." The core variables are colored medium blue in the listing.
- Core w/Qual - Is the same as "Core Vars" above but also includes the quality columns associated with each of the core variables.
- All Vars - This format is comma separated value (csv) ASCII file which will contain all the variables available (the first row of the file contains variable names).
Now select desired years of data. If you want all available years, choose "Select all years", otherwise choose the individual years.
Once you have specified your desired selections, hit the "Submit" button.
- Daily - Values aggregated up to daily values. So that there is one value per variable per day.
- Hourly - The 1/2 hourly values.
This dataset is relatively large. Below are some figures to help you estimate the size of your requested download.
A single site year of data averages:
If you download all site years, the download will be
All Daily Files
Binary = 162MB,
Core Variables = 135MB,
Core w/Quality = 209MB,
All Variables = 459MB
All Hourly Files
Binary = 6.2GB
Core Variables = 7.1GB,
Core w/Quality = 9.7GB
All Variables = 14GB
Binary = 180KB
Core Variables = 150KB
Core w/Quality = 230KB
All Variables = 0.5MB
Binary = 6.9MB
Core Variables = 7.5MB
Core w/Quality = 11MB
All Variables = 16MB
The results of your request will be displayed on the next page. The results page will provide you instructions and base material for completing the download using either ftp or any of the web page download mechanisms. A few of these are described below.
- ftp- At the top of the page you will find instructions for using ftp to transfer the files. However, ftp is only recommended for use if you are getting a small number of files. Most firewalls are set to block ftp so make sure that you have ftp enabled in your firewall. Ftp will download the files to the local directory you started ftp from. The ftp site only allows passive ftp so if things are not working try adding the command "LITERAL PASV" (make it all caps).
- DownThemAll - This is an add-on available in Firefox that will enable reliable download of all the files directly from the web page.
- To add "DownThemAll" - go to https://addons.mozilla.org/en-US/firefox/addon/201 and hit the "Install Now" button. If that doesn't work, go to Tools->Add-ons then choose "Get Extensions" in the box that comes up. This will bring up a web page. Enter "DownThemAll" in the search box in the upper right hand corner. Then select "DownThemAll" on the next page and install. You will need to restart your browser after install.
- To use "DownThemAll" - In the Firefox main menu choose "Tools->DownThemAll->DownThemAll". Use the additional filters box to specify the extensions of the files you want to download. This will usually be "csv" unless you are downloading the binary files which need "bin,hdr" or the matlab files "mat". Below is a screen shot of the DownThemAll interface configured to download all the csv files linked from the current page (The view you are seeing is crolled down to the bottom of the file list).
flashgot - This is another Firefox addin that can be installed. However, to make it work you will also have to install a download manager. There are several available and they are listed on the flashgot page. We had varied success with this path so it is not recommended unless you already have it working.
any other download manager - There are a wide variety of web page download managers that will enable you to download all the links on a page. Feel free to use your favorite one.