pyWaPOR

downloads version Open In Colab

This repository contains a Python implementation of the algorithm used to generate the WaPOR datasets. It can be used to calculate evaporation, transpiration and biomass production maps.

Installation

Its recommended to install in a clean conda environment and use conda to install all the important packages from the conda-forge channel.

conda create -n my_pywapor_env --yes -c conda-forge python pip gdal xarray numpy pandas requests matplotlib netcdf4 pyproj scipy pycurl joblib bs4 rasterio bottleneck tqdm dask rioxarray cryptography cachetools cdsapi shapely lxml scikit-learn numba xmltodict

conda activate my_pywapor_env

Then use the package manager pip to install pywapor.

pip install pywapor

Usage

To run the model for one dekad (from 2021-07-01 to 2021-07-11 in this case) for the Fayoum irrigation scheme in Egypt (but feel free to change the boundingbox defined by latlim and lonlim) using mainly VIIRS and Sentinel-2 data, run the following code (run python in your console to activate Python and exit() to deactivate).

import pywapor

# User inputs.
timelim = ["2021-07-01", "2021-07-03"]
latlim = [28.9, 29.7]
lonlim = [30.2, 31.2]
project_folder = r"/my_first_ETLook_run/"
level = "level_2_v3"

# Load a model configuration.
configuration = pywapor.general.levels.pre_et_look_levels(level = level)

# Download and prepare input data.
ds_in  = pywapor.pre_et_look.main(project_folder, latlim, lonlim, timelim, sources = configuration)

# Run the model.
ds_out = pywapor.et_look.main(ds_in)

Check out the documentation and the notebooks below to learn more!

Documentation

Notebooks
Name Colab
1. Introduction colab
2. Passwords colab
3. Enhancers colab
4. Sideloading colab

WaPOR v3

WaPOR v2

WaPOR v1

Acknowledgments

The methodology for WaPOR was developed by the FRAME1 consortium, consisting of eLEAF (lead), VITO, ITC, University of Twente and Waterwatch foundation, commissioned by and in partnership with the Land and Water Division of FAO. The method for calculating evapotranspiration is based on the ETLook model developed by eLEAF in 2010. The method for calculating total biomass production is based on the C-Fix model.

The code in the pywapor.et_look_v2_v3 module of this repository, containing all core physical functions used by ETLook, was written by Henk Pelgrum (eLEAF) and Rutger Kassies (eLEAF). The remaining modules have been developed by Bert Coerver (FAO), Tim Hessels (WaterSat), and, in the framework of the ESA-funded ET4FAO project, Radoslaw Guzinski (DHI-GRAS), Hector Nieto (Complutig) and Laust Faerch (DHI-GRAS).

Contact

For questions, requests or issues with this repository, please contact Bert Coerver at bert.coerver@fao.org or the WaPOR team at wapor@fao.org.

Release Notes

3.4.0 (2023-09-15)
  • Rewritten code to download and process VIIRSL1 data, which is now available from an S3 bucket.
  • Rewritten code to download PROBA-V data, which now comes from the new Terrascope platform (the old portal was deprecated).
  • Rewritten Sentinel downloader, which now uses the new Copernicus Data Space Ecosystem. Most importantly this means that all images are available instantly (so no more tedious requesting from the Long Term Archive for older scenes).
  • Projecting of curvilinear data (VIIRSL1 and Sentinel-3) is now done using gdal-warp.
  • Calculation of terrain slope and aspect is now done using gdal-dem.
  • It is now possible turn off SSL-verification when downloading certain products. Run 'import os; os.environ["PYWAPOR_VERIFY_SSL"] = "NO"' to do so.
  • Simplified installation requirements.
  • Available options in `pywapor.pre_et_look.main` for `sources` are now "level_1", "level_2", "level_3", "level_2_v3" (new default value) and "level_3_v3". The `bin_length' is now set to 1 by default (was `"DEKAD"`).
  • Various other smaller bug fixes.

3.3.0 (2023-04-05)
  • - Option to smooth and interpolate data with a Whittaker smoother.
  • - Downloading of Sentinel-3 data is now faster.
  • - Fixed an issue that could result in an incorrect scale-factor being applied to Sentinel-2 images.

3.2.0 (2023-01-03)
  • - Full Landsat support, automatically download and process Landsat (5-9) data.
  • - Fixed a bug that caused some MODIS data to be missing inside the selected boundingbox.
  • - You can now re-enter a password in case you've provided an incorrect one.
  • - Bugfixes (including more Windows specific ones).
  • - Updated weights to calculate albedo for Landsat and Sentinel.

3.1.0 (2022-09-22)
  • - Added a thermal sharpening algorithm (pyDMS) to increase LST resolution.
  • - Now, when after several failed attempts to download a variable, the code will continue processing other variables.
  • - Improved cloud-masking for VIIRSL1.
  • - Bugfixes (including several Windows specific ones).
  • - More information in the log.

3.0.0 (2022-08-31)
  • - Bugfixes. Most noteably, server side errors when downloading data are now handeled better, i.e. collect tools will retry several times when a download fails, but many other smaller issues have been fixed too.
  • - Performance improvements, mostly due to fewer reprojections.
  • - Better logging. The logs from SENTINEL and ERA5 are now directed to seperate channels and logs now show peak-memory-usage for critical calculation steps.
  • - `et_look` and `se_root` now accept a `chunks` keyword to adjust the chunksizes at which the calculations are done. Increase them if you have a lot of RAM available, decrease them for slower calculations but with less memory usage.
  • - Support for WaPOR v3 methodology. Choose `et_look_version = "v3"` and `se_root_version = "v3"` when running the respective models (`et_look` and `se_root`).
  • - Default configurations for WaPOR v3 input datasets, i.e. choose `sources = "level_2_v3"` when running `pre_et_look` or `pre_se_root`.
  • - New collect functions for COPERNICUS DEM.
  • - The data structure for STATICS is now consistent with the other products.

2.6.0 (2022-08-04)
  • - New collect functions for VIIRS (Level-1), SENTINEL-2, SENTINEL-3 and (ag)ERA5.
  • - pyWaPOR now works with Python versions greater than 3.8.

2.5.0 (2022-06-23)
  • - Rewritten collect tools.
  • - The entire workflow now works with netCDF.
  • - All the netCDF files are formatted to support the rio-acccessor.

2.4.2 (2022-04-26)
  • - New biomass module and NPP calculation.

2.4.1 (2022-03-11)
  • - NetCDF files are now compressed when saved to disk.
  • - Calculation of Total Biomass Production is now turned on by default.
  • - It is no longer required to provide all input variables to et_look, the model will calculate as many variables as possible with the given data. For example, if you are only interested in acquiring interception rates, it would now suffice to only prepare ndvi and p (precipitation) data with pre_et_look.
  • - et_look now automatically generates an interactive network graph visualising the executed computation steps.

2.4.0 (2022-02-03)
  • - Easily apply your own functions to data, i.e. use your own custom filters, gap-fillers etc.
  • - Side-load your own data, i.e. easily incorporate you own datasets.
  • - Added functions to process Landsat Level-2 to “ndvi”, “lst” and “r0”.
  • - Data is now stored and processed as netCDF (using xarray and dask).
  • - Calculations in et_look() and se_root() are now done in chunks, instead of using a for-loop.
  • - Some previously constant parameters now have spatial variability.
  • - Improved logging.
  • - Download functions now show progress and download-speed.
  • - MODIS datasets switched from v6.0 (decommissioned soon) to v6.1.
  • - The lapse-rate correction to temperature data is now more accurate and faster.
  • - VITO and WAPOR passwords are now checked when entered.
  • - Other bug-fixes and performance improvements.

2.3.0 (2021-11-19)
  • - Automatically create input composites before running ETLook.
  • - Choose composite lengths in number of days or dekads.
  • - Option to choose which products to use per variable.
  • - Calculate soil saturation separate from ETLook.
  • - PROBA-V support for NDVI and Albedo inputs.
  • - Define diagnostics pixels, for which extra outputs are created (e.g. charts, maps etc.).
  • - Bug-fixes and performance improvements.

License

APACHE