pyWaPOR vs. WaPOR
In this notebook we’ll make an comparison between dekadal Evapotranspiration data from WaPOR and from pyWaPOR.
First we install pywapor, in case it’s not installed yet.
[1]:
!pip install pywapor --quiet
Next we’ll define a couple of basic variables, which were explained in the introduction notebook.
[1]:
import pywapor
project_folder = r"/Users/hmcoerver/pywapor_notebooks_6"
timelim = ["2021-07-01", "2021-07-11"]
latlim = [28.9, 29.7]
lonlim = [30.2, 31.2]
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
Then we run pre_et_look, to generate an ET map for the first dekad of July 2021.
[2]:
ds_in = pywapor.pre_et_look.main(project_folder, latlim, lonlim, timelim)
> PRE_ET_LOOK
--> Collecting `ndvi` from `MODIS.MOD13Q1.061`.
--> Applying 'mask_qa' to `ndvi`.
--> Saving merged data.
> peak-memory-usage: 4.6MB, execution-time: 0:00:02.407964.
> chunksize|dimsize: [time: 2|2, y: 403|403, x: 506|506]
> timesize: 2 [2021-07-04T00:00, ..., 2021-07-20T00:00]
--> Collecting `ndvi` from `MODIS.MYD13Q1.061`.
--> Applying 'mask_qa' to `ndvi`.
--> Saving merged data.
> peak-memory-usage: 2.3MB, execution-time: 0:00:02.064262.
> chunksize|dimsize: [time: 1|1, y: 403|403, x: 506|506]
> timesize: 1 [2021-07-12T00:00, ..., 2021-07-12T00:00]
--> Collecting `r0` from `MODIS.MCD43A3.061`.
--> Applying 'shortwave_r0' to `r0`.
--> Applying 'mask_qa' to `r0`.
--> Saving merged data.
> peak-memory-usage: 10.4MB, execution-time: 0:00:02.107342.
> chunksize|dimsize: [time: 17|17, y: 207|207, x: 258|258]
> timesize: 17 [2021-06-28T12:00, ..., 2021-07-14T12:00]
--> Collecting `p` from `CHIRPS.P05`.
--> Saving merged data.
> peak-memory-usage: 85.5KB, execution-time: 0:00:02.042592.
> chunksize|dimsize: [time: 17|17, y: 18|18, x: 22|22]
> timesize: 17 [2021-06-28T00:30, ..., 2021-07-14T00:30]
--> Collecting `z` from `SRTM.30M`.
--> Saving merged data.
> peak-memory-usage: 118.8MB, execution-time: 0:00:02.756133.
> chunksize|dimsize: [time: 1|1, y: 2883|2883, x: 3603|3603]
> timesize: 1 [2000-02-11T00:00, ..., 2000-02-11T00:00]
--> Collecting `ra` from `MERRA2.M2T1NXRAD.5.12.4`.
--> Saving merged data.
> peak-memory-usage: 588.0KB, execution-time: 0:00:02.050872.
> chunksize|dimsize: [time: 408|408, y: 4|4, x: 3|3]
> timesize: 408 [2021-06-28T01:00, ..., 2021-07-15T00:00]
--> Collecting `t_air`, `t_air_max`, `t_air_min`, `u2m`, `v2m`, `qv`, `p_air`, `p_air_0`, `wv` from `GEOS5.inst3_2d_asm_Nx`.
--> Downloading data.
> peak-memory-usage: 1.3MB, execution-time: 0:00:42.143596.
> chunksize|dimsize: [time: 129|129, y: 5|5, x: 5|5]
--> Applying 'kelvin_to_celsius' to `t_air`.
--> Applying 'kelvin_to_celsius' to `t_air_max`.
--> Applying 'kelvin_to_celsius' to `t_air_min`.
--> Applying 'pa_to_kpa' to `p_air`.
--> Applying 'pa_to_kpa' to `p_air_0`.
--> Saving netCDF.
> peak-memory-usage: 1.9MB, execution-time: 0:00:02.125794.
> chunksize|dimsize: [time: 129|129, y: 5|5, x: 5|5]
> timesize: 129 [2021-06-28T01:30, ..., 2021-07-14T01:30]
--> Collecting `land_mask`, `rs_min`, `z_obst_max` from `GLOBCOVER.2009_V2.3_Global`.
--> Calculating `land_mask` from `lulc`.
--> Calculating `rs_min` from `lulc`.
--> Calculating `z_obst_max` from `lulc`.
--> Saving 2009_V2.3_Global.nc.
> peak-memory-usage: 4.8MB, execution-time: 0:00:02.076793.
> chunksize|dimsize: [y: 295|295, x: 367|367]
--> Collecting `lw_offset`, `lw_slope`, `z_oro`, `rn_offset`, `rn_slope`, `t_amp_year`, `t_opt`, `vpd_slope` from `STATICS.WaPOR2`.
--> Saving WaPOR2.nc.
> peak-memory-usage: 21.3MB, execution-time: 0:00:02.137228.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> Collecting `se_root` from `se_root.v2`.
> PRE_SE_ROOT
--> Collecting `lst` from `MODIS.MOD11A1.061`.
--> Applying 'mask_bitwise_qa' to `lst`.
--> Applying 'expand_time_dim' to `lst`.
--> Saving merged data.
> peak-memory-usage: 4.5MB, execution-time: 0:00:02.055363.
> chunksize|dimsize: [time: 22|22, y: 104|104, x: 130|130]
> timesize: 22 [2021-06-28T10:54, ..., 2021-07-14T10:54]
--> Collecting `lst` from `MODIS.MYD11A1.061`.
--> Applying 'mask_bitwise_qa' to `lst`.
--> Applying 'expand_time_dim' to `lst`.
--> Saving merged data.
> peak-memory-usage: 6.6MB, execution-time: 0:00:02.075634.
> chunksize|dimsize: [time: 32|32, y: 104|104, x: 130|130]
> timesize: 32 [2021-06-28T12:30, ..., 2021-07-14T12:36]
--> Collecting `ndvi` from `MODIS.MOD13Q1.061`.
> timesize: 2 [2021-07-04T00:00, ..., 2021-07-20T00:00]
--> Collecting `ndvi` from `MODIS.MYD13Q1.061`.
> timesize: 1 [2021-07-12T00:00, ..., 2021-07-12T00:00]
--> Collecting `t_air`, `t_air_max`, `t_air_min`, `u2m`, `v2m`, `qv`, `wv`, `p_air`, `p_air_0` from `GEOS5.inst3_2d_asm_Nx`.
> timesize: 129 [2021-06-28T01:30, ..., 2021-07-14T01:30]
--> Collecting `r0_bare`, `r0_full` from `STATICS.WaPOR2`.
--> Saving WaPOR2.nc.
--> Appending data to an existing file.
> peak-memory-usage: 7.5MB, execution-time: 0:00:02.076309.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> Spatially aligning 2 `lst` products together.
--> Using `MYD11A1.061.nc` as reprojecting example.
> shape: (104, 130), res: 0.0079° x 0.0079°.
--> Spatially aligning 2 `ndvi` products together.
--> Using `MYD13Q1.061.nc` as reprojecting example.
> shape: (403, 506), res: 0.0020° x 0.0020°.
--> Aligning times in `ndvi` (3) with `lst` (54, linear).
> peak-memory-usage: 419.7MB, execution-time: 0:00:06.528576.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 500|506]
> shape: (54, 403, 506), kwargs: [].
--> Aligning times in `p_air` (129) with `lst` (54, linear).
> peak-memory-usage: 590.0KB, execution-time: 0:00:02.041889.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `p_air_0` (129) with `lst` (54, linear).
> peak-memory-usage: 608.7KB, execution-time: 0:00:02.057207.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `qv` (129) with `lst` (54, linear).
> peak-memory-usage: 723.4KB, execution-time: 0:00:02.057910.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `t_air` (129) with `lst` (54, linear).
> peak-memory-usage: 608.6KB, execution-time: 0:00:02.054651.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `t_air_max` (129) with `lst` (54, linear).
> peak-memory-usage: 722.8KB, execution-time: 0:00:02.060097.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `t_air_min` (129) with `lst` (54, linear).
> peak-memory-usage: 134.0KB, execution-time: 0:00:02.059848.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `u2m` (129) with `lst` (54, linear).
> peak-memory-usage: 597.5KB, execution-time: 0:00:02.052905.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `v2m` (129) with `lst` (54, linear).
> peak-memory-usage: 723.3KB, execution-time: 0:00:02.058522.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Aligning times in `wv` (129) with `lst` (54, linear).
> peak-memory-usage: 724.2KB, execution-time: 0:00:02.057240.
> chunksize|dimsize: [time: 54|54, y: 5|5, x: 5|5]
> shape: (54, 5, 5), kwargs: [].
--> Using `MOD13Q1.061.nc` as reprojecting example.
> shape: (403, 506), res: 0.0020° x 0.0020°.
--> Selected `reproject_bulk` for reprojection of MOD11A1.061.nc.
--> Applying `reproject_bulk` to MOD11A1.061.nc:lst (nearest).
> peak-memory-usage: 252.0MB, execution-time: 0:00:02.356091.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of p_air_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.4KB, execution-time: 0:00:00.668064.
--> Saving reprojected data from p_air_i.nc:p_air (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.458436.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of p_air_0_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.3KB, execution-time: 0:00:00.660286.
--> Saving reprojected data from p_air_0_i.nc:p_air_0 (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.363778.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of qv_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.3KB, execution-time: 0:00:00.678404.
--> Saving reprojected data from qv_i.nc:qv (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.339350.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of WaPOR2.nc.
--> Correcting src_ds.
> peak-memory-usage: 6.1MB, execution-time: 0:00:02.074415.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> Warping VRT to netCDF.
> peak-memory-usage: 4.6KB, execution-time: 0:00:00.027683.
--> Saving reprojected data from WaPOR2_fixed.nc:r0_bare (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.055627.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of WaPOR2.nc.
--> Correcting src_ds.
> peak-memory-usage: 6.1MB, execution-time: 0:00:02.065847.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> Warping VRT to netCDF.
> peak-memory-usage: 4.6KB, execution-time: 0:00:00.025086.
--> Saving reprojected data from WaPOR2_fixed.nc:r0_full (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.061849.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_air_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.4KB, execution-time: 0:00:00.674753.
--> Saving reprojected data from t_air_i.nc:t_air (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.507785.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_air_max_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.3KB, execution-time: 0:00:00.670831.
--> Saving reprojected data from t_air_max_i.nc:t_air_max (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.528268.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_air_min_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.4KB, execution-time: 0:00:00.669942.
--> Saving reprojected data from t_air_min_i.nc:t_air_min (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.508515.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of u2m_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.3KB, execution-time: 0:00:00.664672.
--> Saving reprojected data from u2m_i.nc:u2m (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.503099.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of v2m_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.3KB, execution-time: 0:00:00.669495.
--> Saving reprojected data from v2m_i.nc:v2m (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.505445.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of wv_i.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 6.3KB, execution-time: 0:00:00.677106.
--> Saving reprojected data from wv_i.nc:wv (bilinear).
> peak-memory-usage: 420.1MB, execution-time: 0:00:02.515049.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
--> Applying 'rename_meteo'.
--> Applying 'remove_empty_statics'.
--> Applying 'add_constants_new'.
--> Applying 'bt_to_lst'.
--> Applying 'drop_empty_times'.
--> Creating merged file `se_root_in.nc`.
> peak-memory-usage: 799.0MB, execution-time: 0:00:06.414838.
> chunksize|dimsize: [time: 54|54, y: 403|403, x: 506|506]
< PRE_SE_ROOT (0:01:37.937393)
> SE_ROOT
--> Running `se_root` (v2).
--> Calculating `seasonal_correction`.
--> Calculating `declination`.
--> Calculating `day_angle`.
--> Calculating `air_pressure_kpa2mbar`.
--> Calculating `air_pressure_kpa2mbar`.
--> Calculating `vegetation_cover`.
--> Calculating `air_temperature_kelvin_inst`.
--> Calculating `air_temperature_kelvin`.
--> Calculating `vapour_pressure_from_specific_humidity_inst`.
--> Calculating `vapour_pressure_from_specific_humidity`.
--> Calculating `specific_humidity_from_vapour_pressure`.
--> Calculating `moist_air_density_inst`.
--> Calculating `moist_air_density`.
--> Calculating `dry_air_density_inst`.
--> Calculating `dry_air_density`.
--> Calculating `air_density_inst`.
--> Calculating `air_density`.
--> Calculating `wind_speed_blending_height_bare`.
--> Calculating `longitude_rad`.
--> Calculating `latitude_rad`.
--> Calculating `hour_angle`.
--> Calculating `inverse_earth_sun_distance`.
--> Calculating `solar_elevation_angle`.
--> Calculating `solar_elevation_angle_refracted`.
--> Calculating `relative_optical_airmass`.
--> Calculating `rayleigh_optical_thickness`.
--> Calculating `linke_turbidity`.
--> Calculating `extraterrestrial_irradiance_normal`.
--> Calculating `beam_irradiance_normal_clear`.
--> Calculating `beam_irradiance_horizontal_clear`.
--> Calculating `diffuse_irradiance_horizontal_clear`.
--> Calculating `ra_clear_horizontal`.
--> Calculating `atmospheric_emissivity_inst`.
--> Calculating `net_radiation_bare`.
--> Calculating `net_radiation_full`.
--> Calculating `sensible_heat_flux_bare`.
--> Calculating `sensible_heat_flux_full`.
--> Calculating `wind_speed_blending_height_full_inst`.
--> Calculating `friction_velocity_bare_inst`.
--> Calculating `friction_velocity_full_inst`.
--> Calculating `monin_obukhov_length_bare`.
--> Calculating `monin_obukhov_length_full`.
--> Calculating `wind_speed_soil_inst`.
--> Calculating `psi_m`.
--> Calculating `aerodynamical_resistance_forced_convection_soil`.
--> Calculating `aerodynamical_resistance_forced_convection_bare`.
--> Calculating `psi_m`.
--> Calculating `psi_h`.
--> Calculating `aerodynamical_resistance_forced_convection_full`.
--> Calculating `psi_m`.
--> Calculating `psi_m`.
--> Calculating `psi_h`.
--> Calculating `psi_h`.
--> Calculating `maximum_temperature_bare`.
--> Calculating `maximum_temperature_full`.
--> Calculating `wet_bulb_temperature_inst_new`.
--> Calculating `maximum_temperature`.
--> Calculating `wet_bulb_temperature_kelvin_inst`.
--> Calculating `air_temperature_kelvin`.
--> Calculating `minimum_temperature`.
--> Calculating `soil_moisture_from_maximum_temperature`.
/Users/hmcoerver/miniconda3/envs/pywapor/lib/python3.10/site-packages/dask/core.py:119: RuntimeWarning: invalid value encountered in power
return func(*(_execute_task(a, cache) for a in args))
/Users/hmcoerver/miniconda3/envs/pywapor/lib/python3.10/site-packages/dask/core.py:119: RuntimeWarning: invalid value encountered in log
return func(*(_execute_task(a, cache) for a in args))
--> Saving output to `se_root_out.nc`.
> peak-memory-usage: 1.8GB, execution-time: 0:00:04.702192.
> chunksize|dimsize: [time: 1|32, y: 403|403, x: 506|506]
< SE_ROOT (0:00:07.658858)
> timesize: 32 [2021-07-04T10:11, ..., 2021-07-13T13:30]
--> Compositing 26 variables.
--> (1/26) Compositing `ndvi` (mean).
--> Using `MYD13Q1.061.nc` as reprojecting example.
> shape: (403, 506), res: 0.0020° x 0.0020°.
--> Saving `ndvi` composites.
> peak-memory-usage: 14.0MB, execution-time: 0:00:04.057448.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> (2/26) Compositing `r0` (mean).
--> Saving `r0` composites.
> peak-memory-usage: 20.8MB, execution-time: 0:00:02.047886.
> chunksize|dimsize: [time_bins: 1|1, y: 207|207, x: 258|258]
--> (3/26) Compositing `se_root` (max).
--> Saving `se_root` composites.
> peak-memory-usage: 105.8MB, execution-time: 0:00:02.052137.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> (4/26) Compositing `p` (mean).
--> Saving `p` composites.
> peak-memory-usage: 157.9KB, execution-time: 0:00:02.046368.
> chunksize|dimsize: [time_bins: 1|1, y: 18|18, x: 22|22]
--> (5/26) Compositing `z` (None).
--> Saving `z` composites.
> peak-memory-usage: 317.0MB, execution-time: 0:00:02.228660.
> chunksize|dimsize: [y: 2883|2883, x: 3603|3603]
--> (6/26) Compositing `ra` (mean).
--> Saving `ra` composites.
> peak-memory-usage: 126.0KB, execution-time: 0:00:02.047957.
> chunksize|dimsize: [time_bins: 1|1, y: 4|4, x: 3|3]
--> (7/26) Compositing `t_air` (mean).
--> Saving `t_air` composites.
> peak-memory-usage: 80.6KB, execution-time: 0:00:02.041299.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (8/26) Compositing `t_air_max` (max).
--> Saving `t_air_max` composites.
> peak-memory-usage: 78.9KB, execution-time: 0:00:02.034412.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (9/26) Compositing `t_air_min` (min).
--> Saving `t_air_min` composites.
> peak-memory-usage: 79.7KB, execution-time: 0:00:02.051839.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (10/26) Compositing `u2m` (mean).
--> Saving `u2m` composites.
> peak-memory-usage: 80.2KB, execution-time: 0:00:02.049376.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (11/26) Compositing `v2m` (mean).
--> Saving `v2m` composites.
> peak-memory-usage: 82.2KB, execution-time: 0:00:02.049165.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (12/26) Compositing `qv` (mean).
--> Saving `qv` composites.
> peak-memory-usage: 79.9KB, execution-time: 0:00:02.038089.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (13/26) Compositing `p_air` (mean).
--> Saving `p_air` composites.
> peak-memory-usage: 81.9KB, execution-time: 0:00:02.035260.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (14/26) Compositing `p_air_0` (mean).
--> Saving `p_air_0` composites.
> peak-memory-usage: 80.5KB, execution-time: 0:00:02.049169.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (15/26) Compositing `wv` (mean).
--> Saving `wv` composites.
> peak-memory-usage: 79.5KB, execution-time: 0:00:02.037950.
> chunksize|dimsize: [time_bins: 1|1, y: 5|5, x: 5|5]
--> (16/26) Compositing `land_mask` (None).
--> Saving `land_mask` composites.
> peak-memory-usage: 10.9MB, execution-time: 0:00:02.044731.
> chunksize|dimsize: [y: 295|295, x: 367|367]
--> (17/26) Compositing `rs_min` (None).
--> Saving `rs_min` composites.
> peak-memory-usage: 10.9MB, execution-time: 0:00:02.030452.
> chunksize|dimsize: [y: 295|295, x: 367|367]
--> (18/26) Compositing `z_obst_max` (None).
--> Saving `z_obst_max` composites.
> peak-memory-usage: 10.9MB, execution-time: 0:00:02.035580.
> chunksize|dimsize: [y: 295|295, x: 367|367]
--> (19/26) Compositing `lw_offset` (None).
--> Saving `lw_offset` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.047438.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> (20/26) Compositing `lw_slope` (None).
--> Saving `lw_slope` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.039460.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> (21/26) Compositing `z_oro` (None).
--> Saving `z_oro` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.028499.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> (22/26) Compositing `rn_offset` (None).
--> Saving `rn_offset` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.043315.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> (23/26) Compositing `rn_slope` (None).
--> Saving `rn_slope` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.037528.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> (24/26) Compositing `t_amp_year` (None).
--> Saving `t_amp_year` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.032880.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> (25/26) Compositing `t_opt` (None).
--> Saving `t_opt` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.031085.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> (26/26) Compositing `vpd_slope` (None).
--> Saving `vpd_slope` composites.
> peak-memory-usage: 20.1MB, execution-time: 0:00:02.025334.
> chunksize|dimsize: [y: 403|403, x: 493|493]
--> Using `MOD13Q1.061.nc` as reprojecting example.
> shape: (403, 506), res: 0.0020° x 0.0020°.
--> Selected `reproject_chunk` for reprojection of r0_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.021511.
--> Saving reprojected data from r0_bin.nc:r0 (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.043211.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of p_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.019252.
--> Saving reprojected data from p_bin.nc:p (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.041460.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of z_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.281868.
--> Saving reprojected data from z_bin.nc:z (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.041438.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of ra_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.7KB, execution-time: 0:00:00.017997.
--> Saving reprojected data from ra_bin.nc:ra (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.043552.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_air_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.019003.
--> Saving reprojected data from t_air_bin.nc:t_air (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.046233.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_air_max_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.6KB, execution-time: 0:00:00.018447.
--> Saving reprojected data from t_air_max_bin.nc:t_air_max (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.045936.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_air_min_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.018329.
--> Saving reprojected data from t_air_min_bin.nc:t_air_min (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.048257.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of u2m_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.019050.
--> Saving reprojected data from u2m_bin.nc:u2m (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.041612.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of v2m_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.019344.
--> Saving reprojected data from v2m_bin.nc:v2m (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.046137.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of qv_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.020039.
--> Saving reprojected data from qv_bin.nc:qv (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.044186.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of p_air_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.018600.
--> Saving reprojected data from p_air_bin.nc:p_air (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.045454.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of p_air_0_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.4KB, execution-time: 0:00:00.019338.
--> Saving reprojected data from p_air_0_bin.nc:p_air_0 (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.045413.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of wv_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.6KB, execution-time: 0:00:00.018511.
--> Saving reprojected data from wv_bin.nc:wv (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.045915.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of land_mask_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.018446.
--> Saving reprojected data from land_mask_bin.nc:land_mask (nearest).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.038067.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of rs_min_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.018212.
--> Saving reprojected data from rs_min_bin.nc:rs_min (nearest).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.043708.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of z_obst_max_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.020956.
--> Saving reprojected data from z_obst_max_bin.nc:z_obst_max (nearest).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.065467.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of lw_offset_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.029715.
--> Saving reprojected data from lw_offset_bin.nc:lw_offset (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.059265.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of lw_slope_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.031476.
--> Saving reprojected data from lw_slope_bin.nc:lw_slope (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.068699.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of z_oro_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.030293.
--> Saving reprojected data from z_oro_bin.nc:z_oro (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.071099.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of rn_offset_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.030181.
--> Saving reprojected data from rn_offset_bin.nc:rn_offset (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.070886.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of rn_slope_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 5.4KB, execution-time: 0:00:00.030231.
--> Saving reprojected data from rn_slope_bin.nc:rn_slope (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.072817.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_amp_year_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.5KB, execution-time: 0:00:00.032609.
--> Saving reprojected data from t_amp_year_bin.nc:t_amp_year (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.070647.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of t_opt_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 4.6KB, execution-time: 0:00:00.034883.
--> Saving reprojected data from t_opt_bin.nc:t_opt (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.067045.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Selected `reproject_chunk` for reprojection of vpd_slope_bin.nc.
--> Warping VRT to netCDF.
> peak-memory-usage: 5.4KB, execution-time: 0:00:00.030079.
--> Saving reprojected data from vpd_slope_bin.nc:vpd_slope (bilinear).
> peak-memory-usage: 6.2MB, execution-time: 0:00:02.052775.
> chunksize|dimsize: [y: 403|403, x: 506|506]
--> Calculating local means (r = 0.25°) of `z`.
--> Filling 806 missing pixels in 'z'.
--> Applying 'lapse_rate'.
--> Applying 'rename_vars'.
--> Applying 'fill_attrs'.
--> Applying 'calc_doys'.
--> Applying 'remove_empty_statics'.
--> Applying 'add_constants_new'.
--> Creating merged file `et_look_in.nc`.
> peak-memory-usage: 16.6MB, execution-time: 0:00:02.548215.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
< PRE_ET_LOOK (0:06:21.305585)
Then we run et_look.
[3]:
ds = pywapor.et_look.main(ds_in, export_vars = ["et_24_mm"])
> ET_LOOK
--> Running `et_look` (v2).
--> Calculating `declination`.
--> Calculating `inverse_earth_sun_distance`.
--> Calculating `vegetation_cover`.
--> Calculating `leaf_area_index`.
--> Calculating `effective_leaf_area_index`.
--> Calculating `soil_fraction`.
--> Calculating `latitude_rad`.
--> Calculating `sunset_hour_angle`.
--> Calculating `daily_solar_radiation_toa_flat`.
--> Calculating `transmissivity`.
--> Calculating `stress_radiation`.
--> Calculating `air_pressure_kpa2mbar`.
--> Calculating `air_pressure_daily`.
--> Calculating `air_pressure`.
--> Calculating `vapour_pressure_from_specific_humidity_daily`.
--> Calculating `vapour_pressure_from_specific_humidity`.
--> Calculating `saturated_vapour_pressure`.
--> Calculating `vapour_pressure_deficit_daily`.
--> Calculating `vapour_pressure_deficit`.
--> Calculating `stress_vpd`.
--> Calculating `stress_temperature`.
--> Calculating `atmospheric_canopy_resistance`.
--> Calculating `air_temperature_kelvin_daily`.
--> Calculating `air_temperature_kelvin`.
--> Calculating `longwave_radiation_fao`.
--> Calculating `interception_mm`.
--> Calculating `latent_heat_daily`.
--> Calculating `latent_heat`.
--> Calculating `interception_wm2`.
--> Calculating `net_radiation`.
--> Calculating `net_radiation_canopy`.
--> Calculating `stress_moisture`.
--> Calculating `canopy_resistance`.
--> Calculating `obstacle_height`.
--> Calculating `roughness_length`.
--> Calculating `wind_speed`.
--> Calculating `initial_canopy_aerodynamic_resistance`.
--> Calculating `wind_speed_blending_height_daily`.
--> Calculating `wind_speed_blending_height`.
--> Calculating `dry_air_density_daily`.
--> Calculating `dry_air_density`.
--> Calculating `moist_air_density_daily`.
--> Calculating `moist_air_density`.
--> Calculating `air_density_daily`.
--> Calculating `air_density`.
--> Calculating `psychrometric_constant_daily`.
--> Calculating `psychrometric_constant`.
--> Calculating `slope_saturated_vapour_pressure_daily`.
--> Calculating `slope_saturated_vapour_pressure`.
--> Calculating `saturated_vapour_pressure`.
--> Calculating `initial_daily_transpiration`.
--> Calculating `initial_sensible_heat_flux_canopy_daily`.
--> Calculating `displacement_height`.
--> Calculating `initial_friction_velocity_daily`.
--> Calculating `transpiration`.
--> Calculating `transpiration_mm`.
--> Calculating `net_radiation_soil`.
--> Calculating `soil_resistance`.
--> Calculating `initial_soil_aerodynamic_resistance`.
--> Calculating `initial_friction_velocity_soil_daily`.
--> Calculating `soil_thermal_conductivity`.
--> Calculating `volumetric_heat_capacity`.
--> Calculating `damping_depth`.
--> Calculating `bare_soil_heat_flux`.
--> Calculating `soil_heat_flux`.
--> Calculating `initial_daily_evaporation`.
--> Calculating `initial_sensible_heat_flux_soil_daily`.
--> Calculating `evaporation`.
--> Calculating `evaporation_mm`.
--> Calculating `et_actual_mm`.
--> Calculating `net_radiation_grass`.
--> Calculating `et_reference`.
--> Calculating `et_reference_mm`.
--> Calculating `air_temperature_kelvin_daily`.
--> Calculating `air_temperature_kelvin`.
--> Calculating `air_temperature_kelvin_daily`.
--> Calculating `air_temperature_kelvin`.
--> Calculating `mean_temperature_kelvin_daytime`.
--> Calculating `temperature_dependency`.
--> Calculating `affinity_constant_co2`.
--> Calculating `inhibition_constant_o2`.
--> Calculating `co2_o2_specificity_ratio`.
--> Calculating `co2_level_annual`.
--> Calculating `autotrophic_respiration`.
--> Calculating `par`.
--> Calculating `fpar`.
--> Calculating `co2_fertilisation`.
--> Calculating `net_primary_production_max`.
--> Calculating `net_primary_production`.
--> Saving output to `et_look_out.nc`.
> peak-memory-usage: 705.1MB, execution-time: 0:00:03.294189.
> chunksize|dimsize: [time_bins: 1|1, y: 403|403, x: 506|506]
< ET_LOOK (0:00:07.598479)
Now that we have the pyWaPOR ET map, we still need the WaPOR map in order to make a comparison. We can download this particular map like this (check this notebook for a more in-depth explanation of this code).
[7]:
import requests
from osgeo import gdal
import os
import getpass
cubecode = "L1_AETI_D"
rasterid = "L1_AETI_2119"
authorization_request_url = "https://io.apps.fao.org/gismgr/api/v1/iam/sign-in"
authorization_headers = {"X-GISMGR-API-KEY": getpass.getpass("WaPOR API Token:")}
authorization_request_response = requests.post(authorization_request_url, headers = authorization_headers)
authorization_request_response.raise_for_status()
access_token = authorization_request_response.json()["response"]["accessToken"]
tif_request_url = f"https://io.apps.fao.org/gismgr/api/v1/download/WAPOR_2?requestType=MAPSET_RASTER&cubeCode={cubecode}&rasterId={rasterid}"
tif_headers = {"Authorization": f"Bearer {access_token}"}
tif_request_response = requests.get(tif_request_url, headers = tif_headers)
tif_request_response.raise_for_status()
tif_url = tif_request_response.json()["response"]["downloadUrl"]
bounding_box = [lonlim[0], latlim[1], lonlim[1], latlim[0]]
translate_options = gdal.TranslateOptions(projWin=bounding_box, bandList=[1])
wapor_file = os.path.join(project_folder, "wapor_eta.tif")
ds = gdal.Translate(wapor_file, f"/vsicurl/{tif_url}", options = translate_options)
ds.FlushCache()
ds = None
Next, lets have a look at the two datasets side by side.
[9]:
import xarray as xr
import matplotlib.pyplot as plt
wapor_da = xr.open_dataset(wapor_file).isel(band = 0)["band_data"]
pywapor_da = ds.et_24_mm.isel(time_bins=0)
fig, axes = plt.subplots(ncols=2, figsize=(12,4))
wapor_da.plot(ax=axes[0])
pywapor_da.plot(ax=axes[1])
[9]:
<matplotlib.collections.QuadMesh at 0x1079b1420>
As you can see, the values in WaPOR seems to be about 10 times larger than pyWaPOR. We need to apply a scale factor to WaPOR.
[10]:
wapor_da= wapor_da * 0.1
fig, axes = plt.subplots(ncols=2, figsize=(12,4))
wapor_da.plot(ax=axes[0])
pywapor_da.plot(ax=axes[1])
[10]:
<matplotlib.collections.QuadMesh at 0x164859000>
We can also have a look at some other metadata.
[11]:
def print_raster(raster):
print(
f"shape: {raster.rio.shape}\n"
f"resolution: {raster.rio.resolution()}\n"
f"bounds: {raster.rio.bounds()}\n"
f"count: {raster.count().item()}\n"
f"CRS: {raster.rio.crs}\n"
)
print("WaPOR Raster:\n-------------")
print_raster(wapor_da)
print("pyWaPOR Raster:\n---------------")
print_raster(pywapor_da)
WaPOR Raster:
-------------
shape: (358, 448)
resolution: (0.00223214286, -0.00223214286)
bounds: (30.198660777060002, 28.901785714360003, 31.198660778340006, 29.700892858240003)
count: 160384
CRS: EPSG:4326
pyWaPOR Raster:
---------------
shape: (403, 506)
resolution: (0.001982792281130139, -0.001982792281130138)
bounds: (30.198124670407314, 28.90197637470393, 31.201417564659167, 29.70104166399938)
count: 171002
CRS: EPSG:4326
Projections are the same, but shape and pixel sizes are slighly different. Let make them match with eachother, so that we can make a comparison between the two arrays.
[12]:
wapor_da = wapor_da.rio.reproject_match(pywapor_da)
pywapor_da = pywapor_da.where(wapor_da.notnull())
wapor_da = wapor_da.where(pywapor_da.notnull())
[13]:
fig, axes = plt.subplots(ncols=2, figsize=(12,4))
wapor_da.plot(ax=axes[0], vmin = 0, vmax = 12)
pywapor_da.plot(ax=axes[1], vmin = 0, vmax = 12)
print("WaPOR Raster:\n-------------")
print_raster(wapor_da)
print("pyWaPOR Raster:\n---------------")
print_raster(pywapor_da)
WaPOR Raster:
-------------
shape: (403, 506)
resolution: (0.001982792281130139, -0.001982792281130138)
bounds: (30.198124670407314, 28.90197637470393, 31.201417564659167, 29.70104166399938)
count: 171002
CRS: EPSG:4326
pyWaPOR Raster:
---------------
shape: (403, 506)
resolution: (0.001982792281130139, -0.001982792281130138)
bounds: (30.198124670407314, 28.90197637470393, 31.201417564659167, 29.70104166399938)
count: 171002
CRS: EPSG:4326
[14]:
import numpy as np
def plot_hexbin(ax, arrays, xlabel = "", ylabel = ""):
minmax = [np.min(arrays), np.max(arrays)]
ax.plot(minmax, minmax, ":k", label = "1:1")
ax.legend()
hb = ax.hexbin(*arrays, bins = "log")
ax.set_facecolor(plt.cm.get_cmap('viridis')(0.0))
ax.get_figure().colorbar(hb, label = "Number of pixels [-]")
ax.set_xlim(minmax)
ax.set_ylim(minmax)
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)
nse = pywapor.post_et_look.calc_nash_sutcliffe(arrays)[0]
rmse = pywapor.post_et_look.calc_rmse(arrays)[0]
r = pywapor.post_et_look.calc_pearson_correlation(arrays)[0]
ax.set_title(f"r: {r:.3f}, nse: {nse:.3f}, rmse: {rmse:.3f}")
Finally, lets make a scatter plot to see the differences a bit better.
[15]:
arrays = np.array([
pywapor_da.stack({"pixel": ("y", "x")}).dropna("pixel"),
wapor_da.stack({"pixel": ("y", "x")}).dropna("pixel")
])
fig = plt.figure()
ax = fig.gca()
plot_hexbin(ax, arrays,
"ET (pywapor) [mm/day]",
"ET (wapor) [mm/day]")
/var/folders/z8/t3l6wwkd63qdv3c1lb2r63_c0000gn/T/ipykernel_75283/2440719009.py:8: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap(obj)`` instead.
ax.set_facecolor(plt.cm.get_cmap('viridis')(0.0))
Although there is some variance, the two maps are very similar. et_look is identical to the model used to create the WaPOR data, but pre_et_look is not. The solar-radiation (ra_24) product used in pyWaPOR is differnt from WaPOR as well. This variance can thus mainly be explained by differences in the preprocessing of the et_look input data.