❗️ This notebook refers to WaPOR 3.4 and is outdated ❗️

Sideloading

pywapor can download data from different products, if however you wish to use another dataset that is ofcourse possible. Below are shown two methods on how to ingest your own data.

Method 1

Start by loading a default configuration for pre_et_look and pre_se_root.

import pywapor
import xarray as xr
from functools import partial
import pywapor.se_root as se_root

et_look_config = pywapor.general.levels.pre_et_look_levels(level = "level_1")
se_root_config = pywapor.general.levels.pre_se_root_levels(level = "level_1")

OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.

Create a list of the meteorological variables.

meteo_vars = ['t_air', 't_air_max', 't_air_min', 'u2m', 'v2m', 'qv', 'p_air', 'p_air_0', 'wv']

So for each of these we need to adjust the "products" part. For example it looks like this right now for "t_air".

et_look_config["t_air"]

{'products': [{'source': 'GEOS5',
   'product_name': 'inst3_2d_asm_Nx',
   'enhancers': 'default'}],
 'composite_type': 'mean',
 'temporal_interp': 'linear',
 'spatial_interp': 'bilinear'}

We are going to overwrite the list under the "products" key. The value of "source" needs to be a function that returns a xr.Dataset which contains a variable called "t_air". For the product name we can choose a new name.

So we define a function that can return the dataset.

def meteo_sideload(**kwargs):
    fh = r"path/to/my/netcdf_file/meteo_data.nc"
    ds = xr.open_dataset(fh)
    return ds

And then we put it inside the configuration for each variable.

meteo_config = [{"source": meteo_sideload, "product_name": "SAWS_data", "enhancers": []}]
for var in meteo_vars:
    et_look_config[var]["products"] = meteo_config
    se_root_config[var]["products"] = meteo_config

Now the configuration contains our new product called "SAWS_data".

et_look_config["t_air"]

{'products': [{'source': <function __main__.meteo_sideload(**kwargs)>,
   'product_name': 'SAWS_data',
   'enhancers': []}],
 'composite_type': 'mean',
 'temporal_interp': 'linear',
 'spatial_interp': 'bilinear'}

Right now the configuration for "se_root" inside et_look_config still contains the original level_1 configuration.

et_look_config["se_root"]

{'products': [{'source': functools.partial(<function se_root at 0x157ef2520>, bin_length=1, sources='level_1'),
   'product_name': 'v2',
   'enhancers': 'default'}],
 'composite_type': 'max',
 'temporal_interp': None,
 'spatial_interp': 'bilinear'}

We can adjust it like this.

se_root_dler = partial(se_root.se_root, sources = se_root_config)
et_look_config["se_root"]["products"][0]["source"] = se_root_dler

Now the configuration for "se_root" has been updated too.

et_look_config["se_root"]

{'products': [{'source': functools.partial(<function se_root at 0x157ef2520>, sources={'ndvi': {'products': [{'source': 'MODIS', 'product_name': 'MOD13Q1.061', 'enhancers': 'default', 'is_example': True}, {'source': 'MODIS', 'product_name': 'MYD13Q1.061', 'enhancers': 'default'}], 'temporal_interp': 'linear', 'spatial_interp': 'nearest'}, 'lst': {'products': [{'source': 'MODIS', 'product_name': 'MOD11A1.061', 'enhancers': 'default'}, {'source': 'MODIS', 'product_name': 'MYD11A1.061', 'enhancers': 'default'}], 'temporal_interp': None, 'spatial_interp': 'nearest'}, 't_air': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 't_air_max': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 't_air_min': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 'u2m': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 'v2m': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 'qv': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 'wv': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 'p_air': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 'p_air_0': {'products': [{'source': <function meteo_sideload at 0x16203d580>, 'product_name': 'SAWS_data', 'enhancers': []}], 'temporal_interp': 'linear', 'spatial_interp': 'bilinear'}, 'r0_bare': {'products': [{'source': 'STATICS', 'product_name': 'WaPOR2', 'enhancers': 'default'}], 'temporal_interp': None, 'spatial_interp': 'bilinear'}, 'r0_full': {'products': [{'source': 'STATICS', 'product_name': 'WaPOR2', 'enhancers': 'default'}], 'temporal_interp': None, 'spatial_interp': 'bilinear'}}),
   'product_name': 'v2',
   'enhancers': 'default'}],
 'composite_type': 'max',
 'temporal_interp': None,
 'spatial_interp': 'bilinear'}

Finally, we can start pre_et_look as usual.

folder = r""
latlim = lonlim = timelim = [] # You need to define these ofcourse.
input_ds = pywapor.pre_et_look.main(folder, latlim, lonlim, timelim, sources = et_look_config)

Method 2

Another way is to adjust input_ds before calling pywapor.et_look.main. Which would like something like this.

First we remove the meteo variables from the configuration. This is optional, but it avoids downloading data that we aren’t going to use anyway.

et_look_config = pywapor.general.levels.pre_et_look_levels(level = "level_1")

meteo_vars = ['t_air', 't_air_max', 't_air_min', 'u2m', 'v2m', 'qv', 'p_air', 'p_air_0', 'wv']

for var in meteo_vars:
    _ = et_look_config.pop(var, None)

Then we run pywapor.pre_et_look.main as usual.

folder = r""
latlim = lonlim = timelim = []
input_ds = pywapor.pre_et_look.main(folder, latlim, lonlim, timelim, sources = et_look_config)

Next we add the data we want to include before our call to pywapor.et_look.main. Check the documentation of `xarray <https://docs.xarray.dev/en/stable/#>`__ to learn more on how to do this.

input_ds["t_air"] = xr.DataArray(...)
input_ds["u2m"] = xr.DataArray(...)
# ...
# etc.

Then you run et_look.

output_ds = pywapor.et_look.main(input_ds)