{ "cells": [ { "cell_type": "markdown", "id": "4197519b-44fa-49b2-aec8-3fb402248bab", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Copyright CNES\n", "\n", "# Plot a lake's geometries from multiple Lake/River Single Pass products\n", "\n", "In this notebook, we show how to read the SWOT-HR River or Lake Single Pass vector products with geopandas and dask geopandas and how to represent a variable in time.\n", "The example is based on Lake Single Pass Prior products, but it would be the same methodology for all vector products relying on a feature ID" ] }, { "cell_type": "markdown", "id": "f0c5902d-a63b-4a1e-845b-666a7051c4eb", "metadata": {}, "source": [ "## Libraries" ] }, { "cell_type": "code", "execution_count": 1, "id": "da42166b-989a-4a5f-a2bf-4b7c71516808", "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "import glob\n", "\n", "import numpy as np\n", "import geopandas as gpd\n", "from tqdm import tqdm\n", "\n", "from dask import dataframe as dd\n", "import dask_geopandas as dgpd\n", "from datetime import datetime as dt\n" ] }, { "cell_type": "markdown", "id": "ec548ad4-dfa1-43a3-8d54-db049666c6c4", "metadata": {}, "source": [ "## Select all Lake Single-Pass products within our directory" ] }, { "cell_type": "markdown", "id": "345c3d75-f8c4-416e-91d0-f0438ceb38cb", "metadata": {}, "source": [ "### Filter by filename pattern" ] }, { "cell_type": "code", "execution_count": 2, "id": "d0884d32-63a5-4511-8186-e411f1592c23", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "49\n" ] } ], "source": [ "filename_list = glob.glob(\"../docs/data/swot/SWOT_L2_HR_LakeSP_Prior/*PIC0_01/*PIC0_01.zip\")\n", "print(len(filename_list))\n", "# one can use zip files, but note it is slower" ] }, { "cell_type": "markdown", "id": "3cee9499", "metadata": {}, "source": [ "# Load all data\n", "now we want to load all data from all file in lazy mode, otherwise it will not fit into RAM. \n", "For this, we will iteratively read all files with geopandas, and store data in a dask dataframe. \n", "The drawback of this approach is that you cannot keep the geometries. However, if you are fine with that, it is very efficient. \n", "One could hack that by changing geometries into strings and, once the data has been filter, converting back to geometries and a geopandas.Dataframe" ] }, { "cell_type": "code", "execution_count": 3, "id": "bb55e839", "metadata": {}, "outputs": [], "source": [ "def load_layers(files):\n", " \n", " ddf = None\n", " for file in tqdm(files):\n", "\n", " gdf = gpd.read_file(\n", " file,\n", " engine='pyogrio',\n", " use_arrow=True,\n", " ) \n", " \n", " ddf_tmp = dgpd.from_geopandas(gdf, chunksize=10000)\n", " \n", " if ddf is None:\n", " ddf = ddf_tmp\n", " else:\n", " ddf = dd.concat([ddf, ddf_tmp ], ignore_order=True)\n", "\n", " del ddf_tmp, gdf\n", "\n", " return ddf" ] }, { "cell_type": "code", "execution_count": 4, "id": "9dcdc3f4", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "100%|██████████| 49/49 [00:38<00:00, 1.27it/s]\n" ] }, { "data": { "text/html": [ "
\n", " | lake_id | \n", "reach_id | \n", "obs_id | \n", "overlap | \n", "n_overlap | \n", "time | \n", "time_tai | \n", "time_str | \n", "wse | \n", "wse_u | \n", "wse_r_u | \n", "wse_std | \n", "area_total | \n", "area_tot_u | \n", "area_detct | \n", "area_det_u | \n", "layovr_val | \n", "xtrk_dist | \n", "ds1_l | \n", "ds1_l_u | \n", "ds1_q | \n", "ds1_q_u | \n", "ds2_l | \n", "ds2_l_u | \n", "ds2_q | \n", "ds2_q_u | \n", "quality_f | \n", "dark_frac | \n", "ice_clim_f | \n", "ice_dyn_f | \n", "partial_f | \n", "xovr_cal_q | \n", "geoid_hght | \n", "solid_tide | \n", "load_tidef | \n", "load_tideg | \n", "pole_tide | \n", "dry_trop_c | \n", "wet_trop_c | \n", "iono_c | \n", "xovr_cal_c | \n", "lake_name | \n", "p_res_id | \n", "p_lon | \n", "p_lat | \n", "p_ref_wse | \n", "p_ref_area | \n", "p_date_t0 | \n", "p_ds_t0 | \n", "p_storage | \n", "geometry | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
npartitions=165 | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
\n", " | string | \n", "string | \n", "string | \n", "string | \n", "string | \n", "float64 | \n", "float64 | \n", "string | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "int32 | \n", "float64 | \n", "int32 | \n", "int32 | \n", "int32 | \n", "int32 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "string | \n", "int32 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "string | \n", "float64 | \n", "float64 | \n", "geometry | \n", "
\n", " | ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
\n", " | ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
\n", " | ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
\n", " | lake_id | \n", "reach_id | \n", "obs_id | \n", "overlap | \n", "n_overlap | \n", "time | \n", "time_tai | \n", "time_str | \n", "wse | \n", "wse_u | \n", "... | \n", "p_res_id | \n", "p_lon | \n", "p_lat | \n", "p_ref_wse | \n", "p_ref_area | \n", "p_date_t0 | \n", "p_ds_t0 | \n", "p_storage | \n", "geometry | \n", "date | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
24386 | \n", "2940021962 | \n", "no_data | \n", "294094R099897 | \n", "0 | \n", "1 | \n", "7.617207e+08 | \n", "7.617207e+08 | \n", "2024-02-20T05:05:48Z | \n", "117.685 | \n", "0.791 | \n", "... | \n", "-99999999 | \n", "44.416265 | \n", "34.690836 | \n", "-1.000000e+12 | \n", "119.034902 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "MULTIPOLYGON (((44.56503 34.55532, 44.56499 34... | \n", "2024-02-20 05:05:48 | \n", "
19967 | \n", "2940021962 | \n", "no_data | \n", "294094L099732;294094L099863;294094L099955;2940... | \n", "42;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0... | \n", "58 | \n", "7.626653e+08 | \n", "7.626653e+08 | \n", "2024-03-02T03:28:37Z | \n", "113.316 | \n", "0.030 | \n", "... | \n", "-99999999 | \n", "44.416265 | \n", "34.690836 | \n", "-1.000000e+12 | \n", "119.034902 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "MULTIPOLYGON (((44.34098 34.84831, 44.34081 34... | \n", "2024-03-02 03:28:37 | \n", "
27266 | \n", "2940021962 | \n", "no_data | \n", "294094L099739;294094L099973;294094L099832;2940... | \n", "42;1;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0... | \n", "60 | \n", "7.644680e+08 | \n", "7.644680e+08 | \n", "2024-03-23T00:13:40Z | \n", "114.799 | \n", "0.053 | \n", "... | \n", "-99999999 | \n", "44.416265 | \n", "34.690836 | \n", "-1.000000e+12 | \n", "119.034902 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "MULTIPOLYGON (((44.26363 34.82676, 44.26342 34... | \n", "2024-03-23 00:13:40 | \n", "
11426 | \n", "2940021962 | \n", "no_data | \n", "294094L099894;294094L099975;294094L099435;2940... | \n", "42;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0... | \n", "57 | \n", "7.680734e+08 | \n", "7.680734e+08 | \n", "2024-05-03T17:43:47Z | \n", "119.463 | \n", "0.120 | \n", "... | \n", "-99999999 | \n", "44.416265 | \n", "34.690836 | \n", "-1.000000e+12 | \n", "119.034902 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "MULTIPOLYGON (((44.27089 34.81043, 44.27067 34... | \n", "2024-05-03 17:43:47 | \n", "
8346 | \n", "2940021962 | \n", "no_data | \n", "294094L099811;294094L099884;294094L099963;2940... | \n", "42;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0... | \n", "44 | \n", "7.698761e+08 | \n", "7.698761e+08 | \n", "2024-05-24T14:28:53Z | \n", "113.857 | \n", "0.021 | \n", "... | \n", "-99999999 | \n", "44.416265 | \n", "34.690836 | \n", "-1.000000e+12 | \n", "119.034902 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "MULTIPOLYGON (((44.33459 34.84880, 44.33441 34... | \n", "2024-05-24 14:28:53 | \n", "
5 rows × 52 columns
\n", "