{ "cells": [ { "cell_type": "markdown", "id": "4197519b-44fa-49b2-aec8-3fb402248bab", "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "source": [ "Copyright CNES\n", "\n", "# Plot a lake timeseries from multiple Lake/River Single Pass products\n", "\n", "In this notebook, we show how to read the SWOT-HR River or Lake Single Pass vector products with geopandas and dask dataframes and how to represent a variable in time.\n", "The example is based on Lake Single Pass Prior products, but it would be the same methodology for all vector products relying on a feature ID" ] }, { "cell_type": "markdown", "id": "f0c5902d-a63b-4a1e-845b-666a7051c4eb", "metadata": {}, "source": [ "## Libraries" ] }, { "cell_type": "code", "execution_count": 1, "id": "da42166b-989a-4a5f-a2bf-4b7c71516808", "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "import glob\n", "\n", "import numpy as np\n", "import geopandas as gpd\n", "from tqdm import tqdm\n", "\n", "from dask import dataframe as dd\n", "from dask.distributed import Client\n", "\n", "from datetime import datetime as dt\n" ] }, { "cell_type": "markdown", "id": "ec548ad4-dfa1-43a3-8d54-db049666c6c4", "metadata": {}, "source": [ "## Select all Lake Single-Pass products within our directory" ] }, { "cell_type": "markdown", "id": "345c3d75-f8c4-416e-91d0-f0438ceb38cb", "metadata": {}, "source": [ "### Filter by filename pattern" ] }, { "cell_type": "code", "execution_count": 2, "id": "d0884d32-63a5-4511-8186-e411f1592c23", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "49\n" ] } ], "source": [ "filename_list = glob.glob(\"../docs/data/swot/SWOT_L2_HR_LakeSP_Prior/*PIC0_01/*PIC0_01.zip\")\n", "print(len(filename_list))\n", "# one can use zip files, but note it is slower" ] }, { "cell_type": "markdown", "id": "3cee9499", "metadata": {}, "source": [ "# Load all data\n", "now we want to load all data from all file in lazy mode, otherwise it will not fit into RAM. \n", "For this, we will iteratively read all files with geopandas, and store data in a dask dataframe. \n", "With this method, you drop the geometries. If you are fine with that, it is very efficient, otherwise, you can use dask-geopandas instead, see other notebook in the same section." ] }, { "cell_type": "code", "execution_count": 3, "id": "bb55e839", "metadata": {}, "outputs": [], "source": [ "\n", "def get_df(f):\n", " gdf = gpd.read_file(\n", " f,\n", " engine='pyogrio',\n", " use_arrow=True,\n", " )\n", " # RIP geometry or don't drop and use dask_geopandas.from_geopandas instead of dd.from_pandas\n", " gdf = gdf.drop(columns=['geometry']) \n", " return gdf\n", "\n", "\n", "def load_layers(files):\n", " client = Client()\n", " futures = client.map(get_df, files)\n", "\n", " ddf = dd.from_delayed(futures)\n", "\n", " return ddf" ] }, { "cell_type": "code", "execution_count": 4, "id": "9dcdc3f4", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | lake_id | \n", "reach_id | \n", "obs_id | \n", "overlap | \n", "n_overlap | \n", "time | \n", "time_tai | \n", "time_str | \n", "wse | \n", "wse_u | \n", "wse_r_u | \n", "wse_std | \n", "area_total | \n", "area_tot_u | \n", "area_detct | \n", "area_det_u | \n", "layovr_val | \n", "xtrk_dist | \n", "ds1_l | \n", "ds1_l_u | \n", "ds1_q | \n", "ds1_q_u | \n", "ds2_l | \n", "ds2_l_u | \n", "ds2_q | \n", "ds2_q_u | \n", "quality_f | \n", "dark_frac | \n", "ice_clim_f | \n", "ice_dyn_f | \n", "partial_f | \n", "xovr_cal_q | \n", "geoid_hght | \n", "solid_tide | \n", "load_tidef | \n", "load_tideg | \n", "pole_tide | \n", "dry_trop_c | \n", "wet_trop_c | \n", "iono_c | \n", "xovr_cal_c | \n", "lake_name | \n", "p_res_id | \n", "p_lon | \n", "p_lat | \n", "p_ref_wse | \n", "p_ref_area | \n", "p_date_t0 | \n", "p_ds_t0 | \n", "p_storage | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
npartitions=49 | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
\n", " | object | \n", "object | \n", "object | \n", "object | \n", "object | \n", "float64 | \n", "float64 | \n", "object | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "int32 | \n", "float64 | \n", "int32 | \n", "int32 | \n", "int32 | \n", "int32 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "object | \n", "int32 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "float64 | \n", "object | \n", "float64 | \n", "float64 | \n", "
\n", " | ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
\n", " | ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
\n", " | ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
\n", " | lake_id | \n", "reach_id | \n", "obs_id | \n", "overlap | \n", "n_overlap | \n", "time | \n", "time_tai | \n", "time_str | \n", "wse | \n", "wse_u | \n", "... | \n", "xovr_cal_c | \n", "lake_name | \n", "p_res_id | \n", "p_lon | \n", "p_lat | \n", "p_ref_wse | \n", "p_ref_area | \n", "p_date_t0 | \n", "p_ds_t0 | \n", "p_storage | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
27845 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294090L000211;294090L000584;294090L000586;2940... | \n", "6;1;1;0;0;0;0;0;0 | \n", "9 | \n", "7.618071e+08 | \n", "7.618072e+08 | \n", "2024-02-21 05:05:43 | \n", "308.797 | \n", "0.060 | \n", "... | \n", "6.006220e-01 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
29549 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294090R000212;294090R000175;294090R000187;2940... | \n", "2;1;1;1;0;0;0;0;0 | \n", "9 | \n", "7.626653e+08 | \n", "7.626653e+08 | \n", "2024-03-02 03:27:58 | \n", "308.357 | \n", "0.006 | \n", "... | \n", "-2.362345e+00 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
22681 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294219R000180;294219R000160;294219R000194;2942... | \n", "26;14;3;1;1;0;0;0;0;0 | \n", "10 | \n", "7.627129e+08 | \n", "7.627129e+08 | \n", "2024-03-02 16:41:13 | \n", "308.991 | \n", "0.026 | \n", "... | \n", "-9.724420e-01 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
29593 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294090R000695;294090R000580;294090R000609;2940... | \n", "2;1;1;1;0;0;0;0;0;0;0 | \n", "11 | \n", "7.644680e+08 | \n", "7.644680e+08 | \n", "2024-03-23 00:13:01 | \n", "311.566 | \n", "0.006 | \n", "... | \n", "-8.630990e-01 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
27870 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294090L000101;294090L000069;294090L000100;2940... | \n", "4;2;2;2;1;1;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;... | \n", "26 | \n", "7.654126e+08 | \n", "7.654126e+08 | \n", "2024-04-02 22:35:50 | \n", "315.545 | \n", "0.004 | \n", "... | \n", "-1.000000e+12 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
27873 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294090L000068;294090L000127;294090L000258;2940... | \n", "5;1;1;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;... | \n", "27 | \n", "7.672153e+08 | \n", "7.672153e+08 | \n", "2024-04-23 19:20:56 | \n", "320.930 | \n", "0.007 | \n", "... | \n", "6.728950e-01 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
29570 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294090R000446;294090R000379;294090R000393;2940... | \n", "2;1;1;1;0;0;0;0;0;0;0 | \n", "11 | \n", "7.680734e+08 | \n", "7.680734e+08 | \n", "2024-05-03 17:43:09 | \n", "323.098 | \n", "0.003 | \n", "... | \n", "6.785700e-02 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
22680 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294219R000608;294219R000694;294219R000682;2942... | \n", "2;1;1;1;1;0;0;0;0;0;0;0;0;0;0;0 | \n", "16 | \n", "7.681210e+08 | \n", "7.681210e+08 | \n", "2024-05-04 06:56:25 | \n", "323.391 | \n", "0.006 | \n", "... | \n", "4.569000e-03 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
29544 | \n", "2940020983 | \n", "29469100171;29469100181;29469100161;2946910014... | \n", "294090R000222;294090R000171;294090R000182;2940... | \n", "2;1;1;1;0;0;0;0;0;0;0;0;0 | \n", "13 | \n", "7.698761e+08 | \n", "7.698761e+08 | \n", "2024-05-24 14:28:15 | \n", "325.121 | \n", "0.003 | \n", "... | \n", "7.896590e-01 | \n", "no_data | \n", "-99999999 | \n", "42.774543 | \n", "36.726391 | \n", "-1.000000e+12 | \n", "221.719 | \n", "no_data | \n", "-1.000000e+12 | \n", "-1.000000e+12 | \n", "
9 rows × 50 columns
\n", "