{ "cells": [ { "cell_type": "markdown", "id": "1a1aa8bd-d134-452e-b430-ab4660b3b06a", "metadata": {}, "source": [ "# Step by step Tutorial\n", "\n", "In this tutorial we will go throught the entire implementation of spy4cast MCA methodology\n", "\n", "On the first section we will do everything using xarray and on the second one we will introduce the abstractions that spy4cast implements to enhace productivity" ] }, { "cell_type": "markdown", "id": "bcac032e-2dac-4f05-9d26-f111f927e4f0", "metadata": {}, "source": [ "[Using Xarray](#xarray)\n", "- [Configuration](#x-conf)\n", "- [Preprocessing](#x-pre)\n", "- [MCA](#x-mca)\n", "- [Plot results](#x-plot)\n", "\n", "[Using Spy4Cast](#spy4cast)\n", "- [Configuration](#s-conf)\n", "- [Preprocessing](#s-pre)\n", "- [MCA](#s-mca)\n", "- [Plot results](#x-plot)\n", "- [Crossvalidation](#s-cross)\n" ] }, { "cell_type": "markdown", "id": "a82b0adf-9e9f-4f61-9c76-5d9e41e1d669", "metadata": {}, "source": [ "## Using xarray \n", "\n", "In this case of study we will go through the relation between sea surface temperature in the Atlantic ocean and the clorophyl concentration in the shouth coast of the Iberian Peninsula. It is the code used to replicate the results from [Lopez-Parages, Jorge, et al. 2022](https://doi.org/10.3389/fmars.2022.931832)." ] }, { "cell_type": "code", "execution_count": 1, "id": "a5ae3158-2dc5-49f2-828f-7187e20ed503", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/pabloduran/opt/anaconda3/envs/Spy4Cast/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.26.4\n", " warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n" ] } ], "source": [ "import xarray as xr\n", "import os\n", "import numpy as np\n", "import scipy\n", "from scipy import signal, sparse, stats\n", "import matplotlib.pyplot as plt\n", "from matplotlib import gridspec\n", "import cartopy.crs as ccrs" ] }, { "cell_type": "markdown", "id": "4e75afdb-ccf0-423e-8c12-34a81a2ee72b", "metadata": {}, "source": [ "### 1. Configuration " ] }, { "cell_type": "markdown", "id": "49ee0b80-99ca-4197-9a50-831ec21ea666", "metadata": {}, "source": [ "Load the data:" ] }, { "cell_type": "code", "execution_count": 2, "id": "3c057ffb-5b78-4671-9e28-3f597b7e488b", "metadata": {}, "outputs": [], "source": [ "DATASET_DIR = \"datasets\"\n", "\n", "PREDICTOR_NAME = \"HadISST_sst_chopped.nc\"\n", "PREDICTAND_NAME = \"chl_1km_monthly_Sep1997_Dec2020_sliced.nc\"" ] }, { "cell_type": "code", "execution_count": 3, "id": "4d7dd8b7-5efa-4e50-a8d4-7bf7481569dd", "metadata": {}, "outputs": [], "source": [ "sst_ds = xr.open_dataset(os.path.join(DATASET_DIR, PREDICTOR_NAME))\n", "sst = sst_ds[\"sst\"]\n", "chl_ds = xr.open_dataset(os.path.join(DATASET_DIR, PREDICTAND_NAME))\n", "chl = chl_ds[\"CHL\"]" ] }, { "cell_type": "markdown", "id": "66805c70-3d3d-4072-ab58-508b91df6f86", "metadata": {}, "source": [ "Slice the data:\n", "\n", "Predictor Region: Atlantic Ocean (lat = from 5º to 25º, lon = from -75º to -20º)\n", "Predictor Time: 1997 -> 2019\n", "Predictor Season: AUG, SEP\n", "\n", "Predictand Region: Southearn Coast of Iberian Peninsula (lat = from 36º to 37º, lon = from -5.3º to -2º)\n", "Predictand Time: 1998 -> 2020\n", "Predictand Season: MAR, APR" ] }, { "cell_type": "code", "execution_count": 4, "id": "23bc982e-5ef9-4274-9e40-3857e7e84707", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
<xarray.DataArray 'sst' (time: 605, latitude: 180, longitude: 360)> Size: 157MB\n",
"[39204000 values with dtype=float32]\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 5kB 1970-01-16T12:00:00 ... 2020-05-16T1...\n",
" * latitude (latitude) float32 720B 89.5 88.5 87.5 86.5 ... -87.5 -88.5 -89.5\n",
" * longitude (longitude) float32 1kB -179.5 -178.5 -177.5 ... 178.5 179.5\n",
"Attributes:\n",
" standard_name: sea_surface_temperature\n",
" long_name: sst\n",
" units: C\n",
" cell_methods: time: lat: lon: mean<xarray.DataArray 'sst' (time: 46, latitude: 20, longitude: 55)> Size: 202kB\n",
"[50600 values with dtype=float32]\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 368B 1997-08-16T12:00:00 ... 2019-09-16T...\n",
" * latitude (latitude) float32 80B 24.5 23.5 22.5 21.5 ... 8.5 7.5 6.5 5.5\n",
" * longitude (longitude) float32 220B -74.5 -73.5 -72.5 ... -22.5 -21.5 -20.5\n",
"Attributes:\n",
" standard_name: sea_surface_temperature\n",
" long_name: sst\n",
" units: C\n",
" cell_methods: time: lat: lon: mean<xarray.DataArray 'CHL' (time: 92, lat: 296, lon: 257)> Size: 28MB\n",
"[6998624 values with dtype=float32]\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 736B 1998-02-01 1998-03-01 ... 2020-05-01\n",
" * lat (lat) float32 1kB 35.01 35.02 35.03 35.04 ... 37.97 37.98 37.99\n",
" * lon (lon) float32 1kB -5.39 -5.377 -5.364 ... -2.126 -2.114 -2.101\n",
" year (time) int64 736B ...\n",
"Attributes:\n",
" long_name: Monthly Multi-sensor and multi water-type Chlorophyll a c...\n",
" standard_name: mass_concentration_of_chlorophyll_a_in_sea_water\n",
" type: surface\n",
" units: milligram m^-3\n",
" cell_methods: time: mean (interval: 1 month comment: sampled instantan...\n",
" valid_min: 0.01\n",
" valid_max: 100.0\n",
" source: MODISA Aqua - Level2,VIIRSJ JPSS-1 - Level2,VIIRSN Suomi-...\n",
" _ChunkSizes: [ 1 790 1654]<xarray.DataArray 'CHL' (time: 46, lat: 99, lon: 250)> Size: 5MB\n",
"[1138500 values with dtype=float32]\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 368B 1998-03-01 1998-04-01 ... 2020-04-01\n",
" * lat (lat) float32 396B 36.0 36.01 36.02 36.03 ... 36.97 36.98 36.99\n",
" * lon (lon) float32 1kB -5.3 -5.287 -5.274 ... -2.126 -2.114 -2.101\n",
" year (time) int64 368B ...\n",
"Attributes:\n",
" long_name: Monthly Multi-sensor and multi water-type Chlorophyll a c...\n",
" standard_name: mass_concentration_of_chlorophyll_a_in_sea_water\n",
" type: surface\n",
" units: milligram m^-3\n",
" cell_methods: time: mean (interval: 1 month comment: sampled instantan...\n",
" valid_min: 0.01\n",
" valid_max: 100.0\n",
" source: MODISA Aqua - Level2,VIIRSJ JPSS-1 - Level2,VIIRSN Suomi-...\n",
" _ChunkSizes: [ 1 790 1654]<xarray.DataArray 'sst' (year: 23, latitude: 20, longitude: 55)> Size: 101kB\n",
"array([[[29.43972 , 29.324703, 29.236721, ..., 24.867073, 24.746378,\n",
" 24.540201],\n",
" [29.501165, 29.34866 , 29.240013, ..., 24.92178 , 24.772491,\n",
" 24.51123 ],\n",
" [29.448833, 29.289053, 29.198196, ..., 24.95534 , 24.78917 ,\n",
" 24.522488],\n",
" ...,\n",
" [ nan, nan, nan, ..., 27.690258, 27.562199,\n",
" 27.474775],\n",
" [ nan, nan, nan, ..., 27.65865 , 27.54871 ,\n",
" 27.473726],\n",
" [ nan, nan, nan, ..., 27.5221 , 27.39844 ,\n",
" 27.313917]],\n",
"\n",
" [[29.418564, 29.32129 , 29.19013 , ..., 24.786026, 24.727936,\n",
" 24.534008],\n",
" [29.48762 , 29.362225, 29.229086, ..., 24.934086, 24.766254,\n",
" 24.50972 ],\n",
" [29.513603, 29.374004, 29.264664, ..., 25.092678, 24.783533,\n",
" 24.475266],\n",
"...\n",
" [ nan, nan, nan, ..., 27.841267, 27.778168,\n",
" 27.735716],\n",
" [ nan, nan, nan, ..., 27.92116 , 27.866674,\n",
" 27.815903],\n",
" [ nan, nan, nan, ..., 27.870008, 27.823608,\n",
" 27.764965]],\n",
"\n",
" [[29.059797, 29.13901 , 29.207676, ..., 25.025206, 24.790546,\n",
" 24.496622],\n",
" [29.046814, 29.10406 , 29.170261, ..., 25.064781, 24.804836,\n",
" 24.45037 ],\n",
" [29.062769, 29.085262, 29.128931, ..., 25.093243, 24.825397,\n",
" 24.447556],\n",
" ...,\n",
" [ nan, nan, nan, ..., 27.752783, 27.660511,\n",
" 27.626997],\n",
" [ nan, nan, nan, ..., 27.764326, 27.68687 ,\n",
" 27.625046],\n",
" [ nan, nan, nan, ..., 27.736523, 27.646976,\n",
" 27.557606]]], dtype=float32)\n",
"Coordinates:\n",
" * latitude (latitude) float32 80B 24.5 23.5 22.5 21.5 ... 8.5 7.5 6.5 5.5\n",
" * longitude (longitude) float32 220B -74.5 -73.5 -72.5 ... -22.5 -21.5 -20.5\n",
" * year (year) int64 184B 1997 1998 1999 2000 ... 2016 2017 2018 2019\n",
"Attributes:\n",
" standard_name: sea_surface_temperature\n",
" long_name: sst\n",
" units: C\n",
" cell_methods: time: lat: lon: mean<xarray.DataArray 'CHL' (year: 23, lat: 99, lon: 250)> Size: 2MB\n",
"array([[[0.21465224, 0.2473787 , 0.22719526, ..., 0.12090172,\n",
" 0.12297925, 0.1198298 ],\n",
" [0.23239255, 0.22782484, 0.2263079 , ..., 0.12343679,\n",
" 0.11864622, 0.11867309],\n",
" [0.2286636 , 0.20803559, 0.20788139, ..., 0.12761016,\n",
" 0.12068577, 0.12094864],\n",
" ...,\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan],\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan],\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan]],\n",
"\n",
" [[0.16464871, 0.18137816, 0.18113448, ..., 0.310634 ,\n",
" 0.30567884, 0.29548663],\n",
" [0.18559793, 0.18445052, 0.18071373, ..., 0.27648756,\n",
" 0.29144764, 0.27454275],\n",
" [0.18167377, 0.17825404, 0.19277668, ..., 0.29028517,\n",
" 0.28167462, 0.28074667],\n",
"...\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan],\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan],\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan]],\n",
"\n",
" [[0.13607499, 0.13884723, 0.13911843, ..., 0.11104058,\n",
" 0.11072575, 0.11365105],\n",
" [0.14470449, 0.1443741 , 0.14499024, ..., 0.12129846,\n",
" 0.11603758, 0.11563934],\n",
" [0.16214469, 0.1638612 , 0.17014524, ..., 0.14019011,\n",
" 0.12605241, 0.11876828],\n",
" ...,\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan],\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan],\n",
" [ nan, nan, nan, ..., nan,\n",
" nan, nan]]], dtype=float32)\n",
"Coordinates:\n",
" * lat (lat) float32 396B 36.0 36.01 36.02 36.03 ... 36.97 36.98 36.99\n",
" * lon (lon) float32 1kB -5.3 -5.287 -5.274 ... -2.126 -2.114 -2.101\n",
" * year (year) int64 184B 1998 1999 2000 2001 2002 ... 2017 2018 2019 2020\n",
"Attributes:\n",
" long_name: Monthly Multi-sensor and multi water-type Chlorophyll a c...\n",
" standard_name: mass_concentration_of_chlorophyll_a_in_sea_water\n",
" type: surface\n",
" units: milligram m^-3\n",
" cell_methods: time: mean (interval: 1 month comment: sampled instantan...\n",
" valid_min: 0.01\n",
" valid_max: 100.0\n",
" source: MODISA Aqua - Level2,VIIRSJ JPSS-1 - Level2,VIIRSN Suomi-...\n",
" _ChunkSizes: [ 1 790 1654]