Load Iso 2k¶
load data from Iso 2k (https://doi.org/10.5194/essd-12-2261-2020)
Data downloaded from LiPDverse: https://lipdverse.org/iso2k/current_version/, downloaded 31/10/2025 by LL
Author: Lucie Luecke (LL)
30/10/2025 LL: updated Iso2k to version 1.1.2
24/10/2025 LL: tidied up and streamlined for documentation and publication
21/11/2024 LL: added csv saving of compact dataframe, removed redundant output.
Here we extract a dataframe with the following columns:
archiveTypedataSetNamedatasetIdgeo_meanElevgeo_meanLatgeo_meanLongeo_siteNameinterpretation_direction(new in v2.0)interpretation_variableinterpretation_variableDetailinterpretation_seasonality(new in v2.0)originalDataURLoriginalDatabasepaleoData_notespaleoData_proxypaleoData_sensorSpeciespaleoData_unitspaleoData_valuespaleoData_variableNameyearyearUnits
We save a standardised compact dataframe for concatenation to DoD2k
Set up working environment¶
Make sure the repo_root is added correctly, it should be: your_root_dir/dod2k This should be the working directory throughout this notebook (and all other notebooks).
In [1]:
Copied!
%load_ext autoreload
%autoreload 2
import sys
import os
from pathlib import Path
# Add parent directory to path (works from any notebook in notebooks/)
# the repo_root should be the parent directory of the notebooks folder
init_dir = Path().resolve()
# Determine repo root
if init_dir.name == 'dod2k': repo_root = init_dir
elif init_dir.parent.name == 'dod2k': repo_root = init_dir.parent
else: raise Exception('Please review the repo root structure (see first cell).')
# Update cwd and path only if needed
if os.getcwd() != str(repo_root):
os.chdir(repo_root)
if str(repo_root) not in sys.path:
sys.path.insert(0, str(repo_root))
print(f"Repo root: {repo_root}")
if str(os.getcwd())==str(repo_root):
print(f"Working directory matches repo root. ")
%load_ext autoreload
%autoreload 2
import sys
import os
from pathlib import Path
# Add parent directory to path (works from any notebook in notebooks/)
# the repo_root should be the parent directory of the notebooks folder
init_dir = Path().resolve()
# Determine repo root
if init_dir.name == 'dod2k': repo_root = init_dir
elif init_dir.parent.name == 'dod2k': repo_root = init_dir.parent
else: raise Exception('Please review the repo root structure (see first cell).')
# Update cwd and path only if needed
if os.getcwd() != str(repo_root):
os.chdir(repo_root)
if str(repo_root) not in sys.path:
sys.path.insert(0, str(repo_root))
print(f"Repo root: {repo_root}")
if str(os.getcwd())==str(repo_root):
print(f"Working directory matches repo root. ")
Repo root: /home/jupyter-lluecke/dod2k Working directory matches repo root.
In [2]:
Copied!
# Import packages
import lipd
import pandas as pd
import numpy as np
from dod2k_utilities import ut_functions as utf # contains utility functions
from dod2k_utilities import ut_plot as uplt # contains plotting functions
# Import packages
import lipd
import pandas as pd
import numpy as np
from dod2k_utilities import ut_functions as utf # contains utility functions
from dod2k_utilities import ut_plot as uplt # contains plotting functions
Load source data¶
In order to get the source data, run the cell below. This will download a series of LiPD files into the directory lipdfiles
Alternatively skip the cell and directly use the files as provided in this directory (see cell below next).
In [3]:
Copied!
# # Download the file (use -O to specify output filename)
# !wget -O data/iso2k/iso2k1_1_2.zip https://lipdverse.org/iso2k/current_version/iso2k1_1_2.zip
# # Unzip to the correct destination
# !unzip data/iso2k/iso2k1_1_2.zip -d data/iso2k/iso2k1_1_2
# # Download the file (use -O to specify output filename)
# !wget -O data/iso2k/iso2k1_1_2.zip https://lipdverse.org/iso2k/current_version/iso2k1_1_2.zip
# # Unzip to the correct destination
# !unzip data/iso2k/iso2k1_1_2.zip -d data/iso2k/iso2k1_1_2
Load downloaded LiPD files and extract data and metadata from the directory
In [4]:
Copied!
# load LiPD files from the given directory
D = lipd.readLipd(str(repo_root)+'/data/iso2k/iso2k1_1_2/');
TS = lipd.extractTs(D);
len(TS)
os.chdir(repo_root)
# load LiPD files from the given directory
D = lipd.readLipd(str(repo_root)+'/data/iso2k/iso2k1_1_2/');
TS = lipd.extractTs(D);
len(TS)
os.chdir(repo_root)
Disclaimer: LiPD files may be updated and modified to adhere to standards Found: 509 LiPD file(s) reading: CO14WUCL.lpd reading: LS96CUPU.lpd reading: TR18GRPM.lpd reading: IC00OE03.lpd reading: IC02GR17.lpd reading: IC00OEKC.lpd reading: IC03HORB.lpd reading: MS09TCIS.lpd reading: CO95TUNG.lpd reading: CO17MUMA.lpd reading: IC16WE30.lpd reading: LS10WRNA.lpd reading: IC10VICR.lpd reading: MS09DOIP.lpd reading: LS91BEWA.lpd reading: IC13ST12.lpd reading: CO98SWPR.lpd reading: CO04BAFI.lpd reading: SP11STBR.lpd reading: CO17DESC03A.lpd reading: LS07EAGO.lpd reading: IC13STSP.lpd reading: LS13WASU.lpd reading: LS16STPA.lpd reading: LS14LASO.lpd reading: IC09VIA8.lpd reading: CO99DRGB.lpd reading: CO00URMA.lpd reading: IC11LAB4.lpd reading: IC96MOSS.lpd reading: IC06VING.lpd reading: LS07LAHA.lpd reading: IC00STHN.lpd reading: IC99IS89.lpd reading: LS14TAKA.lpd reading: LS09LAEL.lpd reading: SP08HUCN.lpd reading: IC97THGL.lpd reading: CO09NUTB.lpd reading: MS10JTMA.lpd reading: MS96LKSS.lpd reading: SP10BEIN.lpd reading: LS15AIKA.lpd reading: IC13ST01.lpd reading: CO04KIVA.lpd reading: TR13JOAR.lpd reading: LS04JOCH.lpd reading: LS15DOSA.lpd reading: IC02STTA.lpd reading: LS09SCMI.lpd reading: MS11KDMS.lpd reading: IC16WE22.lpd reading: SP13ASBC.lpd reading: MS09TRFD.lpd reading: TR16LAAN.lpd reading: CO13HENG.lpd reading: SP12CACN.lpd reading: CO96QUVA.lpd reading: LS11TICH.lpd reading: LS10STSP.lpd reading: LS13YUSA.lpd reading: IC14VL1K.lpd reading: IC00OE16.lpd reading: SP09GRID.lpd reading: IC00STTY.lpd reading: IC13ST03.lpd reading: SP10MEMX.lpd reading: IC15GANK.lpd reading: CO97CHSY.lpd reading: SP07ASUS.lpd reading: CO04ZIMG.lpd reading: CO98EVXM.lpd reading: TR17EDSW.lpd reading: IC85LAD2.lpd reading: IC10VISD.lpd reading: TR11SAHU00.lpd reading: IC16WE20.lpd reading: IC02MUBI.lpd reading: TR13POMA.lpd reading: LS91HOMI.lpd reading: SP07FLOM.lpd reading: SP07DEUS.lpd reading: LS15LAOH.lpd reading: IC13STWD.lpd reading: CO14ZIMG.lpd reading: IW15MELD.lpd reading: MS99LWSC.lpd reading: IC12WACD.lpd reading: LS05ANJE.lpd reading: SP13PAVU.lpd reading: IC02HEWD.lpd reading: IC99OE10.lpd reading: IC04FIML.lpd reading: MS15VIBC.lpd reading: IC13ST91.lpd reading: IC09VICC.lpd reading: IC16WE17.lpd reading: CO17DESC04A.lpd reading: CO14OSPA.lpd reading: CO06LIFI.lpd reading: CO14ZIHO.lpd reading: IC96ISE9.lpd reading: TR16LAFO.lpd reading: IC05ISAU.lpd reading: LS15SHNE.lpd reading: LS14ATJU.lpd reading: IC13ST15.lpd reading: CO99SWFB.lpd reading: IC99OE07.lpd reading: SP06DYCN.lpd reading: MS09DTNA.lpd reading: SP05MAAT.lpd reading: LS16STCL.lpd reading: IC12RHME.lpd reading: LS02HATI.lpd reading: LS10HEQI.lpd reading: CO92SHPU.lpd reading: LS12THAY.lpd reading: IC00SC29.lpd reading: LS09RUWA.lpd reading: IC06VID3.lpd reading: IC99SCFH.lpd reading: TR14KOSP.lpd reading: LS10CLTA.lpd reading: LS11ANSE.lpd reading: IC10VISA.lpd reading: IC00OE20.lpd reading: TR12BRBO.lpd reading: TR10ANMO.lpd reading: CO17LITW01A.lpd reading: CO14CATI.lpd reading: LS09SASP.lpd reading: LS06JONA.lpd reading: TR08THCO.lpd reading: LS09BAUM.lpd reading: IC06THPR.lpd reading: SP12KEBZ.lpd reading: IC13STN7.lpd reading: LS06MIBO.lpd reading: TR06TRBO.lpd reading: CO03COPM.lpd reading: SP04YUCN.lpd reading: CO93COTW.lpd reading: CO13COTB.lpd reading: GI19POYU.lpd reading: TR11GRTP.lpd reading: IC06KAS1.lpd reading: LS03HAIG.lpd reading: CO94DUUR.lpd reading: IC99OE02.lpd reading: IC12SIWG.lpd reading: CO13CABL.lpd reading: IC01JO01.lpd reading: CO11NUPM.lpd reading: CO17WUBO1G.lpd reading: TR11BAPU.lpd reading: CO17WUBO1B.lpd reading: IC00OE23.lpd reading: CO13COXM.lpd reading: IC99OE03.lpd reading: IC99OE04.lpd reading: LS15DOCH.lpd reading: MS06DLGN.lpd reading: LS13KOLA.lpd reading: MS11CSCN.lpd reading: MS04ICNI.lpd reading: LS01HOCH.lpd reading: CO06MOPE.lpd reading: IC82STVE.lpd reading: CO04MGNG.lpd reading: LS17VAWA01.lpd reading: CO96SWFB.lpd reading: IC07KADF.lpd reading: MS11SENS.lpd reading: SP06DRIT.lpd reading: SP13AYID.lpd reading: MS03LKNS.lpd reading: IC00SC18.lpd reading: MS15VIBM.lpd reading: MS11FAPM.lpd reading: LS15BHAL.lpd reading: CO02KUBE.lpd reading: CO99GUMI.lpd reading: IC16WE19.lpd reading: CO17WUBO1E.lpd reading: TR07REVI.lpd reading: CO96SWBB.lpd reading: IC16WE28.lpd reading: IC95FI87.lpd reading: SP13MCUS.lpd reading: LS14FEZA.lpd reading: LS11MOES.lpd reading: TR16WEXI.lpd reading: TR13SIKO.lpd reading: LS13LUBA.lpd reading: LS09MIKU.lpd reading: TR07RECA.lpd reading: MS14GFSS.lpd reading: CO08KIPR.lpd reading: LS12WOQU.lpd reading: LS09MAAS.lpd reading: SP10WINZ.lpd reading: IC00OE22.lpd reading: IC08VI84.lpd reading: LS02ROSA.lpd reading: CO99KUHO.lpd reading: LS16THQI01.lpd reading: ElJunco.Zhang.2014.lpd reading: ClelandLake.Steinman.2016.lpd reading: IC99OE05.lpd reading: SS05ROTN.lpd reading: IC11STTA.lpd reading: CO17WUBO1D.lpd reading: CO03CHBU.lpd reading: LS11KOMA.lpd reading: MS11KWFS.lpd reading: TR12BECO.lpd reading: LS08LICA.lpd reading: IC02GR07.lpd reading: LS11RYKA.lpd reading: LS14FOTI.lpd reading: LS00WODE.lpd reading: CO04INMI.lpd reading: LS99LASI.lpd reading: IC10VIP2.lpd reading: CO17RAPA.lpd reading: IC13STN8.lpd reading: IC17EKN3.lpd reading: CO12GOVA.lpd reading: SP08ZHCN.lpd reading: IC08HODF.lpd reading: CO18DATO01A.lpd reading: TR11MAPE.lpd reading: MS07KPPB.lpd reading: IC03HOIL.lpd reading: TR15BABO.lpd reading: LS17STNU01.lpd reading: IC10VIP1.lpd reading: IC95THHU.lpd reading: IC00EIGG.lpd reading: LS96VOAM.lpd reading: CO18RELC01A.lpd reading: CO98QUNC.lpd reading: MS03SMAS.lpd reading: IC10VIMI.lpd reading: IC06THDD.lpd reading: LS14KOSA.lpd reading: IC14MAFH.lpd reading: TR13JOJS.lpd reading: LS13BAFL.lpd reading: IC08VI87.lpd reading: MS07LSPG.lpd reading: LS06CRLA.lpd reading: TR11SIMO.lpd reading: IC00OE12.lpd reading: IC13ST13.lpd reading: CO17WUBO1C.lpd reading: TR18XUJA00.lpd reading: TR08HORA.lpd reading: SP14APPE.lpd reading: LS07ROTI.lpd reading: IC92HOML.lpd reading: IC13ST04.lpd reading: LS03RUED.lpd reading: IC98FIPE.lpd reading: TR15YOLL.lpd reading: IC00OE02.lpd reading: LS04YUWA.lpd reading: CO00COKY.lpd reading: TR19LEOU.lpd reading: IC11LIMI.lpd reading: LS16THN301.lpd reading: IC00COPR.lpd reading: CO18HECO01B.lpd reading: LS12GOBA.lpd reading: TR11BAVO.lpd reading: CO07CAFR.lpd reading: CO08GOBE.lpd reading: LS14PRLO.lpd reading: CO05BAFI.lpd reading: IC13OPAN.lpd reading: TR11XUPH.lpd reading: LS11NECA.lpd reading: LS06DENA.lpd reading: TR17GRTP.lpd reading: TR19LEBR.lpd reading: MS07RCNA.lpd reading: LS06STFO.lpd reading: IC17BERI.lpd reading: TR18XUGA00.lpd reading: LS06SHBE.lpd reading: SP08SPUS.lpd reading: IC10VISE.lpd reading: LS97HOKA.lpd reading: SP07WABR.lpd reading: LS09MAZO.lpd reading: CO94LISE.lpd reading: LS12ANBI.lpd reading: CO04PFRE.lpd reading: LS08STCR.lpd reading: IC00OE15.lpd reading: MS13RSAS.lpd reading: IC09EIBE.lpd reading: IC06WAML.lpd reading: TR13SITA.lpd reading: CO17WUBO1F.lpd reading: MS14MOLS.lpd reading: IC08VI79.lpd reading: LS10HOLO.lpd reading: IC13ST24.lpd reading: SP11BOAT.lpd reading: MS12AMMM.lpd reading: IC13ST22.lpd reading: IC13THFE.lpd reading: CO01TUNG.lpd reading: IC99ISS2.lpd reading: SP13CHPE.lpd reading: IC99OE09.lpd reading: IC11BEVL.lpd reading: IC10NAB5.lpd reading: IC00OE18.lpd reading: IC16WENG.lpd reading: LS15DENA.lpd reading: LS13STEL.lpd reading: SP13LAMX.lpd reading: CO98CHPI.lpd reading: SS03HAJA.lpd reading: LS05HOAG.lpd reading: IC17KOEL.lpd reading: IC17EK4K.lpd reading: AKIII.Terwilliger.2013.lpd reading: IC13ST05.lpd reading: CO04LIFI.lpd reading: IC94THDP.lpd reading: LS09SAWA.lpd reading: IC02THKI.lpd reading: LS12ANYE.lpd reading: CO14CABT.lpd reading: SP10CACN.lpd reading: IC00SC16.lpd reading: IC16GOBP.lpd reading: SP09REPE.lpd reading: LS89LATI.lpd reading: SP03BMIL.lpd reading: CO06QUNG.lpd reading: IC17EKPV.lpd reading: LS10HESA.lpd reading: IC02SCGN.lpd reading: CO03CHBA.lpd reading: SL13TEAK.lpd reading: SP13SUSA.lpd reading: MS15BATP.lpd reading: IC99OE08.lpd reading: CO15ABSP.lpd reading: TR17SAMA.lpd reading: MS11ANMN.lpd reading: LS06POVE.lpd reading: LS10ANBL.lpd reading: MS07JRGM.lpd reading: LS02BEPY.lpd reading: LS11BIPU.lpd reading: MS08UAPM.lpd reading: IC08THGZ.lpd reading: LS07ANMA.lpd reading: MS05LKLF.lpd reading: LS09SHJO.lpd reading: MS12CCCH.lpd reading: TR13SAWA00.lpd reading: MS02FLPC.lpd reading: TR11MAJA.lpd reading: IC16WE27.lpd reading: SP05CRBR.lpd reading: IC00OE05.lpd reading: CO13CAHN.lpd reading: IC16WE23.lpd reading: MS06DLDS.lpd reading: LS16WIGH.lpd reading: SH18CASA.lpd reading: IC00SC21.lpd reading: MS01HDAS.lpd reading: MS06DLGS.lpd reading: CO06MOTO.lpd reading: LS12STRE.lpd reading: LS09SAJU.lpd reading: IC11DILO.lpd reading: MS05ESCR.lpd reading: LS15OEPO.lpd reading: LS10GOBL.lpd reading: IC99OE01.lpd reading: SP09FLTR.lpd reading: CO04LIRA.lpd reading: SP12NOBR.lpd reading: TR19LECR.lpd reading: MS07SWGG.lpd reading: TR12SAMU.lpd reading: CO05KUBE.lpd reading: IC06VIGR.lpd reading: MS04BLCB.lpd reading: IC10VIGI.lpd reading: MS97HLCB.lpd reading: CO00KUNI.lpd reading: TR15NACA.lpd reading: CO05ASGU.lpd reading: IC99OE06.lpd reading: LS09LACA.lpd reading: LS15SHBO.lpd reading: CO99LICL.lpd reading: IC13PALD.lpd reading: IC10VIGR.lpd reading: IC11KIPW.lpd reading: CO98BOFP.lpd reading: MS03KTCB.lpd reading: SS05ROES.lpd reading: IW17OPOY.lpd reading: TR13BROA.lpd reading: SP98KUCN.lpd reading: TR13JOPC.lpd reading: SP13KAPE.lpd reading: IC00OE21.lpd reading: LS98CUPE.lpd reading: IC09KANG.lpd reading: CO18HECO01A.lpd reading: IC09VIRE.lpd reading: IC00THDS.lpd reading: LS14ZHJU.lpd reading: IC13ST21.lpd reading: IC07KARB.lpd reading: CO13DESC01A.lpd reading: IC15MANE.lpd reading: LS12KENO.lpd reading: LS06TIST.lpd reading: IC00OE13.lpd reading: IC09OPAN.lpd reading: SP12LAMX.lpd reading: MS07MMNP.lpd reading: MS14MORR.lpd reading: CO08ABSU.lpd reading: SL13TEAG.lpd reading: LS14LESA.lpd reading: IC14EKVK.lpd reading: TR19LEFC.lpd reading: LS14YAHA.lpd reading: CO04PFCH.lpd reading: IC09KAD3.lpd reading: IC12MUJR.lpd reading: IC08VISV.lpd reading: IC00SC26.lpd reading: IC93BAGI.lpd reading: LS10ZHHU.lpd reading: IC00OE11.lpd reading: IC76FIDE.lpd reading: LS11BACH.lpd reading: CO10HEIG.lpd reading: CO00DRBE.lpd reading: SP13SLNA.lpd reading: SP10SUSE.lpd reading: CO06DATZ.lpd reading: LS01HUFA.lpd reading: CO09NAKY.lpd reading: MS03BRNS.lpd reading: CO17XISC01A.lpd reading: IC08VI77.lpd reading: SP10DOCN.lpd reading: IC10VIP3.lpd reading: MS11ANMS.lpd reading: IC97GRGI.lpd reading: IC08DILO.lpd reading: MS13KWFS.lpd reading: TR08VOOL.lpd reading: SP04DEUS.lpd reading: IC00OE19.lpd reading: IC13BOCG.lpd reading: LS12STLI.lpd reading: LS99CUVA.lpd reading: CO94HEAQ.lpd reading: IC00OE14.lpd reading: CO08HEVE.lpd reading: IC17EKN1.lpd reading: SP10HAUS.lpd reading: LS15JOPA.lpd reading: LS15ABSH.lpd reading: IC10VISG.lpd reading: LS17BAHA01.lpd reading: IC10VID3.lpd reading: SP12ERUS.lpd reading: CO00FERA.lpd reading: IC02GR05.lpd reading: SP12FOAT.lpd reading: TR16WEMI.lpd reading: IC98THSA.lpd reading: IC08VIRE.lpd reading: SP11BEMX.lpd reading: SP08VBPE.lpd reading: IC13THQU.lpd reading: TR15YONW.lpd reading: CO17DESC02A.lpd reading: IC10VIRE.lpd reading: MS07LSSP.lpd reading: IC09KA79.lpd reading: TR04EVLI.lpd reading: CO09FEOG.lpd reading: LS00SEJU.lpd reading: IC13ST06.lpd reading: MS10BLMS.lpd reading: MS11CSCS.lpd reading: SH16MEIN.lpd reading: CO17DESC01A.lpd reading: SP99DEUS.lpd reading: CO17WUBO1A.lpd Finished read: 509 records extracting paleoData... extracting: CO14WUCL extracting: LS96CUPU extracting: TR18GRPM extracting: IC00OE03 extracting: IC02GR17 extracting: IC00OEKC extracting: IC03HORB extracting: MS09TCIS extracting: CO95TUNG extracting: CO17MUMA extracting: IC16WE30 extracting: LS10WRNA extracting: IC10VICR extracting: MS09DOIP extracting: LS91BEWA extracting: IC13ST12 extracting: CO98SWPR extracting: CO04BAFI extracting: SP11STBR extracting: CO17DESC03A extracting: LS07EAGO extracting: IC13STSP extracting: LS13WASU extracting: LS16STPA extracting: LS14LASO extracting: IC09VIA8 extracting: CO99DRGB extracting: CO00URMA extracting: IC11LAB4 extracting: IC96MOSS extracting: IC06VING extracting: LS07LAHA extracting: IC00STHN extracting: IC99IS89 extracting: LS14TAKA extracting: LS09LAEL extracting: SP08HUCN extracting: IC97THGL extracting: CO09NUTB extracting: MS10JTMA extracting: MS96LKSS extracting: SP10BEIN extracting: LS15AIKA extracting: IC13ST01 extracting: CO04KIVA extracting: TR13JOAR extracting: LS04JOCH extracting: LS15DOSA extracting: IC02STTA extracting: LS09SCMI extracting: MS11KDMS extracting: IC16WE22 extracting: SP13ASBC extracting: MS09TRFD extracting: TR16LAAN extracting: CO13HENG extracting: SP12CACN extracting: CO96QUVA extracting: LS11TICH extracting: LS10STSP extracting: LS13YUSA extracting: IC14VL1K extracting: IC00OE16 extracting: SP09GRID extracting: IC00STTY extracting: IC13ST03 extracting: SP10MEMX extracting: IC15GANK extracting: CO97CHSY extracting: SP07ASUS extracting: CO04ZIMG extracting: CO98EVXM extracting: TR17EDSW extracting: IC85LAD2 extracting: IC10VISD extracting: TR11SAHU00 extracting: IC16WE20 extracting: IC02MUBI extracting: TR13POMA extracting: LS91HOMI extracting: SP07FLOM extracting: SP07DEUS extracting: LS15LAOH extracting: IC13STWD extracting: CO14ZIMG extracting: IW15MELD extracting: MS99LWSC extracting: IC12WACD extracting: LS05ANJE extracting: SP13PAVU extracting: IC02HEWD extracting: IC99OE10 extracting: IC04FIML extracting: MS15VIBC extracting: IC13ST91 extracting: IC09VICC extracting: IC16WE17 extracting: CO17DESC04A extracting: CO14OSPA extracting: CO06LIFI extracting: CO14ZIHO extracting: IC96ISE9 extracting: TR16LAFO extracting: IC05ISAU extracting: LS15SHNE extracting: LS14ATJU extracting: IC13ST15 extracting: CO99SWFB extracting: IC99OE07 extracting: SP06DYCN extracting: MS09DTNA extracting: SP05MAAT extracting: LS16STCL extracting: IC12RHME extracting: LS02HATI extracting: LS10HEQI extracting: CO92SHPU extracting: LS12THAY extracting: IC00SC29 extracting: LS09RUWA extracting: IC06VID3 extracting: IC99SCFH extracting: TR14KOSP extracting: LS10CLTA extracting: LS11ANSE extracting: IC10VISA extracting: IC00OE20 extracting: TR12BRBO extracting: TR10ANMO extracting: CO17LITW01A extracting: CO14CATI extracting: LS09SASP extracting: LS06JONA extracting: TR08THCO extracting: LS09BAUM extracting: IC06THPR extracting: SP12KEBZ extracting: IC13STN7 extracting: LS06MIBO extracting: TR06TRBO extracting: CO03COPM extracting: SP04YUCN extracting: CO93COTW extracting: CO13COTB extracting: GI19POYU extracting: TR11GRTP extracting: IC06KAS1 extracting: LS03HAIG extracting: CO94DUUR extracting: IC99OE02 extracting: IC12SIWG extracting: CO13CABL extracting: IC01JO01 extracting: CO11NUPM extracting: CO17WUBO1G extracting: TR11BAPU extracting: CO17WUBO1B extracting: IC00OE23 extracting: CO13COXM extracting: IC99OE03 extracting: IC99OE04 extracting: LS15DOCH extracting: MS06DLGN extracting: LS13KOLA extracting: MS11CSCN extracting: MS04ICNI extracting: LS01HOCH extracting: CO06MOPE extracting: IC82STVE extracting: CO04MGNG extracting: LS17VAWA01 extracting: CO96SWFB extracting: IC07KADF extracting: MS11SENS extracting: SP06DRIT extracting: SP13AYID extracting: MS03LKNS extracting: IC00SC18 extracting: MS15VIBM extracting: MS11FAPM extracting: LS15BHAL extracting: CO02KUBE extracting: CO99GUMI extracting: IC16WE19 extracting: CO17WUBO1E extracting: TR07REVI extracting: CO96SWBB extracting: IC16WE28 extracting: IC95FI87 extracting: SP13MCUS extracting: LS14FEZA extracting: LS11MOES extracting: TR16WEXI extracting: TR13SIKO extracting: LS13LUBA extracting: LS09MIKU extracting: TR07RECA extracting: MS14GFSS extracting: CO08KIPR extracting: LS12WOQU extracting: LS09MAAS extracting: SP10WINZ extracting: IC00OE22 extracting: IC08VI84 extracting: LS02ROSA extracting: CO99KUHO extracting: LS16THQI01 extracting: ElJunco.Zhang.2014 extracting: ClelandLake.Steinman.2016 extracting: IC99OE05 extracting: SS05ROTN extracting: IC11STTA extracting: CO17WUBO1D extracting: CO03CHBU extracting: LS11KOMA extracting: MS11KWFS extracting: TR12BECO extracting: LS08LICA extracting: IC02GR07 extracting: LS11RYKA extracting: LS14FOTI extracting: LS00WODE extracting: CO04INMI extracting: LS99LASI extracting: IC10VIP2 extracting: CO17RAPA extracting: IC13STN8 extracting: IC17EKN3 extracting: CO12GOVA extracting: SP08ZHCN extracting: IC08HODF extracting: CO18DATO01A extracting: TR11MAPE extracting: MS07KPPB extracting: IC03HOIL extracting: TR15BABO extracting: LS17STNU01 extracting: IC10VIP1 extracting: IC95THHU extracting: IC00EIGG extracting: LS96VOAM extracting: CO18RELC01A extracting: CO98QUNC extracting: MS03SMAS extracting: IC10VIMI extracting: IC06THDD extracting: LS14KOSA extracting: IC14MAFH extracting: TR13JOJS extracting: LS13BAFL extracting: IC08VI87 extracting: MS07LSPG extracting: LS06CRLA extracting: TR11SIMO extracting: IC00OE12 extracting: IC13ST13 extracting: CO17WUBO1C extracting: TR18XUJA00 extracting: TR08HORA extracting: SP14APPE extracting: LS07ROTI extracting: IC92HOML extracting: IC13ST04 extracting: LS03RUED extracting: IC98FIPE extracting: TR15YOLL extracting: IC00OE02 extracting: LS04YUWA extracting: CO00COKY extracting: TR19LEOU extracting: IC11LIMI extracting: LS16THN301 extracting: IC00COPR extracting: CO18HECO01B extracting: LS12GOBA extracting: TR11BAVO extracting: CO07CAFR extracting: CO08GOBE extracting: LS14PRLO extracting: CO05BAFI extracting: IC13OPAN extracting: TR11XUPH extracting: LS11NECA extracting: LS06DENA extracting: TR17GRTP extracting: TR19LEBR extracting: MS07RCNA extracting: LS06STFO extracting: IC17BERI extracting: TR18XUGA00 extracting: LS06SHBE extracting: SP08SPUS extracting: IC10VISE extracting: LS97HOKA extracting: SP07WABR extracting: LS09MAZO extracting: CO94LISE extracting: LS12ANBI extracting: CO04PFRE extracting: LS08STCR extracting: IC00OE15 extracting: MS13RSAS extracting: IC09EIBE extracting: IC06WAML extracting: TR13SITA extracting: CO17WUBO1F extracting: MS14MOLS extracting: IC08VI79 extracting: LS10HOLO extracting: IC13ST24 extracting: SP11BOAT extracting: MS12AMMM extracting: IC13ST22 extracting: IC13THFE extracting: CO01TUNG extracting: IC99ISS2 extracting: SP13CHPE extracting: IC99OE09 extracting: IC11BEVL extracting: IC10NAB5 extracting: IC00OE18 extracting: IC16WENG extracting: LS15DENA extracting: LS13STEL extracting: SP13LAMX extracting: CO98CHPI extracting: SS03HAJA extracting: LS05HOAG extracting: IC17KOEL extracting: IC17EK4K extracting: AKIII.Terwilliger.2013 extracting: IC13ST05 extracting: CO04LIFI extracting: IC94THDP extracting: LS09SAWA extracting: IC02THKI extracting: LS12ANYE extracting: CO14CABT extracting: SP10CACN extracting: IC00SC16 extracting: IC16GOBP extracting: SP09REPE extracting: LS89LATI extracting: SP03BMIL extracting: CO06QUNG extracting: IC17EKPV extracting: LS10HESA extracting: IC02SCGN extracting: CO03CHBA extracting: SL13TEAK extracting: SP13SUSA extracting: MS15BATP extracting: IC99OE08 extracting: CO15ABSP extracting: TR17SAMA extracting: MS11ANMN extracting: LS06POVE extracting: LS10ANBL extracting: MS07JRGM extracting: LS02BEPY extracting: LS11BIPU extracting: MS08UAPM extracting: IC08THGZ extracting: LS07ANMA extracting: MS05LKLF extracting: LS09SHJO extracting: MS12CCCH extracting: TR13SAWA00 extracting: MS02FLPC extracting: TR11MAJA extracting: IC16WE27 extracting: SP05CRBR extracting: IC00OE05 extracting: CO13CAHN extracting: IC16WE23 extracting: MS06DLDS extracting: LS16WIGH extracting: SH18CASA extracting: IC00SC21 extracting: MS01HDAS extracting: MS06DLGS extracting: CO06MOTO extracting: LS12STRE extracting: LS09SAJU extracting: IC11DILO extracting: MS05ESCR extracting: LS15OEPO extracting: LS10GOBL extracting: IC99OE01 extracting: SP09FLTR extracting: CO04LIRA extracting: SP12NOBR extracting: TR19LECR extracting: MS07SWGG extracting: TR12SAMU extracting: CO05KUBE extracting: IC06VIGR extracting: MS04BLCB extracting: IC10VIGI extracting: MS97HLCB extracting: CO00KUNI extracting: TR15NACA extracting: CO05ASGU extracting: IC99OE06 extracting: LS09LACA extracting: LS15SHBO extracting: CO99LICL extracting: IC13PALD extracting: IC10VIGR extracting: IC11KIPW extracting: CO98BOFP extracting: MS03KTCB extracting: SS05ROES extracting: IW17OPOY extracting: TR13BROA extracting: SP98KUCN extracting: TR13JOPC extracting: SP13KAPE extracting: IC00OE21 extracting: LS98CUPE extracting: IC09KANG extracting: CO18HECO01A extracting: IC09VIRE extracting: IC00THDS extracting: LS14ZHJU extracting: IC13ST21 extracting: IC07KARB extracting: CO13DESC01A extracting: IC15MANE extracting: LS12KENO extracting: LS06TIST extracting: IC00OE13 extracting: IC09OPAN extracting: SP12LAMX extracting: MS07MMNP extracting: MS14MORR extracting: CO08ABSU extracting: SL13TEAG extracting: LS14LESA extracting: IC14EKVK extracting: TR19LEFC extracting: LS14YAHA extracting: CO04PFCH extracting: IC09KAD3 extracting: IC12MUJR extracting: IC08VISV extracting: IC00SC26 extracting: IC93BAGI extracting: LS10ZHHU extracting: IC00OE11 extracting: IC76FIDE extracting: LS11BACH extracting: CO10HEIG extracting: CO00DRBE extracting: SP13SLNA extracting: SP10SUSE extracting: CO06DATZ extracting: LS01HUFA extracting: CO09NAKY extracting: MS03BRNS extracting: CO17XISC01A extracting: IC08VI77 extracting: SP10DOCN extracting: IC10VIP3 extracting: MS11ANMS extracting: IC97GRGI extracting: IC08DILO extracting: MS13KWFS extracting: TR08VOOL extracting: SP04DEUS extracting: IC00OE19 extracting: IC13BOCG extracting: LS12STLI extracting: LS99CUVA extracting: CO94HEAQ extracting: IC00OE14 extracting: CO08HEVE extracting: IC17EKN1 extracting: SP10HAUS extracting: LS15JOPA extracting: LS15ABSH extracting: IC10VISG extracting: LS17BAHA01 extracting: IC10VID3 extracting: SP12ERUS extracting: CO00FERA extracting: IC02GR05 extracting: SP12FOAT extracting: TR16WEMI extracting: IC98THSA extracting: IC08VIRE extracting: SP11BEMX extracting: SP08VBPE extracting: IC13THQU extracting: TR15YONW extracting: CO17DESC02A extracting: IC10VIRE extracting: MS07LSSP extracting: IC09KA79 extracting: TR04EVLI extracting: CO09FEOG extracting: LS00SEJU extracting: IC13ST06 extracting: MS10BLMS extracting: MS11CSCS extracting: SH16MEIN extracting: CO17DESC01A extracting: SP99DEUS extracting: CO17WUBO1A Created time series: 1964 entries
In [5]:
Copied!
# all_keys = []
# for ii in range(len(TS)):
# keys = TS[ii].keys()
# for key in keys:
# if key not in all_keys: all_keys.append(key)
# all_keys = []
# for ii in range(len(TS)):
# keys = TS[ii].keys()
# for key in keys:
# if key not in all_keys: all_keys.append(key)
Create compact dataframe¶
Dataframe construction¶
Create empty dataframe with set of columns for compact dataframe, and populate with the LiPD data
In [6]:
Copied!
col_str=['archiveType', 'dataSetName', 'datasetId', 'geo_meanElev', 'geo_meanLat', 'geo_meanLon', 'geo_siteName',
'originalDataUrl', 'paleoData_notes', 'paleoData_variableName','paleoData_proxy',
'paleoData_archiveSpecies','paleoData_units','paleoData_interpretation',
'paleoData_values', 'year']
df_tmp = pd.DataFrame(index=range(len(TS)), columns=col_str)
df_tmp.info()
col_str=['archiveType', 'dataSetName', 'datasetId', 'geo_meanElev', 'geo_meanLat', 'geo_meanLon', 'geo_siteName',
'originalDataUrl', 'paleoData_notes', 'paleoData_variableName','paleoData_proxy',
'paleoData_archiveSpecies','paleoData_units','paleoData_interpretation',
'paleoData_values', 'year']
df_tmp = pd.DataFrame(index=range(len(TS)), columns=col_str)
df_tmp.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1964 entries, 0 to 1963 Data columns (total 16 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 archiveType 0 non-null object 1 dataSetName 0 non-null object 2 datasetId 0 non-null object 3 geo_meanElev 0 non-null object 4 geo_meanLat 0 non-null object 5 geo_meanLon 0 non-null object 6 geo_siteName 0 non-null object 7 originalDataUrl 0 non-null object 8 paleoData_notes 0 non-null object 9 paleoData_variableName 0 non-null object 10 paleoData_proxy 0 non-null object 11 paleoData_archiveSpecies 0 non-null object 12 paleoData_units 0 non-null object 13 paleoData_interpretation 0 non-null object 14 paleoData_values 0 non-null object 15 year 0 non-null object dtypes: object(16) memory usage: 245.6+ KB
Populate dataframe¶
Start by populating paleoData_variableName (paleoData_proxy in dod2k standard terms)
In [7]:
Copied!
# loop over the timeseries and pick those for global temperature analysis
i = 0
for ts in TS: #for every time series
# need to filter these variables in the list
# if ts['paleoData_variableName'] not in ['year', 'd18OUncertainty', 'SrCaUncertainty',
# 'uncertainty', 'uncertainty1s', 'uncertaintyHigh',
# 'uncertaintyHigh1s', 'uncertaintyLow', 'uncertaintyLow1s',
# 'needsToBeChanged', 'deleteThisColumn','notes',
# 'sampleID', 'section', 'site']:
for name in col_str: #for each of the 12 main keys, shove the wanted data into the df
if name in ts:
df_tmp.loc[i, name] = ts[name]
else:
df_tmp.loc[i, name] = np.nan
i += 1
# loop over the timeseries and pick those for global temperature analysis
i = 0
for ts in TS: #for every time series
# need to filter these variables in the list
# if ts['paleoData_variableName'] not in ['year', 'd18OUncertainty', 'SrCaUncertainty',
# 'uncertainty', 'uncertainty1s', 'uncertaintyHigh',
# 'uncertaintyHigh1s', 'uncertaintyLow', 'uncertaintyLow1s',
# 'needsToBeChanged', 'deleteThisColumn','notes',
# 'sampleID', 'section', 'site']:
for name in col_str: #for each of the 12 main keys, shove the wanted data into the df
if name in ts:
df_tmp.loc[i, name] = ts[name]
else:
df_tmp.loc[i, name] = np.nan
i += 1
- Drop rows with missing data:
- all NaNs
- NaNs for
yearorpaleoData_values(no data records)
- Populate missing
paleoData_proxyentries withpaleoData_variableName
In [8]:
Copied!
# drop the rows with all NaNs (those not for global temperature analysis)
print(len(df_tmp))
df = df_tmp.dropna(how='all')
print(len(df))
for subset in ['year', 'paleoData_values']:
df = df.dropna(subset=subset, how='all')
print(len(df))
for ii in df.index:
if not isinstance(df.at[ii, 'paleoData_proxy'], str):
df.at[ii, 'paleoData_proxy'] = df.at[ii, 'paleoData_variableName']
# drop the rows with all NaNs (those not for global temperature analysis)
print(len(df_tmp))
df = df_tmp.dropna(how='all')
print(len(df))
for subset in ['year', 'paleoData_values']:
df = df.dropna(subset=subset, how='all')
print(len(df))
for ii in df.index:
if not isinstance(df.at[ii, 'paleoData_proxy'], str):
df.at[ii, 'paleoData_proxy'] = df.at[ii, 'paleoData_variableName']
1964 1964 1662 1662
Standardise dataframe¶
- Ensure standard terminology for
paleoData_proxy
In [9]:
Copied!
df['paleoData_proxy'] = df['paleoData_proxy'].replace({'d2H': 'dD'})
df['paleoData_proxy'] = df['paleoData_proxy'].replace({'d2H': 'dD'})
Keep only the record types which are included in https://doi.org/10.5194/essd-12-2261-2020 (which is 'a global compilation of dD and d18O records')
In [10]:
Copied!
df = df[(df['paleoData_proxy']=='d18O')|(df['paleoData_proxy']=='dD')]
df = df[(df['paleoData_proxy']=='d18O')|(df['paleoData_proxy']=='dD')]
- Add the missing columns to the dataframe, including
originalDatabase,yearUnits,interpretation_variable(these are added manually and not from the LiPD files)
In [11]:
Copied!
# KF: adding original dataset name and yearUnits
df.insert(7, 'originalDatabase', ['Iso2k v1.1.2']*len(df))
df.insert(len(df.columns), 'yearUnits', ['CE'] * len(df))
df.insert(1, 'interpretation_variable', ['N/A']*len(df))
df.insert(1, 'interpretation_variableDetail', ['N/A']*len(df))
df.insert(1, 'interpretation_seasonality', ['N/A']*len(df))
df.insert(1, 'interpretation_direction', ['N/A']*len(df))
# KF: adding original dataset name and yearUnits
df.insert(7, 'originalDatabase', ['Iso2k v1.1.2']*len(df))
df.insert(len(df.columns), 'yearUnits', ['CE'] * len(df))
df.insert(1, 'interpretation_variable', ['N/A']*len(df))
df.insert(1, 'interpretation_variableDetail', ['N/A']*len(df))
df.insert(1, 'interpretation_seasonality', ['N/A']*len(df))
df.insert(1, 'interpretation_direction', ['N/A']*len(df))
- Rename columns to fit naming conventions
In [12]:
Copied!
df = df.rename(columns={'originalDataUrl': 'originalDataURL',
'paleoData_archiveSpecies': 'paleoData_sensorSpecies'})
df = df.rename(columns={'originalDataUrl': 'originalDataURL',
'paleoData_archiveSpecies': 'paleoData_sensorSpecies'})
In [13]:
Copied!
df.info()
df.info()
<class 'pandas.core.frame.DataFrame'> Index: 620 entries, 0 to 1962 Data columns (total 22 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 archiveType 620 non-null object 1 interpretation_direction 620 non-null object 2 interpretation_seasonality 620 non-null object 3 interpretation_variableDetail 620 non-null object 4 interpretation_variable 620 non-null object 5 dataSetName 620 non-null object 6 datasetId 620 non-null object 7 geo_meanElev 593 non-null object 8 geo_meanLat 620 non-null object 9 geo_meanLon 620 non-null object 10 geo_siteName 620 non-null object 11 originalDatabase 620 non-null object 12 originalDataURL 610 non-null object 13 paleoData_notes 541 non-null object 14 paleoData_variableName 620 non-null object 15 paleoData_proxy 620 non-null object 16 paleoData_sensorSpecies 48 non-null object 17 paleoData_units 616 non-null object 18 paleoData_interpretation 620 non-null object 19 paleoData_values 620 non-null object 20 year 620 non-null object 21 yearUnits 620 non-null object dtypes: object(22) memory usage: 111.4+ KB
In [14]:
Copied!
# check that the datasetId is unique
print('dataset length: ', len(df))
print('unique datasetIds:', len(df.datasetId.unique()))
# make datasetId unique by simply adding index number
df.datasetId=df.apply(lambda x: 'iso2k_'+str(x.name), axis=1)
# check uniqueness - problem solved.
assert len(df)== len(df.datasetId.unique())
# check that the datasetId is unique
print('dataset length: ', len(df))
print('unique datasetIds:', len(df.datasetId.unique()))
# make datasetId unique by simply adding index number
df.datasetId=df.apply(lambda x: 'iso2k_'+str(x.name), axis=1)
# check uniqueness - problem solved.
assert len(df)== len(df.datasetId.unique())
dataset length: 620 unique datasetIds: 427
In [15]:
Copied!
keys = ['interpretation_direction', 'interpretation_seasonality',
'interpretation_variable', 'interpretation_variableDetail']
for ii in df.index:
if type(df_tmp.at[ii, 'paleoData_interpretation'])!=list: continue
for key in keys:
if key.split('_')[-1] in df_tmp.at[ii, 'paleoData_interpretation'][0].keys():
df.at[ii, key] = df_tmp.at[ii, 'paleoData_interpretation'][0][key.split('_')[-1]]
keys = ['interpretation_direction', 'interpretation_seasonality',
'interpretation_variable', 'interpretation_variableDetail']
for ii in df.index:
if type(df_tmp.at[ii, 'paleoData_interpretation'])!=list: continue
for key in keys:
if key.split('_')[-1] in df_tmp.at[ii, 'paleoData_interpretation'][0].keys():
df.at[ii, key] = df_tmp.at[ii, 'paleoData_interpretation'][0][key.split('_')[-1]]
The entries for interpretation_variable need to be standardised according to our terminology (note that the original variable is still available in interpretation_variableDetail). Original entries:
['N/A' 'circulationIndex' 'circulationVariable' 'deleteMe' 'effectivePrecipitation' 'evaporation' 'hydrologicBalance' 'precipitation' 'precipitationIsotope' 'salinity' 'seasonality' 'streamflow' 'temperature']
In [16]:
Copied!
interpretation_dict = {'precipitation': 'moisture', 'precipitationIsotope': 'moisture', 'effectivePrecipitation': 'moisture',
'evaporation': 'moisture', 'hydrologicBalance': 'moisture', 'salinity': 'moisture',
'circulationIndex': 'N/A', 'circulationVariable': 'N/A', 'deleteMe': 'N/A', 'seasonality': 'N/A',
'streamflow': 'N/A'}
for key in interpretation_dict.keys():
mask = df['interpretation_variable'] == key
df.loc[mask, 'interpretation_variableDetail'] = (
'Original interpretation_variable: '
+ key + ', interpretation_variableDetail: ' + df.loc[mask, 'interpretation_variableDetail']
)
df['interpretation_variable'] = df['interpretation_variable'].replace(
interpretation_dict)
interpretation_dict = {'precipitation': 'moisture', 'precipitationIsotope': 'moisture', 'effectivePrecipitation': 'moisture',
'evaporation': 'moisture', 'hydrologicBalance': 'moisture', 'salinity': 'moisture',
'circulationIndex': 'N/A', 'circulationVariable': 'N/A', 'deleteMe': 'N/A', 'seasonality': 'N/A',
'streamflow': 'N/A'}
for key in interpretation_dict.keys():
mask = df['interpretation_variable'] == key
df.loc[mask, 'interpretation_variableDetail'] = (
'Original interpretation_variable: '
+ key + ', interpretation_variableDetail: ' + df.loc[mask, 'interpretation_variableDetail']
)
df['interpretation_variable'] = df['interpretation_variable'].replace(
interpretation_dict)
In [17]:
Copied!
df = df.drop(labels=['paleoData_interpretation'], axis=1)
df = df.drop(labels=['paleoData_interpretation'], axis=1)
In [18]:
Copied!
df = df.astype({'archiveType': str, 'dataSetName': str, 'datasetId': str, 'geo_meanElev': np.float32, 'geo_meanLat': np.float32, 'geo_meanLon': np.float32, 'geo_siteName': str,
'originalDatabase': str, 'originalDataURL': str, 'paleoData_notes': str, 'paleoData_proxy': str, 'paleoData_units': str, 'yearUnits': str})
drop_inds = []
for ii in df.index:
try:
year = np.array(df.at[ii, 'year'], dtype=float)
vals = np.array(df.at[ii, 'paleoData_values'], dtype=float)
df.at[ii, 'year'] = year[year>=1]
df.at[ii, 'paleoData_values'] = vals[year>=1]
except:
df.at[ii, 'paleoData_values'] = np.array([utf.convert_to_float(y) for y in df.at[ii, 'paleoData_values']], dtype=float)
df.at[ii, 'year'] = np.array([utf.convert_to_float(y) for y in df.at[ii, 'year']], dtype=float)
print(f'Converted values in paleoData_values and/or year for {ii}.')
df = df.astype({'archiveType': str, 'dataSetName': str, 'datasetId': str, 'geo_meanElev': np.float32, 'geo_meanLat': np.float32, 'geo_meanLon': np.float32, 'geo_siteName': str,
'originalDatabase': str, 'originalDataURL': str, 'paleoData_notes': str, 'paleoData_proxy': str, 'paleoData_units': str, 'yearUnits': str})
drop_inds = []
for ii in df.index:
try:
year = np.array(df.at[ii, 'year'], dtype=float)
vals = np.array(df.at[ii, 'paleoData_values'], dtype=float)
df.at[ii, 'year'] = year[year>=1]
df.at[ii, 'paleoData_values'] = vals[year>=1]
except:
df.at[ii, 'paleoData_values'] = np.array([utf.convert_to_float(y) for y in df.at[ii, 'paleoData_values']], dtype=float)
df.at[ii, 'year'] = np.array([utf.convert_to_float(y) for y in df.at[ii, 'year']], dtype=float)
print(f'Converted values in paleoData_values and/or year for {ii}.')
Include Common Era data only
Note that the datasetId is not unique for each record and thus we added an additional array of strings to make the datasetId unique.
mask out nans and set fill value, then later drop
In [19]:
Copied!
# drop all missing values and exclude all-missing-values-rows
for ii in df.index:
dd = np.array(df.at[ii, 'paleoData_values'])
mask = dd==-9999.99
df.at[ii, 'paleoData_values']=dd[~mask]
df.at[ii, 'year']=np.array(df.at[ii, 'year'])[~mask]
drop_inds = []
for ii, row in enumerate(df.paleoData_values):
try:
if len(row)==0:
print(ii, 'empty row for paleodata_values')
elif len(df.iloc[ii]['year'])==0:
print(ii, 'empty row for year')
elif np.std(row)==0:
print(ii, 'std=0')
elif np.sum(np.diff(row)**2)==0:
print(ii, 'diff=0')
elif np.isnan(np.std(row)):
print(ii, 'std nan')
else:
continue
if df.index[ii] not in drop_inds:
drop_inds += [df.index[ii]]
except:
drop_inds+=[df.index[ii]]
print(drop_inds)
df = df.drop(index=drop_inds)
# drop all missing values and exclude all-missing-values-rows
for ii in df.index:
dd = np.array(df.at[ii, 'paleoData_values'])
mask = dd==-9999.99
df.at[ii, 'paleoData_values']=dd[~mask]
df.at[ii, 'year']=np.array(df.at[ii, 'year'])[~mask]
drop_inds = []
for ii, row in enumerate(df.paleoData_values):
try:
if len(row)==0:
print(ii, 'empty row for paleodata_values')
elif len(df.iloc[ii]['year'])==0:
print(ii, 'empty row for year')
elif np.std(row)==0:
print(ii, 'std=0')
elif np.sum(np.diff(row)**2)==0:
print(ii, 'diff=0')
elif np.isnan(np.std(row)):
print(ii, 'std nan')
else:
continue
if df.index[ii] not in drop_inds:
drop_inds += [df.index[ii]]
except:
drop_inds+=[df.index[ii]]
print(drop_inds)
df = df.drop(index=drop_inds)
1 std nan 2 std nan 4 std nan 13 std nan 14 std nan 20 empty row for paleodata_values 43 std nan 48 std nan 52 std nan 58 std nan 59 std nan 60 std nan 63 std nan 65 std nan 72 std nan 78 std nan 92 std nan 94 std nan 96 std nan 104 std nan 106 std nan 108 std nan 111 std nan 118 std nan 119 std nan 120 std nan 122 std nan 123 std nan 125 std nan 126 std nan 127 std nan 135 std nan 138 std nan 148 std nan 163 empty row for paleodata_values 165 empty row for paleodata_values 174 std nan 175 std nan 183 std nan 186 std nan 188 std nan 189 std nan 193 std nan 194 std nan 198 std nan 206 empty row for paleodata_values 207 empty row for paleodata_values 208 empty row for paleodata_values 209 empty row for paleodata_values 210 empty row for paleodata_values 211 std nan 212 std nan 213 std nan 214 std nan 215 std nan 216 std nan 217 std nan 222 std nan 223 std nan 231 std nan 234 std nan 238 std nan 246 std nan 254 std nan 257 std nan 259 empty row for paleodata_values 262 std nan 268 std nan 280 std nan 282 std nan 295 std nan 296 std nan 297 std nan 299 std nan 301 std nan 302 std nan 304 std nan 308 std nan 309 std nan 313 std nan 314 std nan 319 std nan 320 std nan 327 std nan 328 std nan 329 std nan 330 std nan 331 std nan 332 std nan 333 std nan 334 std nan 339 std nan 348 std nan 351 empty row for paleodata_values 352 std nan 353 std nan 354 std nan 355 std nan 356 std nan 359 std nan 361 std nan 363 std nan 364 std nan 365 std nan 370 std nan 371 std nan 378 std nan 380 std nan 381 std nan 382 std nan 383 std nan 384 std nan 385 std nan 386 std nan 388 std nan 392 std nan 393 std nan 395 std nan 396 std nan 405 empty row for paleodata_values 412 std nan 413 std nan 414 std nan 415 std nan 416 std nan 419 std nan 420 std nan 422 std nan 423 std nan 425 std nan 427 std nan 429 std nan 433 std nan 435 std nan 436 std nan 439 std nan 440 std nan 448 std nan 459 std nan 466 std nan 467 std nan 469 std nan 470 std nan 475 std nan 476 std nan 483 std nan 497 std nan 498 std nan 499 std nan 501 std nan 502 std nan 513 std nan 516 std nan 517 std nan 518 std nan 519 std nan 520 std nan 521 std nan 522 std nan 523 std nan 524 std nan 530 std nan 536 std nan 543 std nan 550 std nan 553 diff=0 554 empty row for paleodata_values 557 std nan 558 std nan 560 std nan 562 std nan 568 std nan 569 std nan 572 std nan 580 std nan 581 std nan 598 std nan 599 empty row for paleodata_values 610 std nan 611 std nan 612 std nan 613 std nan 619 std nan [np.int64(8), np.int64(10), np.int64(14), np.int64(45), np.int64(48), np.int64(63), np.int64(153), np.int64(190), np.int64(209), np.int64(231), np.int64(232), np.int64(234), np.int64(242), np.int64(246), np.int64(262), np.int64(277), np.int64(317), np.int64(322), np.int64(337), np.int64(355), np.int64(360), np.int64(371), np.int64(378), np.int64(418), np.int64(420), np.int64(423), np.int64(425), np.int64(430), np.int64(438), np.int64(441), np.int64(442), np.int64(461), np.int64(470), np.int64(494), np.int64(530), np.int64(536), np.int64(564), np.int64(565), np.int64(583), np.int64(589), np.int64(594), np.int64(596), np.int64(614), np.int64(621), np.int64(635), np.int64(660), np.int64(663), np.int64(666), np.int64(670), np.int64(673), np.int64(674), np.int64(679), np.int64(681), np.int64(688), np.int64(691), np.int64(692), np.int64(694), np.int64(711), np.int64(713), np.int64(755), np.int64(763), np.int64(776), np.int64(816), np.int64(839), np.int64(847), np.int64(851), np.int64(861), np.int64(876), np.int64(904), np.int64(907), np.int64(968), np.int64(970), np.int64(972), np.int64(978), np.int64(981), np.int64(984), np.int64(993), np.int64(1002), np.int64(1004), np.int64(1022), np.int64(1025), np.int64(1048), np.int64(1050), np.int64(1071), np.int64(1073), np.int64(1074), np.int64(1075), np.int64(1076), np.int64(1077), np.int64(1078), np.int64(1079), np.int64(1103), np.int64(1136), np.int64(1154), np.int64(1159), np.int64(1162), np.int64(1163), np.int64(1164), np.int64(1167), np.int64(1176), np.int64(1185), np.int64(1191), np.int64(1196), np.int64(1197), np.int64(1207), np.int64(1210), np.int64(1226), np.int64(1232), np.int64(1234), np.int64(1240), np.int64(1243), np.int64(1246), np.int64(1249), np.int64(1253), np.int64(1262), np.int64(1273), np.int64(1276), np.int64(1278), np.int64(1279), np.int64(1298), np.int64(1301), np.int64(1324), np.int64(1327), np.int64(1328), np.int64(1329), np.int64(1338), np.int64(1350), np.int64(1354), np.int64(1364), np.int64(1365), np.int64(1375), np.int64(1383), np.int64(1390), np.int64(1401), np.int64(1409), np.int64(1411), np.int64(1421), np.int64(1423), np.int64(1454), np.int64(1493), np.int64(1522), np.int64(1524), np.int64(1526), np.int64(1527), np.int64(1537), np.int64(1541), np.int64(1561), np.int64(1608), np.int64(1611), np.int64(1614), np.int64(1623), np.int64(1625), np.int64(1656), np.int64(1662), np.int64(1668), np.int64(1673), np.int64(1678), np.int64(1679), np.int64(1681), np.int64(1684), np.int64(1685), np.int64(1686), np.int64(1705), np.int64(1725), np.int64(1742), np.int64(1760), np.int64(1764), np.int64(1767), np.int64(1776), np.int64(1778), np.int64(1788), np.int64(1793), np.int64(1808), np.int64(1810), np.int64(1815), np.int64(1841), np.int64(1843), np.int64(1879), np.int64(1881), np.int64(1928), np.int64(1938), np.int64(1945), np.int64(1947), np.int64(1958), np.int64(1962)]
Now show the final compact dataframe
In [20]:
Copied!
df = df[sorted(df.columns)]
df.reset_index(drop= True, inplace= True)
print(df.info())
df = df[sorted(df.columns)]
df.reset_index(drop= True, inplace= True)
print(df.info())
<class 'pandas.core.frame.DataFrame'> RangeIndex: 435 entries, 0 to 434 Data columns (total 21 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 archiveType 435 non-null object 1 dataSetName 435 non-null object 2 datasetId 435 non-null object 3 geo_meanElev 415 non-null float32 4 geo_meanLat 435 non-null float32 5 geo_meanLon 435 non-null float32 6 geo_siteName 435 non-null object 7 interpretation_direction 435 non-null object 8 interpretation_seasonality 435 non-null object 9 interpretation_variable 435 non-null object 10 interpretation_variableDetail 435 non-null object 11 originalDataURL 435 non-null object 12 originalDatabase 435 non-null object 13 paleoData_notes 435 non-null object 14 paleoData_proxy 435 non-null object 15 paleoData_sensorSpecies 43 non-null object 16 paleoData_units 435 non-null object 17 paleoData_values 435 non-null object 18 paleoData_variableName 435 non-null object 19 year 435 non-null object 20 yearUnits 435 non-null object dtypes: float32(3), object(18) memory usage: 66.4+ KB None
save compact dataframe¶
save pickle¶
In [21]:
Copied!
# save to a pickle file
df = df[sorted(df.columns)]
df.to_pickle('data/iso2k/iso2k_compact.pkl')
# save to a pickle file
df = df[sorted(df.columns)]
df.to_pickle('data/iso2k/iso2k_compact.pkl')
save csv¶
In [22]:
Copied!
# save to a list of csv files (metadata, data, year)
df.name= 'iso2k'
utf.write_compact_dataframe_to_csv(df)
# save to a list of csv files (metadata, data, year)
df.name= 'iso2k'
utf.write_compact_dataframe_to_csv(df)
METADATA: datasetId, archiveType, dataSetName, geo_meanElev, geo_meanLat, geo_meanLon, geo_siteName, interpretation_direction, interpretation_seasonality, interpretation_variable, interpretation_variableDetail, originalDataURL, originalDatabase, paleoData_notes, paleoData_proxy, paleoData_sensorSpecies, paleoData_units, paleoData_variableName, yearUnits Saved to /home/jupyter-lluecke/dod2k/data/iso2k/iso2k_compact_%s.csv
In [23]:
Copied!
# load dataframe
df = utf.load_compact_dataframe_from_csv('iso2k')
# load dataframe
df = utf.load_compact_dataframe_from_csv('iso2k')
Visualise dataframe¶
Show spatial distribution of records, show archive and proxy types
In [24]:
Copied!
# count archive types
archive_count = {}
for ii, at in enumerate(set(df['archiveType'])):
archive_count[at] = df.loc[df['archiveType']==at, 'archiveType'].count()
sort = np.argsort([cc for cc in archive_count.values()])
archives_sorted = np.array([cc for cc in archive_count.keys()])[sort][::-1]
# Specify colour for each archive (smaller archives get grouped into the same colour)
archive_colour, major_archives, other_archives = uplt.get_archive_colours(archives_sorted, archive_count)
fig = uplt.plot_geo_archive_proxy(df, archive_colour)
utf.save_fig(fig, f'geo_{df.name}', dir=df.name)
# count archive types
archive_count = {}
for ii, at in enumerate(set(df['archiveType'])):
archive_count[at] = df.loc[df['archiveType']==at, 'archiveType'].count()
sort = np.argsort([cc for cc in archive_count.values()])
archives_sorted = np.array([cc for cc in archive_count.keys()])[sort][::-1]
# Specify colour for each archive (smaller archives get grouped into the same colour)
archive_colour, major_archives, other_archives = uplt.get_archive_colours(archives_sorted, archive_count)
fig = uplt.plot_geo_archive_proxy(df, archive_colour)
utf.save_fig(fig, f'geo_{df.name}', dir=df.name)
0 GlacierIce 146 1 Coral 115 2 LakeSediment 67 3 Speleothem 41 4 Wood 33 5 MarineSediment 24 6 GroundIce 6 7 Sclerosponge 2 8 MolluskShell 1 saved figure in /home/jupyter-lluecke/dod2k/figs/iso2k/geo_iso2k.pdf
Now plot the coverage over the Common Era
In [25]:
Copied!
fig = uplt.plot_coverage(df, archives_sorted, major_archives, other_archives, archive_colour)
utf.save_fig(fig, f'time_{df.name}', dir=df.name)
fig = uplt.plot_coverage(df, archives_sorted, major_archives, other_archives, archive_colour)
utf.save_fig(fig, f'time_{df.name}', dir=df.name)
saved figure in /home/jupyter-lluecke/dod2k/figs/iso2k/time_iso2k.pdf
Display dataframe¶
Display identification metadata: dataSetName, datasetId, originalDataURL, originalDatabase¶
index¶
In [26]:
Copied!
# # check index
print(df.index)
# # check index
print(df.index)
RangeIndex(start=0, stop=435, step=1)
dataSetName (associated with each record, may not be unique)¶
In [27]:
Copied!
# # check dataSetName
key = 'dataSetName'
print('%s: '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# # check dataSetName
key = 'dataSetName'
print('%s: '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
dataSetName: ['CO14WUCL' 'IC00OEKC' 'MS09TCIS' 'CO95TUNG' 'CO17MUMA' 'CO17MUMA' 'IC16WE30' 'LS10WRNA' 'IC10VICR' 'IC10VICR' 'CO98SWPR' 'CO04BAFI' 'CO04BAFI' 'CO04BAFI' 'SP11STBR' 'CO17DESC03A' 'CO17DESC03A' 'IC13STSP' 'LS13WASU' 'LS13WASU' 'LS16STPA' 'IC09VIA8' 'CO99DRGB' 'CO00URMA' 'CO00URMA' 'IC11LAB4' 'IC96MOSS' 'IC06VING' 'IC00STHN' 'IC99IS89' 'LS09LAEL' 'SP08HUCN' 'IC97THGL' 'CO09NUTB' 'MS96LKSS' 'MS96LKSS' 'SP10BEIN' 'CO04KIVA' 'IC02STTA' 'MS11KDMS' 'IC16WE22' 'MS09TRFD' 'MS09TRFD' 'TR16LAAN' 'CO13HENG' 'CO13HENG' 'SP12CACN' 'CO96QUVA' 'LS11TICH' 'SP09GRID' 'SP09GRID' 'IC13ST03' 'IC15GANK' 'IC15GANK' 'CO97CHSY' 'SP07ASUS' 'CO04ZIMG' 'CO98EVXM' 'IC85LAD2' 'IC10VISD' 'IC10VISD' 'TR11SAHU00' 'IC16WE20' 'TR13POMA' 'LS91HOMI' 'SP07FLOM' 'SP07DEUS' 'SP07DEUS' 'LS15LAOH' 'IC13STWD' 'IC13STWD' 'IC13STWD' 'CO14ZIMG' 'CO14ZIMG' 'IW15MELD' 'IC12WACD' 'IC02HEWD' 'IC04FIML' 'IC09VICC' 'IC16WE17' 'CO17DESC04A' 'CO17DESC04A' 'CO14OSPA' 'CO14OSPA' 'CO06LIFI' 'IC96ISE9' 'IC05ISAU' 'CO99SWFB' 'CO99SWFB' 'SP06DYCN' 'SP05MAAT' 'LS16STCL' 'IC12RHME' 'LS02HATI' 'LS10HEQI' 'LS12THAY' 'IC00SC29' 'IC06VID3' 'IC99SCFH' 'TR14KOSP' 'TR14KOSP' 'LS11ANSE' 'IC10VISA' 'IC10VISA' 'TR12BRBO' 'TR10ANMO' 'CO14CATI' 'LS09SASP' 'LS09SASP' 'LS06JONA' 'TR08THCO' 'LS09BAUM' 'IC06THPR' 'IC06THPR' 'IC06THPR' 'IC13STN7' 'TR06TRBO' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'CO03COPM' 'SP04YUCN' 'CO93COTW' 'CO13COTB' 'GI19POYU' 'GI19POYU' 'GI19POYU' 'GI19POYU' 'TR11GRTP' 'IC06KAS1' 'IC12SIWG' 'CO13CABL' 'CO13CABL' 'IC01JO01' 'CO11NUPM' 'CO11NUPM' 'CO17WUBO1G' 'TR11BAPU' 'CO17WUBO1B' 'CO13COXM' 'MS06DLGN' 'MS06DLGN' 'LS13KOLA' 'CO06MOPE' 'IC82STVE' 'IC82STVE' 'LS17VAWA01' 'CO96SWFB' 'IC07KADF' 'MS11SENS' 'SP06DRIT' 'SP13AYID' 'SP13AYID' 'CO02KUBE' 'CO99GUMI' 'CO99GUMI' 'IC16WE19' 'CO96SWBB' 'IC16WE28' 'IC95FI87' 'SP13MCUS' 'LS11MOES' 'TR16WEXI' 'TR13SIKO' 'MS14GFSS' 'MS14GFSS' 'LS12WOQU' 'SP10WINZ' 'SP10WINZ' 'IC08VI84' 'LS02ROSA' 'CO99KUHO' 'CO99KUHO' 'LS16THQI01' 'ClelandLake.Steinman.2016' 'ClelandLake.Steinman.2016' 'SS05ROTN' 'IC11STTA' 'CO17WUBO1D' 'CO03CHBU' 'LS11KOMA' 'TR12BECO' 'LS08LICA' 'LS11RYKA' 'LS14FOTI' 'LS00WODE' 'IC10VIP2' 'IC10VIP2' 'CO17RAPA' 'IC13STN8' 'IC17EKN3' 'CO12GOVA' 'SP08ZHCN' 'CO18DATO01A' 'TR11MAPE' 'MS07KPPB' 'MS07KPPB' 'IC03HOIL' 'TR15BABO' 'IC10VIP1' 'IC10VIP1' 'IC95THHU' 'IC95THHU' 'IC95THHU' 'IC95THHU' 'IC95THHU' 'IC00EIGG' 'LS96VOAM' 'CO18RELC01A' 'CO98QUNC' 'IC10VIMI' 'IC10VIMI' 'IC06THDD' 'LS14KOSA' 'IC14MAFH' 'IC08VI87' 'LS06CRLA' 'CO17WUBO1C' 'TR08HORA' 'IC92HOML' 'LS03RUED' 'IC98FIPE' 'TR15YOLL' 'CO00COKY' 'IC11LIMI' 'LS16THN301' 'LS16THN301' 'LS16THN301' 'LS16THN301' 'IC00COPR' 'TR11BAVO' 'TR11BAVO' 'CO07CAFR' 'CO08GOBE' 'CO05BAFI' 'IC13OPAN' 'LS06DENA' 'TR17GRTP' 'IC17BERI' 'TR18XUGA00' 'LS06SHBE' 'SP08SPUS' 'IC10VISE' 'IC10VISE' 'SP07WABR' 'CO94LISE' 'CO04PFRE' 'LS08STCR' 'IC09EIBE' 'IC06WAML' 'IC08VI79' 'LS10HOLO' 'SP11BOAT' 'MS12AMMM' 'IC13THFE' 'CO01TUNG' 'CO01TUNG' 'IC99ISS2' 'IC11BEVL' 'IC10NAB5' 'IC00OE18' 'IC16WENG' 'LS15DENA' 'LS13STEL' 'CO98CHPI' 'IC17EK4K' 'CO04LIFI' 'IC94THDP' 'LS09SAWA' 'IC02THKI' 'CO14CABT' 'CO14CABT' 'CO14CABT' 'SP10CACN' 'SP10CACN' 'IC00SC16' 'IC16GOBP' 'LS89LATI' 'SP03BMIL' 'CO06QUNG' 'IC17EKPV' 'IC02SCGN' 'CO03CHBA' 'CO15ABSP' 'TR17SAMA' 'LS10ANBL' 'LS02BEPY' 'MS05LKLF' 'LS09SHJO' 'MS12CCCH' 'TR13SAWA00' 'MS02FLPC' 'TR11MAJA' 'SP05CRBR' 'IC00OE05' 'IC16WE23' 'MS06DLDS' 'MS06DLDS' 'LS16WIGH' 'LS16WIGH' 'SH18CASA' 'IC00SC21' 'MS01HDAS' 'MS06DLGS' 'MS06DLGS' 'CO06MOTO' 'CO06MOTO' 'LS12STRE' 'LS09SAJU' 'LS09SAJU' 'IC11DILO' 'LS15OEPO' 'SP09FLTR' 'CO04LIRA' 'CO04LIRA' 'CO04LIRA' 'SP12NOBR' 'MS07SWGG' 'TR12SAMU' 'TR12SAMU' 'TR12SAMU' 'CO05KUBE' 'IC06VIGR' 'IC10VIGI' 'IC10VIGI' 'CO00KUNI' 'CO00KUNI' 'TR15NACA' 'CO05ASGU' 'LS09LACA' 'LS15SHBO' 'CO99LICL' 'CO99LICL' 'CO99LICL' 'IC13PALD' 'IC10VIGR' 'IC11KIPW' 'CO98BOFP' 'SS05ROES' 'IW17OPOY' 'TR13BROA' 'SP98KUCN' 'IC09KANG' 'IC09VIRE' 'IC00THDS' 'IC00THDS' 'LS14ZHJU' 'IC13ST21' 'IC07KARB' 'CO13DESC01A' 'CO13DESC01A' 'IC15MANE' 'LS06TIST' 'IC09OPAN' 'IC09OPAN' 'IC14EKVK' 'LS14YAHA' 'LS14YAHA' 'CO04PFCH' 'CO04PFCH' 'IC12MUJR' 'IC08VISV' 'IC00SC26' 'IC93BAGI' 'IC93BAGI' 'IC76FIDE' 'IC76FIDE' 'LS11BACH' 'CO10HEIG' 'CO00DRBE' 'CO00DRBE' 'SP10SUSE' 'CO06DATZ' 'LS01HUFA' 'LS01HUFA' 'CO09NAKY' 'MS03BRNS' 'CO17XISC01A' 'CO17XISC01A' 'IC10VIP3' 'IC10VIP3' 'IC97GRGI' 'SP04DEUS' 'IC13BOCG' 'IC13BOCG' 'IC13BOCG' 'IC13BOCG' 'LS12STLI' 'IC00OE14' 'CO08HEVE' 'SP10HAUS' 'SP10HAUS' 'SP10HAUS' 'IC10VISG' 'IC10VISG' 'LS17BAHA01' 'LS17BAHA01' 'IC10VID3' 'IC10VID3' 'IC10VID3' 'IC10VID3' 'IC10VID3' 'IC10VID3' 'IC10VID3' 'IC10VID3' 'SP12ERUS' 'CO00FERA' 'IC02GR05' 'SP12FOAT' 'TR16WEMI' 'IC98THSA' 'IC08VIRE' 'SP11BEMX' 'SP08VBPE' 'IC13THQU' 'TR15YONW' 'CO17DESC02A' 'CO17DESC02A' 'IC10VIRE' 'IC10VIRE' 'IC09KA79' 'TR04EVLI' 'CO09FEOG' 'CO17DESC01A' 'CO17DESC01A' 'SP99DEUS' 'SP99DEUS'] ["<class 'str'>"] No. of unique values: 327/435
datasetId (unique identifier, as given by original authors, includes original database token)¶
In [28]:
Copied!
# # check datasetId
print(len(df.datasetId.unique()))
print(len(df))
key = 'datasetId'
print('%s (starts with): '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print('datasetId starts with: ', np.unique([str(dd.split('_')[0]) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# # check datasetId
print(len(df.datasetId.unique()))
print(len(df))
key = 'datasetId'
print('%s (starts with): '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print('datasetId starts with: ', np.unique([str(dd.split('_')[0]) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
435 435 datasetId (starts with): ['iso2k_0' 'iso2k_12' 'iso2k_17' 'iso2k_20' 'iso2k_23' 'iso2k_25' 'iso2k_26' 'iso2k_31' 'iso2k_35' 'iso2k_37' 'iso2k_50' 'iso2k_52' 'iso2k_55' 'iso2k_58' 'iso2k_60' 'iso2k_67' 'iso2k_68' 'iso2k_72' 'iso2k_74' 'iso2k_77' 'iso2k_78' 'iso2k_88' 'iso2k_91' 'iso2k_94' 'iso2k_98' 'iso2k_100' 'iso2k_102' 'iso2k_104' 'iso2k_110' 'iso2k_112' 'iso2k_117' 'iso2k_120' 'iso2k_124' 'iso2k_127' 'iso2k_134' 'iso2k_137' 'iso2k_140' 'iso2k_155' 'iso2k_174' 'iso2k_179' 'iso2k_185' 'iso2k_203' 'iso2k_206' 'iso2k_207' 'iso2k_211' 'iso2k_213' 'iso2k_215' 'iso2k_218' 'iso2k_220' 'iso2k_236' 'iso2k_239' 'iso2k_244' 'iso2k_249' 'iso2k_250' 'iso2k_252' 'iso2k_254' 'iso2k_257' 'iso2k_259' 'iso2k_264' 'iso2k_267' 'iso2k_270' 'iso2k_272' 'iso2k_273' 'iso2k_279' 'iso2k_281' 'iso2k_284' 'iso2k_287' 'iso2k_290' 'iso2k_293' 'iso2k_296' 'iso2k_298' 'iso2k_299' 'iso2k_300' 'iso2k_302' 'iso2k_304' 'iso2k_311' 'iso2k_320' 'iso2k_324' 'iso2k_339' 'iso2k_341' 'iso2k_345' 'iso2k_346' 'iso2k_347' 'iso2k_350' 'iso2k_353' 'iso2k_358' 'iso2k_362' 'iso2k_373' 'iso2k_376' 'iso2k_380' 'iso2k_396' 'iso2k_399' 'iso2k_404' 'iso2k_410' 'iso2k_415' 'iso2k_424' 'iso2k_433' 'iso2k_443' 'iso2k_446' 'iso2k_449' 'iso2k_450' 'iso2k_454' 'iso2k_457' 'iso2k_460' 'iso2k_463' 'iso2k_465' 'iso2k_473' 'iso2k_474' 'iso2k_477' 'iso2k_478' 'iso2k_481' 'iso2k_485' 'iso2k_488' 'iso2k_490' 'iso2k_492' 'iso2k_498' 'iso2k_503' 'iso2k_505' 'iso2k_507' 'iso2k_509' 'iso2k_511' 'iso2k_513' 'iso2k_515' 'iso2k_517' 'iso2k_519' 'iso2k_521' 'iso2k_523' 'iso2k_525' 'iso2k_527' 'iso2k_533' 'iso2k_539' 'iso2k_541' 'iso2k_546' 'iso2k_547' 'iso2k_549' 'iso2k_550' 'iso2k_556' 'iso2k_558' 'iso2k_567' 'iso2k_569' 'iso2k_571' 'iso2k_573' 'iso2k_575' 'iso2k_579' 'iso2k_581' 'iso2k_585' 'iso2k_587' 'iso2k_591' 'iso2k_607' 'iso2k_610' 'iso2k_611' 'iso2k_629' 'iso2k_630' 'iso2k_633' 'iso2k_640' 'iso2k_641' 'iso2k_644' 'iso2k_646' 'iso2k_650' 'iso2k_653' 'iso2k_656' 'iso2k_698' 'iso2k_702' 'iso2k_705' 'iso2k_706' 'iso2k_715' 'iso2k_718' 'iso2k_723' 'iso2k_726' 'iso2k_742' 'iso2k_745' 'iso2k_747' 'iso2k_757' 'iso2k_762' 'iso2k_766' 'iso2k_772' 'iso2k_775' 'iso2k_778' 'iso2k_781' 'iso2k_786' 'iso2k_788' 'iso2k_799' 'iso2k_806' 'iso2k_811' 'iso2k_818' 'iso2k_821' 'iso2k_824' 'iso2k_826' 'iso2k_828' 'iso2k_834' 'iso2k_836' 'iso2k_841' 'iso2k_844' 'iso2k_848' 'iso2k_857' 'iso2k_859' 'iso2k_864' 'iso2k_866' 'iso2k_868' 'iso2k_870' 'iso2k_873' 'iso2k_879' 'iso2k_880' 'iso2k_883' 'iso2k_886' 'iso2k_889' 'iso2k_892' 'iso2k_897' 'iso2k_899' 'iso2k_900' 'iso2k_902' 'iso2k_903' 'iso2k_906' 'iso2k_909' 'iso2k_910' 'iso2k_913' 'iso2k_917' 'iso2k_921' 'iso2k_928' 'iso2k_931' 'iso2k_932' 'iso2k_935' 'iso2k_938' 'iso2k_954' 'iso2k_965' 'iso2k_975' 'iso2k_979' 'iso2k_990' 'iso2k_995' 'iso2k_998' 'iso2k_1000' 'iso2k_1010' 'iso2k_1014' 'iso2k_1019' 'iso2k_1033' 'iso2k_1036' 'iso2k_1039' 'iso2k_1044' 'iso2k_1054' 'iso2k_1056' 'iso2k_1057' 'iso2k_1060' 'iso2k_1068' 'iso2k_1069' 'iso2k_1083' 'iso2k_1086' 'iso2k_1099' 'iso2k_1102' 'iso2k_1106' 'iso2k_1107' 'iso2k_1111' 'iso2k_1114' 'iso2k_1118' 'iso2k_1124' 'iso2k_1130' 'iso2k_1132' 'iso2k_1148' 'iso2k_1151' 'iso2k_1170' 'iso2k_1173' 'iso2k_1178' 'iso2k_1190' 'iso2k_1199' 'iso2k_1201' 'iso2k_1203' 'iso2k_1205' 'iso2k_1212' 'iso2k_1214' 'iso2k_1216' 'iso2k_1218' 'iso2k_1221' 'iso2k_1223' 'iso2k_1229' 'iso2k_1255' 'iso2k_1264' 'iso2k_1267' 'iso2k_1269' 'iso2k_1277' 'iso2k_1283' 'iso2k_1286' 'iso2k_1287' 'iso2k_1288' 'iso2k_1291' 'iso2k_1294' 'iso2k_1296' 'iso2k_1304' 'iso2k_1307' 'iso2k_1311' 'iso2k_1315' 'iso2k_1320' 'iso2k_1322' 'iso2k_1340' 'iso2k_1344' 'iso2k_1358' 'iso2k_1366' 'iso2k_1380' 'iso2k_1384' 'iso2k_1395' 'iso2k_1397' 'iso2k_1398' 'iso2k_1407' 'iso2k_1416' 'iso2k_1419' 'iso2k_1425' 'iso2k_1430' 'iso2k_1434' 'iso2k_1439' 'iso2k_1440' 'iso2k_1446' 'iso2k_1449' 'iso2k_1458' 'iso2k_1462' 'iso2k_1466' 'iso2k_1467' 'iso2k_1470' 'iso2k_1474' 'iso2k_1476' 'iso2k_1480' 'iso2k_1481' 'iso2k_1488' 'iso2k_1495' 'iso2k_1498' 'iso2k_1500' 'iso2k_1502' 'iso2k_1504' 'iso2k_1519' 'iso2k_1525' 'iso2k_1528' 'iso2k_1529' 'iso2k_1530' 'iso2k_1534' 'iso2k_1542' 'iso2k_1545' 'iso2k_1554' 'iso2k_1556' 'iso2k_1557' 'iso2k_1559' 'iso2k_1563' 'iso2k_1566' 'iso2k_1569' 'iso2k_1571' 'iso2k_1573' 'iso2k_1575' 'iso2k_1577' 'iso2k_1579' 'iso2k_1581' 'iso2k_1587' 'iso2k_1590' 'iso2k_1593' 'iso2k_1595' 'iso2k_1619' 'iso2k_1626' 'iso2k_1628' 'iso2k_1631' 'iso2k_1633' 'iso2k_1637' 'iso2k_1639' 'iso2k_1643' 'iso2k_1644' 'iso2k_1645' 'iso2k_1653' 'iso2k_1658' 'iso2k_1660' 'iso2k_1690' 'iso2k_1695' 'iso2k_1698' 'iso2k_1701' 'iso2k_1704' 'iso2k_1708' 'iso2k_1710' 'iso2k_1713' 'iso2k_1716' 'iso2k_1719' 'iso2k_1727' 'iso2k_1729' 'iso2k_1732' 'iso2k_1735' 'iso2k_1738' 'iso2k_1740' 'iso2k_1745' 'iso2k_1748' 'iso2k_1750' 'iso2k_1753' 'iso2k_1754' 'iso2k_1756' 'iso2k_1762' 'iso2k_1763' 'iso2k_1770' 'iso2k_1772' 'iso2k_1779' 'iso2k_1790' 'iso2k_1795' 'iso2k_1798' 'iso2k_1799' 'iso2k_1800' 'iso2k_1801' 'iso2k_1811' 'iso2k_1813' 'iso2k_1817' 'iso2k_1820' 'iso2k_1823' 'iso2k_1832' 'iso2k_1835' 'iso2k_1837' 'iso2k_1839' 'iso2k_1846' 'iso2k_1848' 'iso2k_1850' 'iso2k_1851' 'iso2k_1852' 'iso2k_1853' 'iso2k_1854' 'iso2k_1855' 'iso2k_1856' 'iso2k_1861' 'iso2k_1862' 'iso2k_1864' 'iso2k_1867' 'iso2k_1869' 'iso2k_1872' 'iso2k_1875' 'iso2k_1883' 'iso2k_1885' 'iso2k_1898' 'iso2k_1901' 'iso2k_1902' 'iso2k_1903' 'iso2k_1906' 'iso2k_1916' 'iso2k_1918' 'iso2k_1922' 'iso2k_1950' 'iso2k_1951' 'iso2k_1952' 'iso2k_1955'] ["<class 'str'>"] datasetId starts with: ['iso2k'] No. of unique values: 435/435
originalDataURL (URL/DOI of original published record where available)¶
In [29]:
Copied!
# originalDataURL
key = 'originalDataURL'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([kk for kk in df[key] if 'this' in kk]))
print(np.unique([str(type(dd)) for dd in df[key]]))
# 'this study' should point to the correct URL (PAGES2k)
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# originalDataURL
key = 'originalDataURL'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([kk for kk in df[key] if 'this' in kk]))
print(np.unique([str(type(dd)) for dd in df[key]]))
# 'this study' should point to the correct URL (PAGES2k)
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
originalDataURL: ['This compilation' 'http://doi.pangaea.de/10.1594/PANGAEA.711835' 'http://doi.pangaea.de/10.1594/PANGAEA.735717' 'http://doi.pangaea.de/10.1594/PANGAEA.738188' 'http://doi.pangaea.de/10.1594/PANGAEA.776444' 'http://doi.pangaea.de/10.1594/PANGAEA.780423' 'http://www.iceandclimate.nbi.ku.dk/data/Vinther_etal_2010_data_02feb2010.xls' 'http://www1.ncdc.noaa.gov/pub/data/paleo/paleolimnology/europe/germany/ammersee_1999.txt' 'http://www1.ncdc.noaa.gov/pub/data/paleo/paleolimnology/europe/uk/lough-na-shade2010.txt' 'http://www1.ncdc.noaa.gov/pub/data/paleo/treering/isotope/asia/russia/' 'http://www1.ncdc.noaa.gov/pub/data/paleo/treering/isotope/southamerica/' 'https://agupubs.onlinelibrary.wiley.com/action/downloadSupplement?doi=10.1002%2F2016JC012458&file=jgrc22050-sup-0002-2016JC012458-ds01.xlsx' 'https://ars-els-cdn-com.libezp.lib.lsu.edu/content/image/1-s2.0-S1367912016304138-mmc1.xlsx' 'https://doi.org/10.1594/PANGAEA.864869' 'https://doi.org/10.1594/PANGAEA.869839' 'https://doi.org/10.1594/PANGAEA.869840' 'https://doi.org/10.1594/PANGAEA.892303' 'https://doi.org/10.25921/6e73-as97' 'https://doi.pangaea.de/10.1594/PANGAEA.104862' 'https://doi.pangaea.de/10.1594/PANGAEA.104880' 'https://doi.pangaea.de/10.1594/PANGAEA.104886' 'https://doi.pangaea.de/10.1594/PANGAEA.104889' 'https://doi.pangaea.de/10.1594/PANGAEA.728240' 'https://doi.pangaea.de/10.1594/PANGAEA.760166' 'https://doi.pangaea.de/10.1594/PANGAEA.770159' 'https://doi.pangaea.de/10.1594/PANGAEA.786354' 'https://doi.pangaea.de/10.1594/PANGAEA.786356' 'https://doi.pangaea.de/10.1594/PANGAEA.824732' 'https://doi.pangaea.de/10.1594/PANGAEA.829390' 'https://doi.pangaea.de/10.1594/PANGAEA.842166' 'https://doi.pangaea.de/10.1594/PANGAEA.849161' 'https://doi.pangaea.de/10.1594/PANGAEA.857573' 'https://doi.pangaea.de/10.1594/PANGAEA.866368' 'https://doi.pangaea.de/10.1594/PANGAEA.871279' 'https://doi.pangaea.de/10.1594/PANGAEA.873563' 'https://doi.pangaea.de/10.1594/PANGAEA.874205' 'https://doi.pangaea.de/10.1594/PANGAEA.880012' 'https://doi.pangaea.de/10.1594/PANGAEA.880396' 'https://doi.pangaea.de/10.1594/PANGAEA.880528' 'https://www.ncdc.noaa.gov/cdo/f?p=519:1:0::::P1_STUDY_ID:13174' 'https://www.ncdc.noaa.gov/cdo/f?p=519:1:::::P1_STUDY_ID:5472' 'https://www.ncdc.noaa.gov/cdo/f?p=519:1:::::P1_STUDY_ID:8647' 'https://www.ncdc.noaa.gov/paleo-search/study/11173' 'https://www.ncdc.noaa.gov/paleo-search/study/11180' 'https://www.ncdc.noaa.gov/paleo-search/study/15072' 'https://www.ncdc.noaa.gov/paleo-search/study/17798' 'https://www.ncdc.noaa.gov/paleo-search/study/21171' 'https://www.ncdc.noaa.gov/paleo-search/study/22471' 'https://www.ncdc.noaa.gov/paleo-search/study/22479' 'https://www.ncdc.noaa.gov/paleo-search/study/22543' 'https://www.ncdc.noaa.gov/paleo-search/study/22589' 'https://www.ncdc.noaa.gov/paleo-search/study/2431' 'https://www.ncdc.noaa.gov/paleo-search/study/2436' 'https://www.ncdc.noaa.gov/paleo-search/study/2448' 'https://www.ncdc.noaa.gov/paleo-search/study/2466' 'https://www.ncdc.noaa.gov/paleo-search/study/2492' 'https://www.ncdc.noaa.gov/paleo-search/study/2519' 'https://www.ncdc.noaa.gov/paleo-search/study/5416' 'https://www.ncdc.noaa.gov/paleo-search/study/5425' 'https://www.ncdc.noaa.gov/paleo-search/study/5427' 'https://www.ncdc.noaa.gov/paleo-search/study/5441' 'https://www.ncdc.noaa.gov/paleo-search/study/5541' 'https://www.ncdc.noaa.gov/paleo-search/study/5966' 'https://www.ncdc.noaa.gov/paleo-search/study/6182' 'https://www.ncdc.noaa.gov/paleo-search/study/9741' 'https://www.ncdc.noaa.gov/paleo-search/study/9742' 'https://www.ncdc.noaa.gov/paleo/study/10373' 'https://www.ncdc.noaa.gov/paleo/study/10455' 'https://www.ncdc.noaa.gov/paleo/study/10670' 'https://www.ncdc.noaa.gov/paleo/study/10750' 'https://www.ncdc.noaa.gov/paleo/study/10808' 'https://www.ncdc.noaa.gov/paleo/study/10889' 'https://www.ncdc.noaa.gov/paleo/study/11131' 'https://www.ncdc.noaa.gov/paleo/study/11148' 'https://www.ncdc.noaa.gov/paleo/study/11197' 'https://www.ncdc.noaa.gov/paleo/study/12426' 'https://www.ncdc.noaa.gov/paleo/study/12841' 'https://www.ncdc.noaa.gov/paleo/study/12885' 'https://www.ncdc.noaa.gov/paleo/study/12891' 'https://www.ncdc.noaa.gov/paleo/study/12893' 'https://www.ncdc.noaa.gov/paleo/study/12916' 'https://www.ncdc.noaa.gov/paleo/study/12942' 'https://www.ncdc.noaa.gov/paleo/study/12994' 'https://www.ncdc.noaa.gov/paleo/study/13079' 'https://www.ncdc.noaa.gov/paleo/study/13114' 'https://www.ncdc.noaa.gov/paleo/study/13175' 'https://www.ncdc.noaa.gov/paleo/study/13195' 'https://www.ncdc.noaa.gov/paleo/study/13439' 'https://www.ncdc.noaa.gov/paleo/study/13540' 'https://www.ncdc.noaa.gov/paleo/study/13543' 'https://www.ncdc.noaa.gov/paleo/study/13625' 'https://www.ncdc.noaa.gov/paleo/study/13670' 'https://www.ncdc.noaa.gov/paleo/study/13672' 'https://www.ncdc.noaa.gov/paleo/study/13954' 'https://www.ncdc.noaa.gov/paleo/study/14129' 'https://www.ncdc.noaa.gov/paleo/study/14174' 'https://www.ncdc.noaa.gov/paleo/study/14510' 'https://www.ncdc.noaa.gov/paleo/study/14652' 'https://www.ncdc.noaa.gov/paleo/study/15444' 'https://www.ncdc.noaa.gov/paleo/study/15794' 'https://www.ncdc.noaa.gov/paleo/study/16139' 'https://www.ncdc.noaa.gov/paleo/study/16155' 'https://www.ncdc.noaa.gov/paleo/study/16216' 'https://www.ncdc.noaa.gov/paleo/study/16339' 'https://www.ncdc.noaa.gov/paleo/study/16357' 'https://www.ncdc.noaa.gov/paleo/study/16438' 'https://www.ncdc.noaa.gov/paleo/study/17289' 'https://www.ncdc.noaa.gov/paleo/study/17378' 'https://www.ncdc.noaa.gov/paleo/study/17380' 'https://www.ncdc.noaa.gov/paleo/study/17576' 'https://www.ncdc.noaa.gov/paleo/study/17796' 'https://www.ncdc.noaa.gov/paleo/study/17919' 'https://www.ncdc.noaa.gov/paleo/study/18315' 'https://www.ncdc.noaa.gov/paleo/study/18355' 'https://www.ncdc.noaa.gov/paleo/study/1839' 'https://www.ncdc.noaa.gov/paleo/study/1842' 'https://www.ncdc.noaa.gov/paleo/study/1843' 'https://www.ncdc.noaa.gov/paleo/study/1844' 'https://www.ncdc.noaa.gov/paleo/study/1845' 'https://www.ncdc.noaa.gov/paleo/study/1846' 'https://www.ncdc.noaa.gov/paleo/study/1847' 'https://www.ncdc.noaa.gov/paleo/study/1853' 'https://www.ncdc.noaa.gov/paleo/study/1855' 'https://www.ncdc.noaa.gov/paleo/study/1856' 'https://www.ncdc.noaa.gov/paleo/study/1857' 'https://www.ncdc.noaa.gov/paleo/study/1859' 'https://www.ncdc.noaa.gov/paleo/study/1861' 'https://www.ncdc.noaa.gov/paleo/study/1866' 'https://www.ncdc.noaa.gov/paleo/study/1867' 'https://www.ncdc.noaa.gov/paleo/study/1872' 'https://www.ncdc.noaa.gov/paleo/study/1875' 'https://www.ncdc.noaa.gov/paleo/study/1876' 'https://www.ncdc.noaa.gov/paleo/study/1881' 'https://www.ncdc.noaa.gov/paleo/study/1885' 'https://www.ncdc.noaa.gov/paleo/study/1886' 'https://www.ncdc.noaa.gov/paleo/study/18895' 'https://www.ncdc.noaa.gov/paleo/study/1890' 'https://www.ncdc.noaa.gov/paleo/study/1891' 'https://www.ncdc.noaa.gov/paleo/study/1901' 'https://www.ncdc.noaa.gov/paleo/study/1903' 'https://www.ncdc.noaa.gov/paleo/study/1911' 'https://www.ncdc.noaa.gov/paleo/study/1913' 'https://www.ncdc.noaa.gov/paleo/study/1914' 'https://www.ncdc.noaa.gov/paleo/study/1915' 'https://www.ncdc.noaa.gov/paleo/study/1916' 'https://www.ncdc.noaa.gov/paleo/study/19179' 'https://www.ncdc.noaa.gov/paleo/study/1920' 'https://www.ncdc.noaa.gov/paleo/study/1924' 'https://www.ncdc.noaa.gov/paleo/study/1925' 'https://www.ncdc.noaa.gov/paleo/study/19421' 'https://www.ncdc.noaa.gov/paleo/study/20002' 'https://www.ncdc.noaa.gov/paleo/study/20126' 'https://www.ncdc.noaa.gov/paleo/study/20350' 'https://www.ncdc.noaa.gov/paleo/study/20610' 'https://www.ncdc.noaa.gov/paleo/study/20904' 'https://www.ncdc.noaa.gov/paleo/study/20930' 'https://www.ncdc.noaa.gov/paleo/study/21250' 'https://www.ncdc.noaa.gov/paleo/study/22120' 'https://www.ncdc.noaa.gov/paleo/study/22409' 'https://www.ncdc.noaa.gov/paleo/study/22477' 'https://www.ncdc.noaa.gov/paleo/study/22502' 'https://www.ncdc.noaa.gov/paleo/study/22519' 'https://www.ncdc.noaa.gov/paleo/study/22531' 'https://www.ncdc.noaa.gov/paleo/study/22532' 'https://www.ncdc.noaa.gov/paleo/study/22541' 'https://www.ncdc.noaa.gov/paleo/study/22542' 'https://www.ncdc.noaa.gov/paleo/study/22547' 'https://www.ncdc.noaa.gov/paleo/study/22548' 'https://www.ncdc.noaa.gov/paleo/study/22549' 'https://www.ncdc.noaa.gov/paleo/study/22592' 'https://www.ncdc.noaa.gov/paleo/study/22602' 'https://www.ncdc.noaa.gov/paleo/study/22712' 'https://www.ncdc.noaa.gov/paleo/study/23076' 'https://www.ncdc.noaa.gov/paleo/study/23081' 'https://www.ncdc.noaa.gov/paleo/study/23092' 'https://www.ncdc.noaa.gov/paleo/study/23095' 'https://www.ncdc.noaa.gov/paleo/study/23231' 'https://www.ncdc.noaa.gov/paleo/study/23232' 'https://www.ncdc.noaa.gov/paleo/study/23233' 'https://www.ncdc.noaa.gov/paleo/study/23850' 'https://www.ncdc.noaa.gov/paleo/study/2424' 'https://www.ncdc.noaa.gov/paleo/study/2447' 'https://www.ncdc.noaa.gov/paleo/study/24630' 'https://www.ncdc.noaa.gov/paleo/study/2494' 'https://www.ncdc.noaa.gov/paleo/study/29432' 'https://www.ncdc.noaa.gov/paleo/study/5423' 'https://www.ncdc.noaa.gov/paleo/study/5428' 'https://www.ncdc.noaa.gov/paleo/study/5431' 'https://www.ncdc.noaa.gov/paleo/study/5433' 'https://www.ncdc.noaa.gov/paleo/study/5537' 'https://www.ncdc.noaa.gov/paleo/study/5596' 'https://www.ncdc.noaa.gov/paleo/study/5968' 'https://www.ncdc.noaa.gov/paleo/study/6087' 'https://www.ncdc.noaa.gov/paleo/study/6089' 'https://www.ncdc.noaa.gov/paleo/study/6095' 'https://www.ncdc.noaa.gov/paleo/study/6111' 'https://www.ncdc.noaa.gov/paleo/study/6115' 'https://www.ncdc.noaa.gov/paleo/study/6116' 'https://www.ncdc.noaa.gov/paleo/study/6180' 'https://www.ncdc.noaa.gov/paleo/study/6184' 'https://www.ncdc.noaa.gov/paleo/study/6242' 'https://www.ncdc.noaa.gov/paleo/study/8608' 'https://www.ncdc.noaa.gov/paleo/study/8609' 'https://www.ncdc.noaa.gov/paleo/study/8629' 'https://www.ncdc.noaa.gov/paleo/study/8632' 'https://www.ncdc.noaa.gov/paleo/study/8635' 'https://www.ncdc.noaa.gov/paleo/study/8637' 'https://www.ncdc.noaa.gov/paleo/study/8639' 'https://www.ncdc.noaa.gov/paleo/study/8640' 'https://www.ncdc.noaa.gov/paleo/study/8700' 'https://www.ncdc.noaa.gov/paleo/study/8725' 'https://www.ncdc.noaa.gov/paleo/study/9739' 'https://www.ncdc.noaa.gov/paleo/study/9745' 'https://www.ncdc.noaa.gov/paleo/study/9790' 'https://www.ncdc.noaa.gov/paleo/study/9792' 'https://www1.ncdc.noaa.gov/pub/data/paleo/icecore/greenland/gisp/dye2/dye2ad77.txt' 'https://www1.ncdc.noaa.gov/pub/data/paleo/icecore/polar/agassiz/a87del18_1yr.txt' 'https://www1.ncdc.noaa.gov/pub/data/paleo/icecore/polar/devon/d7273del_5yr.txt' 'https://www1.ncdc.noaa.gov/pub/data/paleo/treering/isotope/asia/nepal/ganesh2018d18o.txt' 'nan' 'this compilation' 'www.ncdc.noaa.gov/paleo-search/study/27330' 'www.ncdc.noaa.gov/paleo/study/2474'] ['this compilation'] ["<class 'str'>"] No. of unique values: 223/435
originalDatabase (original database used as input for dataframe)¶
In [30]:
Copied!
# # originalDataSet
key = 'originalDatabase'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
# Note: the last two records have missing URLs
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# # originalDataSet
key = 'originalDatabase'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
# Note: the last two records have missing URLs
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
originalDatabase: ['Iso2k v1.1.2'] ["<class 'str'>"] No. of unique values: 1/435
geographical metadata: elevation, latitude, longitude, site name¶
geo_meanElev (mean elevation in m)¶
In [31]:
Copied!
# check Elevation
key = 'geo_meanElev'
print('%s: '%key)
print(df[key])
print(np.unique(['%d'%kk for kk in df[key] if np.isfinite(kk)]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# check Elevation
key = 'geo_meanElev'
print('%s: '%key)
print(df[key])
print(np.unique(['%d'%kk for kk in df[key] if np.isfinite(kk)]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
geo_meanElev:
0 -13.1
1 1439.0
2 178.0
3 -3.0
4 -5.0
...
430 -5.6
431 -6.0
432 -6.0
433 360.0
434 360.0
Name: geo_meanElev, Length: 435, dtype: float32
['-1' '-10' '-1048' '-11' '-12' '-1250' '-1295' '-13' '-136' '-14' '-143'
'-16' '-18' '-2' '-2391' '-25' '-2543' '-3' '-3975' '-4' '-5' '-531'
'-547' '-6' '-620' '-694' '-695' '-7' '-8' '-852' '-875' '-884' '-9'
'-968' '0' '10' '1000' '1030' '104' '1054' '1126' '113' '1140' '1156'
'12' '1200' '124' '1244' '1250' '1260' '1300' '1354' '1363' '1370' '1439'
'150' '1542' '160' '1600' '1626' '1640' '170' '1700' '1713' '1730' '1742'
'178' '1800' '1806' '1835' '1850' '1885' '1900' '1957' '1975' '20' '2002'
'2023' '2030' '2147' '2185' '2200' '2234' '2242' '230' '2315' '2316'
'2350' '240' '2450' '2451' '2480' '250' '2531' '2543' '2598' '265' '2700'
'2733' '2777' '2800' '2820' '2840' '2874' '2882' '2907' '2917' '293'
'294' '2947' '2960' '3' '300' '3018' '3040' '3087' '3092' '3098' '3100'
'3150' '3160' '3174' '3193' '3194' '320' '3200' '3203' '3233' '3238'
'330' '331' '332' '3330' '340' '3406' '3450' '3488' '3500' '3528' '360'
'3600' '3747' '3810' '3848' '3890' '3900' '3950' '4' '40' '400' '4062'
'4078' '415' '4150' '4200' '4250' '44' '4418' '4450' '4500' '4512' '4718'
'477' '48' '4800' '5' '50' '500' '509' '520' '5325' '533' '5340' '550'
'560' '5670' '5680' '570' '5893' '59' '5950' '6' '600' '6048' '6070'
'6200' '626' '63' '630' '6350' '64' '650' '6518' '6542' '657' '670' '68'
'680' '693' '698' '700' '703' '7200' '733' '740' '750' '754' '780' '880'
'9' '913' '94' '97' '976' '990']
["<class 'float'>"]
No. of unique values: 228/435
geo_meanLat (mean latitude in degrees N)¶
In [32]:
Copied!
# # Latitude
key = 'geo_meanLat'
print('%s: '%key)
print(np.unique(['%d'%kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# # Latitude
key = 'geo_meanLat'
print('%s: '%key)
print(np.unique(['%d'%kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
geo_meanLat: ['-1' '-10' '-11' '-12' '-13' '-14' '-15' '-16' '-17' '-18' '-20' '-21' '-22' '-23' '-27' '-28' '-3' '-4' '-41' '-42' '-46' '-5' '-51' '-64' '-66' '-69' '-7' '-70' '-71' '-72' '-73' '-74' '-75' '-76' '-77' '-78' '-79' '-8' '-82' '-84' '-89' '-9' '0' '1' '10' '11' '12' '13' '16' '17' '18' '19' '2' '21' '22' '23' '24' '25' '27' '28' '29' '3' '30' '31' '32' '33' '34' '35' '36' '37' '38' '39' '4' '40' '41' '42' '43' '44' '45' '46' '47' '48' '49' '5' '50' '51' '52' '54' '55' '6' '60' '62' '63' '64' '65' '66' '67' '68' '69' '7' '70' '71' '72' '73' '75' '76' '77' '78' '79' '80' '81'] ["<class 'float'>"] No. of unique values: 293/435
geo_meanLon (mean longitude)¶
In [33]:
Copied!
# # Longitude
key = 'geo_meanLon'
print('%s: '%key)
print(np.unique(['%d'%kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# # Longitude
key = 'geo_meanLon'
print('%s: '%key)
print(np.unique(['%d'%kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
geo_meanLon: ['-1' '-105' '-109' '-110' '-111' '-112' '-113' '-116' '-117' '-118' '-119' '-12' '-122' '-123' '-13' '-133' '-136' '-138' '-140' '-145' '-146' '-149' '-153' '-157' '-159' '-16' '-160' '-161' '-162' '-174' '-22' '-26' '-35' '-36' '-37' '-38' '-39' '-4' '-41' '-42' '-43' '-44' '-45' '-46' '-49' '-50' '-51' '-54' '-57' '-6' '-60' '-61' '-64' '-66' '-67' '-68' '-69' '-70' '-71' '-73' '-74' '-75' '-76' '-77' '-79' '-8' '-80' '-82' '-83' '-84' '-85' '-86' '-88' '-89' '-9' '-91' '-92' '-94' '-95' '-99' '0' '1' '10' '100' '102' '104' '105' '106' '108' '109' '11' '110' '112' '113' '115' '117' '119' '12' '120' '122' '123' '124' '126' '134' '14' '142' '143' '144' '145' '148' '151' '152' '153' '159' '162' '165' '166' '167' '169' '17' '172' '173' '179' '18' '20' '24' '29' '30' '31' '34' '35' '37' '39' '4' '40' '43' '5' '54' '55' '6' '60' '64' '65' '7' '70' '71' '74' '76' '77' '79' '8' '81' '82' '85' '86' '89' '9' '90' '91' '93' '94' '95' '96' '97'] ["<class 'float'>"] No. of unique values: 288/435
geo_siteName (name of collection site)¶
In [34]:
Copied!
# Site Name
key = 'geo_siteName'
print('%s: '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# Site Name
key = 'geo_siteName'
print('%s: '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
geo_siteName: ['Clipperton Atoll' 'DML Plateau (Kottas Camp) FB9802' 'Ionian Sea' 'Madang, Papua New Guinea' 'Doangdoangan Besar' 'Doangdoangan Besar' 'B30' 'Nam Co' 'Crete' 'Crete' 'Principe, Gulf of Guinea' 'Savusavu Bay, Fiji' 'Savusavu Bay, Fiji' 'Savusavu Bay, Fiji' 'Lapa Grande Cave, Brazil' 'Hainan Island' 'Hainan Island' 'SPRESSO (South Pole)' 'Lake Sugan' 'Lake Sugan' 'Paradise' 'Agassiz A84-A87' 'Abraham Reef, Great Barrier Reef, Australia' 'Maiana Atoll, Republic of Kiribati' 'Maiana Atoll, Republic of Kiribati' 'IND 22B4 Coastal DML' 'Siple Station' 'NGRIP' 'Hercules NA©vA©' 'Eastern DML L89 (DML Ice divide traverse L)' 'Laguna de Salvador' 'Heshang Cave, China' 'Guliya' 'Fanning Island' 'Sargasso Sea' 'Sargasso Sea' 'Dandak Cave, India' 'Malo Channel, Espiritu Santo Island, Vanuatu' 'TD96 Talos Dome' 'Makassar strait' 'B22' 'Feni Drift' 'Feni Drift' 'Angouleme, France' 'Misima Island, Papua New Guinea' 'Misima Island, Papua New Guinea' 'Tianmen Cave, China' 'Tangoa Island, Vanuatu' 'Lake Challa' 'Liang Luar Cave, Indonesia' 'Liang Luar Cave, Indonesia' 'US-ITASE-2000-3' 'Noijin Kangsang Glacier' 'Noijin Kangsang Glacier' 'Beau Vallon Bay, Mahe Island, Republic of the Seychelles' 'pink Panter Cave, New Mexico' 'Ifaty Reef, SW Madagascar' 'Kiritimati' 'DYE2' 'Site D' 'Site D' 'Humla, Nepal' 'B20' 'Mackenzie Delta, NWT, Canada' 'Miragoane' 'Qunf Cave, Oman' 'Devils Icebox Cave, Missouri' 'Devils Icebox Cave, Missouri' 'Ohrid' 'WDC05A' 'WDC05A' 'WDC05A' 'Tulear Reef, SW Madagascar' 'Tulear Reef, SW Madagascar' 'Lena River Delta' 'Col du Dome' 'Windy Dome' 'Mt Logan (PRCol)' 'Camp Century' 'B17' 'Hainan Island' 'Hainan Island' 'Palau' 'Palau' 'Savusavu Bay, Fiji' 'DML Ritscherflya E91' 'Austfonna' 'Lignumvitae Basin, Florida Bay' 'Lignumvitae Basin, Florida Bay' 'Dongge Cave, China' 'Spannagel Cave, Austria' 'Cleland Lake' 'Mt Erebus Saddle - MES' 'tibetanus' 'Qinghai' 'Ayr Lake' 'B29' 'DYE-3' 'Fiescherhorn' 'Lake Estanyola de Gerber, Spanish Pyrenees' 'Lake Estanyola de Gerber, Spanish Pyrenees' 'Seven mile' 'Site A' 'Site A' 'Purisima, Northern Bolivia' 'Monteverde Cloud Forest, Costa Rica' 'Ombai Strait, Timor Island' 'Spooky Lake' 'Spooky Lake' 'Lake Nar' 'Columbia Icefield, British Columbia, Canada' 'Lago Umayo' 'Puruogangri ice cap' 'Puruogangri ice cap' 'Puruogangri ice cap' 'NUS 07-1' 'Boibar Valley, Northern Pakistan' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Palmyra (composite)' 'Dongge Cave, China' 'Tarawa Atoll, Republic of Kiribati' 'Fanning Island (composite)' 'DHP174 site, Yukon, NW Canada' 'DHP174 site, Yukon, NW Canada' 'DHP174 site, Yukon, NW Canada' 'DHP174 site, Yukon, NW Canada' 'Reting site, China' 'DML Fimbulisen S100' 'Whitehall Glacier WGH' 'Belize' 'Belize' 'EDC Dome C' 'Palmyra' 'Palmyra' 'Northwestern shore' 'Puerto Maldonado Peru' 'Northwestern shore' 'Kiritimati Island, Line Islands, Kiribati' 'Great Bahama Bank north' 'Great Bahama Bank north' 'Lake Lading' 'Pedra de Lume, Cape Verde Islands' 'Vernagtferner VF79_core1' 'Vernagtferner VF79_core1' 'Wahoo Lake Shelf Core' 'Lignumvitae Basin, Florida Bay' 'Dome F 1993' 'Norwegian Sea' 'Buca della Renella, Italy Flowstone' 'Liang Luar Cave, Indonesia' 'Liang Luar Cave, Indonesia' 'North East Breakers, Bermuda' 'Nauru' 'Nauru' 'B19' 'Alinas Reef, Biscayne National Park, Florida' 'B28' 'Agassiz A87' 'Crystal Cave, California' 'Estanya' 'Xinpu, China' 'Russia, Koksu, Russian Altai' 'Savu Sea' 'Savu Sea' 'Quartz Lake' 'South Island Composite Record, New Zealand' 'South Island Composite Record, New Zealand' 'Agassiz A84' 'Lake Salpeten' 'Houtman Abrolhos Islands' 'Houtman Abrolhos Islands' 'Lake Qinghai' 'Cleland Lake' 'Cleland Lake' 'Tongue of the Ocean, Bahamas' 'TALDICE -Talos Dome' 'Northwestern shore' 'Bunaken Island' 'Lake Malawi' 'Almagre Mountain site, Colorado' 'Cattle Pond, Dongdao Island' 'Kasenda' 'Lake Titicaca' 'Dervanoi' 'PC2' 'PC2' 'Palaui Island' 'NUS 08-7' 'NVFL-3' 'Vanuatu' 'Wanxiang Cave, China' 'Fonoifua Island' 'Perambikulam, Kerala, India' 'Panama Basin' 'Panama Basin' 'Illimani' 'Selva Negra, Bolivia' 'PC1' 'PC1' 'Nevado HuascarA¡n Core 2' 'Nevado HuascarA¡n Core 2' 'Nevado HuascarA¡n Core 2' 'Nevado HuascarA¡n Core 2' 'Nevado HuascarA¡n Core 2' 'Grenzgletscher' 'Ammersee' 'Little Cayman' 'Amedee Island, New Caledonia' 'Milcent' 'Milcent' 'Dunde Ice Cap' 'Sacred Lake' 'Fiescherhorn' 'Agassiz A87' 'Lake Lamongan' 'Northwestern shore' 'Northwestern Russian Tundra' 'Mt Logan (NWCol)' 'Edward' 'Penny Ice cap' 'Allt Lanlas, Ceredigion' 'Malindi Marine Park' 'Miaoergou' 'Lake N3' 'Lake N3' 'Lake N3' 'Lake N3' 'Plateau Remote' 'Volcan Granada, Argentina' 'Volcan Granada, Argentina' 'Coral Sea' 'Bermuda south shore' 'Savusavu Bay, Fiji' 'Akademii Nauk' 'Lake Nar' 'Qamdo, China' 'RICE (Rosevelt Island)' 'Ganesh' 'Berry Pond' 'Buckeye Creek Cave, West Virginia' 'Site G' 'Site G' 'Botuvera Cave, Brazil' 'Secas Island, Gulf of Chiriqui, Panama' 'La Reunion, Southwestern Indian Ocean' 'Crevice' 'Belukha glacier' 'Malan Glacier' 'Agassiz A79' 'Lough-na-Shade' 'Klapferloch Cave, Austria' 'Mediterranean (Minorca)' 'Ferrigno' 'Deplik Tabat Reef, Madang Lagoon' 'Deplik Tabat Reef, Madang Lagoon' 'DML Fimbulisen S20' 'VLG' 'IND 25B5 Coastal DML' 'DML Plateau (DML18) FB9804' 'NG-stack' 'Lake Nar' 'Lago El Grancho' 'Pirotan Island' '400th km' 'Savusavu Bay' 'Dyer Plateau' 'Washington Lake' 'Kilimanjaro SIF2' 'Butaritari Atoll, Gilbert Islands' 'Butaritari Atoll, Gilbert Islands' 'Butaritari Atoll, Gilbert Islands' 'Jiuxian Cave, China' 'Jiuxian Cave, China' 'B16' 'Bruce Plateau' 'Lake Tiaglamimine' 'Soreq Cave, Israel' 'Rabaul' 'PV-10' 'DML Georg-von-Neumayer Station B04' 'Lombok Strait, Bali, Indonesia' 'Taitaitanopo Island' 'Manali' 'Lake Blektjarnen' 'Pyramid' 'Laurentian Fan, western subpolar North Atlantic' 'Jones' 'Cape Hatteras' 'Wache, Bhutan' 'Chilean margin, Southern Ocean (Pacific sector)' 'Jagdalpur, Central India' 'Botuvera Cave, Brazil' 'DML Plateau (DML05) FB9807' 'B23' 'Dry Tortugas, 62MC' 'Dry Tortugas, 62MC' 'Lake Ghirla' 'Lake Ghirla' 'Saloum shell middens' 'B21' 'Arabian Sea' 'Great Bahama Bank south 118MC' 'Great Bahama Bank south 118MC' 'Buccoo Reef, Tobago' 'Buccoo Reef, Tobago' 'Renner' 'El Junco Lake' 'El Junco Lake' 'Lomonosovfonna' 'Laguna Potrok Aike' 'Sofular Cave, Turkey' 'Rarotonga, Cook Islands, South Pacific' 'Rarotonga, Cook Islands, South Pacific' 'Rarotonga, Cook Islands, South Pacific' 'Diva de Maura Cave, Brazil' 'Gulf of Guinea, EEA' 'Mu Cang Chai, Northern Vietnam' 'Mu Cang Chai, Northern Vietnam' 'Mu Cang Chai, Northern Vietnam' 'North East Breakers, Bermuda' 'GRIP Full' 'GISP2 Summer' 'GISP2 Summer' 'Ningaloo Reef, West Australia' 'Ningaloo Reef, West Australia' 'QuebecLabrador' 'Guam Coral' 'Laguna Castilla' 'Lake Bosumtwi' 'Clipperton Atoll' 'Clipperton Atoll' 'Clipperton Atoll' 'DSS Law Dome' 'GRIP' 'Prince-of-Wales' 'Moorea, French Polynesia' 'Exuma Sound Bahamas' 'Oyogos Yar' 'Nizanda, S Mexico' 'ShiHua Cave, China' 'NGRIP' 'Renland' 'Dasuopu' 'Dasuopu' 'El Junco Lake' 'US-ITASE-2002-1' 'East Rongbuk Glacier' 'Hainan Island' 'Hainan Island' 'NEEM' 'Steel' 'Akademii Nauk' 'Akademii Nauk' 'Vostok composite VRS13 ( a stack of 15 individual isotopic records from snow pits and shallow cores recovered in the vicinity of Vostok Station)' 'Hala Lake' 'Hala Lake' 'Peros Banhos Atoll' 'Peros Banhos Atoll' 'James Ross Island' 'San Valentin' 'B26' 'GISP2-B' 'GISP2-B' 'Devon Ice Cap' 'Devon Ice Cap' 'Challa' 'Guadeloupe' 'Northeast Breakers, Bermuda' 'Northeast Breakers, Bermuda' 'Korallgrottan, Sweden' 'Mafia Archipelago, Tanzania' 'Farewell Lake' 'Farewell Lake' 'Malindi' 'Nordic Seas' 'Hainan Island' 'Hainan Island' 'PC3' 'PC3' 'GISP2' 'Crystal Cave, Wisconsin' 'Colle Gnifetti KCI' 'Colle Gnifetti KCI' 'Colle Gnifetti KCI' 'Colle Gnifetti KCI' 'Lime' 'DML Plateau (DML14) FB9815' 'Cayo Sal, Los Roques Archipelago, Venezuela' 'Buckeye Creek Cave, West Virginia' 'Buckeye Creek Cave, West Virginia' 'Buckeye Creek Cave, West Virginia' 'Site E' 'Site E' 'Haklyutvatnet' 'Haklyutvatnet' 'DYE3' 'DYE3' 'DYE3' 'DYE3' 'DYE3' 'DYE3' 'DYE3' 'DYE3' 'Oregon Caves' 'Ras Umm Sidd, Red Sea' 'B32Site DML05' 'Spannagel Cave, Austria' 'MiMei, China' 'Sajama' 'Renland' 'Cueva del Diablo, Mexico' 'Cueva del Tigre Perdido, Peru' 'Quelccaya Ice Cap' 'National Botanic Gardens of Wales' 'Hainan Island' 'Hainan Island' 'Renland' 'Renland' 'Agassiz A79' 'Liberia, Costa Rica' 'Miyanohama' 'Hainan Island' 'Hainan Island' 'Cold Water Cave, Iowa' 'Cold Water Cave, Iowa'] ["<class 'str'>"] No. of unique values: 298/435
proxy metadata: archive type, proxy type, interpretation¶
archiveType (archive type)¶
In [35]:
Copied!
# archiveType
key = 'archiveType'
print('%s: '%key)
print(np.unique(df[key]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# archiveType
key = 'archiveType'
print('%s: '%key)
print(np.unique(df[key]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
archiveType: ['Coral' 'GlacierIce' 'GroundIce' 'LakeSediment' 'MarineSediment' 'MolluskShell' 'Sclerosponge' 'Speleothem' 'Wood'] ["<class 'str'>"] No. of unique values: 9/435
paleoData_proxy (proxy type)¶
In [36]:
Copied!
# paleoData_proxy
key = 'paleoData_proxy'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# paleoData_proxy
key = 'paleoData_proxy'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
paleoData_proxy: ['d18O' 'dD'] ["<class 'str'>"] No. of unique values: 2/435
paleoData_sensorSpecies (further information on proxy type: species)¶
In [37]:
Copied!
# climate_interpretation
key = 'paleoData_sensorSpecies'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# climate_interpretation
key = 'paleoData_sensorSpecies'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
paleoData_sensorSpecies: ['Ceratoporella nicholsoni' 'Diploria labyrinthiformis' 'Diploria strigosa' 'Hydnophora microconos, Porites lobata' 'Montastraea faveolata' 'NA' 'NaN' 'P. australiensis, possibly P. lobata' 'Porites' 'Porites lobata' 'Porites lutea' 'Porites sp.' 'Siderastrea radians' 'Siderastrea siderea' 'heliopora' 'labyrinthiformis' 'lamellina' 'lobata' 'lutea' 'nan'] ["<class 'str'>"] No. of unique values: 20/435
paleoData_notes (notes)¶
In [38]:
Copied!
# # paleoData_notes
key = 'paleoData_notes'
print('%s: '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# # paleoData_notes
key = 'paleoData_notes'
print('%s: '%key)
print(df[key].values)
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
paleoData_notes: ['; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Marine sediment (MS)' '; climateInterpretation_seasonality changed - was originally subannual; archiveType changed - was originally Coral (CO)' 'nan' 'nan' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' 'nan' 'summer signal' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' 'nan' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' 'nan' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed was originally bimonthly; archiveType changed was originally Coral (CO); archiveType changed was originally Coral (CO)' '; archiveType changed was originally Coral (CO); archiveType changed was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally subannual' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Marine sediment (MS); archiveType changed - was originally Marine sediment (MS)' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Marine sediment (MS); archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Marine sediment (MS); archiveType changed - was originally Marine sediment (MS)' '; archiveType changed was originally Tree ring cellulose (TR)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally annual; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed was originally monthly; archiveType changed was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' 'nan' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' 'nan' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally 3 4 5 6 7 8 9 10 11 12 13 14; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally 3 4 5 6 7 8 9 10 11 12 13 14; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice Wedge (IW); archiveType changed - was originally Ice Wedge (IW)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' 'nan' 'nan' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' 'nan' '; archiveType changed - was originally Ice core (IC)' 'nan' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' 'nan' 'nan' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' 'nan' '; archiveType changed was originally Tree ring cellulose (TR)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed was originally Tree ring cellulose (TR)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Tree ring cellulose (TR)' '; climateInterpretation_seasonality changed - was originally subannual; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' 'nan' 'nan' 'nan' 'nan' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' 'nan' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose' 'nan' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Lake sediment (LS)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' 'nan' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Marine sediment (MS); archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Lake sediment (LS)' 'nan' '; archiveType changed - was originally Tree ring cellulose (TR)' '; archiveType changed - was originally Marine sediment (MS); archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Marine sediment (MS); archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' 'nan' 'P/E lake' 'nan' '; archiveType changed - was originally Sclerosponge (SS)' '; archiveType changed - was originally Ice core (IC)' 'nan' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Lake sediment (LS)' 'nan' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' 'nan' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' 'nan' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' 'nan' '; archiveType changed was originally Tree ring cellulose (TR)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed was originally Tree ring cellulose (TR)' '; climateInterpretation_seasonality changed - was originally -12 -11 1 2 3 4 5 6 7 8 9 10; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' 'nan' 'nan' 'nan' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose' '; climateInterpretation_seasonality changed - was originally annual; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally annual; archiveType changed - was originally Coral (CO)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' 'nan' '; archiveType changed - was originally Ice core (IC)' 'nan' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally subannual; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally seasonal; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally seasonal; archiveType changed - was originally Coral (CO); archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' 'nan' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Marine sediment (MS)' 'nan' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose (TR)' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC); archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Marine sediment (MS)' 'nan' 'nan' 'nan' '; archiveType changed was originally Ice core (IC)' 'nan' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Marine sediment (MS)' '; climateInterpretation_seasonality changed - was originally annual; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally annual; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed was originally Lake sediment (LS)' '; archiveType changed was originally Lake sediment (LS)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Marine sediment (MS)' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose' '; archiveType changed - was originally Tree ring cellulose (TR); archiveType changed - was originally Tree ring cellulose' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed was originally Tree ring cellulose (TR)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally annual; archiveType changed - was originally Sclerosponge (SS)' '; archiveType changed - was originally Ice Wedge (IW); archiveType changed - was originally Ice Wedge (IW)' 'nan' '; archiveType changed - was originally Speleothem (SP); archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' 'nan' 'nan' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' 'nan' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS); archiveType changed - was originally Lake sediment (LS)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally -12 -11 -10 -9 -8 1 2 3 4 5 6 7; archiveType changed - was originally Coral (CO)' '; climateInterpretation_seasonality changed - was originally -12 -11 -10 -9 -8 1 2 3 4 5 6 7; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally monthly, bimonthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Lake sediment (LS)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Marine sediment (MS)' 'nan' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Lake sediment (LS)' '; archiveType changed - was originally Ice core (IC)' '; climateInterpretation_seasonality changed - was originally monthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' 'nan' 'nan' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed was originally Ice core (IC)' '; archiveType changed - was originally Speleothem (SP)' '; climateInterpretation_seasonality changed - was originally bimonthly; archiveType changed - was originally Coral (CO)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Speleothem (SP)' 'nan' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC); archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)' '; paleoData_variableName changed - was originally d18OiceSMOW; paleoData_units changed - was originally per mil SMOW; climateInterpretation_seasonality changed - was originally austral winter; archiveType changed - was originally ice core' '; archiveType changed was originally Tree ring cellulose (TR)' 'nan' 'nan' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Ice core (IC)' '; archiveType changed - was originally Tree ring cellulose (TR)' '; climateInterpretation_seasonality changed - was originally bimonthly' 'nan' 'nan' '; archiveType changed - was originally Speleothem (SP)' '; archiveType changed - was originally Speleothem (SP)'] ["<class 'str'>"] No. of unique values: 39/435
climate metadata: interpretation variable, direction, seasonality¶
interpretation_direction¶
In [39]:
Copied!
# climate_interpretation
key = 'interpretation_direction'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# climate_interpretation
key = 'interpretation_direction'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
interpretation_direction: ['Increase' 'N/A' 'NaN' 'T_air (positive), P_amount (negative)' 'T_air (positive), P_amount (negative), SPEI (negative)' 'decrease' 'decrease/increase' 'depends (orbital timescale: More Indian Monsoon moisture-->more enriched. Since 3ka: Indian source has been stable, so amount effect dominates: more rainfall, more intense hydrological cycle -->More depleted)' 'increase' 'negaitive' 'negative' 'positive' 'positive for d18O-temperature relation, negative for d13C-precipiation amount'] No. of unique values: 13/435
interpretation_seasonality¶
In [40]:
Copied!
# climate_interpretation
key = 'interpretation_seasonality'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# climate_interpretation
key = 'interpretation_seasonality'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
interpretation_seasonality: ['Annual' 'Apr-Jul' 'Aug-Jul' 'Dec-Feb' 'Dec-May' 'Growing Season' 'Jan' 'Jan-Apr' 'Jan-Jun' 'Jul-Sep' 'Jun-Aug' 'Jun-Sep' 'Mar' 'Mar-Nov' 'May-Dec' 'May-Oct' 'May-Sep' 'N/A' 'Nov-Feb' 'Nov-Jan' 'Oct-Apr' 'Oct-Dec' 'Oct-Sep' 'Spr-Sum' 'Summer' 'Wet Season' 'Winter' 'deleteMe' 'subannual'] No. of unique values: 29/435
interpretation_variable¶
In [41]:
Copied!
# climate_interpretation
key = 'interpretation_variable'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# climate_interpretation
key = 'interpretation_variable'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
interpretation_variable: ['N/A' 'moisture' 'temperature'] No. of unique values: 3/435
interpretation_variableDetail¶
In [42]:
Copied!
# climate_interpretation
key = 'interpretation_variableDetail'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# climate_interpretation
key = 'interpretation_variableDetail'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
interpretation_variableDetail: ['0.58 +- 0.11ppt/degrees C' 'Changes in annual temperature modulated by oceanic circulation and gradients' 'Maximum air temperature, seasonal' 'Maximum temperature' 'N/A' 'NaN' 'Original interpretation_variable: circulationIndex, interpretation_variableDetail: lake water' 'Original interpretation_variable: circulationVariable, interpretation_variableDetail: Indian monsoon' 'Original interpretation_variable: circulationVariable, interpretation_variableDetail: More negative d18O values correspond to stronger amount' 'Original interpretation_variable: circulationVariable, interpretation_variableDetail: N/A' 'Original interpretation_variable: circulationVariable, interpretation_variableDetail: tropical or North Pacific moisture' 'Original interpretation_variable: deleteMe, interpretation_variableDetail: N/A' 'Original interpretation_variable: deleteMe, interpretation_variableDetail: more positive values of d13C indicate a spread of C4 prairy grasses and decline of C3 forest plants, more positive d18O indicates evaporation of soil water which is stronger in the prairy environment than in the forsest' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: E:P lake water' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: LakeLevel@surface' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: N/A' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: Seasonal' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: air@surface' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: lake level' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: lake water' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: lake, winds in eastern Patagonia' 'Original interpretation_variable: effectivePrecipitation, interpretation_variableDetail: soil moisture' 'Original interpretation_variable: evaporation, interpretation_variableDetail: Aleutian Low/westerly storm trajectories' 'Original interpretation_variable: evaporation, interpretation_variableDetail: Indian Monsoon Strength' 'Original interpretation_variable: evaporation, interpretation_variableDetail: N/A' 'Original interpretation_variable: hydrologicBalance, interpretation_variableDetail: groundwater' 'Original interpretation_variable: hydrologicBalance, interpretation_variableDetail: lake water' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Amount of rainfall change' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Australian-Indonesian Summer monsoon; More negative d18O values correspond to stronger amount' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Australian-Indonesian monsoon rainfall' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Continental Sweden' 'Original interpretation_variable: precipitation, interpretation_variableDetail: ENSO/PDO' 'Original interpretation_variable: precipitation, interpretation_variableDetail: East Asian Monsoon Strength' 'Original interpretation_variable: precipitation, interpretation_variableDetail: East Asian Monsoon Strength; more negative values of d18O are interpreted as indicative of increased monsoon strength' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Indian Summer Monsoon; more negative values of d18O are interpreted as indicative of increased monsoon strength' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Lower precipitation produces higher d13C and Sr/Ca values' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Monsoon strength' 'Original interpretation_variable: precipitation, interpretation_variableDetail: More negative d18O values correspond to stronger amount' 'Original interpretation_variable: precipitation, interpretation_variableDetail: N/A' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Precipitation' 'Original interpretation_variable: precipitation, interpretation_variableDetail: SAM' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Seasonal' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Seasonal, annual' 'Original interpretation_variable: precipitation, interpretation_variableDetail: South China Sea' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Southern Tibet' 'Original interpretation_variable: precipitation, interpretation_variableDetail: The interpretation is made for an older section of the sample. Last 2k data was not the focus of the manuscript' 'Original interpretation_variable: precipitation, interpretation_variableDetail: Variations in NAO (related to the amount of rainfall. Season not specified)' 'Original interpretation_variable: precipitation, interpretation_variableDetail: air@surface' 'Original interpretation_variable: precipitation, interpretation_variableDetail: amount of rainfall' 'Original interpretation_variable: precipitation, interpretation_variableDetail: d18O changes of speleothems reflect effects of temperature on raifnall d18O, rainfall amounts affect cave hydrology and biomass density above the cave, which is recorded in d13C of speleothems' 'Original interpretation_variable: precipitation, interpretation_variableDetail: higher values are related to less rainfall - this can be realted to less moisture influex from the Caribbean due to a southward shift of the ITCZ in phases when high amounts of meltwater enter the cooling north Atlantic Ocean; after ~4.3 ka the connection to the north Atalatic is lost and ENSO becomes more important with warm ENSO events (El Nino) causing higher d18O' 'Original interpretation_variable: precipitation, interpretation_variableDetail: in the southern tropical Andes' 'Original interpretation_variable: precipitation, interpretation_variableDetail: more negative values of d18O are interpreted as indicative of increased summer monsoon precipitation' 'Original interpretation_variable: precipitation, interpretation_variableDetail: more positive d18O values are interpreted to represent wetter conditions' 'Original interpretation_variable: precipitation, interpretation_variableDetail: precipitation' 'Original interpretation_variable: precipitation, interpretation_variableDetail: relative portion of summer (SAM) vs winter rainfall' 'Original interpretation_variable: precipitation, interpretation_variableDetail: variations in paleoprecipitation amount on a multi-annual timescale (On longer timescales, however, the flowstone?s growth dynamics have to be considered)' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: Competing influence of polar and maritime airmasses' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: East Asian Monsoon rainfall' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: minimum temperature' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: moisture' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: of precipitation' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: precipitation' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: precipitation amount' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: rain' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: relative humidity' 'Original interpretation_variable: precipitationIsotope, interpretation_variableDetail: summer monsoon' 'Original interpretation_variable: salinity, interpretation_variableDetail: N/A' 'Original interpretation_variable: salinity, interpretation_variableDetail: sea surface' 'Original interpretation_variable: salinity, interpretation_variableDetail: surface' 'Original interpretation_variable: seasonality, interpretation_variableDetail: N/A' 'Original interpretation_variable: seasonality, interpretation_variableDetail: changes of d18O in speleothems reflect changes of the average d18O of rainfall in the region related to rainfall seasonality' 'Original interpretation_variable: seasonality, interpretation_variableDetail: relative amount of winter snowfall' 'Original interpretation_variable: streamflow, interpretation_variableDetail: N/A' 'Original interpretation_variable: streamflow, interpretation_variableDetail: lake water' 'Seasonal, annual' 'air@condensationLevel' 'air@surface' 'lake water' 'near sea surface' 'regional and hemispheric temperature' 'sea surface' 'sea@surface' 'sub surface (30m)' 'sub surface (~50 m)' 'subsurface (60-80m)' 'subsurface, 136 m' 'subsurface, 143 m' 'surface' 'variations in air temperature due to large-scale atmospheric patterns' 'variations in winter temperature in the Alps'] No. of unique values: 91/435
data¶
paleoData_values¶
In [43]:
Copied!
# # paleoData_values
key = 'paleoData_values'
print('%s: '%key)
for ii, vv in enumerate(df[key][:20]):
try:
print('%-30s: %s -- %s'%(df['dataSetName'].iloc[ii][:30], str(np.nanmin(vv)), str(np.nanmax(vv))))
print(type(vv))
except: print(df['dataSetName'].iloc[ii], 'NaNs detected.')
print(np.unique([str(type(dd)) for dd in df[key]]))
# # paleoData_values
key = 'paleoData_values'
print('%s: '%key)
for ii, vv in enumerate(df[key][:20]):
try:
print('%-30s: %s -- %s'%(df['dataSetName'].iloc[ii][:30], str(np.nanmin(vv)), str(np.nanmax(vv))))
print(type(vv))
except: print(df['dataSetName'].iloc[ii], 'NaNs detected.')
print(np.unique([str(type(dd)) for dd in df[key]]))
paleoData_values: CO14WUCL : -5.8101864 -- -4.7915373 <class 'numpy.ndarray'> IC00OEKC : -35.4 -- -26.84 <class 'numpy.ndarray'> MS09TCIS : -0.333 -- 1.063 <class 'numpy.ndarray'> CO95TUNG : -5.895 -- -4.578 <class 'numpy.ndarray'> CO17MUMA : -6.438 -- -4.634 <class 'numpy.ndarray'> CO17MUMA : -6.37924 -- -4.69934 <class 'numpy.ndarray'> IC16WE30 : -38.5 -- -31.5 <class 'numpy.ndarray'> LS10WRNA : -4.84 -- -3.46 <class 'numpy.ndarray'> IC10VICR : -37.7 -- -30.23 <class 'numpy.ndarray'> IC10VICR : -36.56 -- -24.67 <class 'numpy.ndarray'> CO98SWPR : -4.25 -- -2.91 <class 'numpy.ndarray'> CO04BAFI : -5.45 -- -4.48 <class 'numpy.ndarray'> CO04BAFI : -5.39 -- -4.13 <class 'numpy.ndarray'> CO04BAFI : -0.26 -- 0.36 <class 'numpy.ndarray'> SP11STBR : -7.48 -- -4.89 <class 'numpy.ndarray'> CO17DESC03A : -5.34 -- -4.7 <class 'numpy.ndarray'> CO17DESC03A : -0.03 -- 0.53 <class 'numpy.ndarray'> IC13STSP : -422.0458 -- -370.413 <class 'numpy.ndarray'> LS13WASU : -171.0 -- -140.0 <class 'numpy.ndarray'> LS13WASU : -180.0 -- -142.0 <class 'numpy.ndarray'> ["<class 'numpy.ndarray'>"]
paleoData_units¶
In [44]:
Copied!
# paleoData_units
key = 'paleoData_units'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# paleoData_units
key = 'paleoData_units'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
paleoData_units: ['nan' 'permil' 'unitless' 'yr AD' 'z score'] ["<class 'str'>"] No. of unique values: 5/435
year¶
In [45]:
Copied!
# # year
key = 'year'
print('%s: '%key)
for ii, vv in enumerate(df[key][:20]):
try: print('%-30s: %s -- %s'%(df['dataSetName'].iloc[ii][:30], str(np.nanmin(vv)), str(np.nanmax(vv))))
except: print('NaNs detected.', vv)
print(np.unique([str(type(dd)) for dd in df[key]]))
# # year
key = 'year'
print('%s: '%key)
for ii, vv in enumerate(df[key][:20]):
try: print('%-30s: %s -- %s'%(df['dataSetName'].iloc[ii][:30], str(np.nanmin(vv)), str(np.nanmax(vv))))
except: print('NaNs detected.', vv)
print(np.unique([str(type(dd)) for dd in df[key]]))
year: CO14WUCL : 1874.08 -- 1956.79 IC00OEKC : 1881.0 -- 1997.0 MS09TCIS : 1.43 -- 1975.13 CO95TUNG : 1922.542 -- 1991.292 CO17MUMA : 1986.67 -- 2010.92 CO17MUMA : 1926.95 -- 2011.94 IC16WE30 : 1242.0 -- 1988.0 LS10WRNA : 1456.0 -- 2000.0 IC10VICR : 552.0 -- 1973.0 IC10VICR : 552.0 -- 1973.0 CO98SWPR : 1939.0 -- 1992.0 CO04BAFI : 1940.54 -- 1997.11 CO04BAFI : 1939.92 -- 2001.96 CO04BAFI : 1940.0 -- 2000.0 SP11STBR : 10.0 -- 660.0 CO17DESC03A : 1628.0 -- 1657.0 CO17DESC03A : 1628.0 -- 1657.0 IC13STSP : 1801.0 -- 1998.0 LS13WASU : 293.0 -- 2007.0 LS13WASU : 293.0 -- 2007.0 ["<class 'numpy.ndarray'>"]
yearUnits¶
In [46]:
Copied!
# yearUnits
key = 'yearUnits'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
# yearUnits
key = 'yearUnits'
print('%s: '%key)
print(np.unique([kk for kk in df[key]]))
print(np.unique([str(type(dd)) for dd in df[key]]))
print(f'No. of unique values: {len(np.unique(df[key]))}/{len(df)}')
yearUnits: ['CE'] ["<class 'str'>"] No. of unique values: 1/435
In [ ]:
Copied!