Published in pangeo·Mar 9, 2020Publishing Xarray Datasets via a Zarr compatible REST APIXpublish is a new Xarray extension that makes it easy to publish datasets via a Zarr-compatible REST API. You can test drive Xpublish now in this Binder or install it from Conda or PyPi: $ conda install -c conda-forge xpublish # or $ pip install xpublish Xpublish enables sharing of…Xarray4 min read
Published in pangeo·Mar 3, 2020Introducing Intake-stacWe’re excited to introduce a new Intake driver: Intake-stac. We think this tool will make it much easier to explore SpatioTemporal Asset Catalogs (STAC) and enable interactive data analysis and visualization in a Python environment. Intake-stac provides Intake drivers that support opening STAC Catalogs, Collections, Items and ItemCollections. By combining…Xarray3 min read
Published in pangeo·Feb 28, 2020Streaming with ZarrSimulation models continue on their path to the exascale — motivating the search for a new paradigm for model data analysis. We explore the use of Zarr (a data format for chunked, compressed, N-dimensional arrays) and Redis (a streaming database) to support streaming data from simulation models to analysis environments. …Zarr5 min read
Published in pangeo·Nov 18, 2019Pangeo Guide for AGU 2019Headed to the 2019 AGU Fall Meeting in San Francisco in December? Here’s your Pangeo guide to tutorials, presentations, and meetups for this years meeting. See this discourse post for up to date discussion on this year’s AGU happenings. Tutorials Pangeo: Hands on with JupyterHub and Open-Source Python Tools for Scalable…Data Science3 min read
Published in pangeo·Nov 8, 2019Pangeo and Kubernetes, Part 3: Thoughts on SecurityWritten by Joe Hamman, Scott Henderson, and Rob Fatland, and Amanda Tan Lehr. Tldr; Pangeo’s public cloud-hosted JupyterHubs are not enterprise grade platforms. We recommend not storing access keys, passwords, or private data on these hubs. In previous posts we’ve discussed our design of Pangeo deployments on Kubernetes and some…Kubernetes4 min read
Published in pangeo·Nov 7, 2019The Pangeo PatternIdeas included in this post are based on numerous conversations that have occurred across the Pangeo community. Ryan Abernathey, Guillaume Eynard Bontemps, Joe Hamman, Chris Holdgraf, Fernando Perez, Niall Robinson, Matthew Rocklin, Richard Signell, and Amanda Tan Lehr (alphabetized) made specific contributions to the development of these thoughts. We kicked…Open Source6 min read
Published in pangeo·Oct 9, 2019Analyzing large climate model ensembles in the cloudWritten by Joe Hamman, posted on behalf of the NCAR Science at Scale Team. The Science at Scale Team at the National Center for Atmospheric Research (NCAR) is excited to announce the release of the Community Earth System Model (CESM) Large Ensemble Numerical Simulation (LENS) dataset published in the Amazon…Data Science4 min read
Published in pangeo·May 30, 2019Pangeo and Kubernetes, Part 2: Cluster DesignIn my last post, we looked into Pangeo’s cloud costs and discussed what it would like to budget for a typical Pangeo research cluster. In this post, I’ll present a technical and opinionated design for a typical Pangeo Kubernetes cluster. …Kubernetes6 min read
Published in pangeo·May 14, 2019Pangeo and Kubernetes, Part 1: Understanding costsWhen we talk to people about Pangeo on the Cloud, we’re often presented with questions about costs? How much does it cost? Who pays for it? What’s the long-term plan for Pangeo on the Cloud? Before diving into the details, I think it is important that we revisit why we’re…Cloud Computing7 min read
Published in pangeo·Oct 16, 2018Pangeo applications for NASA Earth Observing DataToday, a new project begins that expands Pangeo’s capability to utilize remote sensing datasets for geoscientific research. Our focus will be on cloud-native data analysis tools and approaches with the lofty goal of changing the way every day scientists interact with satellite observations of Earth. A team of researchers from…Cloud Computing3 min read
