Unidata Technologies Relevant to GO-ESSP: An Update

Update on Unidata Technologies for Data Access Russ Rew 2011-05-10 Overview Standards endorsements C-based netCDF NetCDF/OPeNDAP integration NetCDF-Java and Unidata Common Data Model TDS: THREDDS Data Server Libcf: Gridspec Enhanced transparency of Unidata open-source development

2 Standards endorsements Standards-based data interoperability has typically meant compliance to ISO and OGC standards Now, netCDF also qualifies: 2009: NASA ESDS standard for earth science data

2010: US FGDC endorsed netCDF-3 and netCDF-4 as Common Encoding Standards 2010: NASA ESDS endorsed CF Metadata Conventions as a standard for earth science data 2011: TDS includes ncISO services providing ISO-19115 metadata (from NOAA/NGDC) 2011: OGC approved netCDF as a core binary encoding standard Additional standards work is underway for netCDF-4, NcML, CF conventions C-based netCDF libraries OPeNDAP client integration Refactored library architecture for maintainability, other formats, protocols Improved performance

Generic tools NetCDF/OPeNDAP integration a success OPeNDAP access from C-based clients Benefits from remote access Supports subset access of selected variables, subarrays Subset access using DAP can be much faster than whole file access, such as with FTP NcML aggregation accessing data in other formats

netCDF subset service from remote servers DAP security authorization Earth System Grid HTTP C library refactored for interoperability Common interfaces and code factored into a new dispatch layer Simpler code Easier maintenance Easier handling of additional formats & protocols (I/O plugins) netCDF classic netCDF-4

HDF4, HDF5 (read only) remote subsets (read only) remote subsets (read only) other formats Performance improvements

Refactored read code for large speedup on opening netCDF-4 files with compressed or chunked variables Speedup variable and dimension lookup by name Improved memory allocation to reduce memory footprint Reduced memory when parallel I/O used Improved read code for handling a large number of netCDF-4 attributes and variables Applied intelligent caching to remote access for OPeNDAP client Generic tools Adapted generic tools to netCDF-4 enhanced data model ncdump: converts netCDF data to CDL text form ncgen: converts CDL text to netCDF data or generates program

nccopy: copies, converts, compresses, chunks netCDF data Proved practicality of more complex data model Practical experience with netCDF-4 Most experience to date is with netCDF-4 classic model format uses netCDF-3 classic data model, APIs uses netCDF-4 HDF5-based format provides backward compatibility Enables performance features: compression, multidimensional tiling (chunking), efficient schema changes,

parallel I/O, Adoption proceeding in a 3-step process 1. 2. 3. Relink applications with netCDF-4 library Continue use of classic model, netCDF-3 APIs but with netCDF-4 classic model format to get performance benefits Make use of features of enhanced model, as needed/supported Status of netCDF-4 adoption: Logjam appears to be broken

NetCDF-4 classic model support in analysis and visualization apps: IDL, GrADS, CDAT, MATLAB, IDV, NCO, NCL, CDO, PyNGL, ncview, Panoply, Ferret, OGC WMS and WCS clients Data providers using netCDF-4 classic model format for transparent compression and chunking: within NASA, NOAA, GFDL, COLA, NetCDF-4 enhanced model support in language APIs: C, Java (read only), C++ (beta), Fortran Partial support for netCDF-4 enhanced model also in NCO, NCL, Panoply, Python API, NetCDF-Java and Common Data Model NetCDF-Java provides one of the few generalpurpose CF metadata implementations

Observational data implementation supporting CF Discrete Sampling Geometry conventions now complete for CF-1.6 addition Recent improvements include improved GRIB collection as alternative to FMRC aggregations Caching of Grid dataset info and File System info GRIB processing correctly handles time interval variables THREDDS Data Server (TDS) Catalog and data access services Recent improvements

OPeNDAP OGC web services: WCS and WMS HTTP NetCDF subset service in coordinate space NetCDF API access to other formats (GRIB, HDF, ) FMRC aggregation configuration simplified and speeded up ncWMS integration improved, Godiva2 browser supported as client ncISO metadata services OPeNDAP handling of ESG, HTTP access control Subsetting services for Point Observations extended to Station data Plans

GRIB-2 refactor for efficiency Improve performance of large collections and aggregations Experimental asynchronous / streaming / remote access services (cdmRemote) Libcf: Gridspec additions Subject of Alex Pletzers talk on Thursday morning Proposed CF extension for complex grids, such as cubed-sphere grid Specifies multi-file implementation of host file, grid files, data files, and mosaic files Supports regridding Transparency in development process

Source code repository open: svn.unidata.ucar.edu/ Bug and issue tracking available: www.unidata.ucar.edu/jira Development plans and schedules more visible Unidata developers posting semi-regularly to blog: www.unidata.ucar.edu/blogs/developer These services are experimental, subject to change Goals Make participation in open-source development practical Leverage community efforts

Recently Viewed Presentations

  • FileNewTemplate

    FileNewTemplate

    30 October 2016, Kigali, Rwanda. Country Team Members: Insert names Photo: Mark Kaye/Save the Children. Leveraging the Scaling Up Nutrition Civil Society Network: building regional platforms to promote learning on how to address malnutrition
  • Tips for Using This Template - Wyoming Instructional Network

    Tips for Using This Template - Wyoming Instructional Network

    Multi-tiered systems of support, or MTSS, in Wyoming is a preventative framework for school improvement made up of four core components. The four components are screening, multilevel prevention system (you may have heard this described as tiered instruction, or RTI,...
  • WIN

    WIN

    WIN Career Readiness Courseware Located on the Website: edgecombe-nc. wincshost.com WIN Career Readiness Courseware Win Career Readiness Courseware prepares students to successfully take the ACT WorkKeys® assessments. Why is ECPS doing this? Cross-curricular program to help address our students needs...
  • Lecture 11: Large Cache Design Topics: large cache

    Lecture 11: Large Cache Design Topics: large cache

    Private Caches in Multi-Core Advantages of a shared cache: Space is dynamically allocated among cores No wastage of space because of replication Potentially faster cache coherence (and easier to locate data on a miss) Advantages of a private cache: small...
  • RANDPRESENTATION WATER TITLE Presented by: Name Surname AnnualDirectorate

    RANDPRESENTATION WATER TITLE Presented by: Name Surname AnnualDirectorate

    PRESENTATION TITLE. Presented by: Name Surname. Directorate. Date. RAND . ... Dabi District Municipality (FDDM), Metsimaholo Local Municipality (MLM), SABC. ... Engineering Human Resources Information Technology Legal Process Control Science 4 4 7 5 3 15 2.
  • Cognition at Stanford Jay McClelland & Lera Boroditsky

    Cognition at Stanford Jay McClelland & Lera Boroditsky

    Life after Stanford Steven Kosslyn Harvard David Rumelhart Stanford Larry Barsalou Emory Bob Sternberg Yale Keith Holyoak UCLA Beth Loftus UC Irvine Richard Shiffrin Indiana John Anderson CMU Steve Sloman Brown Brian Ross Illinois Mark Gluck Rutgers Larry Maloney NYU...
  • スライド 1 - Thomas Jefferson National Accelerator Facility

    スライド 1 - Thomas Jefferson National Accelerator Facility

    E. Hiyama (RIKEN) Hypernuclear physics has recently become very exciting . owing to new epoch-making experimental data. Recent progress in hypernuclear physics . from a theorist's viewpoint. ( of hypernuclear structure.) ... Emiko Hiyama Company:
  • Discrete Structures for Computer Science

    Discrete Structures for Computer Science

    Applications of predicate logic and nested quantifiers. Please read section 1.4. Extra slides. Prove: x [P(x) Q(x)] x P(x) x Q(x) We must prove each "direction" of the equivalence. Assume that P and Q have the same domain. ... Discrete...