ncarnate — Legacy Scientific Data to Modern netCDF4

Scientific data outlives the tools that made it. A satellite granule written in 2007 still holds real measurements, but it arrives in an aging container format with its geography left implicit in a block of metadata that general-purpose tools don’t reconstruct on their own. ncarnate reads those files and writes them back as modern netCDF4 with the coordinates reconstructed, so the data opens in xarray, QGIS, or Panoply without special handling.

Context

ncarnate began as a 2020 utility of mine, netcdf_recompressor — a tool to rewrite a netCDF file with different compression. Two things were wrong with it. Its command line had been broken since it was written: a stray debugging line made it silently do nothing. And it only ever handled netCDF, never the HDF4/HDF-EOS2 files its own documentation claimed to support. In 2026 I rebuilt it around a larger goal than compression — read the legacy earth-science formats and emit netCDF4 that today’s tools can use directly.

The problem

There are two hard parts, and they pull in different directions.

The first is fidelity. Recompressing or converting a scientific file must change how it is stored and nothing about what it says. Packed integers, fill values, and scale factors have to survive exactly; a conversion that quietly re-quantizes a measurement has corrupted the data it claims to preserve.

The second is geolocation. HDF-EOS2 doesn’t store latitude and longitude for every pixel. A grid is described by projection parameters; a swath carries coarse geolocation plus rules for interpolating it. Human-readable it is not, and until those coordinates are made explicit and standard, the file won’t line up on a map in a general-purpose tool.

Approach

The fidelity side is a contract enforced mechanically: converting changes storage, never data. Values are copied raw and bit-for-bit, fill and scale factors are carried across as declarations rather than applied, and every output is checked value-for-value against its source before it is allowed to replace anything — the original is never destroyed on a failed run.

The geolocation side is the harder part. ncarnate parses the HDF-EOS2 StructMetadata and reconstructs CF-convention coordinates. Grid projections (polar-stereographic, geographic, and the EASE-Grid equal-area projection) become CF grid mappings plus one-dimensional x/y and two-dimensional lat/lon, inverse-projected through PROJ via pyproj. Swath geolocation is attached as CF coordinates, and where a swath stores its geography at coarser resolution than its data (a 5 km grid under a 1 km field), the missing coordinates are interpolated — through three-dimensional Cartesian space rather than raw latitude and longitude, so the result stays correct across the antimeridian and near the poles, which is exactly where polar-orbiting satellites spend their time.

To check the reconstruction against something other than itself, I compared its output for an AMSR-E sea-ice granule against The HDF Group’s own independent conversion of the same file; the coordinates agree to about a hundred-thousandth of a degree.

Outcome

ncarnate is live on PyPI under the MIT license — pip install ncarnate, CLI and library both. It reads HDF4/HDF-EOS2 and netCDF/HDF5 and writes recompressed, CF-annotated netCDF4, turning a shelf of unreadable legacy granules into data a modern toolchain can open.

It’s also on conda-forge — conda install -c conda-forge ncarnate. That’s the install to reach for on Windows: the HDF4 library ncarnate depends on has no working build in its PyPI package there, so pip alone can’t give Windows users the HDF4 conversion, but conda-forge builds that library correctly on every platform. The recipe went through a conda-forge maintainer’s review before it was merged.