Introducing ncarnate: converting HDF4 and netCDF to netCDF4

ncarnate is a Python tool and library for converting supported HDF4 and netCDF files into netCDF4. It accepts netCDF3 inputs and can recompress existing netCDF4 files. For supported satellite grids, it adds geographic coordinates that compatible tools can use to locate the measurements.

When I worked with the UMBC ACROS Group and interned at NASA, I wrote scripts to parse large amounts of data on supercomputers. Different atmospheric datasets came in different formats, and older mission data often stayed in legacy formats that were never officially migrated. My colleagues and I wrote scripts to convert the portions we needed into formats our tools could read. Those scripts served the analyses at hand, but another analysis could mean writing another conversion. I hate repeating myself, so I wanted to generalize that work.

The project began in 2020 as netcdf4_recompressor, a small Python utility for rewriting netCDF4/HDF5 files with different compression settings. Its early test files included AMSR-E sea-ice data from my work in UMBC’s NSF-funded CyberTraining program, where we studied discrepancies between MODIS and CALIPSO cloud observations. We used sea-ice concentration data to investigate whether bright surfaces were being mistaken for clouds.

After I got laid off from my tech job earlier this year, I had more time to build things. I wanted to revisit some of my old projects, and this was a tool I wished I’d had when I worked in research. I rebuilt it as ncarnate to handle both recompression and conversion of supported HDF4 files to netCDF4. It also reconstructs coordinates for supported satellite grids, such as EASE-Grid. Here’s what it does with an AMSR-E sea-ice file.

An AMSR-E sea-ice example

Download AMSR_E_L3_SeaIce12km_B02_20020619.hdf from the HDF Group’s archived sample collection.

This file contains daily sea-ice concentration on polar stereographic grids for the Arctic and Antarctic. ncarnate reads the projection metadata and calculates projected x/y coordinates and their corresponding latitude and longitude, following the Climate and Forecast (CF) conventions used by netCDF analysis tools.

From the directory containing the downloaded file, run:

ncarnate AMSR_E_L3_SeaIce12km_B02_20020619.hdf

This writes AMSR_E_L3_SeaIce12km_B02_20020619.nc alongside the HDF4 original and keeps the original intact. ncarnate first writes to a temporary file, reopens it, and compares the stored measurement values against the source before putting the output in place.

For netCDF inputs, the single-file command replaces the original after verification, including when converting netCDF3 to netCDF4. Use --no-overwrite to keep the source. HDF4 originals are always preserved.

Here is the converted file’s daily sea-ice concentration plotted with Cartopy, using the projected x/y coordinates and projection metadata in the output:

Text description of the sea-ice maps

The left map shows the Arctic Ocean, with high ice concentrations across much of the central ocean and lower concentrations along parts of the ice edge. The right map shows sea ice surrounding Antarctica, forming an uneven band with lower concentrations toward its outer edge. Both maps show daily sea-ice concentration on 19 June 2002, on a 12.5 km grid. The scale runs from 0% (open water) to 100% (fully ice-covered). Gray land and purple missing-data areas are separate from that scale; a small missing-data area is visible at the North Pole. These are two views of one day, not a comparison over time.

Daily sea-ice concentration, 19 June 2002. These maps use the source polar stereographic grids. Land and missing-data flags are excluded from the percentage scale. Data: NASA/NSIDC AE_SI12, from the HDF Group’s historical sample. Coastlines: Natural Earth.

ncarnate converts HDF4 measurement arrays and their attributes. It reconstructs coordinates for geographic, polar stereographic, and Lambert azimuthal equal-area grids, including supported EASE-Grid products. It also handles selected satellite swaths with latitude/longitude arrays or regular mappings between coordinate and measurement grids.

Tables and separately stored image objects are not yet covered, and other projections and swath layouts may require additional support. Some netCDF4 data types, including compound types, are also unsupported. The documentation describes the supported cases.

Using ncarnate from Python

You can also call ncarnate from a Python script. The convert_file() function handles both recompression and supported format conversion, and returns the path to the verified output. The maps above use projected x/y coordinates; this example converts the same AMSR-E source and reads the output’s latitude and longitude arrays for use in your own analysis:

import numpy as np
from netCDF4 import Dataset
from ncarnate import convert_file

output = convert_file(
    "AMSR_E_L3_SeaIce12km_B02_20020619.hdf",
    dst = "sea_ice.nc",
)

with Dataset(output) as dataset:

    arctic        = dataset.groups["NpPolarGrid12km"]
    concentration = arctic["SI_12km_NH_ICECON_DAY"][:]
    concentration = np.ma.masked_outside(concentration, 0, 100)
    latitude      = arctic["lat"][:]
    longitude     = arctic["lon"][:]

The concentration array and its matching latitude/longitude arrays are now available to the rest of the script. The mask excludes this product’s land and missing-data codes while keeping zero-percent ice as valid open water.

convert_file() requires a separate destination and keeps the source intact. If the output already exists, it refuses to replace it unless you pass overwrite = True.

You can also use convert_file() to change compression settings on an existing netCDF file. Continuing from the example above:

from ncarnate import convert_file

compressed = convert_file(
    "sea_ice.nc",
    dst       = "sea_ice_recompressed.nc",
    complevel = 9,
)

This writes sea_ice_recompressed.nc and retains sea_ice.nc. Both examples use the same verification step as the command-line tool, so conversion can be one step in a larger analysis script.

Larger collections

For a directory named legacy-data, you can audit its files and subdirectories, then convert supported files into a separate modern-data directory:

ncarnate audit ./legacy-data --output manifest.jsonl --checksum sha256
ncarnate convert --manifest manifest.jsonl --out-dir ./modern-data --root ./legacy-data

The audit produces a readiness report and a manifest listing the files. The second command converts entries marked ready to netCDF4, preserving the directory structure and leaving the source files intact. Other entries remain in the manifest for review. The recorded checksums let the converter detect inputs that changed after the audit.

Installation

ncarnate 2.3.1 is available on PyPI and conda-forge.

Install with pip:

pip install ncarnate

or for Anaconda:

conda install -c conda-forge ncarnate

Conda-forge supplies the native HDF4, netCDF, and PROJ libraries together.

With pip, no separate native-library installation is needed when binary wheels are available for your platform and Python version. If pip needs to build pyhdf from source, you will need the HDF4 library and build tools; see the installation guidance.

The release described here uses pyhdf as a dependency. For version 3.0, I plan to build an alternative Rust-based HDF4 core, with broader format support and prebuilt packages for Windows, macOS, and Linux. The goal is to make installation self-contained, without requiring users to install HDF4 separately or compile the reader.

I built ncarnate to spend less time converting files and more time working with the data. If it doesn’t support a dataset you need, open an issue and tell me what you’re trying to do. The product name, file format, and a sample you can share would help me work out what’s missing.