xarray NetCDF autoclose option avoids segfaults

Python code using xarray.open_dataset() or xarray.open_dataarray() or similar functions reading from NetCDF4 should consider using the autoclose=True option, unless you have a specific reason for not using it. The autoclose=True option solves problems including:

  • random segmentation fault while reading NetCDF4 .nc files, where the same file is reopened in the program.
  • OSError from too many open files, where even increasing ulimit doesn’t help

When to use autoclose=False (default)

A reason why autoclose=False is the default is that out of core NetCDF4 operations can get performance boosts from keeping the file open. Also, BytesIO needs autoclose=False.

Many users may benefit from autoclose=True, but if you aren’t having problems with existing code, perhaps the default autoclose=False is fine for your application.

Tags: ,

Categories:

Comments

Written by Michael Hirsch, Ph.D. //