$Header$ -*-text-*-

The netCDF Operators NCO version 4.7.9 art b'rn

http://nco.sf.net (Homepage, Mailing lists, Help)
http://github.com/nco (Source Code, Issues, Releases, Developers)

What's new?

Producing data for submission to CMIP6 has led to many improvemnts to
the two workflow tools ncclimo and ncremap. Dozens of changes have
made both tools more robust, clearer, and better documented. Notable
changes include include improved sub-gridscale remapping, adding depth
coordinates, CDL updates for hidden attributes, and support for GCC9
and OpenMP 5.0.0. 

Work on NCO 5.0.0 has commenced "under the hood". The key leap in that 
release will be support for netCDF4 user-defined types. Printing of
netCDF4 user-defined types ENUM and VLEN is ready now (though
unsupported) with the --udt flag. 5.0.0 will contain the finished
version of that, and progress on native weight generation by ncremap.

Enjoy,
Charlie

NEW FEATURES (full details always in ChangeLog):

A. ncremap now names the SGS map as per the -m map_fl option.
   Previously there was no way to name SGS maps for archival.
   This capability eases the generation of SGS maps for CMIP6,
   since models (like ELM, CTSM) with time-invariant land units can
   regrid all variables with the same SGS map.
   http://nco.sf.net/nco.html#sgs

B. ncremap and ncclimo support MPAS files with a 'depth' dimension,
   and also adding a depth coordinate to raw MPAS output files.
   Both add 'depth' to MPAS files with the new option --dpt_fl:
   ncremap -P mpas --dpt_fl=refBottomDepth_60lyr.nc ...
   ncremap -m mpaso --dpt_fl=refBottomDepth_60lyr.nc ...
   To work, the Python code add_depth.py must be on the $PATH, and
   dpt_fl must contain the MPAS-Ocean field refBottomDepth.
   Thanks to Xylar Asay-Davis of LANL for creating a robust
   add_depth.py. 
   http://nco.sf.net/nco.html#dpt
   http://nco.sf.net/nco.html#dpt_fl

C. The regridder utilizes the lat_dmn_nm and lon_dmn_nm options, when 
   specified, to name the dimensions for regridded logically
   rectangular files. Previously these options were only used to
   specify horizontal dimension names in input file. Consistent
   with this, lat_nm_out and lon_nm_out now specify only the
   horizontal variable names, not the dimension names as before.
   The indepdendent configurability of dimension and variable names
   allows ncremap to regrid files with map projections as expected
   by Panoply and other software. Thanks to Xylar Asay-Davis of LANL
   for suggesting this feature.
   ncremap --rgr_opt='--rgr lat_dmn_nm=x#lon_dmn_nm=y'
   We realize this API is ugly and will create more aesthetic options
   like ncremap --lat_dmn=x --lon_dmn=y if enough users lovingly
   petition our representatives now standing-by at call centers.
   http://nco.sf.net/nco.html#rgr_opt

D. ncclimo writes more explicit information about parallelism,
   including job_nbr, var_nbr, and nd_nbr. This will help users
   tune throughput for large climatology and splitting jobs.
   http://nco.sf.net/nco.html#ncclimo

E. The splitter in ncclimo now outputs progress more clearly,
   and shows the elapsed time to split each timeseries.
   http://nco.sf.net/nco.html#split

F. ncra/ncrcat now perform time-rebasing in rec_apn mode using
   calendar units from the destination not the source file.
   This makes more sense when you think about it.
   Thanks to Ian McHugh of Monash University for prompting this. 
   http://nco.sf.net/nco.html#rbs

G. NCO now writes more intelligible error messages with HINTS for
   likely causes of and responses to NC_EHDFERR and NC_EBADCHUNK
   errors. 
   http://nco.sf.net/nco.html#hdf
   http://nco.sf.net/nco.html#cnk

H. ncks emulates ncdump more closely and prints hidden attribute
   _NoFill (not _NOFILL) values as "true" or "false" not 1 or 0.
   ncks --hidden -m in.nc
   http://nco.sf.net/nco.html#cdl

I. ncks prints CDL for hidden _Filters attribute, if any, with 
   all filter values.
   ncks --hidden -m in.nc
   http://nco.sf.net/nco.html#cdl

J. ncks prints hidden _Endianness attribute for floating point types
   not just multi-byte integers as before.
   ncks --hidden -m in.nc
   http://nco.sf.net/nco.html#cdl

BUG FIXES:

A. Fix a bug exposed in NCO 4.7.1 that could attempt to dereference
   a NULL pointer, resulting in a core dump. The solution is to
   install 4.7.9. The workaround to that is to run a version of NCO
   prior to 4.7.1.

B. ncremap used an inappropriate area to weight SGS variables.
   Thanks to Min Xu of ORNL for reporting and providing the fix.
   SGS regridding is once again BFB with global means.

C. ncremap orders Tempest remap commands in execution-order in the
   history attribute. Previously they were recorded in an ad hoc
   order. Thanks to Ben Hilman of SNL for suggesting this.

D. Cause ncclimo in MPI mode to exit when no nodes are available.

E. Ensure ncclimo writes nd_fl to writable directory

F. ncap2 exits gracefully when no output file is specified

G. ncap2 exits gracefully when no -s or -S argument given

Full release statement at http://nco.sf.net/ANNOUNCE

KNOWN PROBLEMS DUE TO NCO:

   This section of ANNOUNCE reports and reminds users of the
   existence and severity of known, not yet fixed, problems. 
   These problems occur with NCO 4.7.9 built/tested under
   MacOS 10.14.1 with netCDF 4.6.1 on HDF5 1.10.2 and with
   Linux with netCDF 4.6.2-development (20180515) on HDF5 1.8.19.

A. NOT YET FIXED (NCO problem)
   Correctly read arrays of NC_STRING with embedded delimiters in ncatted arguments

   Demonstration:
   ncatted -D 5 -O -a new_string_att,att_var,c,sng,"list","of","str,ings" ~/nco/data/in_4.nc ~/foo.nc
   ncks -m -C -v att_var ~/foo.nc

   20130724: Verified problem still exists
   TODO nco1102
   Cause: NCO parsing of ncatted arguments is not sophisticated
   enough to handle arrays of NC_STRINGS with embedded delimiters.

B. NOT YET FIXED (NCO problem?)
   ncra/ncrcat (not ncks) hyperslabbing can fail on variables with multiple record dimensions

   Demonstration:
   ncrcat -O -d time,0 ~/nco/data/mrd.nc ~/foo.nc

   20140826: Verified problem still exists
   20140619: Problem reported by rmla
   Cause: Unsure. Maybe ncra.c loop structure not amenable to MRD?
   Workaround: Convert to fixed dimensions then hyperslab

KNOWN PROBLEMS DUE TO BASE LIBRARIES/PROTOCOLS:

A. NOT YET FIXED (netCDF4 or HDF5 problem?)
   Specifying strided hyperslab on large netCDF4 datasets leads
   to slowdown or failure with recent netCDF versions.

   Demonstration with NCO <= 4.4.5:
   time ncks -O -d time,0,,12 ~/ET_2000-01_2001-12.nc ~/foo.nc
   Demonstration with NCL:
   time ncl < ~/nco/data/ncl.ncl   
   20140718: Problem reported by Parker Norton
   20140826: Verified problem still exists
   20140930: Finish NCO workaround for problem
   20190201: Possibly this problem was fixed in netCDF 4.6.2 by https://github.com/Unidata/netcdf-c/pull/1001
   Cause: Slow algorithm in nc_var_gets()?
   Workaround #1: Use NCO 4.4.6 or later (avoids nc_var_gets())
   Workaround #2: Convert file to netCDF3 first, then use stride
   Workaround #3: Compile NCO with netCDF >= 4.6.2

B. NOT YET FIXED (netCDF4 library bug)
   Simultaneously renaming multiple dimensions in netCDF4 file can corrupt output

   Demonstration:
   ncrename -O -d lev,z -d lat,y -d lon,x ~/nco/data/in_grp.nc ~/foo.nc # Completes but produces unreadable file foo.nc
   ncks -v one ~/foo.nc

   20150922: Confirmed problem reported by Isabelle Dast, reported to Unidata
   20150924: Unidata confirmed problem
   20160212: Verified problem still exists in netCDF library
   20160512: Ditto
   20161028: Verified problem still exists with netCDF 4.4.1
   20170323: Verified problem still exists with netCDF 4.4.2-development
   20170323: https://github.com/Unidata/netcdf-c/issues/381
   20171102: Verified problem still exists with netCDF 4.5.1-development
   20171107: https://github.com/Unidata/netcdf-c/issues/597
   20190202: Progress has recently been made in netCDF 4.6.3-development
   More details: http://nco.sf.net/nco.html#ncrename_crd

C. NOT YET FIXED (would require DAP protocol change?)
   Unable to retrieve contents of variables including period '.' in name
   Periods are legal characters in netCDF variable names.
   Metadata are returned successfully, data are not.
   DAP non-transparency: Works locally, fails through DAP server.

   Demonstration:
   ncks -O -C -D 3 -v var_nm.dot -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc # Fails to find variable

   20130724: Verified problem still exists. 
   Stopped testing because inclusion of var_nm.dot broke all test scripts.
   NB: Hard to fix since DAP interprets '.' as structure delimiter in HTTP query string.

   Bug tracking: https://www.unidata.ucar.edu/jira/browse/NCF-47

D. NOT YET FIXED (would require DAP protocol change)
   Correctly read scalar characters over DAP.
   DAP non-transparency: Works locally, fails through DAP server.
   Problem, IMHO, is with DAP definition/protocol

   Demonstration:
   ncks -O -D 1 -H -C -m --md5_dgs -v md5_a -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc

   20120801: Verified problem still exists
   Bug report not filed
   Cause: DAP translates scalar characters into 64-element (this
   dimension is user-configurable, but still...), NUL-terminated
   strings so MD5 agreement fails 

"Sticky" reminders:

A. Reminder that NCO works on most HDF4 and HDF5 datasets, e.g., 
   HDF4: AMSR MERRA MODIS ...
   HDF5: GLAS ICESat Mabel SBUV ...
   HDF-EOS5: AURA HIRDLS OMI ...

B. Pre-built executables for many OS's at:
   http://nco.sf.net#bnr

