Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add thread-safe execution support to netcdf-c #2505

Open
wants to merge 226 commits into
base: main
Choose a base branch
from

Conversation

DennisHeimbigner
Copy link
Collaborator

@DennisHeimbigner DennisHeimbigner commented Sep 14, 2022

This PR is experimental and should be used with caution.

It implements thread-safety with a mechanism similar to that used by HDF5. There is a single global mutex object that controls access to the netcdf API. It effectively serializes all calls to the API. For *nix* operating systems, it uses pthreads and more specifically the pthread_mutex functionality. For Windows it uses the built-in CriticalSection API.

The controlling option is "--enable-threadsafe" in Automake and "-DENABLE_THREADSAFE=on" in CMake. In this PR, it is enabled by default, but if it ever moves into production, it will be disabled by default.

Github Actions

Local build and test of this PR is known to work on Ubuntu-21 and Visual Studio. Build and test on Github Actions is currently failing for all systems. Fixing will require interactive access to the relevant platform.

Testing

This PR has had minimal testing; I am hoping that others can help out. There is one test case included: nc_test/run_threads.sh, which uses nc_test/tst_threads.c.

The primary functionality is in libdispatch/dmutex.c. It has some debug flags enabled to match lock with unlock to detect paths that bypass locking.

Warnings

  • This implementation depends heavily on using recursive mutex objects.

Documentation

For more detailed documentation see netcdf-c/docs/threadsafe.md

Addendum (5/31/2023)

Add some more improvements.

  • Make --enable-threadsafe default to disabled.
  • Make threadsafe work with automake on MinGW and Cygwin.
  • Add threadsafe testing to Ubuntu, MinGW, and Cygwin github actions.
  • Cleanup the library search for PTHREADS4W in CMakeLists.txt.
  • Add a test in nc_test -- run_thread_nc_test.sh and thread_nc_test.c -- that modifies the nc_test.c to use multi-threading.
  • Add a new installed header: netcdf_threadsafe.h.
  • Rename some internal header and .c files.
  • Re-enable DAP4.
  • Remove "const" attribute from the dispatch table pointers because it fails under multi-threading.
  • Unify definitions of e.g. nulldup.
  • Fix some potential memory leaks.

@DennisHeimbigner DennisHeimbigner marked this pull request as ready for review June 2, 2023 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants