TBB deprecation fix (partial) #209

acxz · 2020-01-11T19:10:59Z

This PR addresses #208.
tbb::mutex is replaced with std::mutex

tbb::task_scheduler_init is removed since it is deprecated and not used in the codebase. For tests it is still kept since it adds functionality, specifically specifying the number of threads. It needs to be replaced with tbb:task_arena in those cases. See discussion below for more details.

tbb::exception is slightly harder to get rid of, mostly because I am not sure if the C++11 std::exception has the thread synchronous properties that tbb::exception has. Specifically if std::exceptions can be propagated across threads.
exception_ptr may be the likely substitute for tbb::exception
Based on discussion below, it seems tbb::exceptions do so it is fine.

tbb/task.h is also deprecated but this PR does not address it, hence the partial in the name.

Useful TBB deprecation information

Note based on this tbb::exceptions are properly propagated. Consider looking at tbb::exception changes in this PR.

These changes require C++11 which should be fine for the rest of the codebase.

This change is

chrisbeall · 2020-01-11T19:26:04Z

gtsam/geometry/Unit3.cpp

@@ -80,12 +80,10 @@ static Point3 CalculateBestAxis(const Point3& n) {

 /* ************************************************************************* */
 const Matrix32& Unit3::basis(OptionalJacobian<6, 2> H) const {
-#ifdef GTSAM_USE_TBB


I would consider leaving these #ifdef GTSAM_USE_TBB guards in place. They have the benefit of making single-threaded builds (without TBB) completely mutex-free.

hmm, I see, yeah that would be useful. But I don't like keeping the TBB name since it is not a TBB function/class. Maybe renaming the guard to GTSAM_MULTITHREADED would be more accurate?

I agree it would be more accurate, but let's keep that change until TBB is not the only way to do multi-threading. A change like this has the potential to break some setups of users in ways we can't anticipate. Better to do this as part of a version bump with release docs - I propose this (awesome!) PR should simply addresses the warnings.

Okay I'll add the guards back, but I really hope we don't forget. Maybe we can keep an issue and PR open specifically for this until we decide to add it in.

ProfFan · 2020-01-11T19:41:57Z

In modern TBB we should be able to use plain std::exception.

acxz · 2020-01-11T19:43:37Z

Do you mean modern C++? But what is modern C++? C++11,14, or 17?

dellaert · 2020-01-11T19:53:14Z

gtsam/geometry/Unit3.cpp

@@ -80,12 +80,10 @@ static Point3 CalculateBestAxis(const Point3& n) {

 /* ************************************************************************* */
 const Matrix32& Unit3::basis(OptionalJacobian<6, 2> H) const {
-#ifdef GTSAM_USE_TBB


I agree it would be more accurate, but let's keep that change until TBB is not the only way to do multi-threading. A change like this has the potential to break some setups of users in ways we can't anticipate. Better to do this as part of a version bump with release docs - I propose this (awesome!) PR should simply addresses the warnings.

dellaert · 2020-01-11T19:54:34Z

PS, is tbb::exception giving warnings? If not I guess it's a non-issue. If so, we have to figure it out...

acxz · 2020-01-11T19:59:53Z

It is, basically the tbb::exception class is deprecated so we need to get rid of it. But I fear that we might lose functionality if we use std::exception, since I am not sure if the C++11 std::exception can propagate exceptions throughout threads. But with the help of std::exception_ptr it should be fine. Although based on what @ProfFan is saying maybe we do not need to worry about it. Here is a stackoverflow discussion regarding this: https://stackoverflow.com/questions/233127/how-can-i-propagate-exceptions-between-threads

…hanges

ProfFan · 2020-01-11T20:36:36Z

TBB uses std::exception_ptr internally to propagate normal std::exceptions if there is C++11 support (but definitely need a unit test to be sure)

acxz · 2020-01-11T20:39:56Z

Ahh I understand. Thats p cool. Okay then, I'll try to remove tbb:exception and see if I can get rid of the associated warnings.

acxz · 2020-01-11T21:40:45Z

I have gotten rid of the tbb_exception warnings. The biggest change was instead of using tbb_exception we are using std_exception, but still using tbb's allocator if GTSAM_USE_TBB is true. Note not all the warnings are removed yet. Still getting something like this:

In file included from /home/acxz/vcs/git/github/acxz/gtsam/gtsam/base/treeTraversal/parallelTraversalTasks.h:25,
                 from /home/acxz/vcs/git/github/acxz/gtsam/gtsam/base/treeTraversal-inst.h:19,
                 from /home/acxz/vcs/git/github/acxz/gtsam/gtsam/inference/BayesTree-inst.h:25,
                 from /home/acxz/vcs/git/github/acxz/gtsam/gtsam/symbolic/SymbolicBayesTree.cpp:23:
/usr/include/tbb/tbb.h:21:154: note: #pragma message: TBB Warning: tbb.h contains deprecated functionality. For details, please see Deprecated Features appendix in the TBB reference manual.
   21 | #pragma message("TBB Warning: tbb.h contains deprecated functionality. For details, please see Deprecated Features appendix in the TBB reference manual.")
      |                                                                                                                                                          ^

Haven't dug into this one, but it's weird that it doesn't explicitly state what functionality in tbb.h is deprecated. I'll try to dig around, but about 90% of TBB warnings are gone with these series of commits.

remove deprecated tbb::task_scheduler_init

acxz · 2020-01-11T23:16:01Z

Fixed the tbb.h warnings. Basically it's bad practice to include huge libraries. The needed tbb.h functionality was split into the required includes. Removing the tbb.h include also meant the undef min/max/ERROR are unnecessary, since we are only including what we want. No more warnings during the compilation of GTSAM now! (well one unused variable warning but that is in CCOLAMD 3rdparty project)

ProfFan · 2020-01-12T00:34:38Z

It appears TBB is not currently in the Travis builds?

acxz · 2020-01-12T12:13:21Z

You're right, just created a PR to add tbb into the Travis builds.

dellaert · 2020-01-12T15:16:48Z

OK, let’s wait until that PR passes and then merge that branch in here. That will then be proof things work as expected - provided we have enough coverage, which is unclear.

acxz · 2020-01-12T15:20:35Z

Sounds appropriate.

chrisbeall · 2020-01-12T17:09:33Z

examples/TimeTBB.cpp

@@ -153,7 +154,6 @@ int main(int argc, char* argv[])
  for(size_t n: numThreads)
  {
    cout << "With " << n << " threads:" << endl;
-    tbb::task_scheduler_init init((int)n);


Without this line this entire timing script is pointless. Is there not a replacement for this deprecated function? Otherwise, could we just suppress the deprecation warning?

Why do you say that? The task_scheduler_init was optional in the first place. If we don't provide one, TBB uses one internally. We are still timing how fast the parallelization occurs. From what I can tell the functionality of the script has not changed one bit.
Suppressing the deprecation warning is just hiding the problem. In future releases this will become an error, instead of a warning.

I do understand that it may be hard to support both TBB 4.4 and TBB 2020 and if that is the case, we can just add a requirement during the installation procedure that warns the use of TBB > 2020. Although I think we can get GTSAM to support both versions.

No. Read the for loop more carefully. The purpose of this script is to time the performance of TBB with different grain sizes, and this default grain size varies with number of threads. In order to read this default grain size, this loop initializes tbb with 1, 4, 8 threads, and reads the grain size.
Removing tbb::task_scheduler_init init((int)n); causes number of threads to be equal to number of cores for all cases, rendering the for loops in this test redundant and pointless.

Ahh I see now, yes we are specifying the number of threads each iteration of the for loop. Hmmm, well this is a predicament.

Let me link the TBB documentation: https://software.intel.com/en-us/node/506296 regarding task_scheduler_init

Some quotes from it:

An optional parameter to the constructor and method initialize allows you to specify the number of threads to be used for task execution. This parameter is useful for scaling studies during development, but should not be set for production use.

Tip
The reason for not specifying the number of threads in production code is that in a large software project, there is no way for various components to know how many threads would be optimal for other threads. Hardware threads are a shared global resource. It is best to leave the decision of how many threads to use to the task scheduler.

To minimize time overhead, it is best to rely upon automatic creation of the task scheduler, or create a single task_scheduler_init object whose activation spans all uses of the library's task scheduler. A task_scheduler_init is not assignable or copy-constructible.

But I mean this is just saying don't use it in production. Basically we want a feature to specify the number of threads we create.

I believe we can use task_arena instead. (https://software.intel.com/en-us/node/506359)

I'm not sure on how to replicate this script without task_scheduler_init so I am going to leave this script be, hopefully it will get tackled before the feature gets removed.

chrisbeall · 2020-01-12T17:12:04Z

examples/SolverComparer.cpp

  if(nThreads > 0) {
    cout << "Using " << nThreads << " threads" << endl;
-    init.reset(new tbb::task_scheduler_init(nThreads));


Same here. This line was important to the test.

Just because we do not explicitly create a task_scheduler_init does not mean, TBB or C++11 is not using its internal one. n threads are still created and the test is run in parallel if TBB is being used, although not needed now, since C++11 has concurrency.

If we always wanted max number of threads (i.e. threads = number of cores) that might be ok, but nThreads is an optional parameter to this program, so that the user can test timing for a specific number of threads. As I see it, removing this line effectively turns nThreads into an on/off switch.

Yep, thats right. Thx for the thorough review! I'll add a commit that adds this deprecated feature for now until we figure out how to handle it in the future.

ToniRV · 2020-03-11T04:06:39Z

Ping. Seems like tests are passing. Should we merge?

chrisbeall · 2020-03-11T05:33:03Z

Ping. Seems like tests are passing. Should we merge?

I think SolverComparer.cpp still has an issue. See my comment above.

acxz · 2020-03-25T05:22:38Z

I think now we just have to get the CI working with TBB, #210 or #250

varunagrawal · 2020-03-28T22:34:46Z

We got TBB into the travis build! 🎉
However, the TBB deprecation fix is a big deal with lots of potential places to break. I suggest we create an issue where we can track all the relevant information and fixes so that this enhancement (as well as potential future patches) can be tackled easily.

Basically, this PR is NOT ready to merge.

Here's the section in the TBB reference manual
https://software.intel.com/en-us/node/841348

acxz · 2020-03-28T22:58:10Z

We can just use #208 which this PR attempts to resolve.

Is there anything specific you would like me to do? For example explain the changes behind each changed line in this PR, and point to the respective TBB documentation?

varunagrawal · 2020-03-28T23:05:31Z

Ah I missed the link in your PR description, thank you.

I think this PR needs a concrete plan of attack, something which we don't have the time for right now, unfortunately, since this is a large engineering task due to the various things involved. Maintaining backwards compatibility is the number one priority (unless we do a version bump).

I am going to read up a bit more on how to fix all the deprecations using C++11 and then we can go from there.

In the meantime, if you wish to suppress the warnings, you can just add this line to the CMakeLists.txt

add_compile_definitions(TBB_SUPPRESS_DEPRECATED_MESSAGES=1)

acxz · 2020-03-28T23:09:50Z

Sounds good. Just let me know wherever I can help, I would love to get rid of the warnings and deprecated features.

acxz · 2020-03-28T23:11:03Z

Oh also one more thing, can you force a rebuild for the Travis CI, since building with tbb is now included in the develop branch. (https://stackoverflow.com/questions/17606874/trigger-a-travis-ci-rebuild-without-pushing-a-commit)

varunagrawal · 2020-03-29T01:52:31Z

@acxz already did.

ProfFan · 2020-04-05T21:04:09Z

I took a look at the changes and it aligned with the TBB 2020 manual. So I think after the Actions CI is merged we can also get TBB going by CI-ing

acxz · 2020-04-05T22:10:12Z

Thank you for reviewing my changes! Sounds good. I think it would be a good idea to target the April 23rd since that is when the next Ubuntu LTS release is and that LTS will ship with TBB 2020.

acxz · 2020-04-12T16:55:31Z

I found some time to replace task_scheduler_init with task_arena.
For TimeTBB I also compared the previous task_scheduler_init implementation with the new task_arena one to validate it. Here is the gist with the results. As you can see the results are very similar, note I only have a 2 core machine so the 8 thread metrics prob shouldn't be taken too seriously. @chrisbeall if you could review the new commits that would be great.

I believe this removes all deprecation warnings up to TBB 2020.1 which will land in Ubuntu 20.04. There is still some more deprecations warnings to tackle for TBB 2020.2 so I will leave the partial in the PR name.

ProfFan · 2020-04-13T14:52:28Z

examples/SolverComparer.cpp

@@ -228,6 +234,10 @@ int main(int argc, char *argv[]) {
    runPerturb();
  else if(stats)
    runStats();
+#ifdef gtsam_use_tbb


Should be upper cased?

yikes, thx for catching that. Fixed it up.

I am surprised that it didn't trigger CI fails (or I missed it?)

No it passed, im p sure preprocessor directives can be lowercase, just highly frowned upon since it can much more easily conflict with a variable name.

I mean in your previous file I see two types of macros, one in upper case and one in smaller case.

That is not triggering a CI fail because by default the smaller case is not defined, HOWEVER this means that our test suite is not catching that that piece of code is missing.

acxz · 2020-04-21T18:44:15Z

@varunagrawal have you gone through this PR on a deeper level to make more comments/suggestions?

acxz · 2020-05-06T01:32:51Z

@ProfFan I see that the Github Actions CI PR has been merged in, can we test this PR against it?
My bad, I noticed it was merged to a feature branch.

dellaert · 2020-05-06T14:24:43Z

@varunagrawal have you gone through this PR on a deeper level to make more comments/suggestions?

@varunagrawal I don't know what you want here, either :-)

dellaert · 2020-05-06T14:26:56Z

This is now an old PR and I'd love to merge it. To get things going, I compiled on an Ubuntu machine and had zero TBB-related warnings. Yay!

However, when I tried to run SolverComparer, I had an issue. This is an old script, so I just got the input arguments from the comments in the file, but got:

~/git/acxz/build$ examples/SolverComparer --incremental -d w10000 -o w_inc
Loading dataset w10000
Using threads for all processors
terminate called after throwing an instance of 'tbb::missing_wait'
  what():  wait() was not called on the structured_task_group
Aborted (core dumped)

acxz · 2020-05-06T16:41:15Z

Yikes, don't know how that slipped by me. I guess I forgot to add tg.wait(); before this line: https://github.com/borglab/gtsam/pull/209/files#diff-bb8dc817654b356c1ea808494cab0c65R239

I can fix it up later this afternoon. Thx for catching that @dellaert.

Fixed it up and changed the separate run/wait commands into the combined run_and_wait which is meant to be more efficient. See: https://software.intel.com/en-us/node/506287

SolverComparer does not segfault for me now.

dellaert

Seems to work now :-) I'll merge.
Some timing work might be needed to figure out how we can make TBB really help.

varunagrawal · 2020-05-06T22:57:15Z

Sorry I didn't get a chance to look at this. I trust everything works, and if not, we can open a regression bug.

acxz added 3 commits January 11, 2020 12:39

remove execution file mode

98a9022

change from deprecated tbb::mutex to std::mutex

eb85762

remove unnecessary tbb task_scheduler_init

b2e9331

chrisbeall reviewed Jan 11, 2020

View reviewed changes

dellaert requested changes Jan 11, 2020

View reviewed changes

add TBB guards back for multithread option and potentially breaking c…

1483df7

…hanges

remove deprecated tbb_exception in favor of std_exception

af5d222

remove tbb.h include and specify individual includes needed

aff24bd

remove deprecated tbb::task_scheduler_init

acxz changed the title ~~Tbb deprecation fix regarding mutex and task_scheduler_init~~ TBB deprecation fix Jan 11, 2020

acxz mentioned this pull request Jan 12, 2020

add tbb to travis-ci #210

Closed

chrisbeall reviewed Jan 12, 2020

View reviewed changes

add deprecated task_scheduler_init until alternative is found

bfc32e9

varunagrawal added the enhancement Improvement to GTSAM label Feb 23, 2020

add deprecated task_scheduler_init unitl alternative is found

8096b0e

varunagrawal mentioned this pull request Mar 26, 2020

Add TBB to Travis CI (simple) #250

Merged

acxz changed the title ~~TBB deprecation fix~~ TBB deprecation fix (partial) Mar 28, 2020

acxz mentioned this pull request Apr 5, 2020

TBB Deprecation Warnings #208

Closed

replace task_scheduler_init with task arena/group

b4b4876

ProfFan reviewed Apr 13, 2020

View reviewed changes

replace task_scheduler_init with task arena/group

df9cf86

change to more efficient call of threaded functions

e0cbc76

dellaert approved these changes May 6, 2020

View reviewed changes

dellaert merged commit b4eb0c2 into borglab:develop May 6, 2020

acxz deleted the tbb_fix branch May 7, 2020 00:02

chrisbeall mentioned this pull request Aug 10, 2020

Support thread-safe const marginalCovariance #460

Merged

TBB deprecation fix (partial) #209

TBB deprecation fix (partial) #209

Conversation

acxz commented Jan 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ProfFan commented Jan 11, 2020

acxz commented Jan 11, 2020

Choose a reason for hiding this comment

dellaert commented Jan 11, 2020

acxz commented Jan 11, 2020

ProfFan commented Jan 11, 2020

acxz commented Jan 11, 2020

acxz commented Jan 11, 2020

acxz commented Jan 11, 2020

ProfFan commented Jan 12, 2020

acxz commented Jan 12, 2020

dellaert commented Jan 12, 2020

acxz commented Jan 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acxz Jan 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acxz Jan 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ToniRV commented Mar 11, 2020

chrisbeall commented Mar 11, 2020

acxz commented Mar 25, 2020

varunagrawal commented Mar 28, 2020 • edited Loading

acxz commented Mar 28, 2020 • edited Loading

varunagrawal commented Mar 28, 2020

acxz commented Mar 28, 2020

acxz commented Mar 28, 2020

varunagrawal commented Mar 29, 2020

ProfFan commented Apr 5, 2020

acxz commented Apr 5, 2020

acxz commented Apr 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acxz commented Apr 21, 2020

acxz commented May 6, 2020 • edited Loading

dellaert commented May 6, 2020

dellaert commented May 6, 2020

acxz commented May 6, 2020 • edited Loading

dellaert left a comment

Choose a reason for hiding this comment

varunagrawal commented May 6, 2020

acxz commented Jan 11, 2020 •

edited

Loading

acxz Jan 12, 2020 •

edited

Loading

acxz Jan 12, 2020 •

edited

Loading

varunagrawal commented Mar 28, 2020 •

edited

Loading

acxz commented Mar 28, 2020 •

edited

Loading

acxz commented May 6, 2020 •

edited

Loading

acxz commented May 6, 2020 •

edited

Loading