Cache location config #120

jhiemstrawisc · 2023-04-27T21:44:38Z

See Issue #118

There has been desire to create a method by which users can configure
where to store the key cache without setting the env var $XDG_CACHE_HOME
or relying on the user to have a home dir. This is especially relevant
in cases where the library is embedded in a daemon (e.g. XRootD or HTCondor)
that may have a preferencial key cache location. This commit adds a familiar
config_set_str and config_get_str API for passing a new cache home to
the library through the use of the keycache.cache_home key.

As with config_set_int and config_get_int, these new functions can be
easily updated in the future as the project develops and new
configuration keys are desired. This would entail adding a new check for
the desired key and the logic to handle any values associated with that key.

Some comments/questions:

Do we need to add a mutex/lock to any of this configuration?
Are we interested in more robust error handling in the case that
a garbage path is provided to the configuration API? I made the
assumption that we weren't, since there were no checks on what
was supplied by XDG_CACHE_HOME. Currently, the implementation
tries to create whatever directory is passed through the API if it doesn't
exist, but this could break if the user has insufficient permissions to
write to that location, or the path contains invalid chars (and probably
some other things I'm not thinking of).
I lumped the directory handling/creation methods into the configuration
class. Would we rather those be moved to a new class dedicated to this
type of functionality? I noticed in testing that if XDG_CACHE_HOME is
set to a directory that doesn't already exist, the current code will fail if
any parent dirs need to be created. The new config API attempts to create
those parent dirs, so maybe a stub class to handle these types of things
would be useful.

There has been desire to create a method by which users can configure where to store the key cache without setting the env var `$XDG_CACHE_HOME` or relying on the user to have a home dir. This is especially relevant in cases where the library is embedded in a daemon (e.g. XRootD or HTCondor) that may have a preferencial key cache location. This commit adds a familiar `config_set_str` and `config_get_str` API for passing a new cache home to the library through the use of the `keycache.cache_home` key. As with `config_set_int` and `config_get_int`, these new functions can be easily updated in the future as the project develops and new configuration keys are desired. This would entail adding a new check for the desired key and the logic to handle any values associated with that key.

Getting rid of a few commented blurbs, cleaning up a few extraneous spaces, etc.

src/scitokens.cpp

src/scitokens_cache.cpp

src/scitokens_internal.cpp

src/scitokens_internal.h

test/main.cpp

src/scitokens_cache.cpp

src/scitokens_internal.cpp

src/scitokens_internal.h

bbockelm · 2023-04-27T23:21:53Z

Suddenly I have opinions about allowing the robot to fix, and not just complain about, linter issues!

I would like to make this thread safe. Can you please replace the the storage of this information with a std::shared_ptrstd::string?

src/scitokens.cpp

src/scitokens_internal.cpp

bbockelm · 2023-05-01T12:28:00Z

src/scitokens_internal.cpp

+    std::string component;
+
+    while (std::getline(ss, component, '/')) {
+        pathComponents.push_back(component);


What happens if someone provides a path of ///foo//bar? May need an if (!component.empty()) here to avoid inserting unnecessary items.

You caught my sinful shortcut. Since ultimately this output only gets passed to mkdir, the POSIX standard should suppress the extra slashes (so that the cache would be dumped at /foo/bar even if it was specified as ///foo//bar). But I agree, I should add the check in case this function ever gets used in another spot where someone might not know about that peculiarity.

bbockelm · 2023-05-01T12:31:50Z

src/scitokens_internal.cpp

+    std::vector<std::string> pathComponents = path_split(dir_path);
+    for (const auto &component : pathComponents) {
+        currentLevel += "/" + component;
+        if (!check_dir(currentLevel)) {


Don't check the dir first; just create it and inspect the mkdir return code (if it's EEXIST, then the directory already exists and move on).

For error message creation, it would be better to have an output variable that's the error message so you can say precisely the file operation that failed. The caller of this function currently has something generic that won't be useful for debugging.

The previous code worked when configured to use a cache home like `///foo//bar`, but it wasn't set up to handle this situation robustly -- small future changes could have easily broken the functionality, so I modified the path parsing and directory checking functions to better handle this. I also changed the way error messages are propagated by these new config options to make sure that any errors thrown when calling `mkdir()` make it to the config API's err_msg output variable. This should make debugging a bit easier in any cases where someone unwittingly tries to plop their cache someplace where they don't have read/write access.

src/scitokens_internal.cpp

src/scitokens_internal.h

jhiemstrawisc · 2023-05-03T15:41:26Z

Hmm, any ideas what might be causing the EnforcerTest to fail in the one instance? When I run locally, all tests pass.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

djw8605 · 2023-05-03T15:50:42Z

You say that, but now it looks good. Are you happy with this pull request now?

bbockelm · 2023-05-04T12:39:21Z

@jhiemstrawisc - can you clean up the remaining linter items? Once that's done, I think we're ready to go.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

jhiemstrawisc · 2023-05-04T13:45:02Z

@bbockelm Okay, latest commit made the linter happy, and all the tests are passing.

JaimeFrey · 2023-05-04T20:05:55Z

The symbols config_set_str() and friends seem far too generic for a public library with C bindings.

Unfortunately, the choice of names for the config_set_blah() family of APIs is quite generic and poses the potential to cause C symbol collisions when SciTokens is used by large projects that dynamically load lots of libraries (a la HTCondor). A reasonable plan to address this, since those APIs are in use and changing their names would be API-breaking, seems to be adding a second set of prefixed APIs, with the intention of deprecating the old functions at the next major release.

src/scitokens.cpp

src/scitokens.h

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

JaimeFrey · 2023-05-12T15:04:17Z

With the new config functions prefixed with scitokens_, I suggest changing the unprefixed versions to just call the prefixed ones. That avoids a clone of the code, where the two copies can diverge in functionality by accident.

…_set_blah() To cut down on code duplication, the config_set_blah() family of APIs have been changed to immediately call the scitokens-prefixed family of APIs under the hood.

src/scitokens.h

Per Jaime's comment, since we plan to deprecate the `config_set_blah()` family of functions in favor of the prefixed versions, I've updated the internal API documentation to point people toward the preferred APIs.

jhiemstrawisc · 2023-05-15T14:51:51Z

Anything else on this, or are we ready to wrap things up?

JaimeFrey · 2023-05-15T21:14:30Z

Nothing from my end.

djw8605 · 2023-05-15T21:21:22Z

I wonder how fast we can depreciate the config_* functions. Is HTCondor already using them? I know xrootd isn't. If no one is using them, can we just remove them already?

timtheisen · 2023-05-15T21:42:20Z

Good point. The config_set_str() and config_get_str() were never released. Why release a deprecated API?

HTCondor does not use config_set_int() or config_get_int().

From HTCondor's point of view, it could be removed immediately.

JaimeFrey · 2023-05-16T02:36:43Z

I just noticed that the other functions start with scitoken_, not scitokens_. Most, though not all, are dealing with a single SciToken object.

jhiemstrawisc · 2023-05-16T13:35:54Z

Oh, I thought I had heard that the older config_set/get_int functions were already in use by someone somewhere. If nobody is using them at this point, I'll update to just get rid of them.

…oken It looks like the config_set/get_blah APIs weren't actually in use and hadn't been released, so instead of keeping them around, we decided to just delete them. Also, the `scitokens`-prefixed versions of these APIs were changed to be prefixed by `scitoken` to match other functions in the library.

src/scitokens.cpp

src/scitokens.h

test/main.cpp

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

I mistakenly removed the `config_set/get_int` APIs when removing `config_set/get_str`.

timtheisen · 2023-05-16T15:53:13Z

I think that Brian B wanted to have the C++ deprecated decorator (not just a comment).

https://en.cppreference.com/w/cpp/language/attributes/deprecated

JaimeFrey · 2023-05-16T15:58:08Z

Since scitokens.h contains C declarations, I don't think the C++ deprecated decorator is appropriate (or even allowed).

jhiemstrawisc · 2023-05-16T16:56:24Z

Brian B said to worry about any deprecated decorators in another PR at another time. I think he's looking to have this PR wrapped up expediently.

It looks like the deprecated decorator for C is supported after C23: https://en.cppreference.com/w/c/language/attributes/deprecated, whereas the deprecated decorator for C++ has been supported since C++14.

bbockelm · 2023-05-16T17:46:05Z

Yeah - unfortunately, we’ll need to wrap the deprecated stuff in some macro as it’s only recently standardized in C… hence punting it to a new PR.

jhiemstrawisc · 2023-05-17T14:11:28Z

Since we aren't adding the decorator for deprecation warnings in this PR, is there anything else needed to close this?

timtheisen · 2023-05-17T14:20:50Z

I think that it is good to go.

JaimeFrey · 2023-05-17T15:45:53Z

Nothing from me.

djw8605 · 2023-05-17T17:15:55Z

Is config_get_int and config_set_int being used anywhere? I still think we can remove it without depreciating it.

Otherwise, I think we're in good shape here.

timtheisen · 2023-05-17T18:33:33Z

Let's keep config_get_int and config_set_int for now. If we remove them, we'd have to bump the library to version 2.0 for the ABI change. We need to get this out the door without delay. We can deal with the deprecated methods in the future.

jhiemstrawisc added 2 commits April 27, 2023 20:57

Cleanup a few formatting discrepancies

d802554

Getting rid of a few commented blurbs, cleaning up a few extraneous spaces, etc.

github-actions bot reviewed Apr 27, 2023

View reviewed changes

jhiemstrawisc mentioned this pull request Apr 27, 2023

Configuration API should allow cache location management #118

Open

Incorporate feedback from linter

90552da

github-actions bot reviewed Apr 27, 2023

View reviewed changes

Switch m_cache_home to use be std::shared_ptr<std::string>

2b20b41

github-actions bot reviewed Apr 28, 2023

View reviewed changes

src/scitokens.cpp Outdated Show resolved Hide resolved

src/scitokens_internal.cpp Outdated Show resolved Hide resolved

bbockelm reviewed May 1, 2023

View reviewed changes

github-actions bot reviewed May 3, 2023

View reviewed changes

Apply suggestions from linter review

d0c6e33

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Apply suggestions from linter review

c12e0c2

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

github-actions bot reviewed May 12, 2023

View reviewed changes

src/scitokens.cpp Outdated Show resolved Hide resolved

src/scitokens.h Outdated Show resolved Hide resolved

Apply line wrap suggestions from linter

37e2445

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Change config_set_blah() family of functions to call scitokens_config…

fffb45a

…_set_blah() To cut down on code duplication, the config_set_blah() family of APIs have been changed to immediately call the scitokens-prefixed family of APIs under the hood.

JaimeFrey reviewed May 12, 2023

View reviewed changes

src/scitokens.h Outdated Show resolved Hide resolved

Update internal documentation to emphasize future API deprecation

38af01c

Per Jaime's comment, since we plan to deprecate the `config_set_blah()` family of functions in favor of the prefixed versions, I've updated the internal API documentation to point people toward the preferred APIs.

github-actions bot reviewed May 16, 2023

View reviewed changes

jhiemstrawisc and others added 2 commits May 16, 2023 08:44

Apply suggestions from linter review

8d74f67

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Restore config_set/get_int

6cb9b2c

I mistakenly removed the `config_set/get_int` APIs when removing `config_set/get_str`.

djw8605 merged commit dc2919b into scitokens:master May 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache location config #120

Cache location config #120

jhiemstrawisc commented Apr 27, 2023

bbockelm commented Apr 27, 2023

bbockelm May 1, 2023

jhiemstrawisc May 2, 2023

bbockelm May 1, 2023

jhiemstrawisc commented May 3, 2023

djw8605 commented May 3, 2023

bbockelm commented May 4, 2023

jhiemstrawisc commented May 4, 2023

JaimeFrey commented May 4, 2023

JaimeFrey commented May 12, 2023

jhiemstrawisc commented May 15, 2023

JaimeFrey commented May 15, 2023

djw8605 commented May 15, 2023

timtheisen commented May 15, 2023 •

edited

Loading

JaimeFrey commented May 16, 2023

jhiemstrawisc commented May 16, 2023

timtheisen commented May 16, 2023

JaimeFrey commented May 16, 2023

jhiemstrawisc commented May 16, 2023 •

edited

Loading

bbockelm commented May 16, 2023

jhiemstrawisc commented May 17, 2023

timtheisen commented May 17, 2023

JaimeFrey commented May 17, 2023

djw8605 commented May 17, 2023

timtheisen commented May 17, 2023

Cache location config #120

Cache location config #120

Conversation

jhiemstrawisc commented Apr 27, 2023

bbockelm commented Apr 27, 2023

bbockelm May 1, 2023

Choose a reason for hiding this comment

jhiemstrawisc May 2, 2023

Choose a reason for hiding this comment

bbockelm May 1, 2023

Choose a reason for hiding this comment

jhiemstrawisc commented May 3, 2023

djw8605 commented May 3, 2023

bbockelm commented May 4, 2023

jhiemstrawisc commented May 4, 2023

JaimeFrey commented May 4, 2023

JaimeFrey commented May 12, 2023

jhiemstrawisc commented May 15, 2023

JaimeFrey commented May 15, 2023

djw8605 commented May 15, 2023

timtheisen commented May 15, 2023 • edited Loading

JaimeFrey commented May 16, 2023

jhiemstrawisc commented May 16, 2023

timtheisen commented May 16, 2023

JaimeFrey commented May 16, 2023

jhiemstrawisc commented May 16, 2023 • edited Loading

bbockelm commented May 16, 2023

jhiemstrawisc commented May 17, 2023

timtheisen commented May 17, 2023

JaimeFrey commented May 17, 2023

djw8605 commented May 17, 2023

timtheisen commented May 17, 2023

timtheisen commented May 15, 2023 •

edited

Loading

jhiemstrawisc commented May 16, 2023 •

edited

Loading