force garbage collection on dnf.Base.reset()

[lersek]

The DnfSack object has two properties that together cause an obscure bug:

• DnfSack (very indirectly) owns the libsolv library's "page file file descriptors".
• DnfSack is not immediately released in dnf.Base.reset(), due to circular references -- instead, DnfSack is only released when Python's generational garbage collector thinks it's time to do so.

The result of this is that file descriptors can be leaked for an unspecified amount of time after dnf.Base.reset() -- potentially called from dnf.Base.close() -- returns, until the garbage collector eventually cleans them up. Because the kernel sees those leaked file descriptors, they can interfere with operations on the filesystem they point into. For example, they can prevent livecd-creator from unmounting the /var/cache/dnf filesystem within the install_root (chroot) filesystem.

The commit message on the first patch contains an excruciatingly detailed bug analysis; the code change itself is minimal.

You can read my original analysis / investigation at https://bugzilla.redhat.com/show_bug.cgi?id=2038105, comments 45 through 61.

Please do not edit or squash my commits. If updates are needed, ask me for a force-push please. Thanks.

[rwmjones]

it's a misfeature that should to be removed, as it permits the corruption

should be removed

Also would it be a good idea to add a one-line comment in the code explaining why you're calling the garbage collector there?

[lersek]

should be removed

Yes, thanks.

Also would it be a good idea to add a one-line comment in the code explaining why you're calling the garbage collector there?

I can add something like

# force garbage collection so that we synchronously close libsolv's page file fds here

but I don't know if we want to be that specific there -- after all, this is the dnf source code, and libsolv is not a direct dependency of dnf. dnf depends directly only on libdnf, and libsolv is sort of an "implementation detail" of libdnf.

I kind of see this as a general failure of garbage-collected programming languages. libsolv provides all the necessary destructor functions, providing the client application with full control over resource release. And then comes Python and the circular references, and all that control is willfully discarded.

[lukash]

First off, thanks for doing such an excruciating analysis @lersek. I think what @rwmjones means and what I'd also like to have in the comment is a shorter version of the description from your commit message that would kinda sum it up, but provide the important information about the issue it solves. IMO you should use as many lines as needed to achieve that 🙂. Ideally anyone reading it in a few years will fully understand what's going on from just the comment without having to dig up the commit message.

[lersek]

1:  03715c3c3e05 ! 1:  20bff6f476df Base.reset: plug (temporary) leak of libsolv's page file descriptors
    @@ -128,9 +128,9 @@
                         ts.add_install()
     
             list_installs() creates a list of packages that need to be installed
    -        by DNF. The loop in packagelist_to_pylist(), which constructs the
    -        elements of that list, Py_BuildValue() is called with the "O" format
    -        specifier, and that increases the reference count on "_sack".
    +        by DNF. Inside the loop in packagelist_to_pylist(), which constructs
    +        the elements of that list, Py_BuildValue() is called with the "O"
    +        format specifier, and that increases the reference count on "_sack".
     
             Subsequently, in the _goal2transaction() method, we iterate over the
             package list created by list_installs(), and add each package to the
    @@ -202,10 +202,11 @@
         > normally, using lazy unmount
     
         (Unfortunately, the whole lazy umount idea is misguided in livecd-tools;
    -    it's a misfeature that should to be removed, as it permits the corruption
    -    of the loop-backed filesystem. Now that the real bug is being fixed in
    -    DNF, lazy umount is not needed as a (broken) workaround in livecd-tools.
    -    But that's a separate patch for livecd-tools.)
    +    it's a misfeature that should be removed, as it permits the corruption of
    +    the loop-backed filesystem. Now that the real bug is being fixed in DNF,
    +    lazy umount is not needed as a (broken) workaround in livecd-tools. But
    +    that's a separate patch for livecd-tools:
    +    <https://github.com/livecd-tools/livecd-tools/pull/227>.)
     
         Plug the fd leak by forcing a garbage collection in dnf.Base.reset()
         whenever we cut the "_sack", "_goal" and "_transaction" links -- that is,
    @@ -263,6 +264,44 @@
                  self._transaction = None
              self._update_security_filters = []
     +        if sack and goal:
    ++            # We've just done this, above:
    ++            #
    ++            #      _sack                     _goal
    ++            #         |                        |
    ++            #    -- [CUT] --              -- [CUT] --
    ++            #         |                        |
    ++            #         v                |       v
    ++            #    +----------------+   [C]  +-------------+
    ++            #    | DnfSack object | <-[U]- | Goal object |
    ++            #    +----------------+   [T]  +-------------+
    ++            #      |^    |^    |^      |
    ++            #      ||    ||    ||
    ++            #      ||    ||    ||         |
    ++            #   +--||----||----||---+    [C]
    ++            #   |  v|    v|    v|   | <--[U]-- _transaction
    ++            #   | Pkg1  Pkg2  PkgN  |    [T]
    ++            #   |                   |     |
    ++            #   | Transaction oject |
    ++            #   +-------------------+
    ++            #
    ++            # At this point, the DnfSack object would be released only
    ++            # eventually, by Python's generational garbage collector, due to the
    ++            # cyclic references DnfSack<->Pkg1 ... DnfSack<->PkgN.
    ++            #
    ++            # The delayed release is a problem: the DnfSack object may
    ++            # (indirectly) own "page file" file descriptors in libsolv, via
    ++            # libdnf. For example,
    ++            #
    ++            #   sack->priv->pool->repos[1]->repodata[1]->store.pagefd = 7
    ++            #   sack->priv->pool->repos[1]->repodata[2]->store.pagefd = 8
    ++            #
    ++            # These file descriptors are closed when the DnfSack object is
    ++            # eventually released, that is, when dnf_sack_finalize() (in libdnf)
    ++            # calls pool_free() (in libsolv).
    ++            #
    ++            # We need that to happen right now, as callers may want to unmount
    ++            # the filesystems which those file descriptors refer to immediately
    ++            # after reset() returns. Therefore, force a garbage collection here.
     +            gc.collect()
      
          def _closeRpmDB(self):
2:  b5abcaebef5e = 2:  0c2eff388936 AUTHORS: add Laszlo Ersek

[lersek]

@lukash can I please trouble you to review the updated patch set? As highlighted in #1825 (comment) above, I've updated the commit message (the grammatical error pointed out by @rwmjones , plus a typo I noticed myself), and I've distilled the reasoning to a new comment in the code. No code changes. (Still retested it though.) Thanks!

[lukash]

@lersek you certainly can. LGTM now. Again, thanks for the elaborate analysis, very appreciated!