Remove file system requirements

[mnm678]

The specification refers to files and filesystem operations done on a local filesystem. Some implementations may use TUF on distributed systems or a different non-traditional file backend. To support these implementations, the specification should leave flexibility in how files are represented. This includes changing requirements about paths, directory structure, and files.

For background see discussion in tuf#1009 and #102.

[justincormack]

We also need this for Notary v2 as container registries are basically content addressed stores not filesystems, so we need to directly retrieve by content hash not filename. The consistent snapshots scheme is arguably half way between a file based version and a content hash based version.

[joshuagl]

Thanks for commenting @justincormack!

If I recall context correctly, this issue was originally triggered by an observation that how we describe the TUF metadata is file-centric (i.e. in section 4. Document formats) when it could just as easily be stored in another way, such as a database.

Your comment makes me realise we may also want to think about how we "address" the targets (and other metadata "files") when they are described by metadata. Obvious places in the spec that assume file-paths as addresses are METAPATH in snapshot and timestamp role's "meta" entries, and the TARGETPATH for "targets" entries in the targets role's metadata.
Are there any other aspects of the spec that you think are incompatible with CAS?

I wonder, in a CAS would the name/address of the target (or metadata "file") replace the need to list the target's cryptographic hash in the metadata?

During the PEP 458 implementation we ran into a minor issue where consistent snapshots as specified haven't suited the existing system being integrated with. Having the description of consistent snapshots in the spec be less proscriptive may be useful?

[joshuagl]

Related issue #144

[jku]

I'll add my findings about using rolenames and targetpaths as parts of filenames from theupdateframework/python-tuf#1527:

• Using arbitrary strings (urls, paths, whatever) as rolenames is a feature and we should try not to limit it
• Using those rolenames as filenames is problematic (for both repository and client) as not all strings are actually valid filesystem filenames -- but the spec at times requires using the rolename as filename. Sanitizing the filename is difficult (as collisions must be prevented)
• Using those rolenames as filenames is also problematic because it can lead to path traversal with roles like ../../../filename. This is especially an issue for the client as rolename is input from remote that should not be trusted more than is required (the input is verified by threshold of signatures, but still not something to blindly obey if it tells me to overwrite files in my home dir...)
• There are no known security issues with arbitrary rolenames as files and version numbers, but the area seems ripe for them: is 1.role.json the metadata for the role role or the role 1.role?

Many of these issues apply to targetpaths as well -- although targetpaths are at least guaranteed to be url-path-fragments.

My summary of this is that using rolenames and targetpaths as filenames is unsafe in several ways. So even the implementations that do use a filesystem to store metadata and targets should not be using rolenames and targetpaths as filenames without a lot of caution or additional demands that spec does not currently make

[asraa]

I may have backtracked on my understanding of this, but shouldn't a valid snapshot be pinning these files with their hashes anyway and prevent collisions/incorrect fetches?

[joshuagl]

Another item from this discussion in #150:

we should stop talking about "deleting files", or at least make that a secondary aspect: what is important is that the client should not consider the metadata trusted anymore (in practice, we might want to say out loud why exactly we do this: the existing, now untrusted, metadata should not be used to do rollback checks on the new metadata)