Design repository library

[lukpueh]

[Update 2/14/22]

• ADR 10 shows the updated direction of this issue including further reading (design document, prototype implementation)
• Comments in thread here below point to survey documents for existing repository tooling, based on which ADR 10 was developed

---

[Original issue description]

Identify what functionality from existing repository_tool/repository_lib/roledb/keydb modules should exist in a TUF repository library on top of a metadata model as sketched in #1112.

Start with API functionality.

Coordinate with #1134.

[joshuagl]

I think a good place to start with this would be looking at what scaffolding Warehouse ends up building on top of the metadata API.

pypi/warehouse#7488
pypi/warehouse#8955

[joshuagl]

See also a WIP addition to the specification describes the repository workflow for adding/updating targets which this library should implement: theupdateframework/specification#153

[joshuagl]

See also a WIP addition to the specification describes the repository workflow for adding/updating targets which this library should implement: theupdateframework/specification#153

This has landed and can be referenced here https://theupdateframework.github.io/specification/latest/#adding-updating-targets

[jku]

Linking to #1459 (comment). If this issue is top-down design that one is definitely bottom-up: I'm creating initial repositories in a few places and the code always looks quite ugly but the last comment maybe has a decent idea that would work without lots of new API.

[lukpueh]

Adding the list of surveys of existing TUF repository implementations here so that we can close #1612...

• python-tuf
• repository-editor-for-tuf
• go-tuf and php-tuf
• AWS Tough
• Warehouse

[lukpueh]

Adding more relevant documents ...

• TUF (Re-)Design Document / Section: Repo tool -- high-level directions, potential functionality
• Repo Usage Scenarios -- details required inputs, e.g. knowledge about role metadata, for each scenario
• Repo Tool Design -- basic high-level design with many open questions
• Strawman repo design proposal -- bottom-up design, based on vmware-labs/repository-editor-for-tuf, seems very promising...

Next steps:

• Revise open questions and generalize problems
• Revise strawman design for suitability to answer these questions
• Revise strawman design / prototype implementation for remaining open questions
• Write design document

[lukpueh]

Next steps:

• Revise open questions and generalize problems
• Revise strawman design for suitability to answer these questions
• Revise strawman design / prototype implementation for remaining open questions
• Write design document

See updated issue description above for latest results...

[lukpueh]

Feature request:
Convenience interface that takes a path to local target file(s) and constructs targets metadata, computing length and hashes from the local file and normalizing the path to match the TARGETPATH specification (see #1019 and #1018 for details and challenges)

[prateek041]

I would like to work on this issue, where should I start ?

[lukpueh]

I would like to work on this issue, where should I start ?

Reading the resources referenced above in the issue description and brainstorming designs seems like the way to start.

But as @jku has mentioned on slack, this is likely a long term issue that is going to require experimentation and quite a bit of team work to solve.

[joshuagl]

Another implementation we could look to for inspiration is Rugged, which implements repository and a key manager classes on top of the low-level metadata API.

A key manager/keyring feels like a useful helper to provide.

[jku]

I've decided to just do a PR of this:

• a tiny repository abstraction
• example implementation of that abstraction (a in-memory ephemeral repository a bit like the one in tests)
• demo repository application using that in-memory implementation that simulates a live repository by generating changes every few seconds

I have that 90% ready, will push the PR this week. Future steps would include:

• example developer tool (built using the same building blocks as the repository) that is able to sign changes and submit them to the demo repository application. I have a rough idea, should not be too much work

I have no assumptions that the API would be final in this PR: as minimal as it is, new use cases seem to always bring new complications... So I suggest we

• try to merge something soon
• keep the repository abstraction as unstable API for now
• build more implementations to gather data (potential ones at least RSTUF, repository-editor-for-tuf, repository-playground)
• improve as new ideas come up