Use py-rattler to fetch repodata in proxy mode #677
Conversation
beenje
changed the title
|
@Wackyator Maybe you could think about the API change to be able to get the entire json? |
quetz/tasks/mirror.py
Outdated
| @@ -106,6 +114,32 @@ def json(self): | |||
| return json.load(self.file) | |||
|
|
|||
|
|
|||
| def download_repodata(repository: RemoteRepository, channel: str, platform: str): | |||
There was a problem hiding this comment.
Since this function is basically a wrapper around fetch_repo_data, I'd keep this one as an asynchronous function and do the asyncio.run(...) part in the caller for this function.
There was a problem hiding this comment.
Changed. I used aiofiles to make the files operation async then.
Codecov ReportAttention:
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #677 +/- ##
==========================================
+ Coverage 83.61% 83.66% +0.04%
==========================================
Files 79 79
Lines 6233 6264 +31
==========================================
+ Hits 5212 5241 +29
- Misses 1021 1023 +2 ☔ View full report in Codecov by Sentry. |
|
For testing purpose I added the |
|
@ivergara are you expecting anything more from me on this PR? |
No, all looks good from my side. Hopefully, more people chime in before it gets approved and merged. CC @janjagusch |
rattler is very efficient to download repodata
serve_repo_data fixture copied from rattler
dummy_remote_session_object wasn't cleaning after itself (using return instead of yield)
beenje
force-pushed
the
improve-proxy-repodata
branch
from
5d59e0c
to
876e929
Compare
Test with migration failed with: Error: The action 'Testing server' has timed out after 5 minutes.
beenje
force-pushed
the
improve-proxy-repodata
branch
from
6fd6b37
to
a32a1cd
Compare
|
I rebased to fix conflicts. I had to increase the CI timeout (got Hope this can be merged or closed. |
As mentioned in #660, downloading repodata from a remote server for proxy channels is quite slow right now.
This is a proposal to use
py-rattlerto download repodata, as it's very efficient.I didn't find how to access the full json file from the
SparseRepoDatareturned byfetch_repo_data.I look and return the json file from disk. Maybe there is a better way.
I haven't added any tests yet. Waiting for some feedback first.
@wolfv, any opinion?