Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit concurrency of asyncio for gather_objects_recommendations #58

Closed
lujiajing1126 opened this issue Jun 1, 2023 · 2 comments
Closed
Labels
bug Something isn't working

Comments

@lujiajing1126
Copy link

Describe the bug

When we have a large amount of container/pods in the cluster, e.g. >1000.

async def _gather_objects_recommendations(
self, objects: list[K8sObjectData]
) -> list[tuple[ResourceAllocations, MetricsData]]:
recommendations: list[tuple[RunResult, MetricsData]] = await asyncio.gather(
*[self._calculate_object_recommendations(object) for object in objects]
)

will start the >1000 coroutines to concurrently query Metrics server which cause resource exhaustion, e.g. connection pool in the VictoriaMetrics, local memory larger than 20GB, etc.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior

According to the SO,

async def gather_with_concurrency(n: int, *coros):
    semaphore = asyncio.Semaphore(n)

    async def sem_coro(coro):
        async with semaphore:
            return await coro
    return await asyncio.gather(*(sem_coro(c) for c in coros))

can be used to limit concurrency level.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

@lujiajing1126 lujiajing1126 changed the title Limit coroutine of asyncio for gather_objects_recommendations Limit concurrency of asyncio for gather_objects_recommendations Jun 1, 2023
@LeaveMyYard
Copy link
Contributor

We are working on this issue and will fix it soon
I will update here as soon as it is finished

@LeaveMyYard LeaveMyYard added the bug Something isn't working label Jun 23, 2023
@LeaveMyYard
Copy link
Contributor

#82 was already merged for that issue.
It also added the -w flag to control the exact amount of executors to be used

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants