Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add determinism #145

Draft
wants to merge 20 commits into
base: main
Choose a base branch
from
Draft

Add determinism #145

wants to merge 20 commits into from

Commits on Aug 25, 2023

  1. Add deterministic behavior for calls against a single host: 

    By queuing requests per host we now provide
deterministic behavior of calls.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    1f2d25d View commit details
    Browse the repository at this point in the history

  2. Update linting: 

    Version was old.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    b9b43ac View commit details
    Browse the repository at this point in the history

  3. Fix linting and test failures: 

    Fixes issue where the task is not found.
This was happening because I was creating
the record in the DB in a goroutine and
mocking the action to immediately fail.
This caused the worker to complete and close
before any record could be created.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    10e0e65 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. Add elastic worker pool 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    a148628 View commit details
    Browse the repository at this point in the history

  2. Fix linting issues 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    d459741 View commit details
    Browse the repository at this point in the history

  3. Use NewRunner func 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    6d432f9 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2023

  1. [WIP]: update the design: 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 3, 2023
    Configuration menu
    Copy the full SHA
    f1cc9f8 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2023

  1. Fix orphaned goroutine in worker: 

    The goroutine (in worker) dequeuing tasks was a
blocking call. This meant that after there were
no tasks in the queue, this dequeue call would sit
waiting. This meant that when the rest of the worker
exited this goroutine did not and was orphaned. Workers
are per host id so depending on unique hosts that
requested tasks this would grown and stay at that number.
The resolution was to make the dequeue call non-blocking.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    bf1fe48 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. Clean up: 

    Fix tests, updating for linting issues, etc

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    17f982a View commit details
    Browse the repository at this point in the history

  2. Extend time waiting for task service to be ready. 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    cff1d6c View commit details
    Browse the repository at this point in the history

  3. Fix tests 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    fb4fead View commit details
    Browse the repository at this point in the history

  4. Remove unused code 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    056be60 View commit details
    Browse the repository at this point in the history

  5. Make maxWorkers and workerIdleTimeout configurable: 

    Plumb these through to the cli flags.

Also fixed a channel close so that when workers
are done the goruntimes drops back down to original
levels.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    b90c138 View commit details
    Browse the repository at this point in the history

  6. Fix linting issue 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    1cb6b99 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Add metrics, tune for better ingestion: 

    The ingestor was single threaded.
After adding metrics and watching
the queues and processing happen,
running the ingestor process with a
max goroutine cap yielded better performance.
The per ID queues filled up faster and
allowed processing of BMC interactions
faster.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    5bbd289 View commit details
    Browse the repository at this point in the history

  2. Fix linting issue 

    Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    c4c4e01 View commit details
    Browse the repository at this point in the history

  3. Add a max number of concurrent workers for the ingestion: 

    This keeps the goroutines from growning
uncontrollably.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    5d91df1 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2023

  1. WIP refactor to use channels instead of the queue: 

    This made processing significantly faster.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 9, 2023
    Configuration menu
    Copy the full SHA
    4877220 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2023

  1. Clean up unused code: 

    Update the metric server to handle
slow loris attacks.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 10, 2023
    Configuration menu
    Copy the full SHA
    e9d0f2f View commit details
    Browse the repository at this point in the history

  2. Make task channel smaller: 

    It was too big and made memory
consumption way too high. Also,
the typical/general use-case will be
high number of hosts and low number of
tasks that are requested close together.

Signed-off-by: Jacob Weinstock <[email protected]>
    @jacobweinstock
    jacobweinstock committed Sep 10, 2023
    Configuration menu
    Copy the full SHA
    73de1b2 View commit details
    Browse the repository at this point in the history