Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/concurrency #25

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

youwe-petervanderwal
Copy link
Contributor

@youwe-petervanderwal youwe-petervanderwal commented Mar 10, 2021

Fix concurrency issue in QueueManager::processBatch

Error: Uncaught Amp\Beanstalk\DeadlineSoonException causing the ESB to crash

Caused by: Wrong Beanstalk job deleted at the end of the job
Caused by: Multiple Beanstalk jobs for the same job UUID. With multiple concurrent workers that same job UUID will be assigned to multiple workers. $uuidToBeanstalkIdMap[$job->getUuid()] will then only contain one of the beanstalk job ids, so one of the worker's queue manager won't be able to delete his job after finish.
Caused by: Concurrent calls to QueueManager::processBatch() were possible (and should be possible), but the batch itself was shared over multiple HTTP requests. Causing the same job being inserted twice (or more).

Resolved by: Processing the batch via a local variable instead of the shared property.

Steps to reproduce:

  1. Create a flow with a HttpRequestProducer producing a job (or multiple jobs) on each call
  2. Set the worker.instances config for that flow greater than one (I used 5 in my setup)
  3. Fire some CONCURRENT!!! http calls to your ESB, for example with a bash command
    for n in {1..10}; do curl 'http://localhost:34981/my-url' 2>/dev/null >/dev/null & done
  4. Wait for the workers to pick up the jobs ("Worker reserved a Job" debug messages)
  5. Wait for the job TTR to expire (default $timeout variable in Job is 60 second)
  6. In my setup I got a consistent DeadlineSoonException, but as always with concurrency errors that might only occasionally occur.

Ps. I cherry-picked some commits from my other PR's that were required for the travis-ci checks. The main commit in this PR is of course the first one.

Error: Uncaught Amp\Beanstalk\DeadlineSoonException causing the ESB to crash

Caused by: Wrong Beanstalk job deleted at the end of the job
Caused by: Multiple Beanstalk jobs for the same job UUID. With multiple concurrent workers that same job UUID will be assigned to multiple workers. $uuidToBeanstalkIdMap[$job->getUuid()] will then only contain one of the beanstalk job ids, so one of the worker's queue manager won't be able to delete his job after finish.
Caused by: Concurrent calls to QueueManager::processBatch() were possible (and should be possible), but the batch itself was shared over multiple HTTP requests. Causing the same job being inserted twice (or more).

Resolved by: Processing the batch via a local variable instead of the shared property.
The Combination PHP <7.4 and maglnet/composer-require-checker doesn't work with Composer 2. User Composer version 1 for PHP 7.2 and 7.3.
Newer versions causing errors:

ECS >= 8.3: PHP Fatal error:  Uncaught OutOfBoundsException: Package "symplify/easy-coding-standard" is not installed in phar:///home/youwe/Projects/webgriffe-esb/vendor/phpstan/phpstan/phpstan.phar/vendor/composer/InstalledVersions.php:1081

PhpStan >= 0.12.67:  Ignored error pattern /Unable to resolve the template type .*? in call to function Amp\\call/ was not matched in reported errors. Ignored error pattern /Unable to resolve the template type .*? in instantiation of class Amp\\Producer/ was not matched in reported errors.
PhpStan: Method Webgriffe\Esb\Exception\ElasticSearch\JobNotFoundException::__construct() has parameter $code with no typehint specified.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant