Job not logged in failed_jobs if timeout occurs within database transaction

[graemlourens]

• Horizon Version: v5.10.2
• Laravel Version: v9.35.1
• PHP Version: 8.1.3
• Redis Driver & Version: phpredis 5.3.7

Description:

We had a 'phantom' problem that jobs that failed, were not logged to the failed_jobs table. After some investigation we found out that by mistake we were doing a heavy process within a db transaction which lead the specific job to timeout.

After testing we found out that the issue only happens if the jobs timeout during an open db transaction

Steps To Reproduce:

• Create a job with a timeout of 5 seconds
• In the job handle method sleep within a db transaction

    public function handle(): void
    {
        DB::transaction(function (): void {
            sleep(10);
        });
    }

You will see that in the horizon interface the job is properly listed as failed, but you will NOT see the the job in the failed table.

We're wondering that maybe this is related to https://bugs.php.net/bug.php?id=76742 ?

[driesvints]

We don't know ourselves sorry.. this seems like a real edge case :/

I don't think the PHP bug is related since it's already been fixed and you're using a newer version.

[driesvints]

I'm sorry but since we only gotten one report for this I'm going to close this one. If you ever figure out a solution we'd be happy to receive a PR.

[graemlourens]

@driesvints I'd have appreciated it if you left it open, like that also others would see that this is a rather nasty bug and i'd appreciate the help to find the root of the problem. I hope you reconsider.

[driesvints]

I'm sorry but because only one report was provided we won't re-open this.

[graemlourens]

@crynobone and @NiroDeveloper as you were instrumental in solving the issue laravel/framework#48392 i wonder if you are able to solve this problem here that i described?

I still find this highly painful to know that jobs could be totally lost without a trace in case of timeout during transaction and would hugely welcome any help in finding a solution

[crynobone]

Fixed is already available in latest Laravel 10

[graemlourens]

@crynobone could you refer to the issue/MR that solved this? We're glad to test this out again.

[crynobone]

laravel/framework#48961

[graemlourens]

@crynobone We tested it with recent versions and it still does not work. (The Issue you fixed was related to batches, this has nothing to do with batches, just pure jobs)

• Horizon Version: v5.21.4
• Laravel Version: v10.37.3
• PHP Version: 8.2.7
• Redis Driver & Version: phpredis 6.0.2

Not only is no job logged (still reproduceable the same was as in the initial ticket report) but now the 'JobFailed' event is also not dispatched (which we were relying on to do double-checking if the job was logged and if not, raise an alert internally for us to investigate)

As this is easily reproduceable, we'd appreciate it immensely if you would
a) be able to reproduce it yourself to confirm
b) re-open this ticket (or move it to laravel as it could also be nothing related to horizon
c) maybe find a solution to fix this bug

Thank you very much & Kind regards, Graem

[NiroDeveloper]

@graemlourens I was able to verify the fix with the batches.

I think the failed jobs table is a feature of the laravel framework core.
If i am correct, then you could create an issue there, then it might get more attention.

[graemlourens]

@NiroDeveloper have done so, good idea