How to set up queue_job in odoo sh? #169

elliotxin · 2019-09-09T11:23:37Z

I followed the instruction on the ubuntu server the queue_job working very well, after that, I try to set up the queue_job in odoo sh, unfortunately, its does's work, when i go throw the odoo log, one error message show like this:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8069): Max retries exceeded with url: /queue_job/session (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f25a8441550>: Failed to establish a new connection: [Errno 111] Connection refused',))

Anyone can tell me how to fix this in odoo sh, thanks a lot.

CasVissers-360ERP · 2020-03-25T09:41:27Z

@Cedric-Pigeon @guewen
Any idea on this? Or is Odoo.sh not yet supported?

CasVissers-360ERP · 2020-03-27T10:23:50Z

@elliotxin
I got it running, add this to the Odoo config file:

[queue_job]
channels = root:4
host="https://yourenv.odoo.com"
port=443
http_auth_user="youruser"
http_auth_password="yourpassword"

Good luck!

CasVissers-360ERP · 2020-05-07T11:46:07Z

@Cedric-Pigeon @guewen
Think this issue can be closed. This works.

guewen · 2020-05-07T12:20:53Z

Thanks for the feedback!

acysos · 2020-05-08T10:18:20Z

Hello @CasVissers ,

How do you locate the odoo config file in odoo.sh?

Greetings

CasVissers-360ERP · 2020-05-08T11:16:32Z

@acysos
nano .config/odoo/odoo.conf

acysos · 2020-05-08T11:22:17Z

@CasVissers thank you

NMisko · 2020-07-01T11:18:13Z

@CasVissers Did you have to do anything else except modifying the config file? Does the amount of database workers make a difference? I can't get it working on my odoo.sh (see #244 )

amigrave · 2020-09-16T10:38:28Z

Hi there, I'm Fabien Meghazi, core developer and responsible of the Odoo.sh platform.
I'm sorry to hijack this thread but I'd like to point out that queue_job (in it's current design) is not compatible with Odoo.sh which uses a hybrid model of multiple threaded workers.

queue_job can somewhat work on Odoo.sh when using only one worker but the problem is that for each job it will trigger as many http calls as there are running workers. When an instance is creating a lot of jobs, this translates to a significative drop of performance due to concurrent updates.

We recently listed the projects hosted on Odoo.sh using this module and we are about to contact the problematic ones.

Greets.

pedrobaeza · 2020-09-16T10:49:49Z

Thanks for informing that and I don't consider it as a hijacking, but a good appreciation that closes the discussion.

guewen · 2020-09-16T10:49:58Z

Hi @amigrave,

Thanks for the notice. Do you know any way we could make it work on odoo.sh?

queue_job can somewhat work on Odoo.sh when using only one worker but the problem is that for each job it will trigger as many http calls as there are running workers. When an instance is creating a lot of jobs, this translates to a significative drop of performance due to concurrent updates.

This looks like a misconfiguration issue. The number of parallel jobs (so number of workers busy for jobs at the same time) can be configured using ODOO_QUEUE_JOB_CHANNELS=root:1 or equivalent in the configuration file, the default being 1. Or am I misunderstanding what you mean by "for each job it will trigger as many http calls as there are running workers"?

EDIT: I got what you say.

Is it a peculiarity due to the way the hybrid model works, like, it starts several "jobrunner" workers, because you start several odoo main processes which all fork their own "jobrunner" worker? (if that's the case, it probably can be prevented in queue_job's code).

amigrave · 2020-09-16T13:31:33Z

Is it a peculiarity due to the way the hybrid model works, like, it starts several "jobrunner" workers, because you start several odoo main processes which all fork their own "jobrunner" worker? (if that's the case, it probably can be prevented in queue_job's code).

@guewen Indeed, you nailed it. Unfortunately there's no easy way to make it work. One could think that you just need to let one worker spawn the jobrunner and somehow prevent the other workers to spawn the jobrunner, but this would not work because Odoo.sh recycles workers on some events and there's no "master" process, hence you could end up in a situation where you don't have the jobrunner alive anymore.

In order to make queue_job work on Odoo.sh I guess it would need to support this hybrid mode of multiple threaded workers and setup a communication between the workers so they can decide who is the master (a bit like a derivative of the registry signaling) and in case the master dies another worker becomes the master. I understand that such a thing can't be foreseen in the development of queue_job because:

it seems overly complex
I can't guarantee the current hybrid model won't change in the future
this Odoo.sh hybrid mode is not "standard"

We already discussed internally about having this hybrid mode standardized in Odoo (I love the idea because it would be less code to maintain in Odoo.sh) but nothing has been started yet in that direction so far and some challenged should be tackled first, such as the longpolling routes which are specially handled by Odoo.sh.

Note: This is just a hot reaction, maybe there's an easier way I haven't tough of to make queue_job work on Odoo.sh but if later I think about an easier way I'll post it here.

guewen · 2020-09-16T13:47:32Z

@amigrave Thanks for the very informative and detailed answer!

Actually, the fact that we have to spawn only one jobrunner when we have multiple nodes is kind of annoying outside odoo.sh too, so we could tackle this problem for other uses cases. (At camptocamp, we deploy several nodes on production and the unique jobrunner defeats the improved availability of such setup, regarding jobs).

An idea which doesn't seem that difficult for me, would be that the jobrunner could acquire an advisory lock at loading, and the other nodes would only wait until they can acquire it, the first in the race gets to be the new master. Hot reaction too, I may be overlooking details :)

Anyway, I'll think about this and thanks again for sharing your much helping thoughts.

Starting several odoo (main) processes with "--load=web,queue_job" was unsupported, as it would start several jobrunner, which would all listen to postgresql notifications and try to enqueue jobs in concurrent workers. This is an issue in several cases: * it causes issues on odoo.sh that uses an hybrid model for workers and starts several jobrunners [0] * it defeats any setup that would use several nodes to keep the service available in case of failure of a node/host The solution implemented here is using a PostgreSQL advisory lock, at session level, to lock a database on a specific jobrunner. At loading, the jobrunner tries to acquire the lock. If it can, it initializes the connection and listen for jobs. Every 30 seconds, it tries to acquire again all the databases that it couldn't acquire, in case a concurrent node is stopped. If all the databases have been acquired by a concurrent node, it will do nothing but wait and retry every 30 seconds. Example when a jobrunner is started for databases queue1 and queue2 and another node already works on queue2: INFO ? odoo.addons.queue_job.jobrunner.runner: starting INFO ? odoo.addons.queue_job.jobrunner.runner: initializing database connections INFO ? odoo.addons.queue_job.jobrunner.runner: queue job runner ready for db queue1 DEBUG ? odoo.addons.queue_job.jobrunner.runner: queue job runner already started on another node for db queue2 INFO ? odoo.addons.queue_job.jobrunner.runner: database connections ready for: queue1 Important: new databases need a restart of the jobrunner. This was already the case, and would be a great improvement, but is out of scope for this improvement. [0] OCA#169 (comment)

Support multi-nodes with lock on job runner Starting several odoo (main) processes with "--load=web,queue_job" was unsupported, as it would start several job runner, which would all listen to postgresql notifications and try to enqueue jobs in concurrent workers. This is an issue in several cases: * it causes issues on odoo.sh that uses an hybrid model for workers and starts several job runners [0] * it defeats any setup that would use several nodes to keep the service available in case of failure of a node/host The solution implemented here is using a PostgreSQL advisory lock, at session level in a connection on the "postgres" database, which ensure 2 job runners are not working on the same set of databases. At loading, the job runner tries to acquire the lock. If it can, it initializes the connection and listen for jobs. If the lock is taken by another job runner, it waits and retry to acquire it every 30 seconds. Example when a job runner is started and another one starts: INFO ? odoo.addons.queue_job.jobrunner.runner: starting INFO ? odoo.addons.queue_job.jobrunner.runner: already started on another node The shared lock identifier is computed based on the set of databases the job runner has to listen to: if a job runner is started with ``--database=queue1`` and another with ``--database=queue2``, they will have different locks and such will be able to work in parallel. Important: new databases need a restart of the job runner. This was already the case, and would be a great improvement, but is out of scope for this improvement. [0] OCA#169 (comment)

Starting several odoo (main) processes with "--load=web,queue_job" was unsupported, as it would start several job runner, which would all listen to postgresql notifications and try to enqueue jobs in concurrent workers. This is an issue in several cases: * it causes issues on odoo.sh that uses an hybrid model for workers and starts several job runners [0] * it defeats any setup that would use several nodes to keep the service available in case of failure of a node/host The solution implemented here is using a PostgreSQL advisory lock, at session level in a connection on the "postgres" database, which ensure 2 job runners are not working on the same set of databases. At loading, the job runner tries to acquire the lock. If it can, it initializes the connection and listen for jobs. If the lock is taken by another job runner, it waits and retry to acquire it every 30 seconds. Example when a job runner is started and another one starts: INFO ? odoo.addons.queue_job.jobrunner.runner: starting INFO ? odoo.addons.queue_job.jobrunner.runner: already started on another node The shared lock identifier is computed based on the set of databases the job runner has to listen to: if a job runner is started with ``--database=queue1`` and another with ``--database=queue2``, they will have different locks and such will be able to work in parallel. Important: new databases need a restart of the job runner. This was already the case, and would be a great improvement, but is out of scope for this improvement. [0] OCA#169 (comment)

ArgosVeto · 2021-01-19T11:04:38Z

@CasVissers-360ERP

I tryed your solution and I got this traceback in the log system :

_

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/odoo/src/user/extra-addons/queue_job/jobrunner/runner.py", line 237, in urlopen
response = requests.get(url, timeout=1, auth=auth)
File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='%22https', port=80): Max retries exceeded with url: //name_of_my_app.odoo.com/%22:443/queue_job/runjob?db=name_of_my_database&job_uuid=94b5d0a2-4c5d-4e26-baf5-7b3524f03c73 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8c3f801400>: Failed to establish a new connection: [Errno -2] Name or service not known'))

_

This is my odoo config parameter 👍

[queue_job]
channels = root:4
host = "https://name_of_my_app.odoo.com"
port = 443
http_auth_user = "admin"
http_auth_password = "admin_password"

Do you have any idea ?

guewen · 2021-01-19T12:22:57Z

@ArgosVeto you should have a look at #256, and give feedback (that I need as I'm not using odoo.sh).

Starting several odoo (main) processes with "--load=web,queue_job" was unsupported, as it would start several job runner, which would all listen to postgresql notifications and try to enqueue jobs in concurrent workers. This is an issue in several cases: * it causes issues on odoo.sh that uses an hybrid model for workers and starts several job runners [0] * it defeats any setup that would use several nodes to keep the service available in case of failure of a node/host The solution implemented here is using a PostgreSQL advisory lock, at session level in a connection on the "postgres" database, which ensure 2 job runners are not working on the same set of databases. At loading, the job runner tries to acquire the lock. If it can, it initializes the connection and listen for jobs. If the lock is taken by another job runner, it waits and retry to acquire it every 30 seconds. Example when a job runner is started and another one starts: INFO ? odoo.addons.queue_job.jobrunner.runner: starting INFO ? odoo.addons.queue_job.jobrunner.runner: already started on another node The shared lock identifier is computed based on the set of databases the job runner has to listen to: if a job runner is started with ``--database=queue1`` and another with ``--database=queue2``, they will have different locks and such will be able to work in parallel. Important: new databases need a restart of the job runner. This was already the case, and would be a great improvement, but is out of scope for this improvement. [0] OCA#169 (comment)

ovnicraft · 2021-04-07T19:23:49Z

I share my tests about this:

dbfilter=
server_wide_modules=web,queue_job

[queue_job]
scheme=https
port=443
host=mydb-2350951.dev.odoo.com

And service apparently start:

2021-04-07 19:10:47,634 769 INFO ? odoo.addons.queue_job.jobrunner.runner: starting
2021-04-07 19:10:47,634 769 INFO ? odoo.addons.queue_job.jobrunner.runner: initializing database connections
2021-04-07 19:10:47,646 769 INFO ? odoo.addons.queue_job.jobrunner.runner: queue job runner ready for db mydb-2350951
2021-04-07 19:10:47,646 769 INFO ? odoo.addons.queue_job.jobrunner.runner: database connections ready

BTW jobs dont change to enqueued state, so I updated via SQL to enqueued state, and then run a get request to /queue_job/runjob? from python console and jobs works.

Starting several odoo (main) processes with "--load=web,queue_job" was unsupported, as it would start several job runner, which would all listen to postgresql notifications and try to enqueue jobs in concurrent workers. This is an issue in several cases: * it causes issues on odoo.sh that uses an hybrid model for workers and starts several job runners [0] * it defeats any setup that would use several nodes to keep the service available in case of failure of a node/host The solution implemented here is using a PostgreSQL advisory lock, at session level in a connection on the "postgres" database, which ensure 2 job runners are not working on the same set of databases. At loading, the job runner tries to acquire the lock. If it can, it initializes the connection and listen for jobs. If the lock is taken by another job runner, it waits and retry to acquire it every 30 seconds. Example when a job runner is started and another one starts: INFO ? odoo.addons.queue_job.jobrunner.runner: starting INFO ? odoo.addons.queue_job.jobrunner.runner: already started on another node The shared lock identifier is computed based on the set of databases the job runner has to listen to: if a job runner is started with ``--database=queue1`` and another with ``--database=queue2``, they will have different locks and such will be able to work in parallel. Important: new databases need a restart of the job runner. This was already the case, and would be a great improvement, but is out of scope for this improvement. [0] OCA#169 (comment)

guewen closed this as completed May 7, 2020

guewen mentioned this issue Jun 29, 2020

Jobs stuck in pending in odoo.sh. #244

Closed

guewen mentioned this issue Sep 30, 2020

[13.0] Support multi-nodes with lock on jobrunner #256

Closed

guewen mentioned this issue Dec 15, 2020

[13.0] base_import_async failing with "Wrong value for ir.attachment.type: 'product'" when importing product.template #295

Closed

StefanRijnhart mentioned this issue Apr 12, 2022

[15.0][ADD] queue_job_cron_jobrunner #415

Merged

PCatinean mentioned this issue Sep 9, 2023

New module odoosh_queue_job for running vanilla queue_job on odoo.sh #562

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to set up queue_job in odoo sh? #169

How to set up queue_job in odoo sh? #169

elliotxin commented Sep 9, 2019

CasVissers-360ERP commented Mar 25, 2020

CasVissers-360ERP commented Mar 27, 2020 •

edited

Loading

CasVissers-360ERP commented May 7, 2020

guewen commented May 7, 2020

acysos commented May 8, 2020

CasVissers-360ERP commented May 8, 2020

acysos commented May 8, 2020

NMisko commented Jul 1, 2020

amigrave commented Sep 16, 2020

pedrobaeza commented Sep 16, 2020

guewen commented Sep 16, 2020 •

edited

Loading

amigrave commented Sep 16, 2020 •

edited

Loading

guewen commented Sep 16, 2020

ArgosVeto commented Jan 19, 2021 •

edited

Loading

guewen commented Jan 19, 2021

ovnicraft commented Apr 7, 2021

How to set up queue_job in odoo sh? #169

How to set up queue_job in odoo sh? #169

Comments

elliotxin commented Sep 9, 2019

CasVissers-360ERP commented Mar 25, 2020

CasVissers-360ERP commented Mar 27, 2020 • edited Loading

CasVissers-360ERP commented May 7, 2020

guewen commented May 7, 2020

acysos commented May 8, 2020

CasVissers-360ERP commented May 8, 2020

acysos commented May 8, 2020

NMisko commented Jul 1, 2020

amigrave commented Sep 16, 2020

pedrobaeza commented Sep 16, 2020

guewen commented Sep 16, 2020 • edited Loading

amigrave commented Sep 16, 2020 • edited Loading

guewen commented Sep 16, 2020

ArgosVeto commented Jan 19, 2021 • edited Loading

guewen commented Jan 19, 2021

ovnicraft commented Apr 7, 2021

CasVissers-360ERP commented Mar 27, 2020 •

edited

Loading

guewen commented Sep 16, 2020 •

edited

Loading

amigrave commented Sep 16, 2020 •

edited

Loading

ArgosVeto commented Jan 19, 2021 •

edited

Loading