Celerycam in bootloop on SPC composition

[gannebamm]

Expected Behavior

If I start SPC as composition all containers should start properly.

Actual Behavior

The celerycam container will loop indefinitly with:

RuntimeError: Model class django_celery_monitor.models.WorkerState doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.
[2020-04-20 14:24:29,144: DEBUG/MainProcess] heartbeat_tick : for connection 3599596a105f4a04a8fbd9c5c10bc251
[2020-04-20 14:24:30,145: DEBUG/MainProcess] heartbeat_tick : for connection 3599596a105f4a04a8fbd9c5c10bc251
[2020-04-20 14:24:30,385: ERROR/MainProcess] Error in timer: RuntimeError("Model class django_celery_monitor.models.WorkerState doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 169, in apply_entry
    entry()
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/timer.py", line 130, in _reschedules
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/events/snapshot.py", line 72, in capture
    self.state.freeze_while(self.shutter, clear_after=self.clear_after)
  File "/usr/local/lib/python3.7/site-packages/celery/events/state.py", line 464, in freeze_while
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/events/snapshot.py", line 69, in shutter
    self.on_shutter(self.state)
  File "/usr/local/lib/python3.7/site-packages/django_celery_monitor/camera.py", line 125, in on_shutter
    self.handle_worker(worker)
  File "/usr/local/lib/python3.7/site-packages/django_celery_monitor/camera.py", line 74, in handle_worker
    return self.WorkerState.objects.update_heartbeat(
  File "/usr/local/lib/python3.7/site-packages/django_celery_monitor/camera.py", line 46, in WorkerState
    return symbol_by_name('django_celery_monitor.models.WorkerState')
  File "/usr/local/lib/python3.7/site-packages/kombu/utils/imports.py", line 57, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.7/site-packages/django_celery_monitor/models.py", line 21, in <module>
    class WorkerState(models.Model):
  File "/usr/local/lib/python3.7/site-packages/django/db/models/base.py", line 111, in __new__
    "INSTALLED_APPS." % (module, name)

Steps to Reproduce the Problem

1. pull current master
2. compose up --build the SPC composition
3. inspect spcgeonode_celerycam container

Specifications

• GeoNode version: 8ce66dc
• Installation method: SPCGeoNode Docker
• Platform: Ubuntu 18.04

[edsonflavio]

I'm get the same error after update geonode to last version. I change .env with my hostname and put geonode in https. I used the same process to reproduce the problem.

[deviirmr]

I have also the same issue.
So I just stop that container:
sudo docker stop spcgeonode_celerycam_1
and now that bootloop gone.
But I am not sure how it affecting the rest of the application.
I am uploading data and working with layers and not found any issue.

[t-book]

@gannebamm @deviirmr can you check if the app is in installed apps as the error message says: doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS. From what I read at django_celery_monitor issues the project might be abandoned. If so we should consider using some other monitoring for celery

[t-book]

I'm talking about celery monitor ;)

https://github.com/GeoNode/geonode/blob/master/requirements.txt#L62
https://github.com/GeoNode/geonode/blob/master/scripts/spcgeonode/docker-compose.yml#L87

[deviirmr]

@gannebamm @deviirmr can you check if the app is in installed apps as the error message says: doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS. From what I read at django_celery_monitor issues the project might be abandoned. If so we should consider using some other monitoring for celery

This is the latest log, I don't see an error message

[edsonflavio]

My SPCGeonode continous with this error. I rebuild imagens and verifiy my configurations but the error persist.

[gannebamm]

Can confirm. We should see if the celery monitor is needed. Currently I removed the container from the composition and see if there are any errors.

[gannebamm]

django_celery_monitor was as far as I have investigated not a part of the INSTALLED_APPS. Therefore I am puzzled if this is needed at all. @afabiani Do you guys use the monitor for async tasks?

: edit :
https://github.com/jezdez/django-celery-monitor
The project seems not active anymore. If we do not need it, I would propose to get rid of it.

[edsonflavio]

Can confirm. We should see if the celery monitor is needed. Currently I removed the container from the composition and see if there are any errors.

Hello.
I stoped the container and don't have any errors. You can post changes that you make in your docker-compose or how are you running your SPCGeonode? You can send me in my email.

[afabiani]

This was something introduced by Olivier with SPC. Never used it on GeoNode.

[timlinux]

Getting similar issues here:

root@SADC-GMI-old-deleteme:~/spcgeonode# docker-compose ps
              Name                            Command                  State                        Ports                  
---------------------------------------------------------------------------------------------------------------------------
b30d327465c9_spcgeonode_django_1   /spcgeonode/scripts/spcgeo ...   Up (healthy)   8000/tcp                                
dumper_geonode_data_spcgeonode     /docker-entrypoint.sh /usr ...   Up                                                     
dumper_geonode_spcgeonode          /docker-entrypoint.sh /usr ...   Up                                                     
spcgeonode_celery_1                celery worker --app=geonod ...   Restarting                                             
spcgeonode_celerybeat_1            celery beat --app=geonode. ...   Restarting                                             
spcgeonode_celerycam_1             celery events --app=geonod ...   Restarting                                             
spcgeonode_geoserver_1             /docker-entrypoint.sh bin/ ...   Up (healthy)   8080/tcp                                
spcgeonode_letsencrypt_1           ./docker-entrypoint.sh /bi ...   Up                                                     
spcgeonode_nginx_1                 /docker-entrypoint.sh ngin ...   Up             0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
spcgeonode_pgdumper_1              /docker-entrypoint.sh /usr ...   Up                                                     
spcgeonode_postgres_1              docker-entrypoint.sh postgres    Up             0.0.0.0:5432->5432/tcp                  
spcgeonode_rabbitmq_1              docker-entrypoint.sh rabbi ...   Up             25672/tcp, 4369/tcp, 5671/tcp, 5672/tcp 
spcgeonode_rclone_1                /docker-entrypoint.sh /usr ...   Up                                                     
spcgeonode_smtp_1                  /bin/sh -c /opt/install.sh ...   Up 

Following the logs:

celery_1          |
celery_1          | Error:
celery_1          | Unable to load celery application.
celery_1          | The module geonode was not found.
celery_1          |
celery_1          | Error:
celery_1          | Unable to load celery application.
celery_1          | The module geonode was not found.
celery_1          |

It seems they come up initially fine then fail to restart if you bring the stack down and up for any reason.

[timlinux]

On another host I also get this:

celerycam_1       | Seems we're already running? (pid: 1)
celerycam_1       | ERROR: Pidfile (/celeryev.pid) already exists.
celerycam_1       | Seems we're already running? (pid: 1)

Looks like a stale file lock....

[timlinux]

Update: I found the source of my own issues was that I was using just the spcgeonode folder without the surrounding geonode source tree.

[t-book]

+1 for removing it next to the test GeoNode/geonode-selenium#128 as suggested by @mtnorthcott

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.