Error when executing jobs: Cannot decode JWT token: Not enough segments

[pdelboca]

CKAN version:

master

How to reproduce:

• Create a dataset with a csv resource
• On the command line run: ckan xloader submit <dataset-name>

Errors

CKAN will fail with:

2022-04-28 14:42:38,780 ERROR [ckan.lib.api_token] Cannot decode JWT token: Not enough segments

xloader will fail with (even when it says Not Found, in the backend is a Not Authorized):

HTTP error: 404 Client Error: NOT FOUND for url: http://127.0.0.1:5000/dataset/6e7f2cbd-8c2b-4dbf-9fd9-caee6f7f8321/resource/75f1d79c-7254-42e9-a53e-02b7ff209a7b/download/mini-csv.csv
2022-04-28 14:29:52,237 DEBUG [b5f52e28-893d-47e0-bcef-d1622d92b749] HTTP error: 404 Client Error: NOT FOUND for url: http://127.0.0.1:5000/dataset/6e7f2cbd-8c2b-4dbf-9fd9-caee6f7f8321/resource/75f1d79c-7254-42e9-a53e-02b7ff209a7b/download/mini-csv.csv
2022-04-28 14:29:52,248 ERROR [ckanext.xloader.jobs] xloader error: b'Xloader received a bad HTTP response when trying to download the data file status=404 url=http://127.0.0.1:5000/dataset/6e7f2cbd-8c2b-4dbf-9fd9-caee6f7f8321/resource/75f1d79c-7254-42e9-a53e-02b7ff209a7b/download/mini-csv.csv response=404 Client Error: NOT FOUND for url: http://127.0.0.1:5000/dataset/6e7f2cbd-8c2b-4dbf-9fd9-caee6f7f8321/resource/75f1d79c-7254-42e9-a53e-02b7ff209a7b/download/mini-csv.csv', Traceback (most recent call last):
  File "/home/pdelboca/Repos/ckanext-xloader/ckanext/xloader/jobs.py", line 261, in _download_resource_data
    response = get_response(url, headers)
  File "/home/pdelboca/Repos/ckanext-xloader/ckanext/xloader/jobs.py", line 349, in get_response
    response.raise_for_status()
  File "/home/pdelboca/Repos/ckan/.venv/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: NOT FOUND for url: http://127.0.0.1:5000/dataset/6e7f2cbd-8c2b-4dbf-9fd9-caee6f7f8321/resource/75f1d79c-7254-42e9-a53e-02b7ff209a7b/download/mini-csv.csv

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pdelboca/Repos/ckanext-xloader/ckanext/xloader/jobs.py", line 72, in xloader_data_into_datastore
    xloader_data_into_datastore_(input, job_dict)
  File "/home/pdelboca/Repos/ckanext-xloader/ckanext/xloader/jobs.py", line 151, in xloader_data_into_datastore_
    tmp_file, file_hash = _download_resource_data(resource, data, api_key,
  File "/home/pdelboca/Repos/ckanext-xloader/ckanext/xloader/jobs.py", line 261, in _download_resource_data
    response = get_response(url, headers)
ckanext.xloader.job_exceptions.HTTPError: b'Xloader received a bad HTTP response when trying to download the data file status=404 url=http://127.0.0.1:5000/dataset/6e7f2cbd-8c2b-4dbf-9fd9-caee6f7f8321/resource/75f1d79c-7254-42e9-a53e-02b7ff209a7b/download/mini-csv.csv response=404 Client Error: NOT FOUND for url: http://127.0.0.1:5000/dataset/6e7f2cbd-8c2b-4dbf-9fd9-caee6f7f8321/resource/75f1d79c-7254-42e9-a53e-02b7ff209a7b/download/mini-csv.csv'

More Info:

This error is related to the fact that ckanext-xloader tries to use the site_user API Key to download the resource when it has been removed from CKAN core.

[pdekraker-epa]

I encountered this and fixed it by adding ckanext.xloader.api_token to ckan.ini

First, there should be some check on start that a proper token was provided

Second, my my case (running with the current CKAN master) xloader didn't error out, it downloaded the login page HTML and xloadered it.

[aminumoha]

@pdekraker-epa I too ran into this error. However, setting ckanext.xloader.api_token gives me a python EOL error, I guess because of a long api_token text that can't be read at once ? Did you get this error ?? and the solution??

[ThrawnCA]

This isn't the only extension that will need such a token; archiver, qa, harvest, validation, etc, will all need something similar. Should there be a more generic key name, something like ckan.jobs.api_token, to be shared between them?

[yaythecloud]

@pdekraker-epa I too ran into this error. However, setting ckanext.xloader.api_token gives me a python EOL error, I guess because of a long api_token text that can't be read at once ? Did you get this error ?? and the solution??

I am currently facing similar issues did you ever get this resolved?

[ThrawnCA]

I am currently facing similar issues did you ever get this resolved?

We're running this without problems, but instead of putting the token directly in the file, we're injecting it with ckanext-ssm-config, so you might be seeing a file parsing problem?

Are there any special characters in your token?