API - issue reported by the user for API v0 and V1

[magdalenadrafiova]

@briri When trying to get the full list of plans through API V0 or V1, I do not seem to be able to collect the most recent plans, even when processing through all pages as suggested in the documentation. V1 provides the most comprehensive list of plans (330 entries) but the latest plan I seem to be able to read was created back in mid-February, and the number of items in this list has been stable since I first tried the API in February.

[briri]

The logic as to which plans are returned is slightly different between the two APIs. Below is an explanation of each. I'm not sure why your numbers/counts would be static.

The logic behind which plans you see for API V0 can be found here in it's plans_controller
To summarize it:

• Get all plans for the current user's Org (where one of the Org's users is an owner or coowner of the plan)
• it then restricts the results based on criteria you specify (if applicable)
  • specific users (where the user is an owner, coowner or editor for the plan)
  • date range
  • specific templates
  • specific plan ids
  • whether or not tests should be included (they are by default)

The logic behind which plans you see for API V1 is different can be found here in it's plans_policy
To summarize this one:

• it always returns publicly visible plans
• if the requestor is an Api Client (using their client_id and client_secret to authenticate) then the following applies:
  • all plans created by that Api Client
  • all plans associated with the Api Client's Org (assuming the Api Client was assigned an Org)
• if the requestor is a User (using their email and api_token to authenticate) then the following applies:
  • all of the User's Org's organizationally visible plans (where one of the Org's users is an owner or coowner of the plan)
  • all of the plans the User is associated with regardless of their role
  • if the user is an Org Admin then it will include all plans associated with that Org (where one of the Org's users is an owner or coowner of the plan)

Note that we can update the logic for either/both if needed.

Also note that there is some weird behavior with accessing an Org's plans. The code overrides Rails inherent behavior and we should probably address that at some point: #2724

[magdalenadrafiova]

@briri apologies i forgot to change 'i' this was reported by one of the users

[magdalenadrafiova]

thank you for the clarification though - I have shared the link for this ticket with the user, closing this here for now

[FredericBrochu]

Hi,

Thanks for these explanations.

We are using powerBI to send GET requests to the API V0 to get plan data and answers to the questions.
From your answers, a GET to the target URL :
https://dmp.npl.co.uk/api/v0/plans?created_after=2019-11-01
should return all test plans created by users from our organisation after the 1st of November last year.
It does not return anything, except the following error message when using python's requests instead of powerBI:
ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I am using these headers as well:
headers = {"Authorization": f"Token token=MY_TOKEN","Content-Type":"application/json"}
and turn off SSL verification .

[magdalenadrafiova]

@briri we got further comment above - do you have any further thoughts there? thank you!!!

[briri]

Its hard for me to tell @magdalenadrafiova. I do not have access to that server's logs.

@FredericBrochu do you receive a response if you leave the query parameter off? (e.g. https://dmp.npl.co.uk/api/v0/plans)

[FredericBrochu]

Yes, we are getting a normal json dump with 10 plans' data and no error (status code 200 )

[FredericBrochu]

And if we go through the pages using the ?page=X parameter, we get more data, until eventually we get this error again . This way, we only get plans up to November 2020 but not after.

[briri]

Hi @FredericBrochu try escaping the question mark character. I just tried it with curl and had to add a '\' before the '?' to get it to work. I'm on OSX using zsh. I don't recall having to do this in the past (perhaps a change in the recent OSX patch?) but your powerBI may be having a similar issue.

> curl -vL -H "Authorization: Token token=12345" http://localhost:3000/api/v0/plans?created_after=2019-11-01 
zsh: no matches found: http://localhost:3000/api/v0/plans?created_after=2019-11-01

> curl -vL -H "Authorization: Token token=12345" http://localhost:3000/api/v0/plans\?created_after=2019-01-01
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3000 (#0)
> GET /api/v0/plans?created_after=2019-01-01 HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/7.64.1
> Accept: */*
> Authorization: Token token=12345
> 
< HTTP/1.1 200 OK
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin-when-cross-origin
< Content-Type: application/json; charset=utf-8
< X-bullet-footer-text: ["user: briley  USE eager loading detected    Plan =\u003e [:funder]\n  Add to your query: .includes([:funder])","user: briley  AVOID eager loading detected    Plan =\u003e [:phases, :users]\n  Remove from your query: .includes([:phases, :users])"]
< X-bullet-console-text: ["user: briley\nUSE eager loading detected\n  Plan =\u003e [:funder]\n  Add to your query: .includes([:funder])\nCall stack\n  /Users/briley/Documents/workspace/roadmap/app/views/api/v0/plans/index.json.jbuilder:20:in `block (2 levels) in _app_views_api_v__plans_index_json_jbuilder___2453906840107351138_70282306606400'\n  /Users/briley/Documents/workspace/roadmap/app/views/api/v0/plans/index.json.jbuilder:19:in `block in _app_views_api_v__plans_index_json_jbuilder___2453906840107351138_70282306606400'\n  /Users/briley/Documents/workspace/roadmap/app/views/api/v0/plans/index.json.jbuilder:8:in `_app_views_api_v__plans_index_json_jbuilder___2453906840107351138_70282306606400'\n  /Users/briley/Documents/workspace/roadmap/app/controllers/api/v0/plans_controller.rb:99:in `index'\n\n","user: briley\nAVOID eager loading detected\n  Plan =\u003e [:phases, :users]\n  Remove from your query: .includes([:phases, :users])\nCall stack\n\n"]
< Cache-Control: no-store, must-revalidate, private, max-age=0
< Set-Cookie: _dmp_roadmap_session=0mpF9yJsYQH76AIv98IZhsvBCEOsehbzuq1NBkG%2FvpeA76MFIziwUBk%2Buq4Jp6mrsJoeGFU7otnPjcAMOTA%2FGaSbp%2FbJEOAxkWR2ELLlee7DWfQdn1bKbInoZmCUbqKfUYnMVFhqRr%2Ft--zmbleEfXeHuj0r7C--FIPa0OUUjUYdjGWLMY6gYw%3D%3D; path=/; HttpOnly; SameSite=Lax
< Set-Cookie: __profilin=p%3Dt; path=/; HttpOnly
< X-Request-Id: 9bd93a55-d3c8-433e-86c6-58f7e7671153
< X-Runtime: 0.249032
< X-MiniProfiler-Original-Cache-Control: max-age=0, private, must-revalidate
< X-MiniProfiler-Ids: 71tz8b0t1tvl26wmxife,8cylm5yynvjxpzt0r8z7,7jkj92r3tahj7gbvkgq2,c6wdsljv9hqu60ask139,64rrv0osvkkstktmew1n,c3kriz7wg7qaoglfatbg,q14ejuskfdy889tpnont,uj081fh4c3uaegu189j6,6huuhpybth0qrcwky61q,nqrwns5z2odikxw4j7sk,ovx9tiu1uurq0yeyd26m,rjwcd2s0f842so3z3gwu,fzgxxp3dptiyvbepkirg,5yje8dkfqjlsn5soe1z9,vww159vbfuekrxz62v1x,f6bzdxcmgpt47qjebrv,cj8u5rx462mta6gl405v
< Transfer-Encoding: chunked
< 
[
  {
    "id": 38646,
    "title": "Federal Road Charge Tax Administration Process",
    "grant_number": "",
    "last_updated": "2020-04-24 22:27:33 UTC",
    "creation_date": "2019-01-12 08:48:18 UTC",
    "test_plan": false,
    "template": {
      "title": "National Center for Sustainable Transportation - Project Data Management Plan",
      "id": 1950908230
    },

   ...

[FredericBrochu]

The question mark worked fine for requests with the page parameter (...?page=1 for instance) .
I tried anyway:
https://dmp.npl.co.uk/api/v0/plans\?created_after=2019-11-01
I got a cleaner answer: status_code 404, reason="Not found", but we do have a few test plans generated after this date.

[briri]

Hmm. page=n doesn't work for me without escaping the question mark when using curl.

@magdalenadrafiova I think someone on your side will need to check the logs to see if its reporting an error and why it's not returning any results.

[FredericBrochu]

Making some progress, it is true, but closing is premature. We have test plans up to May 2021 (see attached snapshot), yet I can't get anything past November 2020:

Using:
target_url = "https://dmp.npl.co.uk/api/v0/plans?created_after=2021-01-01"
with the same headers as before:

r = requests.get(target_url, headers=headers, verify=False)
C:\Users\fb10\AppData\Local\Continuum\anaconda3\lib\site-packages\urllib3\connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Traceback (most recent call last):

File "", line 1, in
r = requests.get(target_url, headers=headers, verify=False)

File "C:\Users\fb10\AppData\Local\Continuum\anaconda3\lib\site-packages\requests\api.py", line 75, in get
return request('get', url, params=params, **kwargs)

File "C:\Users\fb10\AppData\Local\Continuum\anaconda3\lib\site-packages\requests\api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)

File "C:\Users\fb10\AppData\Local\Continuum\anaconda3\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)

File "C:\Users\fb10\AppData\Local\Continuum\anaconda3\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)

File "C:\Users\fb10\AppData\Local\Continuum\anaconda3\lib\site-packages\requests\adapters.py", line 498, in send
raise ConnectionError(err, request=request)

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[FredericBrochu]

Hi,

I just wanted to check on progress on this issue, and see if there was anything I can do to help, like providing more information?

[magdalenadrafiova]

@FredericBrochu nothing on our end at the moment. We have not start picking up the API issues/bug yet we will update the information once we look into this

[johnpinto1]

@magdalenadrafiova @briri @raycarrick-ed Like for https://github.com/DigitalCurationCentre/DMPonline-Service/issues/505 and https://github.com/DigitalCurationCentre/DMPonline-Service/issues/462. I can replicate issue locally. With the fix in branch bug_462_api_not_possible_to_get_plans which was fix for bug #462 (comment) I can retrieve Plans for 2021 in a query like this
http://uoe.lvh.me:3000/api/v0/plans?created_after=2019-11-01

[briri]

believe this is fixed in v3.0.3. Thanks for sorting this one out @johnpinto1