Migrate populate-dev-data to a Python script

[lukebjerring]

Fixes #241

It's a little song-and-dance to use the AppEngine SDK from a standalone script, and thus we introduce the first python2.7 script in the util dir.

[lukebjerring]

This should eventually be updated to re-use the code from #242

[rwaldron]

Non-blockers!

[rwaldron]

$ ./util/populate_dev_data.py
usage: populate_dev_data.py [-h] [--log LOG] [--sdk-root SDK_ROOT]
                            [--creds CREDS_PATH] --server SERVER_URI
                            [--secure SECURE]
populate_dev_data.py: error: argument --server is required

[lukebjerring]

Not sure what the comment here is getting at - do you think that more helpful message(s) are needed? The --help output should cover confusion?

[rwaldron]

Sorry, I was just illustrating that the "Example usage:" snippet doesn't work as written, but like I said it's not a blocker

[rwaldron]

What do you think about adding this link? https://developers.google.com/identity/protocols/application-default-credentials

[lukebjerring]

Done.

[rwaldron]

What do you think about adding this link? https://cloud.google.com/appengine/downloads

[lukebjerring]

Done

[rwaldron]

👍

[rwaldron]

Sorry, I just spotted another bit here: need a space after "not", or before "already", otherwise is prints out as "notalready"

[rwaldron]

I needed some time to work through a few things on my end, to try and understand the failures I was getting with this change. I'll try to recap...

• With the master branch, I can follow the README.md (ie. curl http://localhost:8080/tasks/populate-dev-data) and successfully populate dev data to view when running the app locally.
• With this branch, I cannot do so, because: The following field(s) must be defined: refresh_token, but I'm not sure where that comes from. I have /Users/rwaldron/.config/gcloud/application_default_credentials.json which contains the relevant client_id and client_secret, as shown in the credentials that we have access to.

Edit:

Making headway here (All of this needs to be documented somewhere more "officially")

• Run gcloud config set project wptdashboard -> chose authorized user -> "Allow"
• Relevant result:

  You are now logged in as [[email protected]].
  Your current project is [wptdashboard].
  

• Run gcloud auth application-default login -> chose authorized user -> "Allow"
• Relevant result:

  Credentials saved to file: [/Users/rwaldron/.config/gcloud/application_default_credentials.json]
  
  These credentials will be used by any library that requests
  Application Default Credentials.
  

• But...

  $ ./util/populate_dev_data.py --server localhost:9999 --sdk-root /Users/rwaldron/Downloads/google-cloud-sdk --creds /Users/rwaldron/.config/gcloud/application_default_credentials.json
  INFO:root:Adding path /Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine
  INFO:oauth2client.client:Attempting refresh to obtain initial access_token
  INFO:oauth2client.client:Refreshing access_token
  Traceback (most recent call last):
    File "./util/populate_dev_data.py", line 175, in <module>
      main(args)
    File "./util/populate_dev_data.py", line 51, in main
      secure=args.secure)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 972, in ConfigureRemoteApiForOAuth
      rpc_server_factory=rpc_server_factory)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 1057, in ConfigureRemoteApi
      app_id = GetRemoteAppIdFromServer(server, path, rtok)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 762, in GetRemoteAppIdFromServer
      response = server.Send(path, payload=None, **urlargs)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc_httplib2.py", line 246, in Send
      url, method=method, body=payload, headers=headers)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/lib/oauth2client_devserver/oauth2client/util.py", line 129, in positional_wrapper
      return wrapped(*args, **kwargs)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/lib/oauth2client_devserver/oauth2client/client.py", line 533, in new_request
      redirections, connection_type)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 1626, in request
      (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 1368, in _request
      (response, content) = self._conn_request(conn, request_uri, method, body, headers)
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 1288, in _conn_request
      conn.connect()
    File "/Users/rwaldron/Downloads/google-cloud-sdk/platform/google_appengine/lib/httplib2/httplib2/__init__.py", line 950, in connect
      raise socket.error, msg
  socket.error: [Errno 61] Connection refused
  

[lukebjerring]

Did you restart your app_devserver.py with --api_port=9999? 'Connection refused' doesn't sound very auth-related.

[lukebjerring]

OK so after some to-and-fro in IM chat, we diagnosed this as docker not forwarding :9999 (fixed in 1c5171b).

Though there are some hoops to jump through around creds, I think that re-using the instructions for the needed parts is better than maintaining step-by-step docs in this repo?

[mdittmer]

LGTM