Support AE dev appserver

[aozarov]

We need to understand what is the desired expectation when using gcloud-java on AE dev appserver.

The options are:

1. use local services
2. use production services

An issue with option (1) is that currently local dev app services such as Datastore are not accesible via the cloud interface (apiary or gRpc).

An issue with option (2) is that the current implementation detects that we are running on AE and automatically chooses the projectid and authentication mode.
This is nice and convenient in production but is an issue with dev appserver as the AppEngineAuthCredentials is based on AppIdentity which
will not work against production when used by dev appserver.

[aozarov]

/cc @ludoch @pcostell

[aozarov]

Note that it is possible to use gcloud-java with dev appserver by explicitly specifying projectId (if different from the local appid) and using ApplicationDefaultAuthCredentials but I think the goal should be to make the same code work in both modes (prod or dev).

Also, choosing this option seems to omit a warning.

[ludoch]

When you say AE dev app server, do you refer to the Python Devappserver that can also support Java Web apps for Cloud SDK integration or the current Java GAE SDK classic?

[aozarov]

I think we have the same issue for both types of dev appserver and I think we should provide the same solution for both cases.

[aozarov]

Though if we can only deal with one than I would say the non-classic is more important.

[johnshunfan]

/cc @lesv @tmatsuo @saturnism

[lesv]

Why won't DefaultAuthCredentials work if gcloud init has been run? That has both user credentials and a project ID.

[saturnism]

+1 to DefaultAuthCredentials. however iirc, appengine credential is a service account where you can add additional scopes, but gcloud auth login could've been a 3LO oauth2 where the scopes were predetermined during the authorization request. there is no way to add another scope except to re-auth...

[aozarov]

What do you think about the option of routing the calls to local services (fakes/emulators) when we detect that we run in a local dev appserver environment?

[saturnism]

i feel that's a good default behavior, what if they do want to run it against a remote server?

[aozarov]

I think they should be able to do that by explicitly specifying the production host of the service,
its project id and using ApplicationDefaultAuthCredentials (which works today).

[lesv]

There was supposed to be a conversation about Cloud Fakes / Emulators, not sure of the status of that. I don't think this blocks GA.

[garrettjonesgoogle]

This issue is a nebulous design issue that I don't think should be tracked in github any more, so I'm going to close it out.

[GabiAxel]

Where else can this be tracked?

[garrettjonesgoogle]

@GabiAxel is there a specific scenario you are interested in?

[GabiAxel]

I am working on an GAE app with one standard service with Objectify and one flexible service with the Datastore Java library with some of the same Datastore entities, and as far as I understand this is pretty much impossible to do locally.

[garrettjonesgoogle]

@lesv do you have any thoughts on @GabiAxel 's situation?

[lesv]

Yes - different versions of the emulator.

@GabiAxel Do you need to run them both locally at the same time? It might be possible to share the local_db.bin file.

[GabiAxel]

@lesv yes - both services access the same entities in some flows.

[lesv]

@GabiAxel I think what I'm asking is, could you run your tests serially for example, GAE Std app first using the GAE Datastore emulator, then run your flex app using the Datastore emulator.

The other option is to use a real datastore instance (probably on a test project) and for your Flex app, just write to it directly. On your J8 Std app, use the remote api. You'll need to recognize that you are running locally and setup for it. Using the java system property com.google.appengine.runtime.environment

[GabiAxel]

@lesv specifically for the tests, if both GAE devserver and the Datastore emulator use the same filesystem storage format, I suppose it could work, but for properly setting up a local environment in which I could run the application, it's not practical to constantly stop and start the devserver/emulator when switching between the cases where I need each of them.

I am familiar with the remote API and it can be used for this case, but it misses the point of a local development environment (I mean the GAE devserver and Datastore emulator exist for the exact reason - being able to run everything locally and not need to use the real Datastore).

I do appreciate the suggestions, but I think a proper solution with a good developer experience should be either a unification of the GAE devserver and Datastore emulator (not necessarily merge them into one product, but something like adding the Datastore REST API to the GAE devserver) or having them both work with the same files simultaneously.

[lesv]

Yep - that's being worked on, I was trying to get you up and running now. I'm meeting with the team tomorrow to discuss.

[GabiAxel]

@lesv Thanks, much appreciated.

[lesv]

@GabiAxel Can you write me @google.com

[konrad-garus]

It is a pretty big issue, since it prevents use of many libraries that use the "cloud" APIs. One example is Objectify v6 for the Datastore.

I hate having to choose between having an efficient and rich dev environment, and being able to use some nice libraries.