Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add grpc transcoding + tests #146

Closed
wants to merge 2 commits into from
Closed

Conversation

yon-mg
Copy link
Contributor

@yon-mg yon-mg commented Feb 9, 2021

To provide REST/JSON transport for the python-gapic-generator, it is necessary to implement the gRPC transcoding rules outlined here,
https://github.com/googleapis/googleapis/blob/master/google/api/http.proto#L44-L312.

gRPC transcoding essentially outlines how to convert a gRPC request to an HTTP one. In this PR is only an implementation the pattern matching aspects of gRPC transcoding. It does not have any logic to process requests as proto messages, perform JSON encoding or make the request itself. That is left the generated clients.

@yon-mg yon-mg requested a review from a team as a code owner February 9, 2021 02:03
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Feb 9, 2021
Comment on lines +194 to +197
[[['get','/v1/{name}','']], {'foo':'bar'}],
[[['get','/v1/{name}','']], {'name':'first/last'}],
[[['get','/v1/{name=mr/*/*}','']], {'name':'first/last'}],
[[['post','/v1/{name}','data']], {'name':'first/last'}],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add comments explaining these test cases (like the ones above for test_transcode)

@busunkim96
Copy link
Contributor

It looks like lint is failing. You can run flake8 locally by doing nox -s lint.

nox > flake8 google tests
google/api_core/path_template.py:199:1: E302 expected 2 blank lines, found 1
google/api_core/path_template.py:227:27: E231 missing whitespace after ':'
google/api_core/path_template.py:228:23: E231 missing whitespace after ':'
google/api_core/path_template.py:228:31: E231 missing whitespace after ','
tests/unit/test_path_template.py:121:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:121:35: E231 missing whitespace after ','
tests/unit/test_path_template.py:122:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:122:15: E231 missing whitespace after ':'
tests/unit/test_path_template.py:123:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:123:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:123:33: E231 missing whitespace after ','
tests/unit/test_path_template.py:123:36: E231 missing whitespace after ','
tests/unit/test_path_template.py:123:43: E231 missing whitespace after ':'
tests/unit/test_path_template.py:126:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:126:31: E231 missing whitespace after ','
tests/unit/test_path_template.py:127:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:127:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:128:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:128:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:128:28: E231 missing whitespace after ','
tests/unit/test_path_template.py:128:31: E231 missing whitespace after ','
tests/unit/test_path_template.py:130:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:130:35: E231 missing whitespace after ','
tests/unit/test_path_template.py:131:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:131:21: E231 missing whitespace after ':'
tests/unit/test_path_template.py:131:37: E231 missing whitespace after ':'
tests/unit/test_path_template.py:132:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:132:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:132:28: E231 missing whitespace after ','
tests/unit/test_path_template.py:132:31: E231 missing whitespace after ','
tests/unit/test_path_template.py:132:38: E231 missing whitespace after ':'
tests/unit/test_path_template.py:135:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:135:33: E231 missing whitespace after ','
tests/unit/test_path_template.py:136:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:136:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:137:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:137:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:137:28: E231 missing whitespace after ','
tests/unit/test_path_template.py:137:31: E231 missing whitespace after ','
tests/unit/test_path_template.py:139:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:139:39: E231 missing whitespace after ','
tests/unit/test_path_template.py:140:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:140:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:140:36: E231 missing whitespace after ','
tests/unit/test_path_template.py:140:42: E231 missing whitespace after ':'
tests/unit/test_path_template.py:141:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:141:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:141:38: E231 missing whitespace after ','
tests/unit/test_path_template.py:141:41: E231 missing whitespace after ','
tests/unit/test_path_template.py:141:48: E231 missing whitespace after ':'
tests/unit/test_path_template.py:144:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:144:34: E231 missing whitespace after ','
tests/unit/test_path_template.py:145:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:145:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:146:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:146:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:146:31: E231 missing whitespace after ','
tests/unit/test_path_template.py:146:34: E231 missing whitespace after ','
tests/unit/test_path_template.py:148:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:148:41: E231 missing whitespace after ','
tests/unit/test_path_template.py:149:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:149:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:149:49: E231 missing whitespace after ':'
tests/unit/test_path_template.py:150:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:150:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:150:44: E231 missing whitespace after ','
tests/unit/test_path_template.py:150:47: E231 missing whitespace after ','
tests/unit/test_path_template.py:150:54: E231 missing whitespace after ':'
tests/unit/test_path_template.py:153:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:153:40: E231 missing whitespace after ','
tests/unit/test_path_template.py:154:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:154:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:155:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:155:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:155:41: E231 missing whitespace after ','
tests/unit/test_path_template.py:155:44: E231 missing whitespace after ','
tests/unit/test_path_template.py:157:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:157:50: E231 missing whitespace after ','
tests/unit/test_path_template.py:158:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:158:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:158:47: E231 missing whitespace after ':'
tests/unit/test_path_template.py:158:62: E231 missing whitespace after ':'
tests/unit/test_path_template.py:159:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:159:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:159:50: E231 missing whitespace after ','
tests/unit/test_path_template.py:159:53: E231 missing whitespace after ','
tests/unit/test_path_template.py:159:60: E231 missing whitespace after ':'
tests/unit/test_path_template.py:162:18: E231 missing whitespace after ','
tests/unit/test_path_template.py:162:36: E231 missing whitespace after ','
tests/unit/test_path_template.py:163:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:163:16: E231 missing whitespace after ':'
tests/unit/test_path_template.py:163:22: E231 missing whitespace after ':'
tests/unit/test_path_template.py:163:32: E231 missing whitespace after ':'
tests/unit/test_path_template.py:163:45: E231 missing whitespace after ','
tests/unit/test_path_template.py:163:51: E231 missing whitespace after ':'
tests/unit/test_path_template.py:164:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:164:16: E231 missing whitespace after ','
tests/unit/test_path_template.py:164:34: E231 missing whitespace after ','
tests/unit/test_path_template.py:164:40: E231 missing whitespace after ':'
tests/unit/test_path_template.py:164:50: E231 missing whitespace after ':'
tests/unit/test_path_template.py:164:63: E231 missing whitespace after ','
tests/unit/test_path_template.py:164:70: E231 missing whitespace after ':'
tests/unit/test_path_template.py:166:18: E231 missing whitespace after ','
tests/unit/test_path_template.py:166:48: E231 missing whitespace after ','
tests/unit/test_path_template.py:167:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:167:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:167:28: E231 missing whitespace after ','
tests/unit/test_path_template.py:167:35: E231 missing whitespace after ':'
tests/unit/test_path_template.py:167:48: E231 missing whitespace after ','
tests/unit/test_path_template.py:167:55: E231 missing whitespace after ':'
tests/unit/test_path_template.py:167:61: E231 missing whitespace after ':'
tests/unit/test_path_template.py:167:71: E231 missing whitespace after ':'
tests/unit/test_path_template.py:167:84: E231 missing whitespace after ','
tests/unit/test_path_template.py:167:90: E231 missing whitespace after ':'
tests/unit/test_path_template.py:168:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:168:16: E231 missing whitespace after ','
tests/unit/test_path_template.py:168:44: E231 missing whitespace after ','
tests/unit/test_path_template.py:168:50: E231 missing whitespace after ':'
tests/unit/test_path_template.py:168:60: E231 missing whitespace after ':'
tests/unit/test_path_template.py:168:73: E231 missing whitespace after ','
tests/unit/test_path_template.py:168:80: E231 missing whitespace after ':'
tests/unit/test_path_template.py:171:18: E231 missing whitespace after ','
tests/unit/test_path_template.py:171:48: E231 missing whitespace after ','
tests/unit/test_path_template.py:172:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:172:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:172:28: E231 missing whitespace after ','
tests/unit/test_path_template.py:172:35: E231 missing whitespace after ':'
tests/unit/test_path_template.py:172:48: E231 missing whitespace after ','
tests/unit/test_path_template.py:172:55: E231 missing whitespace after ':'
tests/unit/test_path_template.py:172:61: E231 missing whitespace after ':'
tests/unit/test_path_template.py:172:71: E231 missing whitespace after ':'
tests/unit/test_path_template.py:172:84: E231 missing whitespace after ','
tests/unit/test_path_template.py:172:90: E231 missing whitespace after ':'
tests/unit/test_path_template.py:173:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:173:16: E231 missing whitespace after ','
tests/unit/test_path_template.py:173:44: E231 missing whitespace after ','
tests/unit/test_path_template.py:173:52: E231 missing whitespace after ':'
tests/unit/test_path_template.py:173:58: E231 missing whitespace after ':'
tests/unit/test_path_template.py:173:68: E231 missing whitespace after ':'
tests/unit/test_path_template.py:173:81: E231 missing whitespace after ','
tests/unit/test_path_template.py:173:87: E231 missing whitespace after ':'
tests/unit/test_path_template.py:173:94: E231 missing whitespace after ','
tests/unit/test_path_template.py:176:18: E231 missing whitespace after ','
tests/unit/test_path_template.py:176:48: E231 missing whitespace after ','
tests/unit/test_path_template.py:176:71: E231 missing whitespace after ','
tests/unit/test_path_template.py:176:101: E231 missing whitespace after ','
tests/unit/test_path_template.py:177:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:177:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:177:28: E231 missing whitespace after ','
tests/unit/test_path_template.py:177:35: E231 missing whitespace after ':'
tests/unit/test_path_template.py:177:48: E231 missing whitespace after ','
tests/unit/test_path_template.py:177:55: E231 missing whitespace after ':'
tests/unit/test_path_template.py:177:61: E231 missing whitespace after ':'
tests/unit/test_path_template.py:177:71: E231 missing whitespace after ':'
tests/unit/test_path_template.py:177:84: E231 missing whitespace after ','
tests/unit/test_path_template.py:177:90: E231 missing whitespace after ':'
tests/unit/test_path_template.py:178:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:178:16: E231 missing whitespace after ','
tests/unit/test_path_template.py:178:44: E231 missing whitespace after ','
tests/unit/test_path_template.py:178:52: E231 missing whitespace after ':'
tests/unit/test_path_template.py:178:58: E231 missing whitespace after ':'
tests/unit/test_path_template.py:178:68: E231 missing whitespace after ':'
tests/unit/test_path_template.py:178:81: E231 missing whitespace after ','
tests/unit/test_path_template.py:178:87: E231 missing whitespace after ':'
tests/unit/test_path_template.py:178:94: E231 missing whitespace after ','
tests/unit/test_path_template.py:180:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:180:47: E231 missing whitespace after ','
tests/unit/test_path_template.py:180:51: E231 missing whitespace after ','
tests/unit/test_path_template.py:180:58: E231 missing whitespace after ','
tests/unit/test_path_template.py:180:89: E231 missing whitespace after ','
tests/unit/test_path_template.py:181:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:181:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:181:28: E231 missing whitespace after ','
tests/unit/test_path_template.py:181:34: E231 missing whitespace after ':'
tests/unit/test_path_template.py:182:9: E128 continuation line under-indented for visual indent
tests/unit/test_path_template.py:182:15: E231 missing whitespace after ','
tests/unit/test_path_template.py:182:43: E231 missing whitespace after ','
tests/unit/test_path_template.py:182:46: E231 missing whitespace after ','
tests/unit/test_path_template.py:182:53: E231 missing whitespace after ':'
tests/unit/test_path_template.py:189:1: W293 blank line contains whitespace
tests/unit/test_path_template.py:194:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:194:30: E231 missing whitespace after ','
tests/unit/test_path_template.py:194:43: E231 missing whitespace after ':'
tests/unit/test_path_template.py:195:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:195:30: E231 missing whitespace after ','
tests/unit/test_path_template.py:195:44: E231 missing whitespace after ':'
tests/unit/test_path_template.py:196:17: E231 missing whitespace after ','
tests/unit/test_path_template.py:196:37: E231 missing whitespace after ','
tests/unit/test_path_template.py:196:51: E231 missing whitespace after ':'
tests/unit/test_path_template.py:197:18: E231 missing whitespace after ','
tests/unit/test_path_template.py:197:31: E231 missing whitespace after ','
tests/unit/test_path_template.py:197:49: E231 missing whitespace after ':'
tests/unit/test_path_template.py:209:32: E231 missing whitespace after ':'
tests/unit/test_path_template.py:209:51: E231 missing whitespace after ':'
tests/unit/test_path_template.py:215:17: E231 missing whitespace after ':'
tests/unit/test_path_template.py:216:14: E231 missing whitespace after ':'
tests/unit/test_path_template.py:217:23: E231 missing whitespace after ':'
nox > Command flake8 google tests failed with exit code 1
nox > Session lint failed.

Copy link
Contributor

@vchudnov-g vchudnov-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments on test cases. If any need implementation changes, it's fine to add TODOs so we remember to come back in a later PR.

{'field':'parent/p1'},
['get','/v1/parent/p1',{},{}]],

[[['get','/v1/{field=a/**/b/**}','']],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi: ** can only happen at the end of the URL (only the verb can come after)
https://cloud.google.com/endpoints/docs/grpc-service-config/reference/rpc/google.api#grpc-transcoding
However, I don't think this affects you; this should be filtered upstream when the annotations are checked for validity.

['get','/v1/parent',{},{}]],

[[['get','/v1/{field.sub}','']],
{'field.sub':'parent', 'foo':'bar'},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if you have this?

[[['get','/v1/{field.sub}','']],
'field': {'name': 'alice', 'sub':'parent'}

We should get

['get','/v1/parent',{},{'field.name':'alice'}]]

[[['post','/v1/{field=a/*}/b/{name=**}','data']],
{'field':'a/parent','name':'first/last','data':{'id':1, 'info':'some info'},'foo':'bar'},
['post','/v1/a/parent/b/first/last',{'id':1, 'info':'some info'},{'foo':'bar'}]],

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another test case: where a URI field is a member of body:

        [[['post','/v1/{foo.name}/b','foo']],
        {'foo': {'name':'bob', 'state':'wa'}},
        ['post','/v1/bob/b',{'name':'bob', 'state':'wa'},{}]],

['post','/v1/a/parent/b/first/last',{'data':{'id':1, 'info':'some info'},'foo':'bar'},{}]],

# Additional bindings
[[['post','/v1/{field=a/*}/b/{name=**}','extra_data'], ['post','/v1/{field=a/*}/b/{name=**}','*']],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the second pattern, I suggest changing /b/ to /c/ to make it super clear where the answer is coming from

@vchudnov-g
Copy link
Contributor

BTW, once this PR goes in, we'll need to refactor some of the functionality introduced in googleapis/gapic-generator-python#702 to do the transcoding here, correct?

@parthea
Copy link
Collaborator

parthea commented Aug 14, 2021

Hi @yon-mg , I'm going to close this PR due to inactivity but please feel free to re-open it.

@parthea parthea closed this Aug 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants