Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxied Requests #38

Merged
merged 4 commits into from
May 5, 2017
Merged

Proxied Requests #38

merged 4 commits into from
May 5, 2017

Conversation

ingalls
Copy link
Member

@ingalls ingalls commented May 4, 2017

I've been working my way through a ton of potential Michigan sources and seeing a common occurrence of the ArcGIS URL being proxied behind a server which appends the auth token server-side before hitting the actual ArcGIS server.

So a traditional ArcGIS URL Like:

http://example.com/arcgis/rest/services/portAddressPoint/FeatureServer/0

Is proxied through another server like:

https://example.com/proxy.ashx?http://example.com/arcgis/rest/services/portAddressPoint/FeatureServer/0

Since the scraper uses the python requests module to add the param: f=json it correctly detects the previous ? and assumes the f=json are additional params and appends &f=json.

the proxy however expects a double ? as it simply pops the second half url and passes it along. This results in a failed query.

This PR adds an optional --proxy arg to the CLI that allows the scraper to handle double args.

@iandees
Copy link
Member

iandees commented May 4, 2017

I'd like to get the tests fixed and then will merge this. Thanks @ingalls!

@ingalls ingalls self-assigned this May 4, 2017
@iandees iandees merged commit 8a31573 into master May 5, 2017
@iandees iandees deleted the proxyRequest branch May 5, 2017 00:34
@ingalls ingalls mentioned this pull request Mar 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants