Add proxy support

[T0jan]

A small change concerning parameter handling. Consists of two parts:

• The ECCN information from Digikey will be processed as parameter like it is for Element14
• load_category_parameters is restructured to support parameter mappings for categories further down the tree as well as multiple mappings in a tree.

also contains the commits of #146 because I am too stupid for correct branching 😅

[eeintech]

Hello @T0jan

Do you mind pulling main branch into yours now that I've just released 1.0.0?

What about if we closed #146 and add the description of proxy changes here instead? That way I'll check out this PR only 😃

[T0jan]

@eeintech congrats on the release :D

I just synced my branch so no problems on this side.

The only reason I splitted this into a different merge request is, if the change to the parameter mapping conflicts with anything you meant to do with it or if you don't like my solution there, then we could include the proxy stuff independently. So do as you please :D

[eeintech]

@T0jan Thanks, I'll have a look this week 👍

[eeintech]

Hello @T0jan

I'm trying it out however for the proxy part, I don't really have a good way to test if it works.

The settings view looks good to me:

Does it needs only the two extra fields HTTP and HTTPS to work?

I am able to connect to my instance when both fields are empty, then I tested:

1. Adding a valid URL in HTTP field and it does create quite some log on InvenTree side:

[17/Apr/2023 11:52:00] "GET http://127.0.0.1:8000/api/ HTTP/1.1" 302 0
[17/Apr/2023 11:52:00] "GET http://127.0.0.1:8000/accounts/login/?next=/http:/127.0.0.1:8000/api/ HTTP/1.1" 302 0
[17/Apr/2023 11:52:00] "GET http://127.0.0.1:8000/accounts/login/?next=/http:/127.0.0.1:8000/accounts/login/ HTTP/1.1" 302 0
[17/Apr/2023 11:52:00] "GET http://127.0.0.1:8000/accounts/login/?next=/http:/127.0.0.1:8000/accounts/login/ HTTP/1.1" 302 0
[17/Apr/2023 11:52:00] "GET http://127.0.0.1:8000/accounts/login/?next=/http:/127.0.0.1:8000/accounts/login/ HTTP/1.1" 302 0

but eventually fails and gives me the error message:

(note: could it be customized with a more targeted proxy message or specifics on which part of the proxy fails?)

I have tried adding a random string and it keeps showing me the error message so that's good 👍

2. If I enter anything in the HTTPS field but leave HTTP field empty, then it seems completely ignored and logs in with the traditional method. Is it expected? Can a proxy run only on HTTPS? Do you want to catch this use case?

3. Would putting a HTTPS URL in the HTTP field cause any issue using the proxy? On my setup it does give me the error message so at least it doesn't crash 👍

[T0jan]

@eeintech

Adding a valid URL in HTTP field and it does create quite some log on InvenTree side: [...] but eventually fails and gives me the error message:

this is to be expected. basically the GUI now sends proxy request to your localhost server which this one can't interpret as it is not configured as a proxy server. if you want to test with a proxy server you can use any from a free proxy server, e.g. one from here

(note: could it be customized with a more targeted proxy message or specifics on which part of the proxy fails?)

we could add a special error message based on the requests module error case Cannot connect to proxy not sure if it would trigger in this case tho.

If I enter anything in the HTTPS field but leave HTTP field empty, then it seems completely ignored and logs in with the traditional method. Is it expected? Can a proxy run only on HTTPS? Do you want to catch this use case?

Yes, this is expected. if you want to connect to an HTTP-server it will use the HTTP-proxy and for a HTTPS-server the HTTPS-proxy. thinking about it probably would make sense here to give the user only one proxy input field here and letting the program decide (based on the server URL) if it is HTTP or HTTPS. do you think this would make things clearer for the user?

Would putting a HTTPS URL in the HTTP field cause any issue using the proxy? On my setup it does give me the error message so at least it doesn't crash +1

no. server communications just wouldn't work as the program then sends an HTTPS (proxy) request to an HTTP server which this one obviously couldn't decrypt. would throw the same Cannot connect to proxy in the background. if you use a socks-proxy this problem wouldn't occur as these proxies use the same address for HTTP and HTTPS connection.

[eeintech]

this is to be expected. basically the GUI now sends proxy request to your localhost server which this one can't interpret as it is not configured as a proxy server. if you want to test with a proxy server you can use any from a free proxy server, e.g. one from here

Alright so I used one of the servers, put a random token (else it asks for username) and I'm getting this:

Server connection error: <class 'requests.exceptions.SSLError'>
[TREE]	Successfully connected to InvenTree server (ENV=DEVELOPMENT)

So not sure if that actually works...
Do you think we can implement a proxy test in run_tests.py using one of those free proxy servers?

we could add a special error message based on the requests module error case Cannot connect to proxy not sure if it would trigger in this case tho.

I guess for now the general message is clear, something is wrong with the InvenTree settings.
Let's focus on this later.

Yes, this is expected. if you want to connect to an HTTP-server it will use the HTTP-proxy and for a HTTPS-server the HTTPS-proxy. thinking about it probably would make sense here to give the user only one proxy input field here and letting the program decide (based on the server URL) if it is HTTP or HTTPS. do you think this would make things clearer for the user?

One field would definitely be clearer than two and lead to less confusion I believe. If you can use a generic name like Proxy URL that could do it for me.
One question I have for you, if you enter a Proxy URL, do you need to enter anything else?

[eeintech]

Hello again @T0jan

Regarding those two items:

• The ECCN information from Digikey will be processed as parameter like it is for Element14
• load_category_parameters is restructured to support parameter mappings for categories further down the tree as well as multiple mappings in a tree.

I understand you'd like the export_control_class_number to be part of the Digi-Key search data, however I think your implementation is quite crude and not cleanly expandable to other data fields, and it's all my fault not to have made it more flexible so I would like to propose to fix it 😃
Looking into it, I think adding a separate dataset of extra_fields that can be controlled via each supplier _config.yml configuration file could work. In the config file, one would just have to add export_control_class_number to the list under the "EXTRA_FIELDS" key (inside the config file) to get the result as part of the search result, and all those extra fields would be treated as parameters.
What do you think?

Regarding the load_category_parameters, it seems fine to me and looks like a sensible change, thanks for adding this!

[T0jan]

hi @eeintech

Alright so I used one of the servers, put a random token (else it asks for username) and I'm getting this:

have you tried with different proxy servers and to which inventree server do you wanted to connect? The servers in the list often don"t work so you have to try a bit to find a working one. I managed to get a connection to the inventree-demo server with:

Server Address: https://demo.inventree.org
Username: reader
Password: readonly
HTTPs Proxy: https://167.172.173.210:36917

as well as:

HTTPs Proxy: socks5h://104.237.134.201:59166

Do you think we can implement a proxy test in run_tests.py using one of those free proxy servers?

I am not sure how reliable these proxy servers are. I usually use them only for immediate testing and have sometimes to search a while to find a working one. So we could often get errors in run_tests.py only because the proxy is not working.

One field would definitely be clearer than two and lead to less confusion I believe. If you can use a generic name like Proxy URL that could do it for me.

Maybe it would be even a better idea to hide this field behind an additional switch, similar to the input fields for the IPN. 🤔

One question I have for you, if you enter a Proxy URL, do you need to enter anything else?

No you just have to construct the URL correctly like in my two examples above: protocol://address:port

Looking into it, I think adding a separate dataset of extra_fields that can be controlled via each supplier _config.yml configuration file could work. In the config file, one would just have to add export_control_class_number to the list under the "EXTRA_FIELDS" key (inside the config file) to get the result as part of the search result, and all those extra fields would be treated as parameters.
What do you think?

Sounds like a lot cleaner approach then my try to press it into the current structure 😄

The only problem I see here is that you then get two config files (the supplier_parameters.yaml and the supplier_config.yaml) the user needs to modify to get the data actually into inventree. but maybe the supplier_config.yaml can hold the most common extra fields a user would need by default so only special cases would have to look into this file?

[eeintech]

have you tried with different proxy servers and to which inventree server do you wanted to connect? The servers in the list often don"t work so you have to try a bit to find a working one. I managed to get a connection to the inventree-demo server with:

Server Address: https://demo.inventree.org
Username: reader
Password: readonly
HTTPs Proxy: https://167.172.173.210:36917

as well as:

HTTPs Proxy: socks5h://104.237.134.201:59166

Right both are working for me as well.

I am not sure how reliable these proxy servers are. I usually use them only for immediate testing and have sometimes to search a while to find a working one. So we could often get errors in run_tests.py only because the proxy is not working.

True... would be nice to find a permanently working proxy but we can keep it for a later stage.

Maybe it would be even a better idea to hide this field behind an additional switch, similar to the input fields for the IPN.

I think that's a great idea to avoid basic users (like me) confusion. One switch, one field and proxy can be used.

The only problem I see here is that you then get two config files (the supplier_parameters.yaml and the _supplier__config.yaml) the user needs to modify to get the data actually into inventree. but maybe the _supplier__config.yaml can hold the most common extra fields a user would need by default so only special cases would have to look into this file?

The supplier_parameters.yaml is only used for mapping any search parameter to InvenTree parameters. So supplier_config.yaml can hold the extra fields (as parameters) and then supplier_parameters.yaml map them to InvenTree. So you're right both files need an update but it shouldn't be a big one.

[T0jan]

at some point I will learn to check linting before I push...

@eeintech I just added the discussed hiding of the proxy setting. for the extra_fields do you have already an idea how you want to do it? then I would revert my commits related to the ECCN field and you can work cleanly on it in your repo.

[eeintech]

@eeintech I just added the discussed hiding of the proxy setting. for the extra_fields do you have already an idea how you want to do it? then I would revert my commits related to the ECCN field and you can work cleanly on it in your repo.

I sure can do that, if you don't mind reverting that change.

About the proxy implementation, the settings look good:

Without proxy it works great.
However, I cannot find a proxy which works in this list: http://free-proxy.cz/en/
Did you find a good working one?

I'm getting errors like:

Unhandled server error: <class 'requests.exceptions.InvalidSchema'>

Server connection error: <class 'requests.exceptions.ProxyError'>

Server connection error: <class 'requests.exceptions.ConnectTimeout'>

Also, how do you test connecting with a token only? Where do you get the token from?

[T0jan]

@eeintech ok the rebase went not as good as I hoped but it should be fine now. so the changes for the ECCN fields are gone.

However, I cannot find a proxy which works in this list: http://free-proxy.cz/en/
Did you find a good working one?

unfortunately not, I tried with a few handful and had no luck. but at least with my own server I could connect via proxy so I know it works.

Also, how do you test connecting with a token only? Where do you get the token from?

you can request the token directly from the server by sending a request to your server api with server_api_url/user/token/ and your credentials. with this command:

Ki-nTree/kintree/database/inventree_api.py

Line 29 in 74c19e6

return InvenTreeAPI(server, username=username, password=password)

the gui basically also requests the token in the background, holds it as long as it is running and makes every api request to the database with this token, not the username and password.

the inventree_python function to get the token is part of the InvenTreeAPI class: requestToken()

[eeintech]

Hello @T0jan

Couple comments in your PR, could you please review?

Other than that I'm happy to merge 😃

[eeintech]

Could you also rebase on main? it created a lot of duplicate commits 😕

[T0jan]

@eeintech 2nd rebase went better, should have a clean commit history now.

I modified the README and cleaned up the Proxy settings and definitions in case no proxy is used as requested.

On the token topic: I was thinking about adding a third button next to save to replace the user credentials with the token automatically as all requests in the background are anyway done with the token method. any opinions on this?

[eeintech]

@eeintech 2nd rebase went better, should have a clean commit history now.

I modified the README and cleaned up the Proxy settings and definitions in case no proxy is used as requested.

Thanks, I'll take a look!

On the token topic: I was thinking about adding a third button next to save to replace the user credentials with the token automatically as all requests in the background are anyway done with the token method. any opinions on this?

So a "Get Token" button? But this will overwrite the password/token field and then it won't be valid next time Ki-nTree is opened... am I understand this correctly? Would this button useful for proxy method only, because I don't see why using it when you have your username/password filed in?

[eeintech]

Everything looks good to me and I'm happy to merge as-is unless you really think adding the "Get Token" button is necessary to simplify the proxy workflow.

[T0jan]

@eeintech

So a "Get Token" button?

exactly.

But this will overwrite the password/token field and then it won't be valid next time Ki-nTree is opened...

It would be valid. each user in InvenTree has a persistent token assigned to him which can be used for authentication as long as the user exists (and has the access rights to the specific information). it is not token which is connected to user sessions and therefore would needed to be updated each time you connect to the server. It is just a different kind of authentication method compared to user credentials and not connected to the proxy-topic.

otherwise the addition of saving the token instead of the password wouldn't make much sense 😅

But I think we can skip the button for now. at least as long until my colleagues will start asking me about how they can get their token 🙃

[eeintech]

@T0jan Thanks for the feedback. Let's go with what you proposed so far, we can discuss the extra button for another release, if really necessary.

Thanks a lot for your contribution 👍