-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stdout stream is not flushed deterministcally for mitmdump #5
Comments
Actually, it seems like the flushing is deterministic. If I output the chunks, the all end after HTTP requests in the mitmdump output, e.g. (the numbers are added by me to count the chunks):
I can provoke the flushing if I open an App that connects to mitmproxy, but if we leave the device/emulator alone, we only get the first chunk if the phone happens to make a HTTP request. |
Oh yeah, I remember running into the same problem for the data safety label analysis. Probably something changed in mitmproxy (or Node/execa?) since I originally wrote the code. My "solution" back then was to increase the timeout to a ridiculous value, it did always flush after a fairly long time: |
I mean, you could check that the port is closed beforehand and then wait for it to be opened after launching mitmproxy… But I'm not sure whether "port open" == "mitmproxy ready". |
Weirdly enough, if I add
However, for weird reasons we cannot just add our own streams into that to test if that would solve the problem (see sindresorhus/execa#81). |
I think, the cleanest way to do this would be to write an addon for mitmproxy that emits the lifecycle events as IPC events on a socket. But this would be some additional work. What do you think, @baltpeter? |
This would also enable us to detect errors, such as a misconfigured TLS encryption. |
I mean, that would work and be robust of course. But that seems like a lot of work just to know whether mitmproxy has started. And it would require us to carry around that addon (which is exactly what we would ideally like to avoid at some point with the HAR addon)… |
OK, that's a fair point. It would be especially nice if the user could see when the cert pinning bypass fails. |
I'm not happy about the added complexity, but I'll leave the decision to you. If you think that's the best way forward, that's fine. (Have you checked whether mitmproxy might have a similar feature in core already?) |
I don’t think it should be too complicated. IPC seems easy enough (though I worry about Windows): https://stackoverflow.com/questions/23804434/python-process-forked-by-nodejs-alternative-to-process-send-for-python I’ll give it a try. |
Where should I put the addon? |
Can we make that a separate Python package that the user can install as a dependency with |
Ok, so this stackoverflow made me think using the node IPC funtions is easy, even though it is not offically supported. And it is very easy, in fact. I can just write to the correct file descriptor in python and the message is passed to the parent node process: import os
os.write(3, bytes('{"test" : "This is a test"}\n', "utf8")) The parent looks like this: const { spawn, } = require('child_process');
const proc = spawn('python', ['./child.py'], {
stdio: ['pipe', 'pipe', 'pipe', 'ipc'],
});
proc.on('message', (msg) => console.log(msg)); // { test: "This is a test" } This is all fine and good on linux, but when it comes to Windows it is an entirely different and even less supported story. I am unsure if I should pursue this, whether we even want to support Windows or if I should just use sockets. |
Ok, having investigated a little more, I found out that nodejs uses So, having tried quite a lot, I think we will have to give up Windows support for now. I haven't tested the rest of the code on Windows, yet, anyway. I guess as a workaround we can also skip waiting for |
Because python package are really bad for distributing scripts, I don’t see a great way to distribute the addon, for now I just put it in a different repository: https://github.com/tweaselORG/mitmproxy_ipc_events_addon |
While trying to write a proper example I ran into an issue, where the program would timeout all the time while waiting for mitmproxy to start. After a lot of investigation, it turns out that the problem is detecting whether mitmproxy started already. Right now, we do this by waiting for the start text to appear in the output, e.g. "listening at". But this fails often, because the
stdout
stream is not flushed before the timeout runs out, if nothing new is added to the output, leaving theawaitProcessStart
to timeout. I have not found a way to makeexeca
to flush the stream earlier.To resolve this, I either
The text was updated successfully, but these errors were encountered: