Fatal crashes - "[Errno 32] Broken pipe"

[motin]

In automation/SocketInterface.py in send at line 168

Traceback (most recent call last):
  File "crawler.py", line 102, in <module>
    manager.execute_command_sequence(command_sequence)
  File "/opt/OpenWPM/automation/TaskManager.py", line 541, in execute_command_sequence
    self._distribute_command(command_sequence, index)
  File "/opt/OpenWPM/automation/TaskManager.py", line 341, in _distribute_command
    thread = self._start_thread(browser, command_seq)
  File "/opt/OpenWPM/automation/TaskManager.py", line 408, in _start_thread
    "site_url": command_sequence.url
  File "/opt/OpenWPM/automation/SocketInterface.py", line 168, in send
    sent = self.sock.send(msg[totalsent:])
socket.error: [Errno 32] Broken pipe

This currently prevents larger crawls from completing.

If this issue can't be resolved, it'd be great if it can be caught gracefully and the site visit retried so that the crawl can continue.

[motin]

Maybe revisiting WebSockets (as partially implemented in #221) is a good idea?

[motin]

A crawl with backoffLimit 6 failed after 20 of these exceptions, and got through about 3k sites.
Another crawl with backoffLimit 100 failed after 125 of these exceptions, and got through about 8k sites.

With this very limited data, it seems that these socket-induced crashes occur on average every 64-150 sites crawled, with a result in crashing the container about 30-80% of the times.

[motin]

Relates to #255

[englehardt]

This is almost always thrown when another process that the host process is communicating with has crashed, leaving the other end of the socket dead. Thus, catching this exception and recovering from the crash will be difficult or impossible. For example, the stack trace you included is from https://github.com/mozilla/OpenWPM/blob/9f0442241909fdac273eeb444bf06c0f2beb781c/automation/TaskManager.py#L404-L409. This likely means that the data aggregator process has crashed, which is not going to be easy to recover from. In this particular case, the right fix is in #438.

[englehardt]

Closing this since the cause for this particular set of errors was fixed in #438. I don't think there's anything more to do here, for the reasons stated in my comment above.