Client randomly fails to send POST request

[mkpankov]

I have the following code:

impl Session {
    pub fn post_comment(
        &self,
        mr_id: &MrUid,
        message: &str)
    {
        let client = &self.client;
        let mut headers = Headers::new();

        headers.set(ContentType(Mime(TopLevel::Application, SubLevel::Json, vec![])));
        headers.set_raw("PRIVATE-TOKEN", vec![self.private_token.clone().into_bytes()]);

        debug!("headers: {:?}", headers);
        let &MrUid { target_project_id, id } = mr_id;

        let body = &*format!("{{ \"note\": \"{}\" }}", message);
        debug!("body: {:?}", body);

        info!("Posting comment: {}", message);
        let res = client.post(&*format!("{}/projects/{}/merge_request/{}/comments", self.api_root, target_project_id, id))
            .headers(headers)
            .body(body)
            .send()
            .unwrap();
        assert_eq!(res.status, ::hyper::status::StatusCode::Created);
    }
}

It randomly fails, sometimes, with the following logs and backtrace (*** for token bytes):

DEBUG:shurik::gitlab: headers: Headers { PRIVATE-TOKEN: ***, Content-Type: application/json, }
DEBUG:shurik::gitlab: body: "{ \"note\": \"MR \u{441} id 368 \u{43d}\u{435} \u{43d}\u{430}\u{439}\u{434}\u{435}\u{43d}\" }"
INFO:shurik::gitlab: Posting comment: MR с id 368 не найден
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Io(Error { repr: Custom(Custom { kind: ConnectionAborted, error: StringError("Connection closed") }) })', ../src/libcore/result.rs:738
stack backtrace:
   1:     0x7f072ce13310 - sys::backtrace::tracing::imp::write::h1fe79f3711caca0eUnt
   2:     0x7f072ce15fd5 - panicking::log_panic::_<closure>::closure.39701
   3:     0x7f072ce15a21 - panicking::log_panic::hffc6d029fed602571nx
   4:     0x7f072cdf6673 - sys_common::unwind::begin_unwind_inner::h7045c1c64d9ab8edYgs
   5:     0x7f072cdf6d28 - sys_common::unwind::begin_unwind_fmt::h40ee994bfe85f7a54fs
   6:     0x7f072ce129e1 - rust_begin_unwind
   7:     0x7f072ce43d2f - panicking::panic_fmt::h4c8d12e3c05f3b8cZEK
   8:     0x7f072c963894 - result::_<impl>::unwrap::unwrap::h16876323606442618952
                        at ../src/libcore/macros.rs:28
   9:     0x7f072c97e34f - gitlab::_<impl>::post_comment::h0cb8a363c81728070db
                        at src/gitlab.rs:68
  10:     0x7f072ca542dc - report::h550834463e7ec6bbJrg
                        at src/main.rs:1886
  11:     0x7f072ca53774 - main::_<closure>::closure.22051
                        at src/main.rs:1850
  12:     0x7f072ca5330f - std::thread::_<impl>::spawn::_<closure>::_<closure>::closure.22040
                        at ../src/libstd/thread/mod.rs:271
  13:     0x7f072ca532c9 - sys_common::unwind::try::try_fn::try_fn::h12636286934587722039
                        at ../src/libstd/sys/common/unwind/mod.rs:156
  14:     0x7f072ce12848 - __rust_try
  15:     0x7f072ce0908b - sys_common::unwind::try::inner_try::h0c3b0e63f018cc29wds
  16:     0x7f072ca53233 - sys_common::unwind::try::try::h14092093334741834918
                        at ../src/libstd/sys/common/unwind/mod.rs:126
  17:     0x7f072ca530bc - std::thread::_<impl>::spawn::_<closure>::closure.22037
                        at ../src/libstd/thread/mod.rs:271
  18:     0x7f072ca5391c - boxed::_<impl>::call_box::call_box::h17787366675358503356
                        at ../src/liballoc/boxed.rs:521
  19:     0x7f072ce14ac3 - sys::thread::_<impl>::new::thread_start::h85eb4d682b5d5d4ffGw
  20:     0x7f072b39b181 - start_thread
  21:     0x7f072bbb547c - __clone
  22:                0x0 - <unknown>

I checked for possible connectivity issues and tried doing the same using curl - there doesn't seem to be any problem. I'm using hyper 0.6.15.

Could it be related to #309 by any chance? I see that issue is marked as related to server, but the symptoms seem similar: I actually don't read out the Response and just check the status. But I experience the problem in client so I decided to file another issue.

Another reason I think it might be a problem in hyper is that I only recently started actively using GET requests intermixed with POST requests.

Sorry for not providing a minimal example and not being very specific.

[seanmonstar]

It's crashing because of the call to unwrap. You'd be better off matching on the Err case.

This instance is because a socket in the client connection pool went stale.

[seanmonstar]

Darn mobile, thumbs tapping close buttons...

With blocking IO, checking to see if a Connection was still alive could block the thread. So instead, the error happens the first time you try to write after the connection had been closed.

[mkpankov]

It's crashing because of the call to unwrap. You'd be better off matching on the Err case.

This I understand :) I wonder why the error happens, though

It's crashing because of the call to unwrap. You'd be better off matching on the Err case.

Hyper client keeps pool of sockets for its requests?.. Didn't know that.

With blocking IO, checking to see if a Connection was still alive could block the thread. So instead, the error happens the first time you try to write after the connection had been closed.

Hm, so I should just retry in such case?.. Is that a legitimate solution?

[seanmonstar]

Hyper client keeps pool of sockets for its requests?.. Didn't know that.

Yep! It's likely what most people would want, but it can be customized (or turned off): http://hyper.rs/hyper/hyper/client/struct.Client.html

Hm, so I should just retry in such case?.. Is that a legitimate solution?

This is exactly what Servo does: https://github.com/servo/servo/blob/master/components/net/http_loader.rs#L684-L685

[mkpankov]

Thank you, will try it out after holidays. Closing :)