Improve widom insertion error estimate

[jonaslandsgesell]

The standard error is used as estimate for the error of the mean. The assumption is uncorrelated samples which might result in error estimates being too optimistic. Additionally Widoms method tries to brute force sample rare events.

Peter would like to be able to check how well the assumption for uncorrelated samples was fulfilled. He thinks that it would be more useful to output the whole time-series, and let the user estimate the error using his own favorite method.

But that will not solve the problem of brute force sampling a rare event.

Originally posted by @kosovan in #3254

[jonaslandsgesell]

Alternatively a correlation corrected error could be implemented ... (Making use of the Markov Chain CLT and a criterion for stopping the integral of the ACF or a Blocking analysis)

[kosovan]

As mentioned earlier,

I think that it would be more useful to output the whole time-series, and let the user estimate the error using his own favorite method.

Because correlation-corrected error estimates also rely on some assumptions, and they fail (produce incorrect results) if the assumptions are not fulfilled. To my best knowledge, there is no algorithm to compute correlation-corrected error estimates that could reliably and automatically check for its own consistency without user intervention.

[jonaslandsgesell]

#4003

[KaiSzuttor]

@jonaslandsgesell any plans here?

[jonaslandsgesell]

No plans from my side. Returning the bare values from the accumulator to python could be an option - but I doubt that autocorrelations do pose a big problem here. We saw in the other issue that rater the brute force sampling in widoms method is a problem (in dilute systems it s a rather rare event to obtain a sample which significantly contributes to the estimates). I would close this issue and reopen it if somebody can prove (e.g. via outputting the values in a temporary way) that autocorrelations pose a significant problem - however, due to the brute force sampling in the method a big autocorrelation time in terms of samples for a not too small system would be surprising)