Fix statsd Logging in Python 3 #1010
Conversation
| @@ -224,8 +224,8 @@ def info(self, msg, *args, **kwargs): | |||
| def debug(self, msg, *args, **kwargs): | |||
| self.error_log.debug(msg, *args, **kwargs) | |||
|
|
|||
| def exception(self, msg, *args): | |||
| self.error_log.exception(msg, *args) | |||
| def exception(self, msg, *args, **kwargs): | |||
|
Fixed the two issues you pointed out. Note that if Cheers, |
|
LGTM (thanks for squashing commits by the way!). My only question is that if we test the statsd logging against a real socket, shouldn't we use a different class since the class' name is "MockSocket"? In theory, we should be able to test whether the input is str or not by tweaking the MockSocket.send() method (without creating a real socket). |
|
The idea behind using an actual socket is to catch similar errors in unit tests in the future (if they can even exist). This time, it was a text vs bytes issue, but maybe another issue will crop up some day (I have no idea what that issue might be, but after all that's why we write tests!). I'm fine with renaming the class, though — let me know! Cheers, |
| try: | ||
| if isinstance(msg, six.text_type): | ||
| msg = msg.encode("ascii") |
There was a problem hiding this comment.
why not reusing the bytes_to_str function from the _compat module instead?
There was a problem hiding this comment.
I think we'd need str_to_bytes instead, wouldn't we?
Looking at the definition for bytes_to_str, I'm a bit unclear on what it's supposed to returns. It looks like it gives you bytes on Py2 and text Py3? I'm probably misreading though.
Cheers,
There was a problem hiding this comment.
mmm you're right should probably be the reverse in your case. anyway could be useful to have it in a _compat module, so we don't have to do such things on python 2.x. thoughts?
There was a problem hiding this comment.
Makes sense. Can you just confirm what those methods should return to clear up the confusion above?
There was a problem hiding this comment.
If we are going to add a function like this, we need to make it more general(e.g. you can pass different encodings and error handlers). util.to_bytestring is another alternative, but it uses the UTF-8 encoding by default. I'd say merge this now and create a new issue or PR to discuss details.
There was a problem hiding this comment.
+1 let's merge now and handle the rest separately.
On Thu 30 Apr 2015 at 12:44 Berker Peksag [email protected] wrote:
In gunicorn/instrument/statsd.py
#1010 (comment):try:
if isinstance(msg, six.text_type):msg = msg.encode("ascii")If we are going to add a function like this, we need to make it more
general(e.g. you can pass different encodings and error handlers).
util.to_bytestring is another alternative, but it uses the UTF-8 encoding
by default. I'd say merge this now and create a new issue or PR to discuss
details.—
Reply to this email directly or view it on GitHub
https://github.com/benoitc/gunicorn/pull/1010/files#r29419336.
|
Do you guys need me to rebase or squash further or anything? Just let me know : ) |
|
@krallin if you have time, please squash the commits. If not, I can do it today :) Thanks! |
Bytes must be passed to socket.send. Update tests to ensure an actual socket is used, so that errors like this can be caught in unit tests in the future.
|
@berkerpeksag Sure — done |
Fix statsd Logging in Python 3
|
Thanks! |
The argument is documented in [1], and we've been using it in some cases since 410bcfa (Fix statsd logging to work on Python 3, 2015-04-20, benoitc#1010). This commit goes through all of our log calls and makes sure we're using it consistently. For more examples of using exc_info instead of direct traceback calls, see 0acfb55 (Let logging module handle traceback printing, 2016-02-05, benoitc#1201). The traceback call I'm removing snuck in later with f4842a5 (Log exception occurring during worker exit, 2016-05-04, benoitc#1254). I'm removing the "Error processing SSL request" debug messages, because those will be handled by the subsequent handle_error call. We've had them, for reasons that I'm not clear on, since 1198cc2 (handle ssl connections, 2012-12-14). I've restored logging in the URI-less Worker.handle_error 500 case, since that was broken in 79011d0 (Include request URL in error message, 2015-07-08, benoitc#1071) but not restored in 1ccebab (prevent crash when reporting an error, 2016-03-22, benoitc#1231). [1]: https://docs.python.org/2/library/logging.html#logging.Logger.debug
The argument is documented in [1], and we've been using it in some cases since 410bcfa (Fix statsd logging to work on Python 3, 2015-04-20, benoitc#1010). This commit goes through all of our log calls and makes sure we're using it consistently. For more examples of using exc_info instead of direct traceback calls, see 0acfb55 (Let logging module handle traceback printing, 2016-02-05, benoitc#1201). The traceback call I'm removing snuck in later with f4842a5 (Log exception occurring during worker exit, 2016-05-04, benoitc#1254). I'm removing the "Error processing SSL request" debug messages, because those will be handled by the subsequent handle_error call. We've had them, for reasons that I'm not clear on, since 1198cc2 (handle ssl connections, 2012-12-14). I've restored logging in the URI-less Worker.handle_error 500 case, since that was broken in 79011d0 (Include request URL in error message, 2015-07-08, benoitc#1071) but not restored in 1ccebab (prevent crash when reporting an error, 2016-03-22, benoitc#1231). Tests for the lines I've touched may be more trouble than they're worth. I've worked up one for a broken worker_exit hook to show what they look like, but left the others untested for now. [1]: https://docs.python.org/2/library/logging.html#logging.Logger.debug
The argument is documented in [1], and we've been using it in some cases since 410bcfa (Fix statsd logging to work on Python 3, 2015-04-20, benoitc#1010). This commit goes through all of our log calls and makes sure we're using it consistently. For more examples of using exc_info instead of direct traceback calls, see 0acfb55 (Let logging module handle traceback printing, 2016-02-05, benoitc#1201). The traceback call I'm removing snuck in later with f4842a5 (Log exception occurring during worker exit, 2016-05-04, benoitc#1254). I'm removing the "Error processing SSL request" debug messages, because those will be handled by the subsequent handle_error call. We've had them, for reasons that I'm not clear on, since 1198cc2 (handle ssl connections, 2012-12-14). I've restored logging in the URI-less Worker.handle_error 500 case, since that was broken in 79011d0 (Include request URL in error message, 2015-07-08, benoitc#1071) but not restored in 1ccebab (prevent crash when reporting an error, 2016-03-22, benoitc#1231). Tests for the lines I've touched may be more trouble than they're worth. I've worked up one for a broken worker_exit hook to show what they look like, but left the others untested for now. [1]: https://docs.python.org/2/library/logging.html#logging.Logger.debug
statsdlogging appears to be (silently) broken in Python 3, because it passes it passesunicode(i.e. notbytes) to thesock.sendmethod.This PR:
bytesare usedstatsderrors as warnings (the exceptions being silently swallowed made them difficult to troubleshoot)