Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nailgun client error handling isn't clear to users when nailgun isn't connectable #2238

Closed
kwlzn opened this issue Sep 22, 2015 · 1 comment
Closed

nailgun client error handling isn't clear to users when nailgun isn't connectable #2238

kwlzn opened this issue Sep 22, 2015 · 1 comment
Assignees
@kwlzn
Labels
bug

Comments

@kwlzn
Copy link
Member

kwlzn commented Sep 22, 2015

this bug covers two that were exposed recently when a user had a non-connectable nailgun:

  1. a bug in nailgun_executor.py:226:
Exception caught:
  File ".bootstrap/_pex/pex.py", line 315, in execute
    self._wrap_coverage(self._wrap_profiling, self._execute)
  File ".bootstrap/_pex/pex.py", line 250, in _wrap_coverage
    runner(*args)
  File ".bootstrap/_pex/pex.py", line 282, in _wrap_profiling
    runner(*args)
  File ".bootstrap/_pex/pex.py", line 358, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File ".bootstrap/_pex/pex.py", line 416, in execute_entry
    runner(entry_point)
  File ".bootstrap/_pex/pex.py", line 434, in execute_pkg_resources
    runner()
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/pants_exe.py", line 88, in main
    _run(exiter)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/pants_exe.py", line 81, in _run
    result = goal_runner.run()
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/goal_runner.py", line 222, in run
    result = self._do_run()
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/goal_runner.py", line 285, in _do_run
    result = engine.execute(context, self.goals)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/engine/engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/engine/round_engine.py", line 214, in attempt
    goal_executor.attempt(explain)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/engine/round_engine.py", line 47, in attempt
    task.execute()
  File "twitter/pants_internal/tasks/persisted_schema_compatibility_checker.py", line 129, in execute
    self._compat_check()
  File "twitter/pants_internal/tasks/persisted_schema_compatibility_checker.py", line 118, in _compat_check
    self._run_compat_checker(tmpdir, os.path.join(work_dir, thrift_root))
  File "twitter/pants_internal/tasks/persisted_schema_compatibility_checker.py", line 101, in _run_compat_checker
    args= checker_args)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/backend/jvm/tasks/nailgun_task.py", line 93, in runjava
    workunit_log_config=workunit_log_config)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/util.py", line 47, in execute_java
    workunit_log_config=workunit_log_config)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/util.py", line 80, in execute_runner
    ret = runner.run(stdout=workunit.output('stdout'), stderr=workunit.output('stderr'), cwd=cwd)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/nailgun_executor.py", line 150, in run
    nailgun = self._get_nailgun_client(jvm_options, classpath, stdout, stderr)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/nailgun_executor.py", line 187, in _get_nailgun_client
    return self._spawn_nailgun_server(new_fingerprint, jvm_options, classpath, stdout, stderr)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/nailgun_executor.py", line 260, in _spawn_nailgun_server
    self.ensure_connectable(client)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/nailgun_executor.py", line 226, in ensure_connectable
    sock.close()

Exception message: local variable 'sock' referenced before assignment
  1. an unhelpful exception related to NailgunClient.try_connect() returning None when not connectable:
Exception caught:
  File ".bootstrap/_pex/pex.py", line 315, in execute
    self._wrap_coverage(self._wrap_profiling, self._execute)
  File ".bootstrap/_pex/pex.py", line 250, in _wrap_coverage
    runner(*args)
  File ".bootstrap/_pex/pex.py", line 282, in _wrap_profiling
    runner(*args)
  File ".bootstrap/_pex/pex.py", line 358, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File ".bootstrap/_pex/pex.py", line 416, in execute_entry
    runner(entry_point)
  File ".bootstrap/_pex/pex.py", line 434, in execute_pkg_resources
    runner()
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/pants_exe.py", line 88, in main
    _run(exiter)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/pants_exe.py", line 81, in _run
    result = goal_runner.run()
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/goal_runner.py", line 222, in run
    result = self._do_run()
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/bin/goal_runner.py", line 285, in _do_run
    result = engine.execute(context, self.goals)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/engine/engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/engine/round_engine.py", line 214, in attempt
    goal_executor.attempt(explain)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/engine/round_engine.py", line 47, in attempt
    task.execute()
  File "twitter/pants_internal/tasks/persisted_schema_compatibility_checker.py", line 129, in execute
    self._compat_check()
  File "twitter/pants_internal/tasks/persisted_schema_compatibility_checker.py", line 118, in _compat_check
    self._run_compat_checker(tmpdir, os.path.join(work_dir, thrift_root))
  File "twitter/pants_internal/tasks/persisted_schema_compatibility_checker.py", line 101, in _run_compat_checker
    args= checker_args)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/backend/jvm/tasks/nailgun_task.py", line 93, in runjava
    workunit_log_config=workunit_log_config)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/util.py", line 47, in execute_java
    workunit_log_config=workunit_log_config)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/util.py", line 80, in execute_runner
    ret = runner.run(stdout=workunit.output('stdout'), stderr=workunit.output('stderr'), cwd=cwd)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/nailgun_executor.py", line 153, in run
    return nailgun(main, cwd, *args)
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/nailgun_client.py", line 186, in __call__
    sock = self.try_connect()
  File "/Users/ysaraf/workspace/source/.pex/install/pantsbuild.pants-0.0.44-py2-none-any.whl.448f811d12fc07bcefac32b2d68fb57a61f58685/pantsbuild.pants-0.0.44-py2-none-any.whl/pants/java/nailgun_client.py", line 171, in try_connect
    return sock if sock.connect_ex((self._host, self._port)) == 0 else None
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)

Exception message: [Errno 8] nodename nor servname provided, or not known
@kwlzn kwlzn self-assigned this Sep 22, 2015
@kwlzn kwlzn added the bug label Sep 22, 2015
@kwlzn
Copy link
Member Author

kwlzn commented Sep 22, 2015

it seems item 2 was caused by the user having a broken /etc/hosts where 'localhost' did not resolve. this is the default host the NailgunClient connects to and socket.connect_ex() does not catch socket.gaierror in this case resulting in the strange user-facing error.

@kwlzn kwlzn mentioned this issue Sep 23, 2015
Closed
@kwlzn kwlzn closed this as completed in 35ba6bf Sep 23, 2015
@kwlzn kwlzn removed the in-progress label Sep 23, 2015
peiyuwang pushed a commit to peiyuwang/pants that referenced this issue Sep 24, 2015
@kwlzn @peiyuwang
Improve error handling for nailgun client connection attempts
b48738e 
Fixes pantsbuild#2238

- Use 127.0.0.1 vs localhost for nailgun default host. this prevents a situation where a broken /etc/hosts can affect pants ability to connect to the nailgun (currently raises an unhandled and generic socket.gaierror).
- Add a specialized NailgunClient.NailgunConnectionError exception type.
- Convert NailgunClient.try_connect() from socket.connect_ex() to socket.connect() with added catching of socket.gaierror which is not covered by socket.connect_ex().
- Make NailgunClient.try_connect() raise a helpful NailgunConnectionError exception on failure vs returning None.
- Propagate the helpful NailgunConnectionError in both NailgunClient.__call__() and NailgunExecutor.ensure_connectable().
- Improve debug logging.

Testing Done:
https://travis-ci.org/pantsbuild/pants/builds/81720419 + manual testing.

Before/After for a broken /etc/hosts:

    Exception message: [Errno 8] nodename nor servname provided, or not known
vs
    Exception message: Problem connecting to nailgun server at localhost:55261: gaierror(8, 'nodename nor servname provided, or not known')

Before (with resident ng)/Before (without resident ng)/After for an unconnectable nailgun:

    Exception message: Failed to connect to ng server.
vs
    Exception message: 'NoneType' object has no attribute 'close'
vs
    Exception message: Problem connecting to nailgun server at 127.0.0.1:9999: error(61, 'Connection refused')

Bugs closed: 2238, 2246

Reviewed at https://rbcommons.com/s/twitter/r/2869/
@jsirois jsirois mentioned this issue Oct 8, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kwlzn kwlzn

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kwlzn