Squashed commit of the following:

commit f239399
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 22:34:25 2018 -0700

    Remove stray pdb import

commit 3babfb9
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 22:23:19 2018 -0700

    Add back Py3 tests that now work

commit a46b827
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 22:22:52 2018 -0700

    Fix backend python issues

commit 88fa18a
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 20:08:42 2018 -0700

    Fix backend project_info issues

commit f4d4a2d
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 19:54:39 2018 -0700

    Actually fix console separator issues

commit d0c83aa
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 19:38:39 2018 -0700

    Fix issues with backend native

commit 9a1b3b7
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 19:33:48 2018 -0700

    Fix most issues with backend jvm

commit 62bc2a4
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 14:47:42 2018 -0700

    Fix logging issue with passing object to format()

commit a4bf759
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 08:05:07 2018 -0700

    Decode test's execute_task() output

commit b81cd24
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 08:03:57 2018 -0700

    Fix issues with backend graph_info

commit 5d28b9a
Author: Eric Arellano <[email protected]>
Date:   Sun Aug 12 23:13:06 2018 -0700

    Fix console_task's console_separator unicode issues

commit a8674f9
Author: Eric Arellano <[email protected]>
Date:   Sun Aug 12 22:48:42 2018 -0700

    Fix backend codegen unicode vs bytes issues

commit c893f8c
Author: Eric Arellano <[email protected]>
Date:   Sun Aug 12 22:47:04 2018 -0700

    Return unicode from stdout in pants integration test

commit f4ab07c
Author: Benjy Weinberger <[email protected]>
Date:   Thu Aug 16 19:43:41 2018 -0600

    Make requirements on codegen products optional. (pantsbuild#6357)

    We declare product requirements in various places to force
    codegen to run before, e.g., dependency resolution.

    However currently these requirements are mandatory, so
    deregistering all codegen backends will cause the round engine
    to find no producers of these products, and fail.

    This commit changes these requirements to optional, so
    that if there are no codegen backends, we don't attempt
    to depend on their products.

    Also, minor unrelated documentation nits that weren't worth a
    separate pull request.

commit b0a2469
Author: Eric Arellano <[email protected]>
Date:   Thu Aug 16 12:08:29 2018 -0700

    Python 3 - fixes to get green contrib (pantsbuild#6340)

    All contrib folders should now be green on Python 2 and Python 3, excluding `googlejavaformat` and `python`.

    This fixes multiple issues causing Py3 failures + adds Py3 to Travis.

commit 070a5c5
Author: Daniel Wagner-Hall <[email protected]>
Date:   Wed Aug 15 19:30:15 2018 -0700

    Run clippy with nightly rust on CI (pantsbuild#6347)

    This enables all default lints, and a selection of pedantic ones.

    It also fixes all code to be clean with respect to those lints.

    Each commit is independently reviewable.

    It's a shame that the lint selecting can't be shared across creates in some way.

commit 8ce6297
Author: Daniel Wagner-Hall <[email protected]>
Date:   Tue Aug 14 20:04:02 2018 -0700

    Use --entry-point not -c when building pex (pantsbuild#6349)

    I deterministically get this error:

    RuntimeError: Ambiguous script specification pants matches multiple entry points:pants = pants.bin.pants_loader:main pants = pants.bin.pants_loader:main

    when building with -c.

    pantsbuild#6267 changed this behaviour,
    but seems broken - this commit allows me to successfully build pexes.

commit 30878d8
Author: Daniel Wagner-Hall <[email protected]>
Date:   Tue Aug 14 19:11:19 2018 -0700

    Fix formatting of store.rs (pantsbuild#6350)

    It looks like pantsbuild#6336 merged with bad formatting.

commit 5f2ad63
Author: Daniel Wagner-Hall <[email protected]>
Date:   Tue Aug 14 18:40:32 2018 -0700

    Allow pex download path to be overridden (pantsbuild#6348)

commit b7c607f
Author: Stu Hood <[email protected]>
Date:   Tue Aug 14 17:40:34 2018 -0700

    Recover from cancelled remote execution RPCs (pantsbuild#6188)

    ### Problem

    It's possible for an individual RPC to be cancelled while an entire Operation continues, but this is not currently handled. See the problem description on pantsbuild#6100.

    ### Solution

    Recover from `RpcStatus::Cancelled` by retrying the same operation.

    ### Result

    Without the Fixes pantsbuild#6100.

commit a96001e
Author: Ity Kaul <[email protected]>
Date:   Tue Aug 14 16:19:31 2018 -0700

    Download Directory recursively from remote CAS  (pantsbuild#6336)

    ### Problem

    We dont currently have the ability to download a `Directory`, and its contents, recursively.

    ### Solution

    Add `ensure_local_has_recursive_directory()` to be able to download  a `Directory`, and its contents, recursively.

commit e391cd1
Author: Daniel Wagner-Hall <[email protected]>
Date:   Tue Aug 14 16:11:22 2018 -0700

    Process execution: Create symlink to JDK on demand (pantsbuild#6346)

    This creates a symlink `.jdk` pointing at the passed JDK, if one was
    passed.

    This is a hack to allow for a remote execution platform to provide a
    pointer at its pre-installed JDK without needing to pollute absolute
    paths into the action definition.

    Long-term, we want the JDK to be part of the input directory for the
    action, but that's a lot of work, and we want to experiment with
    pre-installed JDKs in the mean time.

commit d79b7c3
Author: Daniel Wagner-Hall <[email protected]>
Date:   Mon Aug 13 21:26:08 2018 -0700

    Simplify ExecuteProcessRequest construction (pantsbuild#6345)

    Remove create_from_snapshot and create_with_empty_snapshot - they don't
    pull their weight.

    Allow default values for all defaultable values.

commit 33674ce
Author: Daniel Wagner-Hall <[email protected]>
Date:   Mon Aug 13 18:57:20 2018 -0700

    Use forked version of grpcio (pantsbuild#6344)

    This includes tikv/grpc-rs#211 which stops us
    seeing RpcFinished errors when the server responds very quickly to
    requests.

    Unfortunately, this will slow down building pants, as you'll need to
    checkout the git repository, but that should be a once-per-machine cost.

commit ccdf523
Author: Daniel Wagner-Hall <[email protected]>
Date:   Mon Aug 13 18:36:28 2018 -0700

    ci.sh uses positive rather than negative flags (pantsbuild#6342)

    This makes it much easier to reason about what's happening, and keeps
    .travis.yml more concise.

commit 68f4b59
Author: Daniel Wagner-Hall <[email protected]>
Date:   Mon Aug 13 16:12:44 2018 -0700

    Merge directories with identical files (pantsbuild#6343)

    This is useful for merging Snapshots for Targets if multiple targets own
    the same file. This should be rare, but does happen (e.g. multiple
    overlapping resource targets).

commit e189acd
Author: Daniel Wagner-Hall <[email protected]>
Date:   Mon Aug 13 09:50:23 2018 -0700

    Set chunk size in process_executor (pantsbuild#6337)

commit 20d267f
Author: Paul <[email protected]>
Date:   Sun Aug 12 14:41:26 2018 -0700

    added fullpath to fix path concat issue with files when not in git root (pantsbuild#6331)

    ### Problem
    As described in pantsbuild#6301 , when the git root and the build root is not the same, `changed` functionalities can be messed up because `git` returns path relative to the working directory, which produces bad paths when it is [concatenated with the git root](https://github.com/pantsbuild/pants/blob/c9f3af460a7504e082931a4b5a668b66f0cd4ed0/src/python/pants/scm/git.py#L161).

    ### Solution
    By adding the `--full-name` tag to [`git ls-files`](https://github.com/pantsbuild/pants/blob/c9f3af460a7504e082931a4b5a668b66f0cd4ed0/src/python/pants/scm/git.py#L178), the file concatenation is done properly

    ### Result
    Doing `./pants list --changed...` will concatenate file names correctly when it is not run from the git root.

build-support/known_py3_failures.txt

@@ -1,55 +1,9 @@
-tests/python/pants_test/backend/codegen/antlr/java:java
-tests/python/pants_test/backend/codegen/antlr/python:python
-tests/python/pants_test/backend/codegen/jaxb:jaxb
 tests/python/pants_test/backend/codegen/thrift/java:java
 tests/python/pants_test/backend/codegen/thrift/python:python
-tests/python/pants_test/backend/docgen/targets:wiki_page
-tests/python/pants_test/backend/graph_info/tasks:cloc
-tests/python/pants_test/backend/graph_info/tasks:dependees
-tests/python/pants_test/backend/graph_info/tasks:filemap
-tests/python/pants_test/backend/graph_info/tasks:filter
-tests/python/pants_test/backend/graph_info/tasks:listtargets
-tests/python/pants_test/backend/graph_info/tasks:minimal_cover
-tests/python/pants_test/backend/graph_info/tasks:paths
-tests/python/pants_test/backend/graph_info/tasks:sorttargets
-tests/python/pants_test/backend/jvm/subsystems:custom_scala
-tests/python/pants_test/backend/jvm/targets:jvm_app
-tests/python/pants_test/backend/jvm/targets:jvm_binary
-tests/python/pants_test/backend/jvm/tasks/reports:reports
-tests/python/pants_test/backend/jvm/tasks:benchmark_run
-tests/python/pants_test/backend/jvm/tasks:binary_create
-tests/python/pants_test/backend/jvm/tasks:bootstrap_jvm_tools
-tests/python/pants_test/backend/jvm/tasks:bundle_create
-tests/python/pants_test/backend/jvm/tasks:check_published_deps
-tests/python/pants_test/backend/jvm/tasks:checkstyle
-tests/python/pants_test/backend/jvm/tasks:classmap
-tests/python/pants_test/backend/jvm/tasks:consolidate_classpath
 tests/python/pants_test/backend/jvm/tasks:coursier_resolve
-tests/python/pants_test/backend/jvm/tasks:detect_duplicates
-tests/python/pants_test/backend/jvm/tasks:ivy_imports
-tests/python/pants_test/backend/jvm/tasks:ivy_resolve
-tests/python/pants_test/backend/jvm/tasks:ivy_utils
-tests/python/pants_test/backend/jvm/tasks:jar_create
-tests/python/pants_test/backend/jvm/tasks:jar_publish
-tests/python/pants_test/backend/jvm/tasks:jar_task
 tests/python/pants_test/backend/jvm/tasks:junit_run
-tests/python/pants_test/backend/jvm/tasks:jvm_platform_analysis
-tests/python/pants_test/backend/jvm/tasks:jvm_run
-tests/python/pants_test/backend/jvm/tasks:jvm_task
-tests/python/pants_test/backend/jvm/tasks:jvmdoc_gen
 tests/python/pants_test/backend/jvm/tasks:resources_task
-tests/python/pants_test/backend/jvm/tasks:scalafmt
-tests/python/pants_test/backend/jvm/tasks:scalastyle
-tests/python/pants_test/backend/jvm/tasks:unpack_jars
-tests/python/pants_test/backend/native/subsystems:subsystems
-tests/python/pants_test/backend/native/tasks:tasks
-tests/python/pants_test/backend/project_info/tasks:dependencies
-tests/python/pants_test/backend/project_info/tasks:depmap
-tests/python/pants_test/backend/project_info/tasks:export
-tests/python/pants_test/backend/project_info/tasks:filedeps
-tests/python/pants_test/backend/python/tasks:isort_run
 tests/python/pants_test/backend/python/tasks:tasks
-tests/python/pants_test/backend/python:interpreter_cache
 tests/python/pants_test/binaries:binaries
 tests/python/pants_test/build_graph:build_file_parser
 tests/python/pants_test/build_graph:build_graph

src/python/pants/backend/graph_info/tasks/cloc.py

@@ -87,7 +87,7 @@ def console_output(self, targets):
       [exec_result.output_directory_digest]
     )[0].dependencies
 
-    files_content = {fc.path: fc.content for fc in files_content_tuple}
+    files_content = {fc.path: fc.content.decode('utf-8') for fc in files_content_tuple}
     for line in files_content['report'].split('\n'):
       yield line
 

src/python/pants/backend/jvm/argfile.py

@@ -53,7 +53,7 @@ def create_argfile(f):
         if delete and os.path.exists(argfile):
           os.unlink(argfile)
     else:
-      with temporary_file(cleanup=delete) as fp:
+      with temporary_file(cleanup=delete, binary_mode=False) as fp:
         yield create_argfile(fp)
   else:
     yield args

src/python/pants/backend/jvm/ivy_utils.py

@@ -539,8 +539,8 @@ def __eq__(self, other):
   # TODO(python3port): Return NotImplemented if other does not have attributes
   def __lt__(self, other):
     # We can't just re-use __repr__ or __str_ because we want to order rev last
-    return ((self.org, self.name, self.classifier, self.ext, self.rev) <
-            (other.org, other.name, other.classifier, other.ext, other.rev))
+    return ((self.org, self.name, self.classifier or '', self.ext, self.rev) <
+            (other.org, other.name, other.classifier or '', other.ext, other.rev))
 
   def __hash__(self):
     return hash(self._id)

src/python/pants/backend/jvm/targets/jvm_binary.py

@@ -238,9 +238,9 @@ def rules(self):
 
   def fingerprint(self):
     hasher = sha1()
-    hasher.update(self.payload.fingerprint())
+    hasher.update(self.payload.fingerprint().encode('utf-8'))
     for rule in self.rules:
-      hasher.update(rule.fingerprint())
+      hasher.update(rule.fingerprint().encode('utf-8'))
     return hasher.hexdigest() if PY3 else hasher.hexdigest().decode('utf-8')
 
   @property

src/python/pants/backend/jvm/tasks/bootstrap_jvm_tools.py

@@ -46,15 +46,15 @@ def compute_fingerprint(self, target):
     if base_fingerprint is None:
       return None
 
-    hasher.update('version=2')
+    hasher.update(b'version=2')
     hasher.update(base_fingerprint.encode('utf-8'))
 
     # NB: this series of updates must always cover the same fields that populate `_tuple`'s slots
     # to ensure proper invalidation.
-    hasher.update(self._main)
+    hasher.update(self._main.encode('utf-8'))
     if self._custom_rules:
       for rule in self._custom_rules:
-        hasher.update(rule.render())
+        hasher.update(rule.render().encode('utf-8'))
 
     return hasher.hexdigest() if PY3 else hasher.hexdigest().decode('utf-8')
 

src/python/pants/backend/jvm/tasks/coursier_resolve.py

@@ -723,7 +723,7 @@ def compute_fingerprint(self, target):
     hasher.update(target.payload.fingerprint().encode('utf-8'))
 
     for conf in self._confs:
-      hasher.update(conf)
+      hasher.update(conf.encode('utf-8'))
 
     for element in hash_elements_for_target:
       hasher.update(element.encode('utf-8'))

src/python/pants/backend/jvm/tasks/coverage/cobertura.py

@@ -219,7 +219,7 @@ def instrument(self, output_dir):
       for pattern in self._exclude_classes:
         args += ["--excludeClasses", pattern]
 
-      with temporary_file() as tmp_file:
+      with temporary_file(binary_mode=False) as tmp_file:
         tmp_file.write("\n".join(unique_files))
         tmp_file.flush()
 

src/python/pants/backend/jvm/tasks/jar_publish.py

@@ -876,7 +876,7 @@ def exportable(tgt):
 
   def entry_fingerprint(self, target, fingerprint_internal):
     sha = hashlib.sha1()
-    sha.update(target.invalidation_hash())
+    sha.update(target.invalidation_hash().encode('utf-8'))
 
     # TODO(Tejal Desai): pantsbuild/pants/65: Remove java_sources attribute for ScalaLibrary
     if isinstance(target, ScalaLibrary):
@@ -892,7 +892,7 @@ def entry_fingerprint(self, target, fingerprint_internal):
     internal_dependencies = sorted(target_internal_dependencies(target), key=lambda t: t.id)
     for internal_target in internal_dependencies:
       fingerprint = fingerprint_internal(internal_target)
-      sha.update(fingerprint)
+      sha.update(fingerprint.encode('utf-8'))
 
     return sha.hexdigest() if PY3 else sha.hexdigest().decode('utf-8')
 

src/python/pants/backend/jvm/tasks/jvm_platform_analysis.py

@@ -138,7 +138,7 @@ class PlatformFingerprintStrategy(FingerprintStrategy):
     def compute_fingerprint(self, target):
       hasher = sha1()
       if hasattr(target, 'platform'):
-        hasher.update(str(tuple(target.platform)))
+        hasher.update(str(tuple(target.platform)).encode('utf-8'))
       return hasher.hexdigest() if PY3 else hasher.hexdigest().decode('utf-8')
 
     def __eq__(self, other):

src/python/pants/backend/native/targets/native_artifact.py

@@ -29,4 +29,5 @@ def as_shared_lib(self, platform):
 
   def _compute_fingerprint(self):
     # FIXME: can we just use the __hash__ method here somehow?
-    return sha1(self.lib_name).hexdigest() if PY3 else sha1(self.lib_name).hexdigest().decode('utf-8')
+    hasher = sha1(self.lib_name.encode('utf-8'))
+    return hasher.hexdigest() if PY3 else hasher.hexdigest().decode('utf-8')

src/python/pants/backend/python/tasks/pytest_run.py

@@ -240,7 +240,7 @@ def _cov_setup(self, workdirs, coverage_morfs, src_to_target_base):
     # Note that it's important to put the tmpfile under the workdir, because pytest
     # uses all arguments that look like paths to compute its rootdir, and we want
     # it to pick the buildroot.
-    with temporary_file(root_dir=workdirs.root_dir) as fp:
+    with temporary_file(root_dir=workdirs.root_dir, binary_mode=False) as fp:
       cp.write(fp)
       fp.close()
       coverage_rc = fp.name

src/python/pants/backend/python/tasks/python_execution_task_base.py

@@ -7,6 +7,7 @@
 import os
 from builtins import str
 
+from future.utils import binary_type, text_type
 from pex.interpreter import PythonInterpreter
 from pex.pex import PEX
 
@@ -46,7 +47,7 @@ def extra_requirements(self):
     """
     return ()
 
-  class ExtraFile(datatype(['path', 'content'])):
+  class ExtraFile(datatype([('path', text_type), ('content', binary_type)])):
     """Models an extra file to place in a PEX."""
 
     @classmethod
@@ -56,7 +57,7 @@ def empty(cls, path):
       :param str path: The path this extra file should have when added to a PEX.
       :rtype: :class:`ExtraFile`
       """
-      return cls(path=path, content='')
+      return cls(path=path, content=b'')
 
     def add_to(self, builder):
       """Adds this extra file to a PEX builder.

src/python/pants/backend/python/tasks/select_interpreter.py

@@ -34,7 +34,7 @@ def compute_fingerprint(self, python_target):
       return None
     hasher = hashlib.sha1()
     for element in hash_elements_for_target:
-      hasher.update(element)
+      hasher.update(element.encode('utf-8'))
     return hasher.hexdigest() if PY3 else hasher.hexdigest().decode('utf-8')
 
 

src/python/pants/backend/python/tasks/setup_py.py

@@ -13,7 +13,7 @@
 from builtins import map, object, str, zip
 from collections import OrderedDict, defaultdict
 
-from pex.compatibility import string, to_bytes
+from future.utils import PY2, PY3
 from pex.installer import InstallerBase, Packager
 from pex.interpreter import PythonInterpreter
 from twitter.common.collections import OrderedSet
@@ -300,7 +300,7 @@ def collect_reduced_dependencies(current):
 class SetupPy(Task):
   """Generate setup.py-based Python projects."""
 
-  SOURCE_ROOT = b'src'
+  SOURCE_ROOT = 'src' if PY3 else b'src'
 
   PYTHON_DISTS_PRODUCT = 'python_dists'
 
@@ -514,18 +514,23 @@ def write_setup(self, root_target, reduced_dependencies, chroot):
 
     setup_keywords = root_target.provides.setup_py_keywords.copy()
 
-    package_dir = {b'': self.SOURCE_ROOT}
+    package_dir = {'' if PY3 else b'': self.SOURCE_ROOT}
     packages, namespace_packages, resources = self.find_packages(chroot, self.context.log)
 
     if namespace_packages:
       setup_keywords['namespace_packages'] = list(sorted(namespace_packages))
 
     if packages:
+      normalized_package_data = (
+        resources.items()
+        if PY3 else
+        ((package.encode('utf-8'), [v.encode('utf-8') for v in rs])
+         for (package, rs) in resources.items())
+      )
       setup_keywords.update(
           package_dir=package_dir,
           packages=list(sorted(packages)),
-          package_data=dict((str(package), list(map(str, rs)))
-                            for (package, rs) in resources.items()))
+          package_data=dict(normalized_package_data))
 
     setup_keywords['install_requires'] = list(self.install_requires(reduced_dependencies))
 
@@ -546,8 +551,8 @@ def convert(input):
         return out
       elif isinstance(input, list):
         return [convert(element) for element in input]
-      elif isinstance(input, string):
-        return to_bytes(input)
+      elif PY2 and isinstance(input, str):
+        return input.encode('utf-8')
       else:
         return input
 
@@ -571,7 +576,7 @@ def convert(input):
     chroot.write(self._setup_boilerplate().format(
       setup_dict=pprint.pformat(convert(setup_keywords), indent=4),
       setup_target=repr(root_target)
-    ), 'setup.py')
+    ).encode('utf-8'), 'setup.py')
 
     # make sure that setup.py is included
     chroot.write('include *.py'.encode('utf8'), 'MANIFEST.in')

src/python/pants/build_graph/app_base.py

@@ -174,7 +174,7 @@ class BundleField(tuple, PayloadField):
   @staticmethod
   def _hash_bundle(bundle):
     hasher = sha1()
-    hasher.update(bundle.rel_path)
+    hasher.update(bundle.rel_path.encode('utf-8'))
     for abs_path in sorted(bundle.filemap.keys()):
       buildroot_relative_path = os.path.relpath(abs_path, get_buildroot()).encode('utf-8')
       hasher.update(buildroot_relative_path)

src/python/pants/build_graph/intermediate_target_factory.py

@@ -7,7 +7,7 @@
 from builtins import str
 from hashlib import sha1
 
-from future.utils import string_types
+from future.utils import PY3, string_types
 
 from pants.base.exceptions import TargetDefinitionException
 from pants.build_graph.address import Address
@@ -16,9 +16,9 @@
 
 def hash_target(address, suffix):
   hasher = sha1()
-  hasher.update(address)
-  hasher.update(suffix)
-  return hasher.hexdigest()
+  hasher.update(address.encode('utf-8'))
+  hasher.update(suffix.encode('utf-8'))
+  return hasher.hexdigest() if PY3 else hasher.hexdigest().decode('utf-8')
 
 
 class IntermediateTargetFactoryBase(AbstractClass):

src/python/pants/java/jar/manifest.py

@@ -25,7 +25,7 @@ def _wrap(text):
         chunk = fp.read(69)
         if not chunk:
           return
-        yield ' {}'.format(chunk)
+        yield b' ' + chunk
 
   PATH = 'META-INF/MANIFEST.MF'
 

src/python/pants/task/console_task.py

@@ -29,7 +29,7 @@ def register_options(cls, register):
 
   def __init__(self, *args, **kwargs):
     super(ConsoleTask, self).__init__(*args, **kwargs)
-    self._console_separator = self.get_options().sep.decode('unicode_escape')
+    self._console_separator = self.get_options().sep.encode('ascii').decode('unicode_escape')
     if self.get_options().output_file:
       try:
         self._outstream = safe_open(os.path.abspath(self.get_options().output_file), 'wb')
@@ -55,7 +55,7 @@ def execute(self):
         targets = self.context.targets()
         for value in self.console_output(targets) or tuple():
           self._outstream.write(value.encode('utf-8'))
-          self._outstream.write(self._console_separator.encode('utf-8'))
+          self._outstream.write(self._console_separator.encode('ascii'))
       finally:
         self._outstream.flush()
         if self.get_options().output_file: