ARROW-2611: [Python] Fix Python 2 integer serialization #2055

pschafhalter · 2018-05-17T00:26:29Z

Fixes an issue where serialization turns integers into longs in Python 2.

In [1]: import pyarrow as pa

In [2]: value = 1

In [3]: type(value)
Out[3]: int

In [4]: serialized = pa.serialize(value)

In [5]: deserialized = serialized.deserialize()

In [6]: type(deserialized)
Out[6]: long

pcmoritz

+1 LGTM

robertnishihara · 2018-05-17T03:25:40Z

cpp/src/arrow/python/arrow_to_python.cc

+        *result = PyInt_FromSsize_t(checked_cast<const Int64Array&>(arr).Value(index));
+        return Status::OK();
+      }
+#endif


I think it's slightly clearer to do

#if PY_MAJOR_VERSION < 3 // handle Py2 case #else // handle Py3 case #endif

instead of

#if PY_MAJOR_VERSION < 3 // handle Py2 case #endif // handle Py3 case

There is only a Python 2 case here, the "Python 3" case needs to be handled in both Python 2 and Python 3 (in Python 2 there is both int and long).

I see, actually, now that I think about it, we don't need the #if, right?

It's needed to get it to compile, since in Python 3 the function PyInt_FromSsize_t doesn't exist. Arrow has some common code to handle differences like this but I didn't find anything that covers this function.

Ok, sounds good.

pitrou

I think this PR should simply touch the deserialization path without creating a distinct serialization format for int (as opposed to long).

pitrou · 2018-05-17T13:20:37Z

cpp/src/arrow/python/python_to_arrow.cc

@@ -481,7 +490,7 @@ Status Append(PyObject* context, PyObject* elem, SequenceBuilder* builder,
    }
 #if PY_MAJOR_VERSION < 3
  } else if (PyInt_Check(elem)) {
-    RETURN_NOT_OK(builder->AppendInt64(static_cast<int64_t>(PyInt_AS_LONG(elem))));
+    RETURN_NOT_OK(builder->AppendPy2Int64(static_cast<int64_t>(PyInt_AS_LONG(elem))));


I don't think this it the right approach. int and long are pretty much interchangeable in Python 2. So only the deserialization needs to be changed and attempt to build an int before falling back on long.

An issue with this approach is that a serialized long with a small numeric value will be deserialized as an int which could cause issues for applications relying on type checking.

For example:

In [9]: value = 0L In [10]: type(value) Out[10]: long In [11]: serialized = pa.serialize(value) In [12]: deserialized = serialized.deserialize() In [13]: type(deserialized) Out[13]: int

Accepting only long and refusing int sounds like a weird thing to do, though. Can you point an API that works this way?
(there are rare APIs that do the reverse - accept only int and reject long - but usually it's because they were written in C and the programmer couldn't care more)

Sorry, I don't know one off the top of my head -- I agree that it's bad practice. I do think that being consistent with types is important from a correctness point of view.
Pickle, for reference, does not deserialize a small long as an int.

That's true. Let's go with this patch, then. Hopefully we'll ditch Python 2 compatibility soon ;-)

PR is good enough after all

pcmoritz · 2018-05-17T22:04:53Z

Great, thanks for the discussion and @pschafhalter for getting this to work, I'll merge it then :)

[Retriggering the build first]

codecov-io · 2018-05-18T00:02:36Z

Codecov Report

Merging #2055 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2055      +/-   ##
==========================================
+ Coverage   86.28%   86.28%   +<.01%     
==========================================
  Files         242      242              
  Lines       41034    41041       +7     
==========================================
+ Hits        35405    35412       +7     
  Misses       5629     5629

Impacted Files	Coverage Δ
cpp/src/arrow/python/arrow_to_python.cc	`91.71% <100%> (+0.2%)`	⬆️
python/pyarrow/tests/test_serialization.py	`90.33% <100%> (ø)`	⬆️
cpp/src/arrow/python/python_to_arrow.cc	`88.29% <100%> (+0.09%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd54041...7b96b67. Read the comment docs.

wesm · 2018-05-18T14:10:22Z

Looking at this late. This fix seems odd to me but I haven't looked closely enough at the nuances to judge for certain.

Would it be possible to have only a single int64 array in the serialized representation and then convert back to either PyInt or PyLong depending on the size of the value? In Python 2 it seems like int would always be a safe type if the value fits. If this sounds reasonable perhaps we can open a JIRA to investigate further another time

pschafhalter · 2018-05-18T23:02:50Z

I think the discussion with @pitrou touched on this.

The main issue with converting to PyInt or PyLong depending on the size of the value is that if I serialize a long, I expect to receive a long on deserialization. In Python 2, small values may have type long, e.g. type(0L) == long, but choosing PyInt or PyLong depending on the size of the value would deserialize 0L as an int.

Fixes an issue where serialization turns integers into longs in Python 2. ```python In [1]: import pyarrow as pa In [2]: value = 1 In [3]: type(value) Out[3]: int In [4]: serialized = pa.serialize(value) In [5]: deserialized = serialized.deserialize() In [6]: type(deserialized) Out[6]: long ``` Author: Peter Schafhalter <[email protected]> Closes apache#2055 from pschafhalter/fix-python2-int-serialization and squashes the following commits: 7b96b67 <Peter Schafhalter> Fix bug with Python 3 C++ API 5d8ff26 <Peter Schafhalter> Add type checking in assert_equal d5e5e5d <Peter Schafhalter> Fix python2 integer serialization bug

pschafhalter added 2 commits May 16, 2018 17:00

Fix python2 integer serialization bug

d5e5e5d

Add type checking in assert_equal

5d8ff26

pschafhalter mentioned this pull request May 17, 2018

Ray serialization turns ints into longs in Python 2. ray-project/ray#1014

Closed

pcmoritz approved these changes May 17, 2018

View reviewed changes

robertnishihara reviewed May 17, 2018

View reviewed changes

pitrou previously requested changes May 17, 2018

View reviewed changes

Fix bug with Python 3 C++ API

7b96b67

pcmoritz force-pushed the fix-python2-int-serialization branch from e2adf74 to 7b96b67 Compare May 17, 2018 22:10

pcmoritz changed the title ~~[Python] Fix Python 2 integer serialization~~ ARROW-2611: [Python] Fix Python 2 integer serialization May 18, 2018

pcmoritz closed this in 3d41978 May 18, 2018

pschafhalter mentioned this pull request May 18, 2018

Update to latest arrow? ray-project/ray#2084

Closed

pschafhalter deleted the fix-python2-int-serialization branch May 18, 2018 23:03

pcmoritz mentioned this pull request May 19, 2018

Update arrow to latest master ray-project/ray#2100

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-2611: [Python] Fix Python 2 integer serialization #2055

ARROW-2611: [Python] Fix Python 2 integer serialization #2055

pschafhalter commented May 17, 2018

pcmoritz left a comment

robertnishihara May 17, 2018

pcmoritz May 17, 2018

robertnishihara May 17, 2018

pcmoritz May 17, 2018

robertnishihara May 17, 2018

pitrou left a comment

pitrou May 17, 2018

pschafhalter May 17, 2018

pitrou May 17, 2018

pschafhalter May 17, 2018

pitrou May 17, 2018

pcmoritz commented May 17, 2018 •

edited

Loading

codecov-io commented May 18, 2018

wesm commented May 18, 2018

pschafhalter commented May 18, 2018

ARROW-2611: [Python] Fix Python 2 integer serialization #2055

ARROW-2611: [Python] Fix Python 2 integer serialization #2055

Conversation

pschafhalter commented May 17, 2018

pcmoritz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz commented May 17, 2018 • edited Loading

codecov-io commented May 18, 2018

Codecov Report

wesm commented May 18, 2018

pschafhalter commented May 18, 2018

pcmoritz commented May 17, 2018 •

edited

Loading