You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When running DataFrame.Collect() a long value in the dataset (even with a proper schema set to LongType) can be collected as an int.
To Reproduce
This reproduces the issue:
longvalue=1;stringfieldName="SomeLongValue";vardf= _spark.CreateDataFrame(new[]{new GenericRow(newobject[]{ value }),},new StructType(new[]{new StructField(fieldName,new LongType(),false)}));varcollected= df.Collect().First().Get(fieldName);
Assert.Equal(value, collected);// this fails since `value` is a long whereas `collected` is an int
On the other hand, this test would run fine:
longvalue=long.MaxValue;stringfieldName="SomeLongValue";vardf= _spark.CreateDataFrame(new[]{new GenericRow(newobject[]{ value }),},new StructType(new[]{new StructField(fieldName,new LongType(),false)}));varcollected= df.Collect().First().Get(fieldName);
Assert.Equal(value, collected);// this works
It might come from Unpickling or from Spark -> Spark-Python, I have no idea.
Expected behavior
When the Schema specifies a column is of LongType, the collected value should be a long
Desktop (please complete the following information):
OS: Windows 11
Version: 2.1.1
The text was updated successfully, but these errors were encountered:
Describe the bug
When running
DataFrame.Collect()
a long value in the dataset (even with a proper schema set toLongType
) can be collected as anint
.To Reproduce
This reproduces the issue:
On the other hand, this test would run fine:
It might come from Unpickling or from Spark -> Spark-Python, I have no idea.
Expected behavior
When the Schema specifies a column is of
LongType
, the collected value should be along
Desktop (please complete the following information):
The text was updated successfully, but these errors were encountered: