Index.to_frame() seems not working properly.

[itholic]

>>> kdf = ks.DataFrame({"Koalas": [1, 2, 3]}, index=pd.Index([1, 2, 3]))
>>> kdf["NEW"] = ks.Series([100, 200, 300])
>>> kdf
   0    NEW
1  1  200.0
3  3    NaN
2  2  300.0

The above code is working well.

But the same shape of DataFrame which is made from Index.to_frame() seems not work properly.

>>> kdf = ks.Index([1, 2, 3]).to_frame(name="Koalas")
>>> kdf["NEW"] = ks.Series([100, 200, 300])
>>> kdf
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: Reference 'Koalas' is ambiguous, could be: Koalas, Koalas.;

[itholic]

Let me fix this for completing #1630

[ueshin]

What's the problem?

[itholic]

I thought we should always manage the data_spark_columns and index_spark_columns separately because the internal Spark columns should be changed when the external Koalas columns are changed.

For example,

>>> kdf = ks.DataFrame({"Koalas": [1, 2, 3]}, index=pd.Index([1, 2, 3]))
>>> kdf["NEW"] = ks.Series([100, 200, 300])
>>> kdf
   0    NEW
1  1  200.0
3  3    NaN
2  2  300.0

The above code is working well.

But the same shape of DataFrame which is made from Index.to_frame() seems not work properly.

>>> kdf = ks.Index([1, 2, 3]).to_frame(name="Koalas")
>>> kdf["NEW"] = ks.Series([100, 200, 300])
>>> kdf
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: Reference 'Koalas' is ambiguous, could be: Koalas, Koalas.;

Hmm.. but probably It seems not a problem related with internal Spark frame... let me investigate this more carefully

[itholic]

I just changed the title and description of this issue.

[itholic]

Anyway, could I have some additional question about InternalFrame that I often feel confusing?

Don't we need to address the values of internal Spark columns after modifying the external values ?

For example, let's say we have a Series like the below.

>>> kser
0    1
1    2
2    3
Name: 0, dtype: int64

>>> kser._internal.spark_frame.show()
+-----------------+---+-----------------+
|__index_level_0__|  0|__natural_order__|
+-----------------+---+-----------------+
|                0|  1|      25769803776|
|                1|  2|      60129542144|
|                2|  3|      94489280512|
+-----------------+---+-----------------+

And let's modify this by filtering with where, and make a new Series named new_kser.

>>> new_kser = kser.where(kser < 2)
>>> new_kser
0    1.0
1    NaN
2    NaN
Name: 0, dtype: float64

>>> new_kser._internal.spark_frame.show()
+-----------------+---+-----------------+
|__index_level_0__|  0|__natural_order__|
+-----------------+---+-----------------+
|                0|  1|      25769803776|
|                1|  2|      60129542144|
|                2|  3|      94489280512|
+-----------------+---+-----------------+

Even though the new_kser has the values of [1.0, NaN, Nan], the internal Spark frame has the values of [1, 2, 3] which one from original kser.

Isn't there any problem with such behaviour because the InternalFrame is only used as an internal purpose ??

[ueshin]

InternalFrame doesn't work only with spark_frame but with all the metadata.
The data_spark_columns contains all the changes since the last time spark_frame is created.

Even if spark_frame shows the values of [1, 2, 3], data_spark_columns has the operation of kser.where(kser < 2).

>>> new_kser._internal.spark_frame.show()
+-----------------+---+-----------------+
|__index_level_0__|  0|__natural_order__|
+-----------------+---+-----------------+
|                0|  1|      25769803776|
|                1|  2|      60129542144|
|                2|  3|      94489280512|
+-----------------+---+-----------------+

>>> new_kser._internal.data_spark_columns
[Column<b'CASE WHEN CASE WHEN ((0 < 2) IS NULL) THEN false ELSE (0 < 2) END AS `0` THEN 0 ELSE NaN END AS `0` AS `0`'>]

We always use both to show the actual values.

>>> new_kser._internal.spark_frame.select(new_kser._internal.data_spark_columns).show()
+---+
|  0|
+---+
|1.0|
|NaN|
|NaN|
+---+

[ueshin]

The case of Index.to_frame() you mentioned above is maybe an issue of the function or util.align_diff_frames.
I'll take a look.

[itholic]

@ueshin Oh, now It's clear for me. Thanks !! :D

[itholic]

Close since this is resolved.