API: Add string extension type

[TomAugspurger]

This adds a new extension type 'string' for storing string data.

The data model is essentially unchanged from master. String are still
stored in an object-dtype ndarray. Scalar elements are still Python
strs, and np.nan is still used as the string dtype.

Things are pretty well contained. The major changes outside the new array are

1. docs
2. core/strings.py to handle things correctly (mostly, returning string dtype when there's a string input.

No rush on reviewing this. Just parking it here for now.

[jbrockmendel]

Does the .str accessor only show up on string-dtype Series, or also object-coincidentally-all-string Series as in master?

[TomAugspurger]

Also on object-dtype as in master.

This PR doesn't change the current behavior at all (modulo new bugs).

[mroeschke]

May be a more suitable question for an issue, but dtype=str will still point to object in this PR and in the future(?)

[TomAugspurger]

dtype=str will still point to object in this PR and in the future(?)

It'll have to for now (in my first commit I allowed dtype='str' as well as dtype='string', but that of course broke things.)

We'll want to think of a migration path for making 'str' mean StringDtype, and probably we'll want to infer ['a', 'b'] as string rather than object.

[TomAugspurger]

I was curious about how much the string validation (ruling out pd.array(['a', 1], dtype="string") cost.

relative:

absolute:

code

from timeit import default_timer as tic
import pandas as pd
import numpy as np
import pandas.util.testing as tm

ns = [0, 10, 100, 1_000, 10_000, 100_000, 1_000_000]
times = []
for n in ns:
    data = tm.makeStringIndex(n).tolist()
    t0 = tic()
    np.array(data, dtype=object)
    t1 = tic()
    pd.core.arrays.StringArray._from_sequence(data)
    t2 = tic()
    
    times.append((n, t1 - t0, t2 - t1))

df = pd.DataFrame(times, columns=['n', 'numpy', 'pandas']).set_index('n')
df.div(df['numpy'], axis=0).plot(
    logx=True, title="Overhead Relative to NumPy"
);

I believe there are two sources of relative slowdown

1. per-value validation that the scalars are strings or NA
2. Ensuring that we have a consistent NA value (no mix of None and np.nan. Right now, this is done with an isna() (linear scan over the values) and a setitem if needed. I suspect that this could be rolled into the per-value scalar validation if desired, but I'm less concerned about perf right now.

[TomAugspurger]

(CI is passing now).

[jbrockmendel]

are you looking to get this merged quickly? i.e. can this go on the "review later this week" pile or does it belong in the "later today" pile?

[TomAugspurger]

No. This can go in the "when I'm bored" pile. I don't think there's a huge rush. It'd be nice for 1.0 but not essential.

…

On Mon, Aug 19, 2019 at 3:43 PM jbrockmendel ***@***.***> wrote: are you looking to get this merged quickly? i.e. can this go on the "review later this week" pile or does it belong in the "later today" pile? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#27949?email_source=notifications&email_token=AAKAOIULOTZCKS2XW7YFRHDQFMAYPA5CNFSM4IMKKYTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4UHSDI#issuecomment-522746125>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIXGPYKMAKIMN55YAO3QFMAYPANCNFSM4IMKKYTA> .

[jreback]

is the intention to have this have an encoding (maybe parameterized)? or is that a unicode type? IOW shouldn't this be string[utf8] ? (and what you have here is a base class for this), we could certianly default dtype='string' -> string[utf8].

also rebase when you have a chance, have only glanced, but looks pretty good so far.

[TomAugspurger]

is the intention to have this have an encoding (maybe parameterized)? or is that a unicode type?

Mmm, this type is for in-memory strings, so I don't think an encoding is necessary. I don't think we're exposing the actual storage anywhere, which is when the encoding would matter.

If we had a ByteArray, then yes I think that should be parametrized by encoding (with UTF8 as the default).

[pep8speaks]

Hello @TomAugspurger! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-10-04 14:03:39 UTC

[WillAyd]

For the sake of clarity this doesn't offer any real speed or memory savings over the object dtype right? Just a better way to ensure that an array only contains strings?

[jorisvandenbossche]

I am a bit biased as I have clear preference, but trying to summarize arguments for strings vs text:

Reasons to go with "string":

• It matches our own existing terminology (string methods with str accessor)
• It matches Python terminology (str type, the scalar value of this array is a python string)
• It matches what some other libraries / languages do (Arrow, Julia (they also have a Char), C++, ..; R uses character)

Reasons to go with "text":

• A less subtle difference with "dtype=str" (which for now keeps giving the object dtype)
• .. other?

SQL has VARCHAR (so also character), Postgres also has a non-SQL-standard TEXT type (but the equivalent of what we are building here is not TEXT but VARCHAR, AFAIU)

i think would be more amenable to (and alias of string) if we start showing. FutureWarning if passing ‘str’ as a dtype (in preparation for the switch)

IMO, we shouldn't do that now, let's first get some experience with it as an opt-in experimental feature.
But I think we agree the long term goal is to change "str" to mean this StringDtype at some point? For me, that is another reason to go with "string" now, as that will cause less confusion in the future (at some point, "str" and "string" can be aliases then).

[jreback]

yep i’ll retract my support for Text and go with String

but i think we need to make it very clear that for now str is not the same as string

[TomAugspurger]

Yep, I think we'll want dtype='str' to mean this new EA dtype in the future.

And we'll likely want to deprecate .str on object-dtype columns. But that's down the road.

I'll make the switch back to StringDtype / StringArray.

[jreback]

does this pass if you passing a StringArray itself? (and do we infer correctly in is_string_array)?

[TomAugspurger]

does this pass if you passing a StringArray itself?

Done.

and do we infer correctly in is_string_array

We (Cython) actually raises on lib.is_string_array(StringArray) since it's expecting an ndarray. Not sure what's best here. IIUC, we only use len(values) and values.dtype.

[jreback]

does arr.dtype work here?

[TomAugspurger]

Not always: #27953

[jreback]

thanks @TomAugspurger very nice. IIRC there are a couple of followups.

[jreback]

did this have an issue to close?

[TomAugspurger]

There's the main String Dtype issue: #8640. Leaving that open for part 2, which is storing strings natively, rather than as Python objects.

…

On Sat, Oct 5, 2019 at 6:18 PM Jeff Reback ***@***.***> wrote: did this have an issue to close? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27949?email_source=notifications&email_token=AAKAOIRXO5WVDNVFDSNWZILQNEOE3A5CNFSM4IMKKYTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAN53TA#issuecomment-538697164>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIRFZU7SQIODB5URCYDQNEOE3ANCNFSM4IMKKYTA> .