gcloud.datastore should allow explicit specification of whether the field is indexed

[jgeewax]

Summary

Right now, everything is indexed by default (based on the Value protobuf's indexed [default = true] field). See https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/v1beta2-rev1-2.1.0/proto/datastore_v1.proto#L129

This will get crazy expensive. We need to allow people to specify whether a field should be indexed. We should probably also default this to false. But that's a bigger question.

---

To do

• Decide whether to allow explicit indexing or unindexing
• Figure out what the syntax will be for specifying indexing
• Submit a patch for the feature

Options for specifying indexing

List of indexed fields

Unless a field is explicitly in the list of indexed fields, the indexed property will be set to False.

entity = ...
entity.indexed_fields = ['field1', 'field2']

List of unindexed fields

If a field is in this list, it's indexed property will be set to False. Otherwise it will be left unset (and therefore default to True per the protobuf definition.

entity = ...
entity.unindexed_fields = ['field1', 'field2']

CC

@proppy @aliafshar

[pcostell]

Pending some work by kdo@, this may not be necessary (and actually we may prefer not to expose this here).

[dhermes]

@pcostell any update on this?

[pcostell]

Still a work in progress. For exposing indexed, I would recommend instead exposing it as it will be in the next Cloud Datastore release, exclude_from_indexes (the opposed of indexed). It's more verbose, but makes it more clear that this removes the property from both built-in and composite indexes. This field should really only be used for implementing sparse indexes or for values which cannot be indexed (i.e. large blobs).

[silvolu]

In gcloud-node, we implemented @pcostell suggestion: for each property you can pass an excludeFromIndexes; if present, we negate the value an pass it as value for indexed. I think we should do the same here.

[tseaver]

It's tricky here, as we just map an Entity as a dict: there is really no obvious place to store the policy. I guess we could add an 'exclude_from_indexes' attribute to Entity (a set of names?), and then have the marshalling code look at that attribute to decide whether / how to set the protobuf indexed field.

[silvolu]

Sounds good. Basically the 2nd option proposed by @jgeewax, using exclude_from_indexes as the name for the property.