Support pinecone-client to 3.0+

[Taragolis]

Body

Originally pinecone-client limited to <3.0 in #36818 because it not compatible with current version of provider.

We need to have eventually support of SDK v3, the Guidance which might help to migrate Pinecone Provider to this version of client: https://canyon-quilt-082.notion.site/Pinecone-Python-SDK-v3-0-0-Migration-Guide-056d3897d7634bf7be399676a4757c7b

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[rawwar]

I will be working on this. Thanks

[rawwar]

UPDATE: I will be going with approach 2 and give examples in the documentation. If you do have feedback, please comment. Thanks!

In the 3.x client, creating index signature has changed. Now, they use podSpec or serverlessSpec classes to pass configurations w.r.t infra. I've been thinking how to rewrite the hook and am split between two ways and need help in finalizing which one to go with.

Approach 1:

will have two methods in PineconeHook namely create_pod_index and create_serverless_index which will take all possible parameters w.r.t index and the spec and internally will create the Spec object and pass it to the create_index method

method will be as follows for the pod based index creation

def create_pod_index(
        self,
        index_name: str,
        dimension: int,
        environment: str,
        metric: str | None = "cosine",
        replicas: int | None = 1,
        shards: int | None = 1,
        pods: int | None = 1,
        pod_type: str | None = "p1.x1",
        metadata_config: dict[str, str] | None = None,
        source_collection: str | None = "",
        timeout: int | None = None,
    ) -> None:
        pod_spec = PodSpec(
            environment=environment,
            replicas=replicas,
            shards=shards,
            pods=pods,
            pod_type=pod_type,
            metadata_config=metadata_config,
            source_collection=source_collection,
        )
        
        self.conn.create_index(
            name=index_name,
            dimension=dimension,
            spec=pod_spec
            metric=metric,
            timeout=timeout,
        )

Approach 2:

Will have helper methods to create PodSpec and ServerlessSpec objects and then have the create_index method follow the pinecone signature and accept only one of these two types.

def get_pod_spec_obj(self,environment, replicas, shards, pods, pod_type, metadata_config, source_collection):
        return PodSpec(
            environment=environment,
            replicas=replicas,
            shards=shards,
            pods=pods,
            pod_type=pod_type,
            metadata_config=metadata_config,
            source_collection=source_collection,
        )

def create_index(self, name ,dimension,spec,metric,timeout):
    return self.conn.create_index(name=name,dimension=dimension,spec=spec,metric=metric,timeout=timeout)

[Taragolis]

@rawwar difficult to say which approach are better. I can't find any usage into the operators, so assume this might only use into the custom operators or taskflow operators, so maybe create method create_index_from_spec?

[rawwar]

I am picking this back up.