Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pgvector & :copyfrom command not working #3655

Open
embiem opened this issue Oct 15, 2024 · 0 comments
Open

pgvector & :copyfrom command not working #3655

embiem opened this issue Oct 15, 2024 · 0 comments
Labels

Comments

@embiem
Copy link

embiem commented Oct 15, 2024

Version

1.27.0

What happened?

The pgvector support introduced in v1.23.0 doesn't seem to play nicely with the :copyfrom command. I get the following error with the setup described in this issue: ERROR: vector cannot have more than 16000 dimensions (SQLSTATE 54000). Even though the inserted pgvector has the dimension of 1024, same as the column.

When I use the following query using :exec with that same vector, then it works without problems:

-- name: CreateEmbedding :exec
INSERT INTO "embeddings"."embeddings_1024" (node_id, embedding)
VALUES (
  $1, $2
);

Relevant log output

ERROR: vector cannot have more than 16000 dimensions (SQLSTATE 54000)

Database schema

CREATE EXTENSION IF NOT EXISTS vector;

CREATE SCHEMA IF NOT EXISTS "embeddings";

CREATE TABLE "embeddings"."embeddings_1024" (
    node_id BIGINT PRIMARY KEY,
    embedding VECTOR(1024)
);

SQL queries

-- name: CreateEmbeddings :copyfrom
INSERT INTO "embeddings"."embeddings_1024" (node_id, embedding)
VALUES (
  $1, $2
);

Configuration

version: "2"
sql:
  - engine: "postgresql"
    queries: "db/query.sql"
    schema: "db/migrations"
    gen:
      go:
        package: "data"
        out: "data"
        sql_package: "pgx/v5"

Playground URL

can't use pgvector

Alternatively, here is the docker-compose.yml I use for local dev to have pgvector installed:

version: "3.9"

services:
  db:
    image: pgvector/pgvector:pg16
    restart: on-failure
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"

  adminer:
    image: ghcr.io/shyim/adminerevo:latest
    restart: on-failure
    ports:
      - 8080:8080
volumes:
  db_data:

What operating system are you using?

Linux

What database engines are you using?

PostgreSQL

What type of code are you generating?

Go

Workaround

The workaround I'm using to still insert many vectors at once is the :batchexec command, which works fine with pgvector.

@embiem embiem added the bug Something isn't working label Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant