Shape meta data not cleaned up properly on delete

[kevin-dp]

When we delete a shape we are forgetting to clean up some meta data somewhere because the old schema information re-appears if we later create a new shape for the same table.

To reproduce this problem, first create a table:

CREATE TABLE foo (c1 INT PRIMARY KEY); INSERT INTO foo VALUES (1), (2);

Now, fetch the shape (i'm using HTTPie, can be done with curl too):

$ HTTP GET "localhost:3000/v1/shape/foo" offset==-1
HTTP/1.1 200 OK
access-control-allow-methods: GET, POST, OPTIONS
access-control-allow-origin: *
access-control-expose-headers: *
cache-control: max-age=1, stale-while-revalidate=3
content-type: application/json; charset=utf-8
date: Wed, 21 Aug 2024 12:16:36 GMT
etag: 3833821-1724242596930:-1:0_0
server: ElectricSQL/0.3.3
transfer-encoding: chunked
x-electric-chunk-last-offset: 0_0
x-electric-schema: {"c1":{"type":"int4","not_null":true,"pk_index":0}}
x-electric-shape-id: 3833821-1724242596930
x-request-id: F-29gac4MT1aYc4AAAAk

[
    {
        "headers": {
            "operation": "insert"
        },
        "key": "\"public\".\"foo\"/\"1\"",
        "offset": "0_0",
        "value": {
            "c1": "1"
        }
    },
    {
        "headers": {
            "operation": "insert"
        },
        "key": "\"public\".\"foo\"/\"2\"",
        "offset": "0_0",
        "value": {
            "c1": "2"
        }
    },
    {
        "headers": {
            "control": "up-to-date"
        }
    }
]

Ok, we get the shape with the 2 rows. This is fine.
Now drop the table:

DROP TABLE foo;

Now, tell Electric to delete the shape:

$ HTTP DELETE "localhost:3000/v1/shape/foo"
HTTP/1.1 202 Accepted
cache-control: max-age=0, private, must-revalidate
content-length: 0
content-type: application/json; charset=utf-8
date: Wed, 21 Aug 2024 12:17:49 GMT
server: ElectricSQL/0.3.3
vary: accept-encoding
x-request-id: F-29kphNgnGWR_EAAAAj

Ok, the shape is deleted.
Let's recreate the table but with an extra column:

CREATE TABLE foo (c1 INT PRIMARY KEY, c2 TEXT);
INSERT INTO foo VALUES (9, 'bar');

Let's fetch that table:

$ HTTP GET "localhost:3000/v1/shape/foo" offset==-1
HTTP/1.1 200 OK
access-control-allow-methods: GET, POST, OPTIONS
access-control-allow-origin: *
access-control-expose-headers: *
cache-control: max-age=1, stale-while-revalidate=3
content-type: application/json; charset=utf-8
date: Wed, 21 Aug 2024 12:18:11 GMT
etag: 3833821-1724242692287:-1:0_0
server: ElectricSQL/0.3.3
transfer-encoding: chunked
x-electric-chunk-last-offset: 0_0
x-electric-schema: {"c1":{"type":"int4","not_null":true,"pk_index":0}}
x-electric-shape-id: 3833821-1724242692287
x-request-id: F-29l9xY4smjtZsAAABj

[
    {
        "headers": {
            "operation": "insert"
        },
        "key": "\"public\".\"foo\"/\"9\"",
        "offset": "0_0",
        "value": {
            "c1": "9"
        }
    },
    {
        "headers": {
            "control": "up-to-date"
        }
    }
]

Now, the returned data is wrong as the rows only include the c1 column and not the c2 column (also the schema only includes c1).

[kevin-dp]

This seems to be the reason for the bug:

• when an HTTP request arrives, we create the shape
• Shape.new loads the column info via the inspector: Inspector.load_column_info
• The first time, the inspector loads this information from PG and stores it in ETS
• All subsequent load_column_info calls fetch this information from ETS

Thus, we must ensure that the ETS entry is removed when a table is altered.