Bigtable - issue with table.exists when creating a table then adding a family

[dom-white]

Hi, it's great to see the new Bigtable functionality added, but I am having an issue checking and creating tables.

I can successfully use table.create() and then call table.createFamily() from it's callback function, but I get an error on creation of the 'family' if I have just happened to test whether the table exists first.

Environment details

• OS: Ubuntu 16.04
• Node.js version: v6.1.0
• npm version: 3.8.6
• gcloud-node version: v0.37.0

Steps to reproduce

var table = bigtable.table('prezzy');

table.exists(function(err, exists) {
    if (exists) {
        console.log("Table already exists");
    } else {
        table.create(function(err, table, apiResponse) {
            if (err) {
                console.log("Error from table.create(): " + err);
            } else {
                console.log("Table created, now create family");

                table.createFamily('follows', function(err, family) {
                    if (err) {
                        console.log("Error from table.createFamily(): " + err);
                    } else {
                        console.log("family created");
                    }
                });
            }
        });
    }
});

In this case I get

Table created, now create family
Error from table.createFamily(): Error: Failed to read: projects/{12345678}/instances//clusters/my-cluster/tables/prezzy

If I simply remove the 'exists' test it works (obviously if the table does not exist. I am deleting the table between the tests)

var table = bigtable.table('prezzy');

table.create(function(err, table, apiResponse) {

    if (err) {
        console.log("Error from table.create(): " + err);
    } else {
        console.log("Table created, now create family");

        table.createFamily('follows', function(err, family) {
            if (err) {
                console.log("Error from table.createFamily(): " + err);
            } else {
                console.log("family created");
            }
        });
    }
});

This code produces the expected result

Table created, now create family
family created

It would seem that the use of the 'exists' API call is changing the nature of the 'table' object

[dom-white]

If I use the get methods with the autoCreate flag set to true I see the same issue

var table = bigtable.table('prezzy');

table.get({autoCreate: true}, function(err, table, apiResponse) {
    if (err) {
        console.log("Error from table.get: " + err);
    } else {

        var family = table.family('follows');

        console.log("Table Got/Created Ok");
        family.get({autoCreate: true}, function(err, family, apiResponse) {
            if (err) {
                console.log("Error from family.get: " + err);
            } else {
                console.log("Family Got/Created Ok");
            }
        });
    }
});

[callmehiphop]

@dom-white thanks for opening this issue, I was able to produce this locally using your provided steps! Interestingly enough if I use a setTimeout to delay creating the family by about 10 seconds, everything passes, anything less and I receive the same error.

@lesv Any idea why this might be happening?

[lesv]

projects/{12345678}/instances//clusters/my-cluster/tables/prezzy

That doesn't look right.

[dom-white]

projects/{12345678}/instances//clusters/my-cluster/tables/prezzy
That doesn't look right.

Sorry, I should have made it clear. I had redacted my project id for the post

[callmehiphop]

@lesv Is that because of the missing instance parameter? We haven't moved to Bigtable v2 yet, so I'm a little confused as to why it's even there. I'm also curious about why a race condition could be occurring when making a GetTable request for a table that doesn't exist.

[lesv]

I'm curious why it's both there and empty. If your not on v2, it shouldn't be there at all. That might be why things are broken.

[stephenplusplus]

@lesv here's some info about the API call that we make:

• baseUrl: bigtable.googleapis.com
• apiVersion: v1

proto definition file

• BigtableTableService

method call

• service: BigtableTableService
• method: createColumnFamily

payload

{
  name: 'projects/nth-circlet-705/zones/us-central1-b/clusters/gcloud-node/tables/prezzy',
  columnFamilyId: 'follows'
}

response message

Failed to read: projects/{288560394597}/instances//clusters/gcloud-node/tables/prezzy

It seems like we're calling the v1 endpoint, but it's expecting information in a v2 format.

[lesv]

I've asked Eng to comment -- will let you know when they respond. That actually makes some sense.

[lesv]

Engineering thinks you might have found a caching bug:

1. the table.exists() call inserts the table name in the server's table cache as "does not exist"
2. the table creation happens, but does not invalidate the "not found" in the cache
3. the create column family call looks for the table in the cache and gets the "not found" error

As an immediate work-around, you should be able to provide the column families to create in the CreateTable call and skip the separate CreateColumnFamily call entirely.

Alternatively, the cache expiration on point 1 is 5-10 seconds, so as long as there are at least 5-11 seconds between points 2 and 3 it should work OK.

They are working on a more permanent solution - No ETA of a fix is available.

[callmehiphop]

This should be resolved per 1a42246 and v0.38.3.

[dom-white]

that's great news, thanks!

[dom-white]

ok, just to be clear the caching bug still exists, so you have implemented the workaround regarding creating the family in the same call as the table is created. Yes?

[callmehiphop]

@dom-white correct - you can now specify all the column families up front when creating the table.

https://googlecloudplatform.github.io/google-cloud-node/#/docs/bigtable/0.1.1/bigtable?method=createTable

[dom-white]

ok, I have tested my scenario where I am unsure whether the table or the family is created yet and I have got it to work successfully with the updated createTable API. Here's my updated test code

var table = bigtable.table('prezzy');
table.exists(function(err, tableExists) {
    if (err) {
        return console.log("ERROR from table.exists() " + err);
    }

    // if the table doesn't exist yet
    if (!tableExists) {
        console.log("TABLE DOES NOT EXIST");

        // attempt to create it and its family
        bigtable.createTable('prezzy', {families: ['cf1']}, function(err, newTable, apiResponse) {
            if (err) {
                return console.log("ERROR from createTable() " + err);
            }
            table = newTable;
            console.log("TABLE AND FAMILY CREATED.. ATTEMPT TO ADD DATA");
            addData(table);
        });
    } else {
        console.log("TABLE EXISTS");
        // if table does exist, check the family exists within the table
        var family = table.family('cf1');
        family.exists(function(err, familyExists) {
            if (err) {
                return console.log("ERROR from family.exists() " + err);
            }
            // if the family does not exist yet
            if(!familyExists) {
                table.createFamily('cf1', function(err, family, apiResponse){
                    if (err) {
                        return console.log("ERROR from createFamily() " + err);
                    }
                    console.log("FAMILY CREATED.. ATTEMPT TO ADD DATA");
                    addData(table);

                });
            } else {
                // nothing else to do, continue...
                console.log("FAMILY ALREADY EXISTS.. ATTEMPT TO ADD DATA");
                addData(table);
            }
        });
    }
});

Thanks again,
Dom