BigQuery: support /upload method for loading data from local files.

[tswast]

See documentation: https://cloud.google.com/bigquery/loading-data-post-request

Python, PHP, Ruby, and Node.js provide an upload-from-file method on the Table object.

This is blocking documentation samples.

[tswast]

Actually, I see we have a LoadJobConfiguration. http://googlecloudplatform.github.io/google-cloud-java/0.5.1/apidocs/com/google/cloud/bigquery/LoadJobConfiguration.html That might be sufficient.

[mziccard]

Tim have a look at BigQuery.writer(), it allows you to upload a file in chunks (generally a good idea as they tend to be quite big files). It should be rather simple to write a whole file using it.

[tswast]

Thanks for the pointer.

[mziccard]

Can we close this?

[tswast]

Is there a way to get the Job the writer must create? The Python version is able to get the number of rows written in it's sample.

https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/bigquery/cloud-client/load_data_from_file.py#L51

I am trying to improve the snippets for this (there are no integration tests) and having the returned data from the Job would be really useful.

[tswast]

Or even getting the Job ID would be useful. Then I can fetch the job results after finishing my writes.

[mziccard]

Is there a way to get the Job the writer must create? The Python version is able to get the number of rows written in it's sample.

I need to look into this. I don't know any way of getting the job metadata/id when using resumable upload: the open request just returns the Upload ID, all subsequent POST/PUT seem to have empty response body.

[tswast]

Thanks. I'll dig into it, too.

[tswast]

My current thought is making BigQueryRpc.open() return a Job instead of a String. I will attempt to get a PR out for this.

[mziccard]

Tim I was wrong. The last call of a chunked upload returns all the metadata of the just created job. We can put #1394 on hold until I fix this. I will probably make TableDataWriteChannel public and add a getJob() method that returns the job associated to the upload, only once the channel is closed (i.e. the upload was finalized).

[tswast]

@mziccard Thanks. That sounds like a great solution.

[mziccard]

Thanks for catching this, I have always assumed that the job metadata wasn't available and I couldn't be more wrong :)

[tswast]

I'm getting an error:

java.lang.ClassCastException: com.google.cloud.bigquery.JobStatistics$CopyStatistics cannot be cast to com.google.cloud.bigquery.JobStatistics$LoadStatistics

when trying to use this new functionality in the snippets like:

TableDataWriteChannel writer = bigquery.writer(writeChannelConfiguration);
  // Write data to writer
 try {
    writer.write(ByteBuffer.wrap(csvData.getBytes(Charsets.UTF_8)));
  } finally {
    writer.close();
  }
  // Get load job
  Job job = writer.getJob();
  job.waitFor();
  LoadStatistics stats = job.getStatistics();
  return stats.getOutputRows();

[tswast]

I think it should be a Load Job, not a Copy Job.

[tswast]

Ah, setting job = job.waitFor(); fixes it.

Is this desired behavior? @mziccard

[michael-hll]

I tied the example, it seems the issue still there. It happens when my csv data was not prepared well. I'm using cloud version below:

com.google.cloud
google-cloud
0.8.0