Bigquery exception while loading com.google.cloud.bigquery.BigQueryException: Error writing request body to server

[vinochithra]

On of the customer using Striim's BigQueryConnector is experiencing the following exception

2020-02-20 20:50:41,385 @S10_111_12_29 @toko_product.tokopedia_product -ERROR toko_product.bq_tokopedia_product-0 com.striim.bigquery.TransferCSVDataToBQ.transferToBQ (TransferCSVDataToBQ.java:139) Bigquery exception while loading
com.google.cloud.bigquery.BigQueryException: Error writing request body to server
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:99)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:465)
at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:56)
at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:53)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:52)
at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:161)
at java.nio.channels.Channels$1.close(Channels.java:178)
at com.striim.bigquery.TransferCSVDataToBQ.transferToBQ(TransferCSVDataToBQ.java:135)
at com.striim.bigquery.BigQueryIntegrationTask.transferData(BigQueryIntegrationTask.java:58)
at com.striim.bigquery.BigQueryIntegrationTask.execute(BigQueryIntegrationTask.java:88)
at com.striim.dwhwriter.integrator.IntegrationTask.call(IntegrationTask.java:46)
at com.striim.dwhwriter.integrator.IntegrationTask.call(IntegrationTask.java:23)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Error writing request body to server
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3597)
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3580)
at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
at com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:80)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:448)
... 17 more

This is happening while we try to upload a csv file to the BQ table.
Code snippet:

    // Describe the resulting table you are importing to:
    TableId tid = TableId.of(projectId,datasetId, tableId);

    Schema schema = client.getTable(tid).getDefinition().getSchema();
    //Loads of options available which can be exposed later
    CsvOptions csvOptions = CsvOptions
            .newBuilder()
            .setAllowQuotedNewLines(true)
            .setEncoding(encoding)
            // can be exposed but use case is very specific
            .setAllowJaggedRows(false)
            .setFieldDelimiter(columnDelimiter)
            .setQuote(quoteCharacter)
            .setSkipLeadingRows(0)
            .build();

    WriteChannelConfiguration writeChannelConfiguration = WriteChannelConfiguration
            .newBuilder(tid)
            .setFormatOptions(csvOptions)
            .setSchema(schema)
            .setMaxBadRecords(0)
            .setWriteDisposition(JobInfo.WriteDisposition.WRITE_APPEND)
            .setNullMarker(nullMarker)
            .build();

    Path csvFilePath = Paths.get(csvPath);

    JobId jobId = /* build job id */
    TableDataWriteChannel writeChannel = client.writer(jobId,writeChannelConfiguration);

    try(OutputStream stream = Channels.newOutputStream(writeChannel)){
        Files.copy(csvFilePath, stream);
    }
    catch (IOException ex){
        throw new AdapterException("Exception occurred during file upload: " + ex);
    } catch (BigQueryException be) {
        logger.error("Bigquery exception while loading ", be);

Could this be a due to network issues ? I saw a similar issues logged for GCS - googleapis/google-cloud-java#3410.

If its a similar issue has this been fixed for BigQuery already ? This happened with client version 1.35.0

   <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.35.0</version>
    </dependency>

[pmakani]

@vinochithra I have tested above sample code 1k times and used same version of library which you have mentioned but not able reproduced that error. please try latest version of library.

[vinochithra]

Even I was not able to reproduce it in house. This makes the problem more serious when someone has to deploy BQConnector for production use cases. The customer would like to know when such exception - "Error writing request body to server" would be thrown by BQ API. On what circumstances does "at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:465)" throws this exception in version 1.35.0 ?

With latest Striim version we use BQ client version 1.104.

[stephaniewang526]

Investigating.... also 1.35.0 is a very old version - any reason why they're not using the newer/latest version of BQ client?

[yebrahim]

Seems to me this potentially masks an issue with GCS upload before loading into BQ: googleapis/google-cloud-java#3678. I'd try updating the client version.

[stephaniewang526]

Closing this ticket as no follow-up was posted.