copy_to dimension limit

[karoliskascenas]

Is there any reason why this limitation exists? I've rebuilt the package without it and was successfully able to write a 50000x5 table.

[ianmcook]

The only direct means implyr has of copying data from an R session into an Impala table is by running a SQL INSERT ... VALUES() statement, with the values to be inserted represented as literals in the statement. This method of loading data into Impala is kludgy and is inefficient for large amounts of data, and it is potentially difficult to cancel or undo. That's why this limit exists. Whenever possible, I'd encourage you to use a more efficient method of loading data into whatever storage system backs your Impala table. But I recognize that this implyr method might be your best option in some cases. The choice of the number 1000 was somewhat arbitrary. It was based on some informal tests I performed while connecting over the internet to a remote instance of Impala over an average speed internet connection. If you would like to submit a PR to make this limit configurable using options(), I will gladly merge it. Thank you.

[karoliskascenas]

#50