Use bulk INSERT to insert fixtures #29504
Conversation
| @@ -526,8 +526,25 @@ def default_index_type?(index) # :nodoc: | |||
| index.using == :btree || super | |||
| end | |||
|
|
|||
| def insert_fixtures(*) | |||
| without_sql_mode('NO_AUTO_VALUE_ON_ZERO') { super } | |||
There was a problem hiding this comment.
In bulk insert, we rely on queries like INSERT INTO users (id, name) VALUES (DEFAULT, 'David'), (DEFAULT, 'Rafael'). This kind of query doesn't work with NO_AUTO_VALUE_ON_ZERO flag.
On MySQL adapter, we enable flag option by default. The solution is to temporary disable this flag to insert the fixtures.
| @@ -349,6 +349,12 @@ def foreign_keys(table_name) | |||
| end | |||
| end | |||
|
|
|||
| def insert_fixtures(rows, table_name) | |||
There was a problem hiding this comment.
Fallback in Sqlite that doesn't support bulk inserts.
Improves the performance from O(n) to O(1). Previously it would require 50 queries to insert 50 fixtures. Now it takes only one query. Disabled on sqlite which doesn't support multiple inserts.
|
Updated. Thanks for a quick review ❤️ |
rails/rails#29504 implements bulk insert for fixture load. Oracle database needs another syntax. In the mean time this commit just falls back to conventional fixture load. Refer rails/arel#482
rails/rails#29504 implements bulk insert for fixture load. Oracle database needs another syntax. In the mean time, this commit just falls back to conventional fixture load. Refer rails/arel#482
|
FYI, SQLite does support multi-row inserts since version 3.7.11 (released in 2012, see changelog), but in some CIs older versions are being used by default, so you need to upgrade SQLite first (like this: travis-ci/apt-package-safelist#368 (comment)) Maybe it worth adding a version check in implementation for SQLite? |
|
TIL. Thanks for the tip, I'll look into it!
…On Jun 25, 2017 2:03 PM, "Andrey Novikov" ***@***.***> wrote:
FYI, SQLite does support multi-row inserts since version 3.7.11 (released
in 2012, see changelog <http://sqlite.org/releaselog/3_7_11.html>), but
in some CIs older versions are being used by default, so you need to
upgrade SQLite first (like this: travis-ci/apt-package-safelist#368
(comment)
<travis-ci/apt-package-safelist#368 (comment)>
)
Maybe it worth adding a version check in implementation for SQLite?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#29504 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAf3qxSnWtmDxfHJHs8jL7Dtso6tUB6zks5sHqDYgaJpZM4N_Byw>
.
|
|
@Envek I just realized that there was another reason why I couldn't use SQLite for bulk insert. As a placeholder for default values (those columns that were not specified in a fixture), we use I found that SQLite doesn't support |
Since rails#29504, mysql2 adapter lost ability to insert zero value on primary key due to enforce `NO_AUTO_VALUE_ON_ZERO` disabled. That is for using `DEFAULT` on auto increment column, but we can use `NULL` instead in that case.
This patch improves the performance of inserting fixtures from O(n) to O(1).
Previously it would require 50 queries to insert 50 fixtures into one table. Now it takes only one query per table.
Disabled on sqlite which doesn't support multiple inserts.
Some benchmarks:
While 1k fixtures may sound unrealistic, at Shopify scale we have hundreds of tables, each having up to 100 fixtures. In our app this patch dramatically improved the performance of test helper. This patch would be especially helpful for those apps who use
fixtures(:all)(I heard it's the case for Basecamp too).🎉 🎉 🎉
Closes #26901
@rafaelfranca @matthewd