Migrate ID columns to bigint #6563

nickvergessen · 2017-09-19T10:54:51Z

For better scaling

mention-bot · 2017-09-19T10:54:53Z

@nickvergessen, thanks for your PR! By analyzing the history of the files in this pull request, we identified @rullzer, @MorrisJobke and @LukasReschke to be potential reviewers.

rullzer

Fine by me

codecov · 2017-09-19T12:26:30Z

Codecov Report

Merging #6563 into master will increase coverage by 3.19%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #6563      +/-   ##
============================================
+ Coverage     53.06%   56.25%   +3.19%     
============================================
  Files          1417       61    -1356     
  Lines         87809     7574   -80235     
  Branches       1340     1340              
============================================
- Hits          46596     4261   -42335     
+ Misses        41213     3313   -37900

Impacted Files	Coverage Δ	Complexity Δ
apps/files_external/lib/Command/Delete.php
settings/ajax/setquota.php
apps/files_external/lib/Lib/Backend/DAV.php
apps/theming/lib/ThemingDefaults.php
lib/private/Preview/Image.php
apps/dav/lib/CardDAV/Converter.php
lib/private/Files/Cache/StorageGlobal.php
...s/workflowengine/lib/Controller/FlowOperations.php
core/Command/Maintenance/UpdateTheme.php
apps/workflowengine/lib/Check/RequestTime.php
... and 1345 more

LukasReschke

Merge conflict

nickvergessen · 2017-09-19T18:10:07Z

Yeah, as to be expected. I will rebase after #6564 and #6566

nickvergessen · 2017-09-20T07:11:33Z

Rebased, ready for review and merge

nickvergessen · 2017-09-22T07:10:21Z

@LukasReschke please review, so this is available early in dailies and master for testing

MorrisJobke · 2017-09-26T12:13:51Z

apps/dav/lib/Migration/Version1004Date20170919103422.php

+/**
+ * Auto-generated migration step: Please modify to your needs!
+ */
+class Version1004Date20170919103422 extends BigIntMigration {


This one is not executed at all. :/

My migrations table for dav looks like this:

dav | 1004Date20170825134824 dav | 1004Date20170919104507 dav | 1004Date20170924124212

Ah I bet that is ebcause we merged the other migration to dav.

Yeah need to rebase and change the timestamp to be newer...
#JustDevProblems....

Will have a look separate if we can detect not executed migrations and run them individually

Will have a look separate if we can detect not executed migrations and run them individually

isn't that the idea of migrations? xD I thought it will just be checked what was already done and what not and then execute all not executed ones.

No the issue is you need to do it sequentially. So you can't just insert a new one in the middle. Because that could lead to inconsistencies.

Exactly, I would also only allow this for debug,...

Okay actually it should have worked, the problem is another PR bumped the version number for the dav app on master already, therefor no update was done at all.

Fixed it now.

MorrisJobke · 2017-09-27T08:25:37Z

apps/dav/lib/Migration/Version1004Date20170926103422.php

+		return [
+			'addressbooks' => ['id'],
+			'addressbookchanges' => ['id', 'addressbookid'],
+			'calendars' => ['id', 'calendarid'],


Doctrine\DBAL\Schema\SchemaException: There is no column with name 'calendarid' on table 'oc_calendars'.

Stupid copy and paste, but I checked the dav tables and it looks good now.

codecov-io · 2017-09-27T13:19:54Z

Codecov Report

Merging #6563 into master will decrease coverage by 0.01%.
The diff coverage is 34.14%.

@@             Coverage Diff              @@
##             master    #6563      +/-   ##
============================================
- Coverage     53.03%   53.01%   -0.02%     
- Complexity    22583    22606      +23     
============================================
  Files          1417     1422       +5     
  Lines         87873    87954      +81     
  Branches       1341     1341              
============================================
+ Hits          46601    46629      +28     
- Misses        41272    41325      +53

Impacted Files	Coverage Δ	Complexity Δ
...av/lib/Migration/Version1004Date20170926103422.php	`0% <0%> (ø)`	`1 <1> (?)`
version.php	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
...es/lib/Migration/Version1002Date20170926101419.php	`0% <0%> (ø)`	`1 <1> (?)`
core/Migrations/Version13000Date20170926101637.php	`0% <0%> (ø)`	`1 <1> (?)`
lib/public/Migration/BigIntMigration.php	`0% <0%> (ø)`	`4 <4> (?)`
lib/private/DB/ConnectionFactory.php	`58.02% <33.33%> (-0.95%)`	`24 <0> (+1)`
lib/private/DB/OCPostgreSqlPlatform.php	`96.42% <96.42%> (ø)`	`15 <15> (?)`
... and 2 more

MorrisJobke

Tested and works 👍

Fixed.

MorrisJobke · 2017-09-27T13:47:07Z

I also fixed the autoloader.

nickvergessen · 2017-09-29T08:55:21Z

What, postgres fails?

Error while trying to create admin user: An exception occurred while executing 'ALTER TABLE oc_personal_settings ALTER id TYPE BIGSERIAL':

SQLSTATE[42704]: Undefined object: 7 ERROR: type "bigserial" does not exist

Signed-off-by: Joas Schilling <[email protected]>

nickvergessen · 2017-09-29T09:11:51Z

Copied owncloud/core#28364

MorrisJobke

Still working fine 👍

nickvergessen added 3. to review Waiting for reviews enhancement labels Sep 19, 2017

nickvergessen added this to the Nextcloud 13 milestone Sep 19, 2017

nickvergessen requested review from rullzer and MorrisJobke September 19, 2017 10:54

rullzer approved these changes Sep 19, 2017

View reviewed changes

nickvergessen mentioned this pull request Sep 19, 2017

Fix database structure of twofactor backupcodes app to match 12 #6566

Merged

nickvergessen added the downstream label Sep 19, 2017

LukasReschke previously requested changes Sep 19, 2017

View reviewed changes

nickvergessen force-pushed the bigint-ids branch from 4df2a51 to 58b307f Compare September 20, 2017 07:11

nickvergessen force-pushed the bigint-ids branch 2 times, most recently from 4ef49be to beea34b Compare September 20, 2017 07:16

MorrisJobke requested changes Sep 26, 2017

View reviewed changes

nickvergessen force-pushed the bigint-ids branch 2 times, most recently from 53d0734 to 4243f40 Compare September 26, 2017 14:14

MorrisJobke requested changes Sep 27, 2017

View reviewed changes

MorrisJobke approved these changes Sep 27, 2017

View reviewed changes

MorrisJobke added 4. to release Ready to be released and/or waiting for tests to finish and removed 3. to review Waiting for reviews labels Sep 27, 2017

nickvergessen added 2 commits September 29, 2017 11:10

Add a base migration for easy updating to bigints

124b437

Signed-off-by: Joas Schilling <[email protected]>

Migrate all core id columns to bigint

ce3bcf7

Signed-off-by: Joas Schilling <[email protected]>

nickvergessen added 6 commits September 29, 2017 11:10

Migrate the twofactor_backupcodes app to bigint

c58ff29

Signed-off-by: Joas Schilling <[email protected]>

Migrate ids to bigint in the dav app

740d820

Signed-off-by: Joas Schilling <[email protected]>

Restore the sequencial order

c65e591

Signed-off-by: Joas Schilling <[email protected]>

Fix copy paste errors

155d451

Signed-off-by: Joas Schilling <[email protected]>

Fix bigint handling on postgres

a3bdb31

Signed-off-by: Joas Schilling <[email protected]>

Update autoloading

f205412

Signed-off-by: Joas Schilling <[email protected]>

nickvergessen force-pushed the bigint-ids branch from dcea970 to f205412 Compare September 29, 2017 09:11

MorrisJobke approved these changes Sep 29, 2017

View reviewed changes

MorrisJobke merged commit 870d716 into master Sep 29, 2017

MorrisJobke deleted the bigint-ids branch September 29, 2017 10:23

MorrisJobke mentioned this pull request Sep 29, 2017

Numeric value out of range: 167 Out of range value for column 'fileid' #2775

Closed

BernhardPosselt mentioned this pull request Jan 19, 2018

Updating Nextcloud 12.0.5 to 13.0.0.rc2 fails on Postgres #7960

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate ID columns to bigint #6563

Migrate ID columns to bigint #6563

nickvergessen commented Sep 19, 2017

mention-bot commented Sep 19, 2017

rullzer left a comment

codecov bot commented Sep 19, 2017 •

edited

Loading

LukasReschke left a comment

nickvergessen commented Sep 19, 2017 •

edited

Loading

nickvergessen commented Sep 20, 2017

nickvergessen commented Sep 22, 2017

MorrisJobke Sep 26, 2017

rullzer Sep 26, 2017

nickvergessen Sep 26, 2017

MorrisJobke Sep 26, 2017

rullzer Sep 26, 2017

nickvergessen Sep 26, 2017

nickvergessen Sep 26, 2017

MorrisJobke Sep 27, 2017

nickvergessen Sep 27, 2017

codecov-io commented Sep 27, 2017 •

edited by codecov bot

Loading

MorrisJobke left a comment

MorrisJobke commented Sep 27, 2017

nickvergessen commented Sep 29, 2017

nickvergessen commented Sep 29, 2017

MorrisJobke left a comment

Migrate ID columns to bigint #6563

Migrate ID columns to bigint #6563

Conversation

nickvergessen commented Sep 19, 2017

mention-bot commented Sep 19, 2017

rullzer left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 19, 2017 • edited Loading

Codecov Report

LukasReschke left a comment

Choose a reason for hiding this comment

nickvergessen commented Sep 19, 2017 • edited Loading

nickvergessen commented Sep 20, 2017

nickvergessen commented Sep 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Sep 27, 2017 • edited by codecov bot Loading

Codecov Report

MorrisJobke left a comment

Choose a reason for hiding this comment

MorrisJobke commented Sep 27, 2017

nickvergessen commented Sep 29, 2017

nickvergessen commented Sep 29, 2017

MorrisJobke left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 19, 2017 •

edited

Loading

nickvergessen commented Sep 19, 2017 •

edited

Loading

codecov-io commented Sep 27, 2017 •

edited by codecov bot

Loading