Database migration failing on primary key unique constraint

[smoogipoo]

Hello!

I have a database from one of our users (ppy/osu#29413) that seems to be totally unrecoverable from our end, and I'm asking for help.

The database in question: client.realm.zip

It loads fine in Realm Studio and our application, however when a no-op migration to schema 42 is applied, realm crashes with:

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   librealm-wrappers.dylib       	       0x168e73430 realm::Array::init_from_ref(unsigned long) + 64
1   librealm-wrappers.dylib       	       0x169179a28 (anonymous namespace)::has_duplicate_values(realm::Array const&, realm::ClusterColumn const&) + 324
2   librealm-wrappers.dylib       	       0x169179994 (anonymous namespace)::has_duplicate_values(realm::Array const&, realm::ClusterColumn const&) + 176
3   librealm-wrappers.dylib       	       0x1692366a8 realm::Table::validate_column_is_unique(realm::ColKey) const + 256
4   librealm-wrappers.dylib       	       0x169164928 realm::Group::validate_primary_columns() + 136
5   librealm-wrappers.dylib       	       0x168f20170 realm::ObjectStore::apply_schema_changes(realm::Transaction&, unsigned long long, realm::Schema&, unsigned long long, realm::SchemaMode, std::__1::vector<realm::SchemaChange, std::__1::allocator<realm::SchemaChange>> const&, bool, std::__1::function<void ()>) + 3544
6   librealm-wrappers.dylib       	       0x168f6f2a0 realm::Realm::update_schema(realm::Schema, unsigned long long, std::__1::function<void (std::__1::shared_ptr<realm::Realm>, std::__1::shared_ptr<realm::Realm>, realm::Schema&)>, std::__1::function<void (std::__1::shared_ptr<realm::Realm>)>, bool) + 1960
7   librealm-wrappers.dylib       	       0x168ef2398 realm::_impl::RealmCoordinator::do_get_realm(realm::RealmConfig&&, std::__1::shared_ptr<realm::Realm>&, std::__1::optional<realm::VersionID>, realm::util::CheckedUniqueLock&) + 1764
8   librealm-wrappers.dylib       	       0x168ef1bac realm::_impl::RealmCoordinator::get_realm(realm::RealmConfig, std::__1::optional<realm::VersionID>) + 508
9   librealm-wrappers.dylib       	       0x168f6dc50 realm::Realm::get_shared_realm(realm::RealmConfig) + 120
10  librealm-wrappers.dylib       	       0x168e8f61c shared_realm_open + 192

Since the migration is a no-op, it's impossible for it to have added a duplicate value.

I've tried manually searching for the duplicate during migration via:

HashSet<string> ids = new HashSet<string>();

testPrimaryKey<Guid>("Skin", "ID");
testPrimaryKey<Guid>("Score", "ID");
testPrimaryKey<string>("Ruleset", "ShortName");
testPrimaryKey<Guid>("ModPreset", "ID");
testPrimaryKey<Guid>("KeyBinding", "ID");
testPrimaryKey<string>("File", "Hash");
testPrimaryKey<Guid>("BeatmapSet", "ID");
testPrimaryKey<Guid>("BeatmapCollection", "ID");
testPrimaryKey<Guid>("Beatmap", "ID");

void testPrimaryKey<T>(string objectName, string pkColumn)
{
    ids.Clear();

    foreach (var item in migration.OldRealm.DynamicApi.All(objectName))
    {
        if (!ids.Add(item.DynamicApi.Get<T>(pkColumn)!.ToString()!))
        {
            Trace.Fail("This should not be reached.");
        }
    }
}

To no avail - the assertion never gets tripped, so we can't even see any duplicates.

I've also tried to run a compact on the existing schema, prior to running the migration, and that does not solve the issue.

There are two questions I have:

• What, exactly, in the database is corrupted? What are the two duplicates?
• Is it possible for the database to be recovered for the user, at all?
• A last question, and this might be one for the subatomic gremlin particle to answer for, is how did the database get into this state? Is there some pathway that can bypass the uniqueness constraint? Perhaps migration related?

[smoogipoo]

I found the object that causes this:
File { Hash = 651af2ec8a0e9f849dcb3536ec8379abcec22ce956d5bf044e9c9bea18dfbee0 }

Deleting this file makes Realm crash with:

/Users/runner/work/realm-dotnet/realm-dotnet/wrappers/realm-core/src/realm/group_writer.cpp:1091: [realm-core-13.20.1] Assertion failed: prev_ref + prev_size <= ref with (prev_ref, prev_size, ref, i, limit, m_alloc.get_file_path_for_assertions()) =  [3140456, 1304, 3140976, 3653, 3922, "/Users/smgi/Library/Application Support/osu-development/client.realm"]
0   librealm-wrappers.dylib             0x000000016a7ecfa8 _ZN5realm4util19terminate_with_infoEPKcS2_lS2_OSt16initializer_listINS0_9PrintableEE + 400
1   librealm-wrappers.dylib             0x000000016a6e59b8 _ZN5realm4util19terminate_with_infoIJRKmEEEvPKciS5_S5_DpOT_ + 0
2   librealm-wrappers.dylib             0x000000016a6e43e4 _ZN5realm11GroupWriter11write_groupEv + 8740
3   librealm-wrappers.dylib             0x000000016a6c4f68 _ZN5realm2DB16low_level_commitEyRNS_11TransactionEb + 1628
4   librealm-wrappers.dylib             0x000000016a6c480c _ZN5realm2DB9do_commitERNS_11TransactionEb + 132
5   librealm-wrappers.dylib             0x000000016a7c85e8 _ZN5realm11Transaction27commit_and_continue_as_readEb + 104
6   librealm-wrappers.dylib             0x000000016a471530 _ZN5realm5_impl16RealmCoordinator12commit_writeERNS_5RealmEb + 132
7   librealm-wrappers.dylib             0x000000016a4eb190 _ZN5realm5Realm13update_schemaENS_6SchemaEyNSt3__18functionIFvNS2_10shared_ptrIS0_EES5_RS1_EEENS3_IFvS5_EEEb + 1688
8   librealm-wrappers.dylib             0x000000016a46e398 _ZN5realm5_impl16RealmCoordinator12do_get_realmEONS_11RealmConfigERNSt3__110shared_ptrINS_5RealmEEENS4_8optionalINS_9VersionIDEEERNS_4util17CheckedUniqueLockE + 1764
9   librealm-wrappers.dylib             0x000000016a46dbac _ZN5realm5_impl16RealmCoordinator9get_realmENS_11RealmConfigENSt3__18optionalINS_9VersionIDEEE + 508
10  librealm-wrappers.dylib             0x000000016a4e9c50 _ZN5realm5Realm16get_shared_realmENS_11RealmConfigE + 120
11  librealm-wrappers.dylib             0x000000016a40b61c shared_realm_open + 192

And querying the File object set crashes Realm Studio with:
Hash == '651af2ec8a0e9f849dcb3536ec8379abcec22ce956d5bf044e9c9bea18dfbee0'

[jedelbo]

The problem was not a duplicate primary key, but rather that the index on the "class_File" table was corrupted. Unfortunately a cannot say how we could get into this state. As the damage was not too severe, I was able to recover the file.
client1.realm..zip