Introduced a static ownership crawler #3111

FastLee · 2024-11-01T14:55:49Z

replaces #3075
close #2890

github-actions · 2024-11-01T17:21:44Z

❌ 111/114 passed, 3 failed, 5 skipped, 8h30m44s total

❌ test_grant_ownership: AttributeError: 'MockRuntimeContext' object has no attribute 'TEST_SCHEMA_owner_group' (11m36.175s)

... (skipped 43263 bytes)
s.com'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2296939550881230', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2324459211697819', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='2562508731442717', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='2614179680770278', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-rust.write', primary=None, ref='Groups/1011887706650232', type='direct', value='1011887706650232'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2697155103502290', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='role.labs.dbsync.write', primary=None, ref='Groups/890738974305402', type='direct', value='890738974305402'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='role.labs.delta-oms.write', primary=None, ref='Groups/925885353519506', type='direct', value='925885353519506')], id='3272374305027664', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268')], id='3297263947987260', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='3619520043928476', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-dotnet.write', primary=None, ref='Groups/518828574352171', type='direct', value='518828574352171'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4085253546229304', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='4142558850594997', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.blueprint.maintain', primary=None, ref='Groups/1118462350168987', type='direct', value='1118462350168987'), ComplexValue(display='role.labs.watchdog.maintain', primary=None, ref='Groups/362656320875527', type='direct', value='362656320875527'), ComplexValue(display='role.labs.lsql.maintain', primary=None, ref='Groups/583508855493027', type='direct', value='583508855493027'), ComplexValue(display='role.labs.sdk-r.maintain', primary=None, ref='Groups/69514106516856', type='direct', value='69514106516856'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='role.labs.transpiler.maintain', primary=None, ref='Groups/51203042855777', type='direct', value='51203042855777'), ComplexValue(display='role.labs.sdk-dotnet.maintain', primary=None, ref='Groups/422903443390924', type='direct', value='422903443390924'), ComplexValue(display='role.labs.sdk-rust.maintain', primary=None, ref='Groups/400100643391091', type='direct', value='400100643391091'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='labs-oss', primary=None, ref='Groups/852860106691249', type='direct', value='852860106691249'), ComplexValue(display='role.labs.TEST_CATALOG.maintain', primary=None, ref='Groups/891852804756554', type='direct', value='891852804756554'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='direct', value='993102204406282'), ComplexValue(display='role.labs.ucx.maintain', primary=None, ref='Groups/658020652963371', type='direct', value='658020652963371'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.remorph.maintain', primary=None, ref='Groups/75982969078436', type='direct', value='75982969078436'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268')], id='4183391249163402', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='4256879669741685', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4313162842336807', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596')], id='5220576099010372', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='5629836015416408', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.dlt-meta.write', primary=None, ref='Groups/408004270888312', type='direct', value='408004270888312'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268')], id='5818361206914695', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='5934952062729548', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.maintain', primary=None, ref='Groups/929610554024196', type='direct', value='929610554024196'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='6472610608479780', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282')], id='6779888502363704', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs-oss', primary=None, ref='Groups/852860106691249', type='direct', value='852860106691249'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='7344182020334936', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='role.labs.blueprint.write', primary=None, ref='Groups/956728708332625', type='direct', value='956728708332625'), ComplexValue(display='role.labs.remorph.maintain', primary=None, ref='Groups/75982969078436', type='direct', value='75982969078436')], id='7534000974286519', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.migrate.maintain', primary=None, ref='Groups/174848272515934', type='direct', value='174848272515934'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='7603434113437775', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='7816683983824279', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='8077637869895037', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.watchdog.write', primary=None, ref='Groups/1025833911894044', type='direct', value='1025833911894044'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='8609205844338373', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.migrate.maintain', primary=None, ref='Groups/174848272515934', type='direct', value='174848272515934'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8922191418148017', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-r.write', primary=None, ref='Groups/179198484394146', type='direct', value='179198484394146'), ComplexValue(display='admins', primary=None, ref='Groups/993102204406282', type='indirect', value='993102204406282'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8957403854939895', name=None, roles=[], schemas=None, user_name='[email protected]')]
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_all_grant_types: AssertionError: assert {('CATALOG', ...dummy_tg4ht')} == {('ANONYMOUS ..._fucmm'), ...} (12m21.364s)

AssertionError: assert {('CATALOG', ...dummy_tg4ht')} == {('ANONYMOUS ..._fucmm'), ...}
  
  Extra items in the right set:
  ('ANY FILE', None)
  ('ANONYMOUS FUNCTION', None)
  
  Full diff:
    {
  -     (
  -         'ANONYMOUS FUNCTION',
  -         None,
  -     ),
  -     (
  -         'ANY FILE',
  -         None,
  -     ),
        (
            'CATALOG',
            'hive_metastore',
        ),
        (
            'DATABASE',
            'hive_metastore.dummy_skhhh',
        ),
        (
            'TABLE',
            'hive_metastore.dummy_skhhh.dummy_tsvc6',
        ),
        (
            'UDF',
            'hive_metastore.dummy_skhhh.dummy_fucmm',
        ),
        (
            'VIEW',
            'hive_metastore.dummy_skhhh.dummy_tg4ht',
        ),
    }
22:09 INFO [databricks.labs.ucx.install] Creating ucx schemas...
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
22:09 INFO [databricks.labs.ucx.install] Creating ucx schemas...
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.grants] fetching grants inventory
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.grants] crawling new set of snapshot data for grants
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.tables] fetching tables inventory
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.tables] crawling new set of snapshot data for tables
22:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_skhhh] listing tables and views
22:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_skhhh.dummy_tg4ht] fetching table metadata
22:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_skhhh.dummy_tsvc6] fetching table metadata
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.tables] found 2 new records for tables
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.udfs] fetching udfs inventory
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.udfs] crawling new set of snapshot data for udfs
22:10 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_skhhh] listing udfs
22:11 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_skhhh.dummy_fucmm] fetching udf metadata
22:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.udfs] found 1 new records for udfs
22:21 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-668586c2524d209f5beb32903cb7c1d5-47a9ec8abe9366a4-00]
22:22 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-0631454eb49be4469da214a0d62c5554-b0824a577699c76b-00]
22:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.grants] found 189 new records for grants
22:09 INFO [databricks.labs.ucx.install] Creating ucx schemas...
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.grants] fetching grants inventory
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.grants] crawling new set of snapshot data for grants
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.tables] fetching tables inventory
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.tables] crawling new set of snapshot data for tables
22:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_skhhh] listing tables and views
22:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_skhhh.dummy_tg4ht] fetching table metadata
22:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_skhhh.dummy_tsvc6] fetching table metadata
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.tables] found 2 new records for tables
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.udfs] fetching udfs inventory
22:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.udfs] crawling new set of snapshot data for udfs
22:10 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_skhhh] listing udfs
22:11 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_skhhh.dummy_fucmm] fetching udf metadata
22:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.udfs] found 1 new records for udfs
22:21 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-668586c2524d209f5beb32903cb7c1d5-47a9ec8abe9366a4-00]
22:22 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-0631454eb49be4469da214a0d62c5554-b0824a577699c76b-00]
22:22 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svxa6.grants] found 189 new records for grants
[gw7] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_all_grants_for_other_objects: AssertionError: assert {'MODIFY', 'SELECT'} == set() (11m54.517s)

AssertionError: assert {'MODIFY', 'SELECT'} == set()
  
  Extra items in the left set:
  'SELECT'
  'MODIFY'
  
  Full diff:
  - set()
  + {
  +     'MODIFY',
  +     'SELECT',
  + }
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.grants] fetching grants inventory
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 291, in execute
    self._raise_if_needed(result_status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_skh0r`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.grants] crawling new set of snapshot data for grants
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.tables] fetching tables inventory
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_skh0r`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.tables] crawling new set of snapshot data for tables
22:34 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sg19h] listing tables and views
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.tables] found 0 new records for tables
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.udfs] fetching udfs inventory
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_skh0r`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.udfs] crawling new set of snapshot data for udfs
22:34 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_sg19h] listing udfs
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.udfs] found 0 new records for udfs
22:45 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-bbf47ec7df233562217b4105c4e688b7-28923ac014590473-00]
22:45 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-8fd0e37d025f96b25ec0b758dd39ac5a-f4e1ce84076c48b3-00]
22:45 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.grants] found 5 new records for grants
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.grants] fetching grants inventory
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 291, in execute
    self._raise_if_needed(result_status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_skh0r`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.grants] crawling new set of snapshot data for grants
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.tables] fetching tables inventory
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_skh0r`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.tables] crawling new set of snapshot data for tables
22:34 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sg19h] listing tables and views
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.tables] found 0 new records for tables
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.udfs] fetching udfs inventory
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_skh0r`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.udfs] crawling new set of snapshot data for udfs
22:34 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_sg19h] listing udfs
22:34 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.udfs] found 0 new records for udfs
22:45 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-bbf47ec7df233562217b4105c4e688b7-28923ac014590473-00]
22:45 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-8fd0e37d025f96b25ec0b758dd39ac5a-f4e1ce84076c48b3-00]
22:45 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_skh0r.grants] found 5 new records for grants
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from acceptance #7367}

FastLee · 2024-11-05T15:55:45Z

Failing integration test has been validated to work fine.

labs.yml

src/databricks/labs/ucx/config.py

src/databricks/labs/ucx/hive_metastore/grants.py

src/databricks/labs/ucx/hive_metastore/ownership.py

tests/unit/hive_metastore/test_grants.py

src/databricks/labs/ucx/hive_metastore/grants.py

src/databricks/labs/ucx/contexts/application.py

src/databricks/labs/ucx/hive_metastore/grants.py

src/databricks/labs/ucx/workspace_access/groups.py

…ership lookup

FastLee requested a review from a team as a code owner November 1, 2024 14:55

FastLee had a problem deploying to account-admin November 1, 2024 14:55 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin November 1, 2024 15:49 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin November 1, 2024 17:25 — with GitHub Actions Failure

FastLee force-pushed the feat/table-group-ownership branch from 2ccc5e4 to 92e857d Compare November 1, 2024 19:53

FastLee had a problem deploying to account-admin November 1, 2024 19:53 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin November 4, 2024 16:31 — with GitHub Actions Error

FastLee force-pushed the feat/table-group-ownership branch from 0352b19 to 733d3e3 Compare November 4, 2024 16:47

FastLee had a problem deploying to account-admin November 4, 2024 16:47 — with GitHub Actions Error

FastLee force-pushed the feat/table-group-ownership branch from 733d3e3 to 6c52bba Compare November 4, 2024 17:35

FastLee temporarily deployed to account-admin November 4, 2024 17:35 — with GitHub Actions Inactive

FastLee had a problem deploying to account-admin November 4, 2024 20:25 — with GitHub Actions Error

FastLee had a problem deploying to account-admin November 4, 2024 20:37 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin November 5, 2024 15:47 — with GitHub Actions Failure

FastLee force-pushed the feat/table-group-ownership branch from dbce162 to fe90ab6 Compare November 6, 2024 13:44

FastLee had a problem deploying to account-admin November 6, 2024 13:44 — with GitHub Actions Error

FastLee had a problem deploying to account-admin November 6, 2024 15:25 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin November 6, 2024 19:25 — with GitHub Actions Failure

FastLee had a problem deploying to account-admin November 7, 2024 13:52 — with GitHub Actions Error

FastLee force-pushed the feat/table-group-ownership branch from fac8a60 to dfb8eb1 Compare November 7, 2024 13:58

FastLee had a problem deploying to account-admin November 7, 2024 13:58 — with GitHub Actions Failure

nfx requested changes Nov 7, 2024

View reviewed changes

nfx changed the title ~~Separated ownership crawler from the other grant crawlers. Introduced a static ownership crawler.~~ Introduced a static ownership crawler Nov 7, 2024

FastLee marked this pull request as draft November 7, 2024 18:00

nfx requested changes Nov 8, 2024

View reviewed changes

FastLee force-pushed the feat/table-group-ownership branch from 28f6cfc to 6e1681d Compare November 8, 2024 17:18

FastLee marked this pull request as ready for review November 8, 2024 17:30

FastLee had a problem deploying to account-admin November 8, 2024 17:30 — with GitHub Actions Failure

FastLee added 28 commits November 13, 2024 17:07

Added tests

c93615e

Added missing property

c422228

Fixed test coverage.

7e64c92

Added functionality for a static Ownership for tables.

b2cc246

Fixed Unit Tests

bdfff7e

Fixed Unit Tests

8291bcb

Addressed Integration Tests

d93767d

Addressed Integration Tests Issues

a1e65dd

Addressed Integration Tests Issues

f258faa

Added CLI Option

ac6978f

Fixed FMT Issues

bd6f5be

Added unit test for CLI

032d6cc

Addressing Review Comment. Removing SKIP ACLS functionality.

5b22313

Introduced test ownership

a26653c

Add unit tests

4d519e8

Added integration test and converted from username to userid for memb…

c3b0bb3

…ership lookup

Added integration test and converted from username to userid for memb…

4719349

…ership lookup

Rebased

b2e93e4

Added coverage

bb15b2a

Added command description to README.md

603efb0

Fixed README.md

fe47b55

Added test coverage.

a940902

Added test coverage.

3f1e6e8

Added test coverage.

e7042ee

Added test coverage.

9dd1bbf

Addressed review comments

86fdcb9

Added test coverage

0584327

Added test coverage

ab4bc78

FastLee force-pushed the feat/table-group-ownership branch from 09ce49f to ab4bc78 Compare November 13, 2024 22:07

FastLee had a problem deploying to account-admin November 13, 2024 22:08 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduced a static ownership crawler #3111

Introduced a static ownership crawler #3111

FastLee commented Nov 1, 2024 •

edited

Loading

github-actions bot commented Nov 1, 2024 •

edited

Loading

FastLee commented Nov 5, 2024

Introduced a static ownership crawler #3111

Are you sure you want to change the base?

Introduced a static ownership crawler #3111

Conversation

FastLee commented Nov 1, 2024 • edited Loading

github-actions bot commented Nov 1, 2024 • edited Loading

FastLee commented Nov 5, 2024

FastLee commented Nov 1, 2024 •

edited

Loading

github-actions bot commented Nov 1, 2024 •

edited

Loading