Fixed SystemError: AST constructor recursion depth mismatch failing the entire job
#3000
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
✅ 43/43 passed, 2 skipped, 31m22s total Running from acceptance #6845 |
nfx
added a commit
that referenced
this pull request
Oct 17, 2024
* Added `lazy_loader` to known list ([#2991](#2991)). With this commit, the `lazy_loader` module has been added to the known list in the configuration file, addressing a portion of issue [#193](#193), which may have been caused by the discovery or loading of this module. The `lazy_loader` is a package or module that, once added to the known list, will be recognized and loaded by the system. This change does not affect any existing functionality or introduce new methods. The commit solely updates the known.json file to include `lazy_loader` with an empty list, indicating that it is ready for use. This modification will enable the correct loading and recognition of the `lazy_loader` module in the system. * Added `librosa` to known list ([#2992](#2992)). In this update, we have added several open-source libraries to the known list in the configuration file, including `librosa`, `llvmlite`, `msgpack`, `pooch`, `soundfile`, and `soxr`. These libraries are commonly used in data engineering, machine learning, and scientific computing tasks. `librosa` is a Python library for audio and music analysis, while `llvmlite` is a lightweight Python interface to the LLVM compiler infrastructure. `msgpack` is a binary serialization format like JSON, `pooch` is a package for managing external data files, `soundfile` is a library for reading and writing audio files, and `soxr` is a library for high-quality audio resampling. Each library has an empty list next to it for specifying additional configuration related to the library. This update partially resolves issue [#1931](#1931) by adding `librosa` to the known list, ensuring that these libraries will be properly recognized and utilized by the codebase. * Added `linkify-it-py` to known list ([#2993](#2993)). In this release, we have added support for two new open-source packages, `linkify-it-py` and `uc-micro-py`, to enhance the software's functionality and compatibility. The addition of `linkify-it-py` and its constituent modules, as well as the incorporation of `uc-micro-py` with its modules and classes, aims to expand the software's capabilities. These changes are related to the resolution of issue [#1931](#1931), and they will enable the software to work seamlessly with these packages, thereby providing a better user experience. * Added `lz4` to known list ([#2994](#2994)). In this release, we have added support for the LZ4 lossless data compression algorithm, which is known for its focus on compression and decompression speed. The implementation includes four variants: lz4, lz4.block, lz4.frame, and lz4.version, each providing different levels of compression and decompression speed and flexibility. This addition expands the range of supported compression algorithms, providing more options for users to choose from and partially addressing issue [#1931](#1931) related to supporting additional compression algorithms. This improvement will be beneficial to software engineers working with data compression in their projects. * Fixed `SystemError: AST constructor recursion depth mismatch` failing the entire job ([#3000](#3000)). This PR introduces more deterministic, Go-style, error handling for parsing Python code, addressing issues that caused the entire job to fail due to a `SystemError: AST constructor recursion depth mismatch` ([#3000](#3000)) and bug [#2976](#2976). It includes removing the `AstroidSyntaxError` import, adding an import for `SqlglotError`, and updating the `SqlParseError` exception to `SqlglotError` in the `lint` method of the `SqlLinter` class. Additionally, abstract classes `TablePyCollector` and `DfsaPyCollector` and their respective methods for collecting tables and direct file system accesses have been removed. The `PythonSequentialLinter` class, previously handling multiple responsibilities, has also been removed, enhancing code modularity, understandability, maintainability, and testability. The changes affect the `base.py`, `python_ast.py`, and `python_sequential_linter.py` modules. * Skip applying permissions for workspace system groups to Unity Catalog resources ([#2997](#2997)). This commit introduces changes to the ACL-related code in the `databricks labs ucx create-catalog-schemas` command and the `migrate-table-*` workflow, skipping the application of permissions for workspace system groups in the Unity Catalog. These system groups, which include 'admins', do not exist at the account level. To ensure the correctness of these modifications, unit and integration tests have been added, including a test that checks the proper handling of user privileges in system groups during catalog schema creation. The `AccessControlResponse` object has been updated for the `admins` and `users` groups, granting them specific permissions for a workspace and warehouse object, respectively, enhancing the system's functionality in multi-user environments with system groups.
Merged
nfx
added a commit
that referenced
this pull request
Oct 17, 2024
* Added `lazy_loader` to known list ([#2991](#2991)). With this commit, the `lazy_loader` module has been added to the known list in the configuration file, addressing a portion of issue [#193](#193), which may have been caused by the discovery or loading of this module. The `lazy_loader` is a package or module that, once added to the known list, will be recognized and loaded by the system. This change does not affect any existing functionality or introduce new methods. The commit solely updates the known.json file to include `lazy_loader` with an empty list, indicating that it is ready for use. This modification will enable the correct loading and recognition of the `lazy_loader` module in the system. * Added `librosa` to known list ([#2992](#2992)). In this update, we have added several open-source libraries to the known list in the configuration file, including `librosa`, `llvmlite`, `msgpack`, `pooch`, `soundfile`, and `soxr`. These libraries are commonly used in data engineering, machine learning, and scientific computing tasks. `librosa` is a Python library for audio and music analysis, while `llvmlite` is a lightweight Python interface to the LLVM compiler infrastructure. `msgpack` is a binary serialization format like JSON, `pooch` is a package for managing external data files, `soundfile` is a library for reading and writing audio files, and `soxr` is a library for high-quality audio resampling. Each library has an empty list next to it for specifying additional configuration related to the library. This update partially resolves issue [#1931](#1931) by adding `librosa` to the known list, ensuring that these libraries will be properly recognized and utilized by the codebase. * Added `linkify-it-py` to known list ([#2993](#2993)). In this release, we have added support for two new open-source packages, `linkify-it-py` and `uc-micro-py`, to enhance the software's functionality and compatibility. The addition of `linkify-it-py` and its constituent modules, as well as the incorporation of `uc-micro-py` with its modules and classes, aims to expand the software's capabilities. These changes are related to the resolution of issue [#1931](#1931), and they will enable the software to work seamlessly with these packages, thereby providing a better user experience. * Added `lz4` to known list ([#2994](#2994)). In this release, we have added support for the LZ4 lossless data compression algorithm, which is known for its focus on compression and decompression speed. The implementation includes four variants: lz4, lz4.block, lz4.frame, and lz4.version, each providing different levels of compression and decompression speed and flexibility. This addition expands the range of supported compression algorithms, providing more options for users to choose from and partially addressing issue [#1931](#1931) related to supporting additional compression algorithms. This improvement will be beneficial to software engineers working with data compression in their projects. * Fixed `SystemError: AST constructor recursion depth mismatch` failing the entire job ([#3000](#3000)). This PR introduces more deterministic, Go-style, error handling for parsing Python code, addressing issues that caused the entire job to fail due to a `SystemError: AST constructor recursion depth mismatch` ([#3000](#3000)) and bug [#2976](#2976). It includes removing the `AstroidSyntaxError` import, adding an import for `SqlglotError`, and updating the `SqlParseError` exception to `SqlglotError` in the `lint` method of the `SqlLinter` class. Additionally, abstract classes `TablePyCollector` and `DfsaPyCollector` and their respective methods for collecting tables and direct file system accesses have been removed. The `PythonSequentialLinter` class, previously handling multiple responsibilities, has also been removed, enhancing code modularity, understandability, maintainability, and testability. The changes affect the `base.py`, `python_ast.py`, and `python_sequential_linter.py` modules. * Skip applying permissions for workspace system groups to Unity Catalog resources ([#2997](#2997)). This commit introduces changes to the ACL-related code in the `databricks labs ucx create-catalog-schemas` command and the `migrate-table-*` workflow, skipping the application of permissions for workspace system groups in the Unity Catalog. These system groups, which include 'admins', do not exist at the account level. To ensure the correctness of these modifications, unit and integration tests have been added, including a test that checks the proper handling of user privileges in system groups during catalog schema creation. The `AccessControlResponse` object has been updated for the `admins` and `users` groups, granting them specific permissions for a workspace and warehouse object, respectively, enhancing the system's functionality in multi-user environments with system groups.
nfx
added a commit
that referenced
this pull request
Nov 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds more deterministic, Go-style, error handling for parsing Python code
Fix #2976