🚀 Unleashing UCX v0.18.0: A Powerful Release for Seamless Migrations and Improved Compatibility

Greetings, Databricks enthusiasts! 🤝 We’re thrilled to announce the release of UCX v0.18.0, bringing you an array of new features, enhancements, and bug fixes targeted at streamlining your workflows, increasing reliability, and expanding compatibility. Let’s dive into the highlights of this much-anticipated release.

🔒 Legacy Table ACL Grants Migration

The migrate-tables workflow now supports a legacy table ACL grants migration feature, addressing issue #340 and paving the way for future improvements. This enhancement introduces a new method, get_tables_to_migrate, and an optional acl_strategy parameter in the migrate_tables method. The GrantsCrawler class and a new AclMigrationWhat enum class have been implemented as well. The hive_metastore module also received updates, enhancing the migration process for external tables, DBFS root delta tables, and table-level ACLs. (#1054)

⚙️ databricks labs ucx cluster-remap

The new databricks labs ucx cluster-remap command simplifies remapping legacy cluster configurations to UC-compatible ones, enabling a smooth transition of tables to UC. A revert functionality, databricks labs ucx revert-cluster-remap, has also been introduced. (#994)

💻 databricks labs ucx migrate-local-code

The release debuts the databricks labs ucx migrate-local-code command for migrating local code to a Databricks environment, supporting Python and SQL files. This experimental feature aims to assist in managing code migration, maintaining consistency across workspaces, and improving compatibility with Unity Catalog. (#1067)

🗃️ Instance Pool in Cluster Policy

UCX v0.18.0 extends flexibility in configuring workflow clusters by adding the ability to specify an instance pool ID for cluster policies. The update includes a new method, _get_instance_pool_id(), and integration and unit tests to verify table migration jobs using instance pools. (#1078)

🔄 Fixed SYNC Command Failures

Improvements to handling SYNC command failures in the _migrate_external_table method ensure better reliability and error handling for external table migrations in the Hive Metastore. (#1073)

🚀 Additional Improvements

• The ucx move command now supports moving Unity Catalog tables from one schema to another after the table upgrade process. (#1062)
• Fixed integration test failures with new pytest fixtures, policy crawler integration tests, and migration of external tables in Hive Metastore. (#1068)
• Refactoring the installer improves modularity and reusability in the codebase. (#1055)

📝 Additional Notes

• Skipped the test_table_migration_job integration test (#1075)
• Skipped unsupported locations while migrating to external location in Azure (#1066)
• Triggering Assessment Workflow from Installer based on User Prompt (#1007)
• Updated README.md for Service Principal Installation Limit (#1076)
• Updated missing documentation for cluster remap command (#1079)

For the complete details on this release, please refer to Release v0.18.0.

🚀 Let’s embrace UCX v0.18.0 and drive your Databricks migrations to new heights! 🔥

Please share and subscribe to Updates from Databricks Labs newsletter to stay up-to-date with the latest releases from GitHub namespace. Subscribing ensures you’re the first to know about the latest enhancements, bug fixes, and exciting features that will take your Data Intelligence Platform experience to the next level. You‘re encouraged to follow the GitHub org as well ;)