Managed Private Endpoints: To Securely Access Azure Resources from Fabric Spark
Introduction
Private endpoints provide a secure method for workspace admins to access data sources such as Azure SQL DB, Storage Accounts, Azure Synapse Analytics, and Azure Cosmos DB behind a firewall. Private endpoint connections ensure sensitive data remains protected without the need for complex network configurations or exposure to public networks.
Requirements to Get Started
Before diving into setting up managed private endpoints in Fabric Spark, ensure you have:
- An active Azure resource (e.g., Azure SQL Database).
- Select a premium workspace where the private endpoint connection will be created.
Now, we will explore how to achieve this in the upcoming step-by-step guide.
Step 1: Workspace Configuration
- Navigate to Workspace Settings → Network Security.
- Choose the option to create a new private endpoint.
- Fill in details including Endpoint Name, Resource Identifier, Target sub-resource, and Request Message.
Step 2: Activating the Endpoint
- Obtain the Resource ID from your Azure SQL Database.
- Check Activation status under Network Security until it changes from Provisioning to Succeeded.
Step 3: Configuring Azure SQL
- In Azure SQL, disable public access under the networking tab to restrict access exclusively through the private endpoint connection.
- Approve private access for the selected endpoint connection.
Step 4: Validation
- Verify Approval status to ensure the private access request has been approved.
- Test the connection by querying test data using a notebook within the same workspace.
server_name = "<server_name>"
database = "<database>"
port = 1433
username = "<username>"
password = "<password>"
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("TestSecureAzureSQLDB") \
.config("spark.jars.packages", "com.microsoft.azure:azure-sqldb-spark:1.0.2").getOrCreate()
jdbc_url = f"jdbc:sqlserver://{server_name}:{port};database={database}"
connection = {"user": username, "password": password, "driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver"}
df = spark.read.jdbc(url=jdbc_url, table="dbo.Student", properties=connection)
df.show(10)Pros and Cons
Pros:
- Cross-Tenant Access: Supports cross-tenant managed private endpoints, enabling secure connections across different Azure tenants.
- Versatility: Extends support to various Azure data sources beyond SQL DB.
Cons:
- Starter Pool Limitations: Managed private endpoints currently face limitations with starter pools in certain configurations.
- SKU Dependency: Availability limited to specific Fabric capacities (F64 or higher).
- Operational Constraints: Some operations like VACUUM and OPTIMIZE are not yet supported with managed private endpoints.
- Endpoint Limits: Number of managed private endpoints is subject to restrictions based on data source-specific limits.
Conclusion
As of now, we’ve explored the seamless integration of Azure SQL DB with Fabric Spark using managed private endpoints. This approach not only enhances data privacy and compliance but also ensures a secure connection through Microsoft’s backbone. Additionally, it simplifies network management by eliminating the need for complex configurations.
About Us
Bi3 has been recognized for being one of the fastest-growing companies in Australia. Our team has delivered substantial and complex projects for some of the largest organizations around the globe, and we’re quickly building a brand that is well-known for superior delivery.
Website: https://bi3technologies.com/
Follow us on,
LinkedIn: https://www.linkedin.com/company/bi3technologies
Instagram: https://www.instagram.com/bi3technologies/
Twitter: https://twitter.com/Bi3Technologies
