Supercharge your Python Code with Blocks
Introducing Blocks: open source, modular, reusable, and sharable configuration plus code
Everyone who codes can save time by reusing configuration — whether for logging in to cloud providers or databases, configuring Docker containers, or sending notifications.
Install the Prefect library and add a few lines of code and you’ve got all these super powers and more! 🦸
Have you ever wanted to share cloud provider configuration across your Python codebase in a secure way? What are your potential options?
- You could commit your secrets directly in your code. Then if you upload your code to a public code repository or otherwise share it on the internet you are at risk of having your systems compromised. Don’t do this! ⛔️
- You could use environment variables, but they can be insecure, as they are also potentially viewable in logs and when passed around the internet. Environment variables are also tied to a single machine.
- You could use a secrets manager. Secrets managers are nice, but they don’t include other configuration or code.
- You could build your own custom secrets manager. That sounds like more than a weekend project. 😂
- You can share code snippets a number of ways, but you don’t want to be passing secrets in them.
Luckily, Prefect is an open source solution that can do everything you need. 🎉
Introducing Prefect Blocks
Prefect has seen rapid adoption among data and machine learning professionals. The library recently passed 10,000 GitHub stars and has an active Slack community with over 20,000 members.
After two years of learning from users of Prefect 1, Prefect 2 was built from the ground up and recently released to general availability. Blocks are a new feature that nearly every Python programmer can benefit from.
Blocks are Python classes that can be created either via Python code or the Prefect GUI. Blocks live both on the your client machine and the Prefect server. The server can be either the open source version that comes when you install Prefect or the hosted Prefect Cloud platform.
Prefect Blocks provide you a quick way to create reusable, sharable, and secure configuration with code.
Make a block
After you make a block, you can use it with two lines of code. The Prefect GUI even shows you how:
You can share your block with anyone you add to your Prefect Cloud workspace.
So what all can these blocks do for you?
There’s a block for that
Blocks allow you to store configuration such as:
- Credentials for external cloud services such as AWS, Azure, GCP, or dbt Cloud.
- Passwords to databases.
- Key-value pairs for general use.
- Numeric threshold values over which to send a notification or kick-off a process.
- Infrastructure configuration for use in container-based deployments.
- Hyperparameter tuning search values for training ML models.
Block capabilities provide all kinds of benefits including the ability to:
- Rotate credentials because they can be accessed in the workspace GUI or automated via a script.
- Share credentials with other people in your team working within the same workspace.
- Centrally manage infrastructure and storage configuration for deployments and external systems used in your stack.
- Enforce good architecture practices with modular configuration and reusable components.
- Provide visibility into the many external components used in your data stack. This observation capability will soon be extended — see what’s coming here. 👀
Kinds of blocks
Prefect ships with 18 different block types available off the shelf — and more are being added all the time. Prefect Collections are Python packages for integrating other external tools with Prefect. Collections often contain additional block types.
There are blocks for basic data types, notifications, storage, infrastructure, and cloud providers.
Basic data blocks
Basic block types include strings, JSON, date times, and secrets. Secrets are especially popular. All values — or all but the last few values for longer strings — are obfuscated by pydantic under the hood. Blocks are encrypted in the database.
Notifications
Blocks can be used to store Slack or email configuration. Then you can send a Slack message or email notification when something occurs in your code. 🚀
Storage
There are blocks for saving and running code in AWS, GCP, and Azure. A more generic Remote File System block makes it easy to connect with any cloud storage provider that can interface with fsspec.
If you want to run code from a GitHub repository, there’s a block for that — see a video demo here. 🎥
Infrastructure
Containerization is one of the most popular software engineering advances of the 21st Century. Prefect blocks for Docker and Kubernetes can help you manage the configuration that accompanies container orchestration.
I wrote a bunch of posts on Docker and plan to expand my Kubernetes series in the future. Follow me so you don’t miss it!
Prefect block types give you lots of built-in functionality. What if you want to build your own types of blocks?
Custom blocks
If you have a need for other block functionality, you can create your own custom block types and add them to your workspace.
Here’s how you can create a Cube block type and make a block named my_cube.
from prefect.blocks.core import Blockclass Cube(Block):
edge_length_inches: float def get_volume(self):
return self.edge_length_inches**3my_cube = Cube(edge_length_inches=3)
my_cube.save(name="cube-it")
After running this code, the Cube block type shows up in the GUI — it’s automatically registered with the server behind the scenes.
Nesting blocks
Blocks can be nested within other blocks.
This functionality is handy when reusing configuration across many different blocks. For example, you can use a Docker Registry block that stores configuration about your Docker Registry connection inside many different Docker Container blocks. 🤯
Below you can see the GUI interface for selecting a Docker Registry block when creating a Docker Container block.
See how you can make custom nested blocks in the docs.
Block FAQ
What can I name my block?
You can name a block most anything you want, but be careful to save your block name in a valid format: with lowercase letters, numbers, and dashes only.
How do I tell the API server about my new block type
If you create a new block type with a Python file locally, and want to add it to your workspace, you need either save an instance of that block or explicitly register it with the server. You can register the block type with the server from the command line with prefect block register --file my_block_file.py.
How can I see all my blocks from the command line?
prefect block ls will show you all of the blocks you’ve created.
Check out all the possible block commands from the CLI with prefect block.
Where do blocks live?
When a block is created it is stored in the database server that the Prefect client is configured to interact with at that time. If you’re connected to Prefect Cloud, it will be saved in Prefect Cloud’s database. If you’re connected to a local Orion server, the block data is saved in your local SQLite Prefect database. Block data is always encrypted at rest.
How do I edit a block?
If you wish to overwrite an existing block from in a Python script, pass overwrite=True to the block’s save method.
If you want to edit or delete a block from the GUI, click on the three-dot menu for that block under the Blocks menu.
Should I try out blocks?
If you aren’t already using them, ummm, yes. Definitely. Prefect blocks can help you increase your productivity. 🙂
Conclusion
In this post you’ve seen the benefits of Prefect blocks, how to create and use them, and how to extend them. Blocks help you keep your code DRY, secure, and modular. With Prefect Cloud your colleagues can create or update blocks by filling out a form in the GUI.
Check out Prefect blocks and save yourself time and headaches. If you have any questions, join the Prefect Community Slack.
Happy building! 🧱
Big thanks to Anna Geller and Alex Streed for contributions to this post! 🎉
