The Prefect Blog
Published in

The Prefect Blog

Supercharge your Python Code with Blocks

Introducing Blocks: open source, modular, reusable, and sharable configuration plus code

colorful Block cubes
Source: Pixabay
  1. You could commit your secrets directly in your code. Then if you upload your code to a public code repository or otherwise share it on the internet you are at risk of having your systems compromised. Don’t do this! ⛔️
  2. You could use environment variables, but they can be insecure, as they are also potentially viewable in logs and when passed around the internet. Environment variables are also tied to a single machine.
  3. You could use a secrets manager. Secrets managers are nice, but they don’t include other configuration or code.
  4. You could build your own custom secrets manager. That sounds like more than a weekend project. 😂
  5. You can share code snippets a number of ways, but you don’t want to be passing secrets in them.

Introducing Prefect Blocks

Prefect has seen rapid adoption among data and machine learning professionals. The library recently passed 10,000 GitHub stars and has an active Slack community with over 20,000 members.

Make a block

S3 AWS GUI creation screenshot
AWS Bock screenshot of the graphical user interface

There’s a block for that

Blocks allow you to store configuration such as:

  • Credentials for external cloud services such as AWS, Azure, GCP, or dbt Cloud.
  • Passwords to databases.
  • Key-value pairs for general use.
  • Numeric threshold values over which to send a notification or kick-off a process.
  • Infrastructure configuration for use in container-based deployments.
  • Hyperparameter tuning search values for training ML models.
  • Rotate credentials because they can be accessed in the workspace GUI or automated via a script.
  • Share credentials with other people in your team working within the same workspace.
  • Centrally manage infrastructure and storage configuration for deployments and external systems used in your stack.
  • Enforce good architecture practices with modular configuration and reusable components.
  • Provide visibility into the many external components used in your data stack. This observation capability will soon be extended — see what’s coming here. 👀

Kinds of blocks

Prefect ships with 18 different block types available off the shelf — and more are being added all the time. Prefect Collections are Python packages for integrating other external tools with Prefect. Collections often contain additional block types.

stacked colored blocks
Source: Pixabay.com

Basic data blocks

Basic block types include strings, JSON, date times, and secrets. Secrets are especially popular. All values — or all but the last few values for longer strings — are obfuscated by pydantic under the hood. Blocks are encrypted in the database.

Secret block in GUI with code snippet

Notifications

Blocks can be used to store Slack or email configuration. Then you can send a Slack message or email notification when something occurs in your code. 🚀

Storage

There are blocks for saving and running code in AWS, GCP, and Azure. A more generic Remote File System block makes it easy to connect with any cloud storage provider that can interface with fsspec.

Storage block options

Infrastructure

Containerization is one of the most popular software engineering advances of the 21st Century. Prefect blocks for Docker and Kubernetes can help you manage the configuration that accompanies container orchestration.

Docker container block creation GUI interface screenshot

Custom blocks

If you have a need for other block functionality, you can create your own custom block types and add them to your workspace.

from prefect.blocks.core import Blockclass Cube(Block):
edge_length_inches: float
def get_volume(self):
return self.edge_length_inches**3
my_cube = Cube(edge_length_inches=3)
my_cube.save(name="cube-it")
GUI custom Prefect block screenshot

Nesting blocks

Blocks can be nested within other blocks.

Gif of block building on itself
Source: Giphy

Block FAQ

What can I name my block?

You can name a block most anything you want, but be careful to save your block name in a valid format: with lowercase letters, numbers, and dashes only.

How do I tell the API server about my new block type

If you create a new block type with a Python file locally, and want to add it to your workspace, you need either save an instance of that block or explicitly register it with the server. You can register the block type with the server from the command line with prefect block register --file my_block_file.py.

How can I see all my blocks from the command line?

prefect block ls will show you all of the blocks you’ve created.

List of Prefect blocks in the terminal

Where do blocks live?

When a block is created it is stored in the database server that the Prefect client is configured to interact with at that time. If you’re connected to Prefect Cloud, it will be saved in Prefect Cloud’s database. If you’re connected to a local Orion server, the block data is saved in your local SQLite Prefect database. Block data is always encrypted at rest.

How do I edit a block?

If you wish to overwrite an existing block from in a Python script, pass overwrite=True to the block’s save method.

screenshot of GUI email notification block

Should I try out blocks?

If you aren’t already using them, ummm, yes. Definitely. Prefect blocks can help you increase your productivity. 🙂

Conclusion

In this post you’ve seen the benefits of Prefect blocks, how to create and use them, and how to extend them. Blocks help you keep your code DRY, secure, and modular. With Prefect Cloud your colleagues can create or update blocks by filling out a form in the GUI.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jeff Hale

Jeff Hale

17.4K Followers

I write about data science. Join my Data Awesome mailing list to stay on top of the latest data tools and tips: https://dataawesome.com