The Prefect Blog
Published in

The Prefect Blog

Supercharge your Python Code with Blocks

Introducing Blocks: open source, modular, reusable, and sharable configuration plus code

colorful Block cubes
Source: Pixabay
  1. You could commit your secrets directly in your code. Then if you upload your code to a public code repository or otherwise share it on the internet you are at risk of having your systems compromised. Don’t do this! ⛔️
  2. You could use environment variables, but they can be insecure, as they are also potentially viewable in logs and when passed around the internet. Environment variables are also tied to a single machine.
  3. You could use a secrets manager. Secrets managers are nice, but they don’t include other configuration or code.
  4. You could build your own custom secrets manager. That sounds like more than a weekend project. 😂
  5. You can share code snippets a number of ways, but you don’t want to be passing secrets in them.

Introducing Prefect Blocks

Make a block

S3 AWS GUI creation screenshot
AWS Bock screenshot of the graphical user interface

There’s a block for that

  • Credentials for external cloud services such as AWS, Azure, GCP, or dbt Cloud.
  • Passwords to databases.
  • Key-value pairs for general use.
  • Numeric threshold values over which to send a notification or kick-off a process.
  • Infrastructure configuration for use in container-based deployments.
  • Hyperparameter tuning search values for training ML models.
  • Rotate credentials because they can be accessed in the workspace GUI or automated via a script.
  • Share credentials with other people in your team working within the same workspace.
  • Centrally manage infrastructure and storage configuration for deployments and external systems used in your stack.
  • Enforce good architecture practices with modular configuration and reusable components.
  • Provide visibility into the many external components used in your data stack. This observation capability will soon be extended — see what’s coming here. 👀

Kinds of blocks

stacked colored blocks
Source: Pixabay.com

Basic data blocks

Secret block in GUI with code snippet

Notifications

Storage

Storage block options

Infrastructure

Docker container block creation GUI interface screenshot

Custom blocks

from prefect.blocks.core import Blockclass Cube(Block):
edge_length_inches: float
def get_volume(self):
return self.edge_length_inches**3
my_cube = Cube(edge_length_inches=3)
my_cube.save(name="cube-it")
GUI custom Prefect block screenshot

Nesting blocks

Gif of block building on itself
Source: Giphy

Block FAQ

What can I name my block?

How do I tell the API server about my new block type

How can I see all my blocks from the command line?

List of Prefect blocks in the terminal

Where do blocks live?

How do I edit a block?

screenshot of GUI email notification block

Should I try out blocks?

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Jeff Hale

I write about data science. Join my Data Awesome mailing list to stay on top of the latest data tools and tips: https://dataawesome.com