Prefect is a python based orchestration tool. But why Prefect when we already have multiple workflow management system in place ? Why one should prefer Prefect over Airflow, Dragster, ControlM etc. Well, I won’t go into that debate. Lets explore Prefect as of now and see if that suits your requirement.

This guide is here to help you get started with next gen of Prefect — Prefect 2.

The article will guide you how to:

• Install Prefect locally
• Write a basic ETL flow using Python
• Run Flow locally and use Orion Server
• Run Flow using Prefect Cloud

Below code has been run on Windows machine using VS code but will work alike in mac/Linux environment.

Basic Terminology

Tasks — A task is a Python function that handle workload. It optionally takes inputs, performs action and produces output.

Flows — Prefect refers to a workload as a “flow”. Loosely stated — they are like DAGs of Airflow. Flow consists of tasks. It describes the order in which the tasks are executed. Flows can include calls to tasks as well as to child flows, which are “sub flows” in this context.

Prefect Cloud — A hosted UI to orchestrate workflows and provide visualization into the status of different flows and runs.

Agents - Daemon process running on your local/cloud machine which executes flows when scheduled using Prefect Cloud.

Step 1: Installation

Prefect 2 requires Python 3.7 or later. There are multiple ways to install Prefect.

Here, we will create a virtual environment and install requests, pandas and prefect which will be used in the demo script.

# set up virtual environment in Windows machine. 
python -m venv venv # create the env 
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser # Optional
.\venv\Scripts\activate # activate the virtual env

Install Prefect along with modules used in the script using pip

pip install requests pandas prefect

Step 2: Write ETL flow and run it locally

The ETL pipeline will download data from a dummy API, do basic transformation and save the file on the disk. Below is the description of different functions used in script:

extract(url: str) -> dict makes a GET request to the url parameter which returns a dictionary else raises an exception.

transform(data: dict) -> pd.DataFrame — transforms the data i.e. the output returned by extract by keeping just id, userid and title and returns the transformed data as a Pandas dataframe.

load(data: pd.DataFrame, path: str) -> None — does not return anything. It saves the transformed data to a CSV file at path and append a timestamp to the file name.

Let’s create a python file, call it demo.py and a directory data to store the transformed data as CSV.

Code Snippet

import json
import requests
import pandas as pd
from datetime import datetime
from prefect import task, flow


@task(name="Extract Task", description="Makes request.", retries=10, retry_delay_seconds=10)
def extract(url: str) -> dict:
    res = requests.get(url)
    if not res:
        raise Exception('URL seems unavailable!')
    return json.loads(res.content)

@task(name="Transform Task", description="Transform Existing Data.")
def transform(data: dict) -> pd.DataFrame:
    transformed = []
    for user in data:
        transformed.append({
            'ID': user['id'],
            'UserId': user['userId'],
            'Title': user['title']
        })
    return pd.DataFrame(transformed)

@task(name="Load Task", description="Persists data to disk.")
def load(data: pd.DataFrame, path: str) -> None:
    data.to_csv(path_or_buf=path, index=False)

@flow(name="ETL Flow")
def etl_flow(p_url):
    users = extract(url=p_url)
    df_users = transform(users)
    load(data=df_users, path=f'data/posts_{int(datetime.now().timestamp())}.csv')
    return flow

if __name__ == '__main__':
    flow = etl_flow(p_url='https://jsonplaceholder.typicode.com/posts')

Run the flow locally and check the output.

python .\demo.py 

The pipeline is complete and we can see detailed information about each task’s states. Once complete, a CSV file gets created in data directory.

Step 3: Flow Orchestration

Orchestration of Flows helps in scheduling and deploying workflows. Prefect Orion API server and orchestration engine receives state information from workflows and provides flow run instructions for executing deployments. You can spin up an instance of Orion anytime using CLI

prefect orion start

Hit the URL : http://127.0.0.1:4200/ (check the server output) to view the Prefect Dashboard listing the details of the flows and runs.

Scheduling the flow-runs through deployment build.

The deployment build will create a deployment definition etl_flow-deployment.yaml file. Path to the flow is specified in the format path-to-script:flow-function-name — The path and filename of the flow script file, a colon, then the name of the entrypoint flow function. If configured, it will upload the flow files to the configured storage location and will submit the deployment to the work queue demo(will create if it doesn’t exist).

prefect deployment build demo.py:etl_flow -n demo-deployment -q demo --cron "0 0 * * *"
prefect deployment apply etl_flow-deployment.yaml

The deployment file will look similar as below:

Once deployment apply is executed, an deployment entry and work queue is created.

You can edit the deployment configuration by editing the deployment entry.

To execute flow runs from this deployment, start an agent that pulls work from the demo work queue.

prefect agent start -q demo

To run the flows manually you can also trigger it from UI: Deployment > Runs > Run. Once triggered, the agent will execute the flow.

NEXT: Running Flow with Prefect Cloud

To get started with Prefect cloud:

1. Sign In / Register a Prefect cloud account and create a workspace.

Create an API key for authorizing the execution environment. This can be an agent running locally on your machine or into the cloud environment. Login to your cloud account and set the workspace.

prefect cloud login -k <Key>
prefect cloud workspace set --workspace "<accountname>/demo"

Now all the flow run can be administered on the Prefect cloud.

Re-Build and Apply deployment configuration which will create an entry in Prefect Cloud Deployment.

Run the agent locally to execute the workflow. You can also run your agents in the cloud to run your workflows orchestrated in Prefect Cloud.

Please refer to the official Documentation for advanced topics and configurations.

No ! It’s not a regular write up to reveal the already known features of the WhatsApp. It’s purely technical, yet funny stuff dedicated to our “lazy” tech enthusiasts.

Okai ! Let me ask you something. Have your nightmare of not wishing your dear one’s due to your oblivious behavior turned into reality ? Or, Do you ever wished to integrate your business with WhatsApp ? If yes, then Go ahead. Here we will try to send a message to individual or group using python.

Requisites:
1. python 2.7
2. Chrome Driver: Download compatible version of the driver and the chrome/chromium w.r.t. your OS
3. Connection to internet and WhatsApp installed phone

PS: Below snippets has only been run and tested on Windows 10 but should run smoothly on others too with some tweaks.
** If any module is not installed, use pip.

To learn more about chrome drivers → click me.

Now everything is set, lets cook some code.

Import the necessary modules:

Set up the message to be sent and the corresponding xpath. You may need to take help of chrome’s developer tool (in case below doesn’t work).

Integrate the codes and run.
* Change paths accordingly, just reminded. :D

Once you have run the above code, a new instance of the browser will be forked. Below is the page alike that you will see.

On the mobile app, click the “WhatsApp Web” from the right corner three dots under “chats” tab as shown below and scan the code.

Ta Ta-ra-a-an !! You are done.

PS: I don’t take any responsibility if your friend blocks you, or you are evicted from any group, or even WhatsApp blocks you. ;)

Let’s make this world a better place for the eggs yet to be hatched.
Clap if you liked.

I started it out of my interest for IoT and thought of sharing it to the bigger community. It’s a simple blog to setup, deploy and test sample code on your MCU.

Let me know if it was helpful to you in any way.

* All the steps listed below are performed in Windows 10 and on Nodemcu esp8266

Setup your Arduino IDE:
1. Download and install the driver to port codes to MCU

Verify the driver installation: Connect the device (nodemcu) and navigate to Device Manger > Ports. You will see the connected device listed.

2. Download and install the IDE
Go to File > Preferences and type below URL into “Manger URL” text box as shown below. Install the package for esp8266. Restart the IDE.

URL: http://arduino.esp8266.com/stable/package_esp8266com_index.json

Navigate to Tools > Board. Select ESP8266 (shown below).

Test your NodeMCU:
The below sample code snippet makes ESP8266 to blink at regular interval.

Push the code to MCU:

Once Compiled, the LED will start blinking. Done.

Let’s make this world a better place for the eggs yet to be hatched.

Life would have been too easy if everything would have got in place as per your plan. Alas ! There are unexpected impediments on the way and those who don’t plan for it are bound to ‘exit’ before the end. Programming and life does imitate each other.

‘Exceptions’ are ‘RUN TIME ERRORS’ or ‘EXECUTION TIME ERRORS’. When we feel there is a part of code which may not work as expected if conditions are not met, we handle those situations in exceptions.

If a function doesn’t handle it, the exception is passed to the calling function, then that function’s calling function, and so on “up the stack.” If the exception is never handled, your program will crash. Python will print a “traceback” to standard error, and that’s the end of that.

Below are couple of examples when you don’t handle the exceptions and the program exits. No code is executed from the point an error is raised.

But, what if you have to continue execution even if there is an exception just like life doesn’t end if something goes wrong.

There are few built in exceptions which can be raised directly without defining when they need to catch the exceptions.

Common Built-in-Exceptions:

StopIteration: Raised when the next() method of an iterator does not point to any object.
SystemExit: Raised by the sys.exit() function.
ArithmeticError: Base class for all errors that occur for numeric calculation.
OverflowError: Raised when a calculation exceeds maximum limit for a numeric type.
FloatingPointError: Raised when a floating point calculation fails.
ZeroDivisonError: Raised when division or modulo by zero takes place for all numeric types.
AttributeError: Raised in case of failure of attribute reference or assignment.
EOFError: Raised when there is no input from the input() function and the end of file is reached.
ImportError: Raised when an import statement fails.
KeyboardInterrupt: Raised when the user interrupts program execution, usually by pressing Ctrl+c.
IndexError: Raised when an index is not found in a sequence.
KeyError: Raised when the specified key is not found in the dictionary.
NameError: Raised when an identifier is not found in the local or global namespace.
UnboundLocalError: Raised when trying to access a local variable in a function or method but no value has been assigned to it.
ValueError: Raised when the built-in function for a data type has the valid type of arguments, but the arguments have invalid values specified.
FileNotFoundError: Raised when a file or directory is requested but doesn’t exist.
TypeError: Raised when an operation or function is applied to an object of inappropriate type.

But, what if you don’t know the type of exception that will be thrown. Just write an except block for all the unknown error and it’s done.

But, what if we want to catch undefined errors specifically and don’t want to club all the undefined exceptions into one. User defined exceptions comes to rescue. Exceptions need to be derived from the Exception class, either directly or indirectly.

If something goes unplanned, make sure you catch it.

Reference: https://docs.python.org/3/library/exceptions.html

Let’s make this world a better place for the eggs yet to be hatched.
Clap if you liked.

No. No. No. The image has no relevance with the text below. I am not talking about granny’s method to make pickle out of anything under the sun. So, non programmers, my apologies, you can go ahead and click the red-cross-box on the right upper corner. Adieu.

Serialization (or, pickling) is the act of converting a class’s object into a byte stream. The process is also referred to as deflating, flattening or marshalling. The opposite operation, i.e. extracting a data structure from a series of bytes, is deserialization (also called inflating or unmarshalling or unpickling).

Okai ! That’s cool. But, why we need serialization ?

Whenever we need persistence i.e. want to save the state of an object between user sessions (or, even users), or pass it over the network, share it with other processes, serialization comes to rescue.

In python, inbuilt “pickle” class supports serialization.

Great ! But, How can I do it ?

pickle.dump() function in the pickle module takes a serializable* python data structure, serializes it into a binary, Python-specific format using the latest version of the pickle protocol, and saves it to an opened file.
*Not every Python data structure can be serialized by the pickle module.

pickle.load() function takes the serialized data stream and returns the restructured data structure.

Let’s pickle now.

pickle.dumps() function instead of taking a stream object and writing the serialized data to a file on disk, it simply returns the serialized data.

pickle.loads() function instead of taking a stream object and reading the serialized data from a file, it takes a bytes object containing serialized data, such as the one returned by the pickle.dumps() function.

When the Pickler encounters an unpicklable object, it raises a PicklingError. This class inherits from PickleError. Best way to avoid an error is to know, what we can pickle and unpickle.

1. All python supported datatypes : numbers, bools, strings, bytes, streams, numbers, tuples, sets, lists, dictionary.
2. Built-in functions, user defined functions (using def, not lambda) and classes defined at module’s top level

Happy Pickling !!

Let’s make this world a better place for the eggs yet to be hatched.
Clap if you liked.

Roses are red,
Red is the blood that it’s thorns give you.

Red is the sky when its dawn,
When its dusk, its still red.

Red signals danger,
Red denotes life too.

Anger turns you red,
Red are you, when you blush.

Red is the color of passion,
It’s the same red when you denote prohibition.

Red is your hand when you are being married,
Red is your hand when you end a life.

Red are the stains on the sheet when you are born,
Red is it when you end yourself.

Let’s make this world a better place for the eggs yet to be hatched.
Clap if you liked.

You may call these notions as ‘Facts of Life’ or may call them just ‘Life’. We sabotage the reality by sugarcoating the facts and try to live in an illusion created by ourselves. But, the truth is, life is quite gross and will always be the same. Quite brutal.

My notions/opinions may look quite absurd and contradictory on the plane. But, once you start digging deep in your life, you will encounter it’s grains of wisdom. Lets dare to face the truths from my perspective in the most crude way. Hope, I am able to do justice with them.

1. No One Cares. Pained ? Depressed ? Broken or Broke ? Guess What ! No one cares. Why should anyone ? Everyone has already lived the pain sometime, somewhere in his/her life in some or the other form. Pain is meant to be felt. Get used to it. Free yourself from the shackels of expectations.
2. Out of Sight is Out of Mind. “We will be together forever”, “You are in my thoughts wherever I go”, “Our friendship will always be the same” bla bla — I wish I could say they hold true irrespective of timelines. But, Alas ! Life shows too many shades, bright and dull in such a short split of time, that promises remain the same, just the promisee changes. Its not anyone’s fault. Its just the priorities of life are much defined by the proximity of space, time and relationship. Don’t complain.
3. Learn to say ‘NO’. Don’t mistake me to have negative approach to everything that comes in your life. But, do say ‘NO’ to all the things for which you are convinced is not a good idea to follow currently. Don’t get swayed away by emotions. If you are important to them, they will empathize. If they don’t, ‘NO’ was the best you can bestow.
4. No Escape from Karma. There is no such thing as ‘Let Bygones be Bygones’. If your past was turbulent, make peace with it. Don’t run. Don’t hide. Face it. Your past will keep revisiting you in your present. Best way to escape your past is make your present worth memorable. Remember, nothing f**ks harder than time, and past was the time which was once your present.
5. Don’t be Credulous. Don’t mistake me in being cynical of everything. Just don’t reveal your secrets to everyone who empathize with your emotions. Be selective. You never know when a friend may empathize with your foe. Remember ! You won’t be shown mercy for your naivety.
6. Stop being imprisoned to the notion of Perfection. Perfection is an illusion. You won’t be ever able to achieve it. There is a reason to it. Everyone has his/her version of being perfect. So, Stop trying being perfect. Just try being a better individual every passing day.
7. Pleasing Everyone is sure shot mantra for despondency. Just don’t try. Rather please yourself. There is nothing wrong in it until you become sadist. Loving yourself eliminate the chances of being ignored for your efforts.
8. Money don’t buy happiness. Rather than earning pennies, earn memories, a good one. When you will look back 10 years hence, you will have reason to smile. No matter whatever be your pace, money will always be faster. Remember ! When you are at the deathbed, those nights and holidays spent in office, away from your family, will be for nothing.
9. Practice Gratitude. Don’t end your day without a Thanks for the abundance you have been bestowed with. Find a reason to thank, even if it is just for a beautiful smile which she/he gave you.
10. You have been bluffed with the notion of Eternal Happiness. You wont find it. Stop searching for it. Happiness is quite relative. The joy of riding a bike may not be the same five years hence. The perfect time to be happy is — Now. Be happy, spread happiness.
11. Don’t mistake crush for love. You may screw your life. There is no litmus test to distinguish between them. You can only know when time has elapsed. If the collywobbles dies with time, it was crush. Just give yourself — Time.
12. Don’t try to copy someone else’s life. You owe a different set of shoes. Neither you have walked down his path. So, be original version of you. You are unique in this whole cosmos. Create your version of life. Live life from yours perspective.
13. Be revengeful but only in dreams. It’s wicked. It’s sadist. Hate for hate will leave the whole world hateful — without love.Vengeance is fed by ego and ego only gives temporary joy and permanent misery.
14. Just Love is not Enough. Why it should be ? Did it solve your financial problem or your relationship problems ? Were you spared of your debts since you were in love ? NO. Love does not guarantee compatibility, neither empathy nor ‘eternal’ commitment.
15. Looks matters (but not always and isn’t everything). It’s often said — First impression is last impression (BTW, I don’t agree). And, looks does the same. It paves the path to break the ice. You may call it ‘Appearance’ or whatever, and I know it’s quite superficial, but it does matter. Bullshit is the theory that — ‘Beauty is in eyes of beholder’. Yeah, there will be differences on the degree of beauty but everyone will agree with the beauty of Marilyn Monroe ! Right ?
16. Dream. And Work for it. Dreaming is important but don’t ‘just’ dream. Don’t wait for a Genie to give wings to your dreams. Dream to build the castle but be ready to put the effort to lay down its foundation.
17. Spend time with yourself for yourself. By ‘Yourself’ — I mean your priorities. In noway I am suggesting to not shoulder your responsibilities, but decide the priorities of your life and work accordingly. No matter how hard you try, there will always be work to be completed. It will be easier to decide when you answer, ‘Why you exist and for whom ?’ .
18. People are like seasons. Some may last long and some may vanish in the blink of an eye. Those who are short lived will come with a purpose. They are God sent. They will just clean the mess in your life and on some fine day, without any reason will tiptoe away from your life. You won’t be able to hold them in your life. Then, there will be some who will stay with you no matter what the circumstances are. They are your life lines. Take care of them. And, then there will be some who will be just like seasons. They will keep visiting you. You just need to be cautious about them. Some may come with good omen and some may jinx.
19. You cannot pee when you sneeze. :D
20. Don’t take life too seriously. No one has escaped. Everything comes with an expiry date. A waiter panics if he spills the coffee on customer’s shirt. But do you really think will it matter five years down the line ? If , howsoever he remembers, he can only smile on that silliness. Just, apologize and move on. Remember, Everything you like will cease to exist. Everyone you love will die eventually before or after you. Help them live. Help them to pursue their definition of ‘happiness’. Help them create a better version of themselves.
21. Final slumber is inevitable. Death is only truth . Accept it. It is coming and it won’t stop. It won’t wait — No matter how many dreams are yet to be ticked. No matter how many confessions you still want to convey. Live before you die. Live today ! Not in yesterday , not in tomorrow. You don’t know when you will meet your last breathe. Better stop waiting.

That’s LIFE — “ Living as IF you are on Edge” or “Live beFore it Ends”. Everything will work out eventually. Just keep going. Chill. Best Wishes. Peace.

Let’s make this world a better place for the eggs yet to be hatched.
Clap if you liked.