Beerway Oriented Programming in F# [ 1 ]

11 min readJul 15, 2017

Introduction

In this blog post, I’ll be taking a detour from my regular Astrophysics related posts and write about a side project I have been working on to alert me if there is an update in the beer selection of my favorite brewery: Tired Hands. Similar to my previous posts, I’ll be using F# with VS Code, Mono, Ionide, PAKET and FAKE on a Mac.

I plan to split discussing the development of this project into 2 blogposts: the first one builds the foundation via Railway Oriented Programming and defines the domain and basic functions in the pipeline. The second one highlights how to make the code a lot more generic to make it conducive to add more breweries in addition to introducing other accoutrements such as to improve monitoring, scheduling and testing.

We’ll be developing the basic pipeline involving:

Scraping the latest list of beers from the Tired Hand’s website
Comparing the difference between the previous and latest scrape
Alerting via text if there is a difference.

Thank you, F#: it’s pretty cool to improve my beer drinking endeavors using functional programming concepts [ M*nads, Algebraic Data Types, Pattern Matching to list a few ].

Impetus

Inspired by Scott Wlaschin’s posts about Railway Oriented Programming on his awesome website, F# for Fun And Profit, I wanted to incorporate the idea of segregating the “Unhappy Path” from the “Happy” one in this project along with making use of the other gifts incorporating ROP brings.

Additionally, I wanted a way to get an alert whenever Tired Hand’s beer selection from their Fermentaria changed. Hence, the name just had to be a portmanteau of Beer and Railway Oriented Programming.

Through developing this project, I was successfully able to earn a competitive advantage over my fellow beer-drinking cohorts who also frequent this brewery by knowing way in advance which new beers to try, beforehand.

Setting Up

We begin by creating a new folder and console application based project in VS Code called “BeerwayOrientedProgramming”. Creating the new project can be done by opening up the Command Pallet [ Cmd + Shift + P ] and following these steps:

The result of creating the new project makes our newly created directory look something like:

We’ll be making use of the following libraries, which’ll be used in this as well as the next blogpost:

FSharp.Data’s HtmlTypeProvider for easily scraping the website to create domain specific records of the latest beer information.
Chiron, a library for working with JSON in F# for the deserialization of the beer information of the previous scrape and serialization to persist the current beer information of the most recent scrape.
Twilio API to send out a text in case there were differences found between the previous and current scrape.
Logary, a bazooka of a logging library that provides metrics as well. [ Used in the next post ]
FSharp.Configuration to parse the configuration file for the Twilio API Account SID and Auth Token as well as the interval time between timer callbacks. [ Used in the next post ]

We start by adding all the aforementioned dependencies using PAKET after first opening up the fsproj file. To demonstrate adding dependencies via PAKET, the example that I am using is that of adding Chiron to the project.

We can check if the dependency was successfully added by making sure the most recently added reference made it through to the fsproj, paket.references and paket.dependency files.

Railway Oriented Programming

Railway Oriented Programming is a clean, robust, functional way of dealing with the unhappy path. The unhappy path is the path of the entire smorgasbord of exceptions, errors and tears. A presentation that does true justice explaining this topic by Scott Wlaschin can be found here.

In a nutshell, Railway Oriented Programming involves segregating the Happy and Unhappy Path of the proposed pipeline. The output of each ROP function, or a function in this pipeline, will return a Result type that’s either a Success or Failure.

If the result is a Failure, we bypass the rest of the functions in the pipeline and return the Failure else, return a Success output that’ll serve as the input to the next function in the pipeline.

As a result, the failure cases are determined beforehand providing documentation and compile time type checks of the defined unhappy path that is now elevated to the status of being a first class citizen in our pipeline.

We add a new file to the project by creating a new file called ROP.fs that’ll contain all our Railway Oriented Programming functions:

And add the file to the project. Since this file will be used by other modules, we’ll be moving the file higher up in the compilation order.

The code from this module looks like the following; I nicked this from the presentation and blogpost. More information can be found there; as highlighted before, there are no better resources in my opinion than Mr. Wlaschin’s post and presentation.

The Domain

We define our main record type, “BeerInfo”, to consist of “TimeOfScrape”, a DateTime, and “Beers”, a list of strings representing all the Beers from that scrape.

To make use of Chiron, we have to define two static members: ToJson and FromJson to Serialize to and Deserialize the BeerInfo type from respectively. More information about how Chiron works can be found here.

The Goal

Once we have defined our domain, we define our goal to be able to do the following in order:

Scrape the Tired Hand’s website for the beer selection. Success is if we successfully retrieve the BeerInfo record of the latest scrape. Failure is a ScrapeError that can be as a result of a Web Exception such as a request timeout or unauthorized access to the website.
Compare the results of the latest scrape to the previous one and find differences. Success is if the difference between the previous scrape and latest one. Failure is a NoDifference error if there is no difference between the scrapes or a CompareError in the case of an IO exception while deserializing the previous scrape results or serializing the latest scrape results.
Alert us via Text Message incase a difference in the beer selection has been found. Success is the process of successfully sending out a text and Failure is an AlertError if we encounter an API Exception in the case of incorrect inputs or reaching some throttle limits.

Our pipeline should read like:

scrape >=> compare >=> alert

We’ll add The Errors of each step of the way by creating a new file: Common.fs containing modules of our common components moved below ROP.fs in the compilation order:

Scrape: Scraping and Type Providers

Type Providers were one of the main reasons I decided to start learning how to develop in F#. Inspired after a mind blowing presentation by Rachel Reese on them [ December 2015 in Philadelphia ], I set out on a tireless journey by enlisting myself as an enemy of Uncontrolled Mutable State using F#.

We make use of the awesome Html Type Provider in the FSharp.Data library that we should currently have downloaded via our PAKET excursion while setting up our project. We start by adding a new file, “TiredHandsScraper.fs”, containing our TiredHands scrape specific functions and then move this file above BeerwayOrientedProgramming.fs containing our entry point and main function.

To get the list of all the beers, we traverse the required descendants of the DOM and filter out the undesired ones. This process involved looking up all the divs with names “menu-item-title”, removing some edge cases and cleaning up the filtered results. The code ends up looking like:

I have to admit, this was the trickiest part of the project since I had no idea how the markup of this website was structured but after some trial and error, I got the data fairly smoothly.

We now have all our components to create the scrape function, which returns a Success of a BeerInfo if there are no scrape exceptions or a Failure of the scrape exception message; this function looks like:

Here is an example of the result of the scrape function on July 15th, 2017:

Success
  {TimeOfScrape = 7/15/2017 2:25:35 AM;
   Beers =
    ["HopHands "; "SaisonHands "; "Wish Fulfillment "; "Alien Church ";
     "Progression Through Unlearning II "; "Aloha Ra "; "Trendler Pilsner ";
     "Ridiculoid "; "Elaborate Expectations "; "Gatherer 2017 ";
     "Forced Harmony "; "Awake Minds Coffee "; "Iced Darjeeling Tea";
     "PopHands Soda Pop"; "In House Bottle Selections"; "Ourison"; "Shambolic";
     "Eyes Closed & Open"; "Undertow"; "Hysterical Obelisk"; "Obliterative";
     "Heavy Gem Virga"];}

Yes, I know, I should be asleep.

Compare: Deserialization, Set Differences and Serialization

Our next step is to write our Compare function that:

Deserializes previous scrapes; to keep things simple I am simply serializing and writing the JSON to a file called “TiredHandsScrape.json”. If this file isn’t found, we automatically have it created by step 4
Compares the Beers of the previous scrape to the current one by using Set Differences that get us the beers present in the current scrape that aren’t in the previous one
Outputs either a Success consisting of a difference or a Failure of either a ComparerError or NoDifference
Serializes the result of the scrape to serve as the previous scrape BeerInfo for the next one and saves it to the TiredHandsScrape.json file.

We move the functions pertinent to this Compare step into its own module in Common.fs called Compare.

Overall, this module looks like:

The JSON file, TiredHandsScrape.json, from a later scrape with the compare function looks like:

{
  "Beers": [
    "HopHands ",
    "SaisonHands ",
    "Wish Fulfillment ",
    "Alien Church ",
    "Progression Through Unlearning II ",
    "Aloha Ra ",
    "Trendler Pilsner ",
    "Ridiculoid ",
    "Elaborate Expectations ",
    "Gatherer 2017 ",
    "Forced Harmony ",
    "Awake Minds Coffee ",
    "Iced Darjeeling Tea",
    "PopHands Soda Pop",
    "In House Bottle Selections",
    "Ourison",
    "Shambolic",
    "Eyes Closed & Open",
    "Undertow",
    "Hysterical Obelisk",
    "Obliterative",
    "Heavy Gem Virga"
  ],
  "TimeOfScrape": "2017-07-15T13:21:00.8194460Z"
}

I know hardcoding the name of text file is probably not the best idea but don’t worry, we’ll make this module way more generic in the next blog post.

Alert: Sending a Text via the Twilio API

Developing the last item in the pipeline is fairly simple since the Twilio API is pretty easy to use.

Twilio is a “cloud communications platform for building SMS, Voice & Messaging applications on an API built for global scale.”

We start create a new module called “Alert” in our Common.fs file and make use of the C# Twilio API to send our text in case differences between the two scrapes exist. A great place to get started is here. After you are done setting up an account and get the phone number to use, the Account SID and Auth Token, you should be good to start sending text messages.

The code for this module looks like:

Again, we’d very much like to create a configuration file rather than hardcode the information from Twilio to send out a text.

Side note: I tried to add the parameter names to the Create function but couldn’t since the first parameter’s name is ‘to’, which is an F# language keyword used for looping. More details here. Maybe there is a way to do this?

The Pipeline: Tying it All Together

Now, we aim to combine all the modules we have been developing to get us the latest scrape difference, if valid, via a text.

This combination process is extremely straightforward: all we need to do is apply some Railway Oriented Programming magic to this melting pot of module functions. Overall, BeerwayOrientedProgramming.fs looks like:

Beautifully simple, yet powerful enough to handle all our errors, isn’t it?

To build, we use FAKE to build our project by opening up the Command Pallet [ Cmd + Shift + P ]:

We can confirm the success of the compilation by taking a look at the build messages and double checking if we get an Ok from our Build Time Report.

Next, we run the program, using Mono in the following manner from our shell by changing directories to get to our top level BeerwayOrientedProgramming directory; the build folder will contain our freshly baked dlls and main exe:

mono build/BeerwayOrientedProgramming.exe

If no differences in the beer selection are found, we get:

Failure NoDifference

If either this is the first time we run our program or differences are found from the previous scrape, we get the following from the command line:

Success "New Message Sent Sucessfully"

And if the stars have aligned, we get a text similar to the following:

Troubleshooting

The order of the compilation in our fsproj file should look like the following:

if compilation fails, we need to ensure that the compilation order is similar to this; note that BeerInfo.fs and ROP.fs can be interchanged since they are independent of any other module.

Another gotcha is incorrectly entering the Twilio API details which should be easy to identify through our robust pipeline.

Next Steps

Now that we have created a basic pipeline, we want to add a couple of more features namely:

Configuring the Literals so none of this hardcoded mumbo-jumbo continues
Logging the results along the way; this will be especially helpful for situations where the markup changes significantly and we don’t see any results. Additionally, we’ll eventually want to make use of regression to predict the patterns of beer releases based on the log information
Scheduling the process to run on a timely basis
Generalizing the scheduler to run the pipeline for multiple breweries
Testing our ROP functions
Adding Async Workflows for all our IO interactions.

We’ll be discussing these in an upcoming post, so, stay tuned.

Conclusion

In this blogpost, we explored how to make use of Railway Oriented Programming to alert us in the case the beer selection has been changed from the Tired Hands brewery. In the next post we plan to build on top of this foundation to add a lot more features as mentioned above.

The code of this project can be found here. I have added code specific to this blogpost in the “blogpost1” branch.

As always, any feedback will be greatly appreciated! I had a great time writing code for this project; improving it in any way will make me an even happier camper.

An additional thanks to my fellow mad man, Nick A for introducing me to this idea and Tired Hands Beer.