Automated Status Checker
Web Automation — An Experiment
The Problem
Textbooks are an important resource required by many courses within an undergraduate program. Such resources help explain the content in more detail with examples and exercises to serve as a companion to the lecture delivered by the instructor. So you would assume, that everyone rushes to the bookstore to get new copies each year? Not quite so. As important as they are, they can be pretty expensive and heavy to carry around. Most students resort to online resources or detailed notes that are sometime provided by the course itself. Similarly, such a scenario occurred when I was enrolled in my Concurrent System (SE 3BB4) course and discovered that the author of the textbook had an online section consisting of slides of each chapter with sample code. It was going all well, just until I found that the slides were not as detailed as I hoped. In addition, the use of a physical copy would be helpful for the upcoming final exam. At that moment, I knew I needed to get my hands on that textbook. Luckily enough, the library on campus keeps at least one textbook from each course as a reference, which can be made available to any student. So I logged on to the library’s catalogue and began my search. The status came back as Checked Out, so I thought I would check back again later. Same message appeared after a few hours, which turned into days. Finding no success and running out of time, I decided to take action. And at that moment, it just hit me…
Eureka!
I remembered playing around with Selenium, a testing framework used for web automation. Now all I needed was a script that would constantly check the catalogue and alert me whenever the textbook becomes available. Then I would simply head over to the library and check it out before anyone else does. Essentially, automation would simply outsource the rigorous process of me constantly checking the catalogue, where a script can compute more effectively throughout multiple times a day. How nice, eh? Ideas like these, with their simplicity in technology and composed of few lines of code, can go a long way in achieving a solution and potentially making life easier. This is what I enjoy most about Software Engineering.
Requirements
Let’s define the application as a System consisting of various software modules in order to achieve the requirements. For me to get my textbook in time, the system would need to achieve the following set of objectives:
- The system shall open the library catalogue and search for the required SE 3BB4 textbook (Concurrency: state models & Java programs).
- The system shall parse the status of the textbook to determine whether it is available or checked out.
- If the textbook is available, the system shall prompt the user with a success message via Terminal and keep the browser active.
- If the textbook is checked out, the system shall close the browser and any corresponding test process.
- The system shall execute the script at 9:00 AM, 6:00 PM and 9:00 PM everyday until the textbook becomes available.
The design flow diagrams provide the behaviour of the system in relation to the various subsystems that need to be executed in the given order:
Selenium Test Case
The first subsystem consists of defining a test case within a new Selenium project. Selenium is a powerful browser automation framework, which allows the creation of robust regression test for web applications. In particularly, the extend use of WebDriver will allow us to define the desired test case and parse it on to the browser. In such acase, we would want to know the content of the element which the status holds and translate it as a passed or failed test. Let’s put some code to the requirements we have initialized earlier:
Open desired web page url
final String bookURL = "http://catalogue.mcmaster.ca/catalogue/Search/...submit=FIND";WebDriver webDriver = new ChromeDriver();
webDriver.get(bookURL);
Find the specified element which holds the content
/* Use WebElement to parse the inspect element of site*/
WebElement webElement;
webElement = webDriver.findElement(By.cssSelector("*[class^='callnumAndLocation']"));/* Find element that are embedded within table*/
List<WebElement> tableRows;
tableRows = webElement.findElements(By.tagName("tr"));
Transform it into a test case
boolean bookStatus = false;@Before
public void setTestParam(){
//initialize webDriver and open browser
}@Test
public void getBookStatusTest(){
//find specified element and its content
String tableArray[] = (tableRows.getText()).split(" ");
int N = tableArray.length;
if (tableArray[N-1].contains("Available"){
bookStatus = true;
} Assert.assertTrue("Book Available: ", bookStatus);
}
Shell Script
The second subsystem is a shell script which acts as a controller and monitor. When executed, the script navigates to where Maven is stored on the local machine and runs the Maven command against the test folder in the Selenium project, which holds the first subsystem. With the completion of a test case, a passed or failed status is reported back for the script to act accordingly to show a success message or end the test process.
Caution: Make sure you have Maven installed!
Execute the following command to confirm Maven is installed
mvn -versionTo run a test case
#Where CheckStatus is the Java class you want to testmvn -Dtest=CheckStatus test
Extra Credit: Considering the first two subsystem, is somewhat of a prerequiste for Continuous Integration.
Launchd
The third and final subsystem is the governing factor on when to execute the script. To allow the script to execute automatically, tools like Cron job or Window’s Task Scheduler come in handy. As for Mac, Launchd offer the same services in creating, starting and stopping the execution of applications and scripts. All we have to do now is to create a Launchd configuration of the desired times and days, which will get assigned to the script.
Configuration
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.flareapp.job</string>
<key>ProgramArguments</key>
<array
<string>/Users/.../FlareApp/run.sh</string>
</array> <key>StartCalendarInterval</key>
<array>
<dict>
<key>Hour</key>
<integer>9</integer>
<key>Minute</key>
<integer>0</integer>
<key>Weekday</key>
<integer>1</integer>
</dict>
...
<array>
</dict>
Installation procedure
Enablecp com.flareapp.job.plist /Library/LaunchDaemons/launchctl load -w /Library/LaunchDaemons/com.flareapp.job.plistChecklaunchctl listDisablelaunchctl unload -w /Library/LaunchDaemons/com.flareapp.job.plistrm /Library/LaunchDaemons/com.flareapp.job.plist
Demo
Success!
Next Steps
The following section includes enhancements to the functionality of the system, if it continues to evolve into a product. With the addition of new features, a detailed set of requirements will need to be refined for build 2.0. But for now, we will simply discuss the following enhancements as stated:
- Broader scope of usability — Instead of hardcoding the url and checking the attributes of inspect element, what if users were able to attach a flare* by just highlighting the content. For this example, I would highlight the field where it says available or checked out, to let the application know that I wish to monitor this value.
- Enhanced alert messages — The script displays a success message in the Terminal and keeps the browser active when the textbook is available. What if this trigger was enhanced to an email, social media post or phone call.
- Stand-alone account based application — Most people would not have the required technology to compile the application, so what if the application was placed on a central server with an universal dashboard for each users and their flares*. This will reduce the prerequisites for the users and have a dedicated server to conduct more frequent checks.
*Flare is simply an object that is being monitored. In this example, the flare was attached to the status of the textbook in the library’s catalogue. In same reference, flare is also the name of the application.
Product Vision
As the experiment comes to an end, I am glad that everything worked out. As for most experiments and prototypes, such as this one, they are only meant for specific function in a certain environment. Considering how a few lines of code helped me with my personal task, I wonder how many more people out there might need something similar in their lives. And upon that thought, this leads to my favourite part of the project, which is simply governed by one simple question. How do we turn this into a product?
If we consider how websites are formulated with content, we can define the specific elements as either text or status. Text are simply plain content that are displayed as a header or paragraph. Status, however are linked to the backend to display some condition that the system set out to do. With the example of the catalogue, titles such as Location and Call Number are simply text fields. The values for availabilty of the textbook will be known as a status, which is retrieved from the database correlating to a process of signing out or returning a book administered by a librarian. Considering the terminology set, we can search for other examples in which this experiment could evolve as a product into the real world.
Price tag or availability on eBay — Being alerted when the price reduces to a desired amount or when the product is made available to the market.
Anything that dynamically changes — The scope of use is open to any content on any website. If a value is determined to change and the corresponding output requires to be monitored, then this application is essential to set off a trigger to prompt the user.
GitHub
Considering trying it out yourself or want to get in on the fun? Full source code of the Automated Status Checker is available on GitHub.
