LiveAPI — A Visual Data Extraction Tool

Image for post
Image for post

LiveAPI is a developer tool that can be used to turn any website’s public data into an API in minutes. It consists of two parts: a Chrome Extension that facilitates selection of elements to extract and a user hosted server that extracts the data on an interval, caches and serves it at user-created API endpoints.

If you have not installed the LiveAPI server on your computer, or a cloud-hosting solution like AWS, read Part 1 to setup the server.

If you have not created one or more user accounts on your server, read Part 2 on authentication.

Once steps one and two are finished, install the LiveAPI Chrome Extension from the Chrome Extension webstore.

To activate the extension, navigate to the target URL you wish to scrape and click the browser action icon (right side on browser top bar).

The UI bar will move you through the necessary steps for designing your endpoint.

Image for post
Image for post

Step 1: Click on a DOM element of your choice that contains one of the properties that you want in your data objects; you will see your selection highlighted in yellow. Select another of the same kind on the page and if the elements are structurally similar, LiveAPI will highlight the remaining items of that kind on the page. Deselect anything you do not wish to include. Enter a name for your new property (such as “title”) then click “Save”. Repeat as desired for additional elements.

Note: Use names with no spaces that are camelCased or snake_cased. Please do not use spaces or special characters besides dashes. These will be your keys in your key value pairs of each object.

Step 2 and 3 are not functional at the moment and are in development.

Step 4: Authenticate yourself with the credentials you created on the LiveAPI server. The address field is where you indicate the URL and port of your server. In most cases the port will be 4443.

Step 5: Create a name for your endpoint. Observe the same rules for naming as above because this name will be part of your endpoint URI. Enter an interval measured in seconds and click “Create Endpoint”. You will then have the choice to navigate to your endpoint to verify accuracy of your data or you can elect to begin another endpoint design.

And that’s it!

Don’t hesitate to reach out with questions, suggestions or issues you have about LiveAPI. We’re excited to keep expanding it.

If you’re interested in learning more, check out the project’s github or our website. We’ve written a bit on how it works under the hood as well.

Current contributors: Penn Wu | Melissa Schwartz | Brett Beekley

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store