Getting started with LibCurl
Using LibCurl to download webpage using C
LibCurl is an open-source client-side URL transfer utility supporting multiple protocols. LibCurl is supported by almost every conceivable common platform, making it one of the most versatile libraries of its kind.
In this post, I will show you how to use LibCurl in C to make simple HTTP requests with its built-in cookie processor to programmatically log in to websites and download web pages.
Setup / Installation
Download the LibCurl for your platform here. I would be using it on Windows with MinGW compiler and hence downloaded Win32 — Generic build. I extracted the zip file and copied the contents of “include” and “lib” folder to MinGW’s “include” and “lib” folders respectively. That’s it, you are ready to write your C program.
Retrieving a Webpage
The program which I am going to write will POST the login data to a login page, retrieve the cookie and use it for downloading webpage.
First we need to include LibCurl’s header file
Now we need to initialize the LibCurl globally as below
curl_global_init(CURL_GLOBAL_ALL);
This will initialize the LibCurl’s module. Now we need to create a handle to make HTTP requests.
CURL *myHandle;
CURLcode result; // We’ll store the result of CURL’s webpage retrieval, for simple error checking.
myHandle = curl_easy_init();
Now we will define our user agent as some websites reject HTTP requests without a proper user agent. We’ll also specify for LibCurl to follow redirects, as many login pages redirect users to a home screen. Remember, we only need to do this once. LibCurl will keep these settings in effect unless we change them.
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/4.0");
curl_easy_setopt(curl, CURLOPT_AUTOREFERER, 1 );
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1 );
Next, we’ll enable LibCurl’s automatic processing of cookies.
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "");
Now we’ll tell LibCurl which URL to fetch.
curl_easy_setopt(myHandle, CURLOPT_URL, "http://www.example.com");
Let’s visit the login page once to obtain a PHPSESSID cookie.
curl_easy_perform(myHandle);
Next forge the HTTP referer field, or website will deny the login.
curl_easy_setopt(myHandle, CURLOPT_REFERER, http://www.example.com);
Now we'll prepare the data to post to login form.
char *data=”username=your_username_here&password=your_password_here”;
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
You may have to change data string according to the structure of login page. Next we'll make the second HTTP POST request to obtain the real page.
curl_easy_perform(myHandle);
And lastly we'll cleanup.
curl_easy_cleanup(myHandle);
Below is the complete code.
Next we'll compile the code.
gcc downloader.c -o downloader.exe -lcurl -lwsock32 -lidn -lwldap32 -lssh2 -lrtmp -lcrypto -lz -lws2_32 -lwinmm -lssl
The executable file thus generated can be executed to fetch the webpage.
Epilogue
LibCurl is used in numerous real-world applications. Refer this page for a list of such applications. You can customize the code above to make applications suited to your need.
Originally published at pranavprakash.net on March 20, 2014.