In programming, interfacing with various languages and their frameworks often involves the need for a developer to communicate with servers and APIs via HTTP requests. While this task can be accomplished through code, there’s also a convenient tool that stands out when it comes to interacting with web services directly from the command line. It’s widely embraced by the web development community and goes by the name Curl. In this article, we’ll delve into Curl’s capabilities and explore why it is so popular in the world of programming.
Objectives that this will cover
- Introduction to CURL
- Basic Usage of the Command
- Supported Protocols
- Advanced Features
The CURL Command
The command line tool is used for the primary purpose of transferring data between the client(usually the computer you are using) and a server via the various network protocols. These protocols include:
HTTP
HTTPS
FTP
FTPS
SCP
SMB
We will dive deeper into this later in the article.
Let’s put it like this, Curl acts as a bridge between your computer and remote servers which in turn facilitates the exchange of information.
Installation via different Operating Systems
Ubuntu
sudo apt-get install curlMac Os
brew install curlWindows
For Windows, I would recommend following the official documentation for Curl:
The Sytax looks something like this:
curl [options] [URL...]The breakdown of the above syntax:
- curl: command itself. When you run
“curl”In your terminal or command prompt, you’re invoking the Curl tool. - [options]: These are various command-line options or flags that you can include to modify the behavior of Curl. Options start with a hyphen (-) or double hyphen ( — ), followed by a specific command or setting. For example, you might use “-o” to specify an output file or “ — header” to set custom HTTP headers.
- [URL…]: This part represents one or more URLs (Uniform Resource Locators) that you want to interact with using Curl. URLs can point to web resources, files, or other network locations. You can provide multiple URLs separated by spaces if you want to perform operations on multiple resources in a single Curl command.
Basic Commands to Use with The Curl Command
Input
curl https://medium.com The Output
Running the curl command with the URL "https://medium.com" will initiate an HTTP GET request to the Medium website and display the HTML content of the page in your terminal.
Note: The output is quite extensive if you need to save it to a file, all you need to do is to define a file where that output will go. Something like this:
curl -o medium.html https://medium.comOutput
The curl command executed successfully downloaded the HTML content of the Medium website and saved it to a file named "medium.html" in your current directory. Here's what the output means:
100%: This indicates that the download is complete, and all the data has been retrieved successfully.120k: The downloaded data size is approximately 120 kilobytes.259k: This represents the average download speed during the transfer. In this case, it's 259 kilobytes per second.
Headers using the Curl Command
The curl -I command is used to send an HTTP HEAD request to a URL, which retrieves only the headers of the response without downloading the actual content. When you run the command with the website URL “https://medium.com”. Curl will give out an output displaying the website's headers.
Input
curl -I https://medium.comOutput
The output basically displays all the headers ranging from the server to its cookies.
Supported Protocols
- HTTP/HTTPS: Curl does a pretty good job in making HTTP and HTTPS requests which is the backbone of web communication. It can retrieve web pages, interact with web APIs, and perform various HTTP operations like GET, POST, PUT, DELETE, and more.
- FTP/FTPS: For file transfer needs, Curl supports both FTP (File Transfer Protocol) and FTPS (FTP Secure). It facilitates efficient file transfers between clients and servers, ensuring data integrity and security.
- Telnet: Curl even offers Telnet support, allowing remote terminal access to network-connected devices. This is particularly useful for tasks like configuring network equipment.
- Others: Curl also supports various other protocols, such as RTMP (Real-Time Messaging Protocol), RTSP (Real-Time Streaming Protocol), LDAP, MQTT (Message Queuing Telemetry Transport), and more.
Downloading Files
Picture this scenario, You work for a company that produces daily sales reports in CSV format. You need to automate the process of uploading these reports to an FTP server for further processing and analysis.
How would you go about this? Well, one of the ways would be to automate the file upload process to the FTP server. Here’s how you can achieve this:
- Scripting the Upload
Create a script (e.g., a Bash or PowerShell script) that generates the daily sales report in CSV format and then uses Curl to upload the file to the FTP server. The script would look something like this (Linux example):
# Generate the daily sales report (example: sales_report_20230901.csv)
generate_sales_report.sh
# Use Curl to upload the file to the FTP server
curl -T sales_report_20230901.csv ftp://ftp.example.com/sales/In this example, -T specifies the local file to upload to the FTP server, and ftp://ftp.example.com/sales/ is the FTP server's URL where you want to place the file.
2. Automation
Schedule this script to run daily using a task scheduler (e.g., cron on Linux or Task Scheduler on Windows). By doing so, you ensure that the daily sales report is automatically generated and uploaded to the FTP server without manual intervention.
3. Error Handling
Enhance the script to include error handling and notifications. For example, if the upload fails, the script can send an email alert to notify the relevant personnel about the issue.
Interacting with APIs
Get Requests
The basic syntax for a GET request would be:
curl -X GET https://urlIn this example, we will be using the genderize.io website to achieve this:
Input
curl -X GET https://api.genderize.io?name=Kim Breakdown of the command:
-X GET: Specifies that the request method is GET. While specifying GET explicitly is not necessary, it makes the command more explicit.https://api.genderize.io?name=Kim: This is the URL you are accessing, where "https://api.genderize.io" is the base URL, and "?name=Kim" is the query parameter. You are passing the name "Kim" as the input to the API to predict the associated gender.
Output
{
"name": "Kim",
"gender": "female",
"probability": 0.7,
"count": 83361
}Post Requests
To make a basic POST request, all you need to do is type in the command below:
curl -X POST https://example.com/Additional fields
curl -d "user=user1&pass=abcd" -X POST https://example.com/loginBreakdown of the command below:
-d "user=user1&pass=abcd": This flag specifies the data to be included in the request body. In this case, we are sending a form-encoded data string with two fields: "user" with the value "user1" and "pass" with the value "abcd".-X POST: This flag explicitly specifies that the HTTP request method is POST. While specifying POST explicitly is not necessary becausecurldefaults to a GET request if you don't specify a method, it's included here to make the request method explicit.
When you run this curl command, it will send a POST request to "https://example.com/login" with the provided data in the request body. This is a common way to simulate submitting a login form to a web server.
Specifying the Content-Type in the POST request
The -H flag can be used to send a specific data type or header with curl. The following command sends a JSON object with the request.
curl -d '{json}' -H 'Content-Type: application/json' https://example.com/login-d '{json}': This flag specifies the data to be included in the request body. In this case, you have'json'enclosed in single quote, which suggests that you intend to send JSON data. However, you should replace'json'it with the actual JSON data you want to send.
For Example:
-d '{"user": "user1", "pass": "abcd"}'This represents a JSON object with two key-value pairs, “user” and “pass.”
-H 'Content-Type: application/json': This flag sets the HTTP request header "Content-Type" to "application/JSON." This header informs the server that the data in the request body is in JSON format.
When you run this curl command with valid JSON data, it will send a POST request to "https://example.com/login" with the provided JSON data in the request body, and the "Content-Type" header will indicate that the content is JSON-formatted. The server will process the request accordingly, typically handling the login authentication process and responding with a login status or an authentication token, depending on the server's functionality.
Sending a file using curl
curl --form "fileupload=@my-file.txt" https://example.com/resource.cgiBreakdown code:
--form "fileupload=@my-file.txt": This flag specifies a file upload as part of a multipart form request. It uses the@symbol to indicate that you want to attach a file to the form. "file upload" is the form field name, and "my-file.txt" is the local file you want to upload.
When you run this curl command, it will initiate a POST request to "https://example.com/resource.cgi" and include the "my-file.txt" file as part of the multipart form data in the "file upload" field. The server will receive the file and can process it as needed. The actual behavior and response from the server will depend on the server's implementation and the purpose of the "resource.cgi" script or endpoint.
Resources to Use
While this article provides a solid overview of `curl` features, it’s essential to know that `curl` is a versatile tool with extensive capabilities. To further your knowledge about CURL I would recommend visiting the official documentation.
