Step 3: Web Requests
So our last little venture as an intro to networking went quite smoothly except for the whole subnet masking soire. I’m assuming I was just too tired but I’ll come back to it when I actually need it.
Today… web requests! This is the next step in our little ‘hold me accountable for my learning’ thing. Thanks to the new followers/readers, I appreciate you guys. I realise this is not interesting at all and I appreciate it, but yeah this stream of conciousness/note taking is for me, but it’s nice to see others are following along.
I’m working through the pre-requisites for the ‘Getting Started,’ module for HTB Academy. Just need to do Web Requests and Introduction to Web Applications. Apparently this should take about 15 hours… maybe that’s why I didn’t understand the networking topic… ah well WE MOVE FORWARD!
Probably gonna have to do this in two (maybe three) sittings. Four hours apparently and it’s currently 9am on the 5th of June and I get up at 5am for work! Ah the, ‘work on a career switch while still working full time and arranging moving to a new country with your girlfriend while still talking to her as she is back with her parents for the summer because her academic year ended earlier than yours because her boss wanted to save money but not paying her the last two months of her salary.’
HyperText Transfer Protocol (HTTP)
Okay so just stuff I already knew, bar HTTP default communication port being 80… cool beans. Definitely need to remember the difference in FQDN and URL.
URL
Scheme — Identifies the protocol http://https://
User info — optional component with the @ after it to precede host
Port — if not specified http defaults to 80 and https to 443
Path — indicates resource being accessed, usually a file or folder.
Query string — starts with a question mark with a parameter (eg. login) and a value (eg true). Multiple parameters can be split with ampersands.
Fragments — processed by the browsers client-side to locate sections within the primary resource
Main mandatory fields are scheme and host.
HTTP Flow
cURL
Web requests sent through a browser (Chrome/Firefox) and the cURL command line tool.
cURL (client URL) is a command-line tool that mainly supports HTTP. Ideal for scripts and automation. Ideal for web pentesting.
cURL does not render HTML/Javascript/CSS cose but prints raw format. As pentesters, we are focused on the request and response context which makes this more convenient.
Can also use the cURL to download a page or file and output the content into a file using the -O flag. Can use lower case ‘o’ to specify the name.
s flag silences the status and -h flag is your normal help flag.
A little capture the flag reminding me of PicoCTF (the reason I stumbled into cybersecurity from machine learning)…
Well that was easy enough… thanks PicoCTF!
So I’ve had a few distractions but I’m looking at the next section and VERBOSE… so I’ll leave it there for now. I think I’ll just continue this section because I’d really like to leave each Medium entry as one module, even if it ends up being several hours of content. This is more for my organisation than ease of reading (not that people should be reading this haha, I am doing this for me, not to garner an audience, although the accountability is nice).
Hypertext Transfer Protocol Secure (HTTPS)
Alright… we’re back. It’s the next day. It’s 7pm the day after and I’m feeling good. Just had a shower to ‘wash off the day,’ and I’m ready to get stuck in.
So HTTP is transferred in clear text and is more prone to Main-in-the-middle (MiTM) attacks to view transferred data. HTTP is effectively being phased out due to the lack of encryption. Funny enough one of my Year 7 classes today was asking me why they haven’t learned about e-safety (a crime in this day and age). We got to talking about the double ended encryption Whatsapp uses. Anyway…
Spot the difference…
Even though data sent over HTTPS protocol may be encrypted, the request may still reveal visited URL if it contacted a clear-text DNS server. So… use an encrypted DNS server or a VPN to ensure fully encrypted traffic.
HTTPS Flow
If you use HTTP instead of HTTPS to visit a site that enforces HTTPS the above diagram shows what happens. This is called a 301 Moved Permanently response code… coming soon.
HTTP downgrade attacks can happen which takes HTTPS down to HTTP by setting up a MiTM proxy to transfer traffic through attacker’s host without user’s knowledge. Most modern browsers however and servers as well as web applications have protections against this attack.
cURL for HTTPS
cURL should automatically handle HTTPS communication and perform the secure handsahek then encrypt and decrypt automatically. However, if you try a website with invalid SSL certificates or an outdated one, then cURL would not proceed to protect against MiTM attacks.
You can skip the certificate check with cURL using the -k flag.
HTTP Requests and Responses
HTTP version 1.x sends requests as clear-text, and uses a new-line character to split different fields and requests. HTTP version 2.x on the other hand sends requests as binary data in dictionary form.
cURL (again)
the -v (verbose flag) will print request and response as shown below:
vvv flag gives even MORE info…oh dear. We’ll come back to that later.
Browser DevTools
Modern browsers come with build-in developer tools. As web pentesters we will use these too. Already I’m seeing I’m going to need to commit some time to some basic html… that’s fine. Should be easy as the sheer number of hours I spent messing around with html in the myspace days was TERRIFYING.
These questions were easy peasy. What’s the HTTP method (GET) and what’s the version of Apache on the server.
Simple little -v flag on the curl request and you’re golden!
HTTP Headers
Five categories of headers: general, entity, request, response and security
General — both requests and responses. Describe the message rather than its contents.
Entity — common to both request and response. Describe the content transferred by the message. Found typically in responses and POST or PUT (huh?) requests.
Request — HTTP transaction from client side. Do note relate to content of the message. More found here (TEXT WALL DUMP incoming https://datatracker.ietf.org/doc/html/rfc7231#section-5 )
Security — response headers used to specify certain rules and policies followed by a browser whilst accessing a website. (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers)
cURL… we meet again
v flag shows full details of the HTTP request and response. -I flag is for response headers by sending a HEAD request. Can use -i (lower case) to display both headers and response body (eg HTML code).
H flag set requests headers. For example -A ‘Mozilla/5.0’
Browser DevTools… hello old friend
Network tab lets us see requests made by a page. Click to view details.
This question was simple. ctrl+shift+I to open web devtools in firefox. Network tab and reload the page and you get a flag.txt extention of the server to you paste into a new tab and it loads the raw text flag for you to take… Hopefully I described that correctly! Makes sense and was done quickly with no pain!
That’s enough for today I think. I’m feeling good and read up a lot more info than I typed up here. Keep it nice and chill today so I can get a decent night sleep.
HTTP Methods and Codes
TODAY… we finish web requests come hell or high water!
I do think long-term this has been better than cramming it all. I am definitely recalling more as it’s been broken into smaller sections.
Most modern web apps rely on GET and POST. Apps that use REST APIs rely more on PUT and DELETE.
HTTP status codes…
More boring stuff you just read as you need or look up when it happens I think but I’ve seen it now… MOVING ON.
HTTP Basic Auth
Handled directly by webserver to protect a specific page/directory without direct interaction with the web application.
The -u flag with cURL allows us to access using credentials
Okay these took me longer than I care to admit. BASICALLY, go to the URL… it opens a search. You search for ‘flag’ as if it were a city, bring up devtools with ctrl+shift+k then network tab and resend the request. Right click the request to copy as cURL for terminal use. then throw your login credentials as shown below ‘admin:admin’ followed by the url with the search.php request for flag… TA DAAAA
Oo I enjoyed that little one. All those fancy tacks and it was ‘log in with credentials and the question literally said search flag… doh!
Post
Used for transferring files or moving user parameters from the URL.
POST places user parameters within the HTTP request body. Three main benefits here:
Lack of logging: POST may transfer large files, it wouldn’t be efficient for server to log all uploaded files as part of the requested URL as would be the case with a GET request.
Less Encoding requirements: URLs are designed to be shared. POST places data in the body which accept binary data. The only characters to be encoded are those used to separate parameters
More data cen be sent: Mximum URL length varies between browsers, servers and content delivery networks.
Login Forms
PHP login form instead of basic HTTP basic auth.
We use the -X POST flag to send a POST request. The -d flag adds the data we need after it as shown below.
Authenticated Cookies
The -v or -i flags will have the Set-Cookie header.
Storage tab in devtools, click cookies to view current cookies on the given site.
JSON Data
POST data can be in JSON format. Request must have the content-type header to be application/json. Right click, copy > copy request headers:
Used the -X POST -d string with cURL to get the cookie id ‘PHPSESSID=mqg8vh224314agilgb5ku3vru8’ then used the code below for a POST json request and easy peasy!
CRUD API
The end is nigh…
Many APIs interact with a database such that we can specify the requested table and row within the API query then use HTTP method to perform what we need.
CRUD
Create, read, updated and delete which will be seen more in the next module (I think) we’re planning on doing. I’m just gonna dive in. A lot of this reading seems fairly redundant here. The key is that these CRUD operations may not inheretently be allowed for all users as it is a vulnerability if anyone can modify or delete data entries. You would have certain users given read/write privileges… hence the beauty of a privilege escalation attack.
HACKIN DUH APIS
This one, painfully took me longer to realise. I kept trying to do fancy things rather just literally do what the question said to do.
PUT let me update the London URL ( it did mention earlier how to list every city that was in the database but for ease I used London.) Renamed London URL to flag.
Then we deleted Leeds. Sorry Leeds, Liverpool is the only true Northern English city we need!
Print the api.php/city/flag url with “ | jq”. To be honest i’m entirely sure what the jq stands for. So, let’s google it. The lesson just has it in green as a key term but didn’t explain it so…
This makes sense!
WOW What a doozy of a module. I MASSIVELY MASSIVELY enjoyed this more than I thought I would. I’m starting to feel like I’m actually getting somewhere. Don’t get me wrong, I also understand I have not even begun to scratch the surface.
I think next up is Introduction to Web Applications, which apparently is only 3 hours so should be good.
