HDFS Commands — The GUI Way
Assumptions:
- You’ve successfully installed Cloudera Manager
- You know how to startup/shutdown your environment, if not please take a look at how to do so here
- The project’s idea is to learn how to use our pipeline here. For more details, do visit the project’s page here.
HDFS NameNode UI
Please note, most functionalities are limited due to permissions. However, to view and understand the layout, go ahead.
- Go to HDFS from Cloudera Manager and navigate to this drop-down option here
- This opens http://instance-1.us-central1-......internal:9870/
- If your instance-1 is standby, use instance-2 (active)
- If you get an error, add the IP address of <instance 1> instead of domain name (i.e. the part before “:”)
If you still get an error, add the port to your firewall settings in Cloud Console. How? — here you go (under specified protocols and ports, add 9870)
If you still get an error, try the <instance-2> url and repeat above steps if required
- Now under utilities, click Browse the file system
- This is the default HDFS directory. Click on “user” and create a new directory for “root” from the top-right options. Explore some more options here.
- For more details about a file, simply click on it. Details on the dataset used can be found in the footnote of this post. The next line really.
The Dataset
The Homepage — https://www.adsbexchange.com/
The data — https://history.adsbexchange.com/downloads/samples/
While you can go ahead and download any zipped file, I have chosen to go with the last one, i.e. unzip 2020–01–01.zip. (Latest Data + Latest fields)
More details on the fields can be found here. Explanation with some data? Here you go.
These datasets are free of cost, without any need for registration/password, but please be sure to give the appropriate credit.
If you are using free-credits provided by your choice of cloud-vendor, please do ensure you shut down your applications & VMs in order to save credits. More details here
