Reverse engineering my Fiber-to-the-Home GPON device

This story is about my effort to get in my FTTH (Fiber-to-the-Home) GPON device in order to get access for more options than it originally offered.

You can read the basics of the FTTH networks here.

Disclaimer: I made the privilege escalation and configuration change with permission from my ISP. I did NOT use it to get higher speed internet or similar. I am not responsible for any damage you cause to your device or ISP by following this guide. By the way I don’t tell the exact type of the device on purpose. These devices may be configured/customized differently within the different ISPs networks, so following the steps may have different effect in your case. Do this at your own risk!

The story begins 1.5 years ago when we moved to a new place, where there was an ISP who offered FTTH based internet/TV/phone access. A single device is installed in your home and it terminates an optical cable. It is first of all a NAT kind of router, but far more: you can connect your TV(s) with DVB-C, your regular fixed phone, your LAN devices to it, but it even has WLAN inside. So we had super-stable internet so far. With the default credentials of this device I could set all parameters what I wanted, especially I was able to forward all TCP/UDP ports from the the public IP to my LAN device. My only LAN device connected to the GPON is actually another router/NAS, but this is not really important. The point is that I could access my data, my security cameras, my irrigation system, etc. from remote, thanks to the port forwarding. The original GPON was a type “640” from the undisclosed manufacturer. We were happy until… a few months ago some TV channels image started to be noisy, so I called the ISP to fix it. The fix was to install a new GPON device, which is actually a newer one, according to the part number: it is “660”, which should be 20 better than the “640”, right? :) Well, it was’t. With the default user/password it had a very limited set of options in the WebUI. Practically I could change the Wi-Fi password and SSID, that’s it. No port-forwarding, no firewall settings, nothing fancy.

GPON end-user device

I wrote a claim letter to the ISP, explaining my pain. After a month silence they replied with a single sentence that “I should use my own router device to get things working”. Now this shows that the back-office guy who replied did not even check that I am an FTTH customer, so I cannot simply buy and install my own FTTH device, it must be theirs. I gave them a call, explaining again why this is important and why the response letter is a bullshit, so to say. They somewhat understood and promised that they will try to get me an old, 640 type of device, so I can get my SYN packets in again :). Meanwhile the service guy who changed the 640 device to 660 gave me some secret idea on phone how I should be able to get in an admin WebUI account, but without any success. He also said that if I am able to solve this, it is also fine for them. So I felt I got the permission to play if I don’t brick the device, or the ISP network :). Let’s get our hands dirty…

Inspecting the limited user account on the WebUI.

The default credentials are on the two-page user manual, so I won’t publish it here. Anyway you can guess or search for it :). So there are some Wi-Fi options, some packet statistics, a reboot option and a configuration backup/restore function. I downloaded a config backup, but it was an encrypted, or at least binary 70kB file. Not that useful so far.

Port scanning

I checked the LAN IP address of the GPON device to step further. It had obiously the default http port open, but there were a couple more, for example 22.

$ nmap 192.168.55.1
Starting Nmap 7.40 ( https://nmap.org ) at 2018-04-24 20:40 CEST
Nmap scan report for 192.168.55.1
Host is up (0.013s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
22/tcp open ssh
23/tcp open telnet
53/tcp open domain
80/tcp open http
Nmap done: 1 IP address (1 host up) scanned in 0.26 seconds
$

Indeed, it has an SSH server! So there shall be Linux inside.

$ ssh 192.168.55.1 -luser                                              Unable to negotiate with 192.168.55.1 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

Okay, so it offers a weak key exchange method only. Good start! :) Let’s enable it:

$ ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 192.168.55.1 -luser user@192.168.55.1's password:
#

I was pretty sure it won’t let me in with the same default credentials as I had for the WebUI, but actually it did! Moreover, can you see the # in the prompt? Woah, am I root with a username, called user? Yes I am. Printing out the passwd file confirmed this:

# cat /etc/passwd
admin:$1$$xxxxxxxxxxxxxxxxxxxx/:0:0:root:/:/bin/sh
user:$1$$yyyyyyyyyyyyyyyyyyyy/:0:0:root:/:/bin/sh
guest:$1$$yyyyyyyyyyyyyyyyyyyy/:0:0:root:/:/bin/sh

I masked out the passwords, but the hashes are in the passwd file. According to the hashes, “admin” has a different one than “user” and “guest”. The zeroes at the middle of the lines means that all 3 users in the system is treated as root, only with different usernames. Well, I don’t feel safe… but we are in!

Exploring the device

After some exploration I learned the following things:

# uname -a
Linux tc 2.6.36 #2 SMP Thu Jan 28 17:27:09 KST 2016 mips unknown
# mount
/dev/mtdblock6 on / type squashfs (ro,relatime)
proc on /proc type proc (rw,relatime)
ramfs on /tmp type ramfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/mtdblock8 on /data type jffs2 (rw,relatime)
# ls -la /data
drwxr-xr-x 4 0 0 0 .
drwxr-xr-x 12 0 0 208 ..
drwxrwxrwx 2 0 0 0 log
# df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/mtdblock6 9728 9728 0 100% /
/dev/mtdblock8 2048 492 1556 24% /data
#

Basics:

  • This is a MIPS architecture, Ralink MT751020 SOC with 128MB RAM and with some Flash storage.
  • 2.6.36 kernel with Busybox.
  • There are a dozen of userspace processes running with unfamiliar names.

Filesystems, storage:

  • The root filesystem is a read-only squashfs.
  • There is a read-write ramfs, mounted under /tmp.
  • An interesting directory under /userfs, which is on the same partition as /.
  • Ther is another read-write FS mounted from the Flash storage under /data. It stores only a single log file.
  • There are alot more unmounted partitions on the internal flash:
# ls -la /dev | grep mtdblock
brw-r--r-- 1 0 0 31, 0 mtdblock0
brw-r--r-- 1 0 0 31, 1 mtdblock1
brw-r--r-- 1 0 0 31, 10 mtdblock10
brw-r--r-- 1 0 0 31, 2 mtdblock2
brw-r--r-- 1 0 0 31, 3 mtdblock3
brw-r--r-- 1 0 0 31, 4 mtdblock4
brw-r--r-- 1 0 0 31, 5 mtdblock5
brw-r--r-- 1 0 0 31, 6 mtdblock6
brw-r--r-- 1 0 0 31, 7 mtdblock7
brw-r--r-- 1 0 0 31, 8 mtdblock8
brw-r--r-- 1 0 0 31, 9 mtdblock9

Some other findings:

  • The device is resetting it’s filesystem back in case of reboot. I think this is common for such embedded devices.
  • The device not surprisingly manages the network traffic with iptables, but with a hell lot of rules, chains, kernel bridges, VLANs, etc.

At this point I was thinking about to understand the iptables chains and rules, than adding a few rules to realize my port forwardings. Since I am root I could do it. But that may clash with the intent of the management softwares on the device. Most probably they flush the chains here and there when the device occupies a different WAN IP address, or when device is rebooted or remotely reconfigured by ISP and so on. So I still wished for more legal user interface options than I had on the WebUI by default.

Fail #1: I changed the admin user’s password (with passwd admin) to the same as the default user’s and tried to get in to the WebUI with that. Although it changed the SSH password, no luck on the WebUI.

Searching for the WebUI service

Unfortunately the Busybox netstat did not have -p option to show who is sitting on TCP port 80, so let’s leave it for now. The device has a single rc script spawned directly from /etc/inittabduring boot. After reading some funny comments in the scripts, left behind by the authors, I found that the webserver type is boa. The UI is implemented using javascripts on the client side and ASP CGI scripts on the server side, embedded in the HTML sources. I am not an ASP expert, though the code appeared to be very simple. It generally uses some unknown tcWebAPI_get(), set() and commit() functions for the config data pieces and for the UI string constants. But more on that later… The boa server config was not any interesting. The interesting part is that there are two symlinks between the static contents:

# ls -la /boaroot/html/
drwxr-xr-x 3 0 0 445 .
drwxr-xr-x 4 0 0 94 ..
-rwxr-xr-x 1 0 0 165 async.js
-rwxr-xr-x 1 0 0 361 chksel.js
-rwxr-xr-x 1 0 0 1057 cross.gif
[...]
-rwxr-xr-x 1 0 0 1026 pen.gif
-rwxr-xr-x 1 0 0 5707 pvc.js
lrwxrwxrwx 1 0 0 24 romfile.cfg -> /tmp/var/romfile_enc.cfg
-rwxr-xr-x 1 0 0 7763 s524-7.js
[...]
lrwxrwxrwx 1 0 0 18 syslog.log -> /data/log/messages
-rwxr-xr-x 1 0 0 8041 val.js
-rwxr-xr-x 1 0 0 3722 wanfunc.js
#

The syslog.log is obviously there to expose the logs for the WebUI users. The other one’s name is rather promising… This is the romfile.cfg which points to /tmp and it suggests that the real file is encrypted. As it turned out, this encrypted file is the one which I was able to download when I requested a backup on the WebUI. Interestingly this symlink is broken after boot until the first backup request. So it is not the master copy… Let’s see the containing directory:

# ls -la /tmp/var/
drwxrwxrwx 8 0 0 0 .
drwxr-xr-x 7 0 0 0 ..
drwxrwxrwx 2 0 0 0 lock
drwxrwxrwx 2 0 0 0 log
drwx------ 2 0 0 0 net-snmp
-rw-r--r-- 1 0 0 77879 romfile.cfg
drwxrwxrwx 2 0 0 0 run
drwxrwxrwx 2 0 0 0 tftpboot
drwxrwxrwx 2 0 0 0 tmp
-rw-r--r-- 1 0 0 2 vport_result
#

Bingo! The cleartext romfile.cfg is there. It indeed holds all the possible configurations of the device as an XML document. And also the user accounts!

[...]
<Account>
<Entry0 username="admin" web_passwd="xxxxxxxxxxx=" console_passwd="xxxxxxxxxxxxxxxxxxxxxx==" display_mask="FF FF FF BF BF DF FF FF FF" UseWan="Yes" UseLan="No" />
<Entry1 username="user" web_passwd="yyyyyy==" console_passwd="yyyyyy==" display_mask="7E 8C 6 8C 8C C 8C 8C 8C" UseWan="No" UseLan="Yes" />
<Entry2 username="guest" web_passwd="yyyy" display_mask="7E 8C 6 8C 8C C 8C 8C 8C" UseWan="No" UseLan="No" />
</Account>
[...]

I masked the passwords again, but the fun fact is that some of them are base64 encoded, while others are clear-text here! :)

What can we see from this file regarding the user accounts? Actually a lot:

  • We can decode the base64 encoded passwords, including the one for “admin”.
  • All users has two options, whether they are allowed to login from WAN or from LAN side. The WebUI is not accessible from the WAN IP by the way. Well, luckily. Anyway, now we know why admin is always forbidden on WebUI… It’s UseLan setting is set to No.
  • All users have a cryptic display_mask value which is probably the visible and allowed features which the subject user account can change. It seems to be a multi-byte bitmask and clearly “admin” has more bits set to 1 than the “guest” and “user” users.

Fail #2: At this point I thought that this romconfig.cfg is the real master config storage, where all the management processes, like the WebUI getting from and setting the data. And this is on a writable filesystem! So I changed the UseLan option to Yes for the “admin” and went back to login with it on the WebUI. No luck. Still forbidden. And BTW the file is reset to it’s original state on reboot.

So the file is again only a secondary copy, similarly to the encrypted version.

Fail #3: My next idea here was that maybe the encrypted version of the file is always generated from the clear-text version. Why not. What if I change again the clear-text version, than requesting the encrypted version for downloading and restoring it back? It shall blindly apply my altered configuration, right? Well it didn’t :(. Seems the encrypted version is always generated based on the master data, which is stored at an unknown place, maybe on one of the unmounted partitions in the flash.

Inspecting the web content once again

As I could not get the job done, I tried to understand once more how the web content is interacting with the config. So the server side ASP CGI code is full of with this tcWebAPI_… function calls. I could not find the actual implementation for those in any readable format, so it is maybe directly understood by the webserver.

# cat /boaroot/cgi-bin/index.asp
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=<%tcWebApi_get("String_Entry","textTTNETCharSet","s")%>">
<meta http-equiv=Content-Script-Type content=text/javascript>
<meta http-equiv=Content-Style-Type content=text/css>
<script language="JavaScript" type="text/JavaScript">
/******************For Multi-Language Strings Start*************************/
var ary_strings = [
['DesIPInvalid', '<% tcWebApi_get("String_Entry", "JSDesIPInvalid", "s")%>'],
['SorIPInvalid', '<% tcWebApi_get("String_Entry", "JSSorIPInvalid", "s")%>'],
['IPIsEmpty', '<% tcWebApi_get("String_Entry", "JSIPIsEmpty", "s")%>'],
['DesNetInvalid', '<% tcWebApi_get("String_Entry", "JSDesNetInvalid", "s")%>'],
[...]

It seems every parameter has a kind of class (like String_Entry) than a key and a value. If I would know how the display_mask is called by the tcWebAPI_set function, most probably I could sooner or later invent an ASP script to change it for the admin user, store it on a writable place, restart the boa webserver by configuring it to run the script and so on… Or… I kept checking the CGI scripts and I found some, which are not ASP, but regular Shell scripts.

# cat /boaroot/cgi-bin/pppoe_pwd.cgi
#!/bin/sh
echo -e "Content-Type: application/json\n"
PASSWD=`/userfs/bin/tcapi get Wan_PVC PASSWORD`
if [ "$PASSWD" = "N/A" ]
then
[...]

So, it seems the magic tc API has a console version too. I already checked the /userfs/bin directory, found great tools like tcpdump and gdb there, but that time the tcapi binary did not look particularly interesting. Let’s run it:

# /userfs/bin/tcapi
set
unset
get
show
commit
save
read
readAll
staticGet
#

What a descriptive help! What can be the actual options? The command did not help, so I grepped all the CGI content with the account XML tag, known from the romfile.cfg:

# cat /boaroot/cgi-bin/*.asp | grep -i account
[...]
<font color="#000000"><strong><% tcWebApi_get("Account_Entry1","username","s") %></strong></font></td>
[...]

There were a lot of matches, like the one above. The tcapi command’s show option can kindly list all key-value pair under a certain node, if we know the node name. Let’s try it with Account_Entry1 .

# /userfs/bin/tcapi show "Account_Entry1"
username=user
web_passwd=xxxxxx==
display_mask=7E 8C 6 8C 8C C 8C 8C 8C
UseWan=No
UseLan=Yes

(I masked the base64 encoded password as before). This time we are really getting closer! We fetched the user account details exactly how the user facing tools does it. It must be coming from the master copy now, where-ever it is.

Since tcapi has set and commit parameter, maybe we can alter what we want. So let’s do it by assigning the same display_mask to user, which the admin has by default:

# /userfs/bin/tcapi set Account_Entry1 display_mask 'FF FF FF BF BF DF FF FF FF'
# /userfs/bin/tcapi commit Account_Entry1
#

Tadaam! Now I had luck. After logging in with the same regular “user” account, I can see all the firewall and other “advanced” settings.

As a reward I could set my original port-forwardings. That’s it!

After I sent all these instructions to the ISP service guy, he confirmed that it quickly caused reactions in the company management. After all, they appreciated my effort and said that they will change all the managed device configurations remotely. They will enable more settings to the end-user based on my findings in their network instead of i.e. suing :).

Conclusion

I don’t know what I should think about this level of security hardening in these devices. It was fun but after all also worrying. The device is basically offering root access towards at least the LAN devices with a documented password. After breaking in to any end-user LAN or Wi-Fi device, one could install permanent backdoors or any other malicious things on the GPON device, which is crazy.

Another post explains more concerns here.