Multiple 3G USB modems with so-called virtualization for World Domination
Lately my Company has struggled with one specific problem — how to have few residential IP’s on one location for testing purposes which would cost us few bucks monthly (we are talking two or three digit numbers here). Furthermore, how to control usage of that network remotely, how to use remote applications on available IP’s and how to make deployment of new IP’s easy and blazing-fast.
Since that problem was not so urgent, I had few weeks to think about it and to design a solution that would cover all our needs in that specific project.
Croatian mobile carriers are lately cheaper and cheaper, we have mobile carrier called “BonBon” that is specifically for young people — in other words it means that there are plenty of 3G/4G mobile packages with high-available bandwidth. After thinkering I came up with pretty simple solution.. Hey, let’s buy few USB 3G modems, connect them to one server and use any virtualization software to control them through dedicated Windows XP installations. Anyway, anyhow we need only usable browser like Chrome or Firefox. If we need new IP’s, we will just connect new 3G/4G modem to USB hub, fire new instance and that’s it. Also RDP/VNC will be used for remote connection to instances, every instance will have residential IP for internet access and problem is pretty much solved.. BUT! It’s not so simple as it sounds. Let me try to explain what dragons I faced while trying to set everything up and how in the end I managed to fix it elegantly with only open source software.
Day 0. — Hardwarezzz
Logically, we need some hardware to get it rollin’.. since project is tight-on-budget (alpha/PoC/testing/call it how you want phase), I need to reuse some old hardware I had in my basement.. so I found nice AMD Athlon 3200+ with 3GB of DDR2 memory lying around in some mid-tower black case.. Perfect fit for my office! Well, even if I thought at the first that old Athlon would be enough to run all that, it was just too crappy to even install any of free available virtualization software (VMWare ESXi, XenServer). Actually, I managed to install XenServer successfully (while VMWare failed with CPU not supported) but it was literally working like a crap.. also later I found out that it would be pain in the ass to manage all those USB devices since XenServer is not so-great as VMWare when it comes to USB pass-through. Previously I had only experience with VMWare ESXi which crushes in USB pass-through aspect.
So, clearly, “pure” virtualization was off.. I just refused to buy any additional hardware just to test this out for project which may or may not be profitable.
Day 1. — Re(sleep)boot myself
Since “only” thing we need is browser that needs to be isolated (did you ever heard for Evercookie?) another solution was only to virtualize browser itself — so I thought.. we should use Docker! There is so much hype lately about Docker.. and I knew that it could work. Docker will isolate only browser and if I’m lucky enough there is already available image with Chrome + VNC server ready to go.
First things first, we need to install pure Linux distribution and as I prefer Debian based distro choice was logical — Ubuntu 14.04.
In my head I had network design looking something like this;
Office is connected to internet via FTTH (200/200 MBit), there is some LAN network in place and one ol’ Ubuntu server with XY 3G USB modems. Docker containers should be accessible remotely via FTTH network (to save 3G bandwith). Pretty simple, huh? Only problem that I could face is routing since that’s something I’m really bad at — I just didn’t had any need and/or time to learn it properly. Everything else should be pretty simple and straightforward.
Day 1. pretty much finished with installing and updating Ubuntu. Also few packets (like WVdial) that were essential to continue were also installed. We were getting closer to world domination.
Day 2. — 3G modems, WVdial and driver dragons
We had up&running clean installation of Ubuntu 14.04. and that was pretty much it until now — it’s time to get our 3G sticks up&running. I should note that every single 3G USB modem has few modes and one of it is USB mass storage where in normal circumstances you would find connection software installation and driver (either for Windows or OS X). So, when you first plug your 3G modem to your computer it gets up as a removable USB media.. and that means that there should be some kind of “software” also to switch it to modem mode.
That piece of software on Linux is called usb_modemswitch. Nowadays it’s really easier to get things up&running on Linux (thank you usb_modeswitch) since pretty much all hardware is supported in some way, but there are also exceptions.. and you could guess, that exception costed me whole day. From few 3G modems I had lying around — one was Icon 225 and rest of them were Huawei’s which were working straight out of the box.
To load Icon 225 driver, one has to unload specific kernel module which is used by Huawei 3G modems and I was dumb enough to spend 6 hours trying to fix it rather to buy few more Huawei’s (it’s about $10/each) from local hardware store. When I finally decided to take a little walk outside of office I just bought few new Huawei’s and plugged them in — and they all were working. Let’s call it a day and take a little rest.
Day 3. — Let’s route all those interfaces!
I could connect all 3G USB modems separately to Internet (yaaaaas!) and everything was working.. but once I rebooted server everything just stopped and no connection could be initiated. Why? Well — answer was pretty simple.
To initiate connection via WVdial, one has to define config file (/etc/wvdial.conf) and specify modem ports — at first I was using /dev/ttyUSB0–10 [pretty standard usage].. but that was bad idea in my situation. When server reboots those ports get mixed up depending on which 3G modem gets initialized first.. So instead of this, we just used path /dev/serial/by-id/* which doesn’t change since devices are being mapped by id.
Take a look at sample WVDial config in the following picture;
Once connection is initiated, for it to became usable, you need to set up routing rules. If you have XY active internet connections on one host, there should be some rules in which situation which interface is used, right? Even if I thought that I would face big problems here, it was pretty simple. We are using following BASH script (/etc/ppp/ip-up.local) that sets up everything automatically once you initiate WVdial to connect.
Now, all interfaces are usable and even after server reboot we have our uplinks ready to be used. Let’s again call it a day.
Day 4. — Everybody loves Squid.
Even if I could go with direct routing of interfaces to Docker containers I just didn’t like that solution since I knew that I can push existing configuration even further. We used Squid. Squid is a caching proxy transparent server. Squid offers caching and we have 3G modems that are tight on bandwidth.. Can we use Squid for caching and routing? Oh yes, yes we can! There is tcp_outgoing_address directive that can be used in Squid config. If there are multiple interfaces on host where Squid is installed, you could just define tcp_outgoing_address 220.127.116.11 and all traffic would be routed through that IP.
So, first let’s define ACL’s with tcp_outgoing_address directive;
For example, user that makes access to proxy and is using IP 172.17.0.2 (local Docker container) would be routed on internet through IP 18.104.22.168. That’s pretty simple…
Let’s also set up some crazy caching rules to save 3G bandwidth..
Now we have fully working proxy server that is routing users on multiple 3G modems based on their IP and is also using crazy long caching rules to save every penny on bandwidth it can.
Day 5. — Docker hype is (maybe) real.
Now when we have pretty much everything up&running, only thing we need is remote accessible browser which is isolated from the rest of the server. As I mentioned before, we are going to use Docker.
Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications.
There are public images for Docker you can just pull [download] and reuse and since I only needed browser and remote desktop, I was lucky enough to find siomiz/chrome docker image which is exactly that. You just fire it up and viola — you have remote Chrome instance that can be controlled from your device. I also found neat feature called Chrome Remote Desktop which can be used to remotely control Chrome instance from your local installation of Chrome. It’s easier and faster than any VNC server. Furthermore, you don’t have to deal with port forwarding, tricky firewall rules, nothing. It just works.
Conclusion — Eternal Happiness
After 5 days of configuring that solution, I guess you’re wondering how it works and why do we need it— well, let me just say this — it works awesome! On that shitty old Athlon there is literally no load, everything works smoothly and it’s usable. I’m also running some network monitoring tools (darkStats) on same host and web server to monitor usage of 3G bandwidth (and to send SMS how much bandwidth do we have left) and it’s pretty much awesome & automated. It will do the job for sure!
Why do we need it? Let’s just say it’s top secret. It’s for World Domination :-) Keep on reading my medium and eventually, our top-secret project will be revealed!
Don’t forget to follow my Co-Founder buddy Igor Furjan who bullshits about business ;p