Lessons learned 4 years on
Introduction
In August 2016 I created my first media server. Nearly 4 years have passed since then and I’ve learned a lot along the way. My hope with this post is that anyone thinking of starting up their own media server may learn from my mistakes by making sensible decisions about architecture, cost, storage, performance, and security (amongst other things!).
Decide what your motivations are and plan ahead
Initially I was just excited to learn new things. This is a good motivation to have, but what if you get a nice setup going and you decide to show it off to your mates? They’re interested too, and want a piece of that action. You’re a generous person and we’re in the midst of a pandemic, so you share your library with them. Then they slide into your DMs sending you a request for a TV show they’re itching to watch because it’s not on Netflix. You oblige. A week later, same again. Soon you’ve shared your library with a few friends. Some of them ask if their friends can get an invite too. Then your server starts to slow down and run out of space so you have to upgrade, which costs money. Things start to get a little expensive so to cut costs you decide to start accepting donations, then switch to a subscription-based model, and before you know it you’re running a small business.
If you’re dead set on staying small-scale (i.e. just for yourself or your household), set those boundaries early on. Otherwise, plan for scale. Poor architecture decisions you make now could make scaling up more painful in the future when your library gets bigger and your server sees more use.
Planning your media server setup
Storage
When I got started maintaining a media library I couldn’t imagine using up more than 4 terabytes of storage. This may be plenty for some, especially if you’ve chosen not to share your media server with anyone else, but there’s a good chance you’ll find those 4 terabytes get gobbled up fast. At time of writing, my media library consists of about 250 shows and 2500 films, totalling 42 terabytes. Ultimately I ended up storing all my media on Google Drive (more on that in a future blog post), but it may make sense to have one or more hard drives.
If you choose to set up a disk array with something like unRAID, ZFS, or some other RAID configuration, you will need to think about things like parity and caching. What happens if one of the disks in your array dies or the data becomes corrupt? How will that affect the data on the other drives? What are the performance implications of the RAID configuration you’ve chosen? Do you care about backups? What will you do when you run out of space? Does your storage solution allow for more capacity to be added retroactively?
Cost
Hardware is expensive. Renting servers can be expensive. Decide how much you are willing to spend and design your solution within the limitations of your budget. Renting a seedbox to torrent media will cost a monthly fee. Building your own server at home could be a large up-front cost. Can you postpone buying certain components in your server (RAM, an SSD, etc.) until your next payday? Will you make money back from your users, and will that be enough to cover the monthly costs of your seedbox, electricity bill, Usenet provider, domain name, etc.?
Performance
Disks
Media libraries are often stored on hard drives because they’re relatively affordable and high-capacity, but they’re slow. This can cause performance issues especially if your OS is running on the same HDD as your media. If possible, use SSDs for your server’s OS and any configuration directories for automation software you’re using. Software like Sonarr and Plex use SQLite databases for indexing your media library which doesn’t perform particularly well on spinning disks. While some RAID configurations can help squeeze more performance out of your disks by spreading reads and writes across your array, I’d strongly recommend keeping HDDs for your media library only.
Transcoding
Client apps for media servers like Plex and Emby are not designed equally. Some players can play back the MP4 film on your hard drive natively, while others will need it to be converted to a compatible format on-the-fly. This is called transcoding and it is a CPU intensive process. If you plan on having a lot of users with a variety of players, you will need a decent CPU to transcode your media for them. If you’re made of money, you can offload transcoding to a compatible graphics card and support a lot more concurrent transcodes. If you decide to use a low-spec device like a Raspberry Pi you might want to convert all your media to a format that can be played directly (i.e. no transcoding required).
Network
Your server’s network connection dictates how well you can stream media remotely. If you’re keeping everything in your LAN then you don’t need to worry too much about this, but if you plan on supporting remote access, you will need a sufficient uplink to avoid dreaded buffering. It might make sense to rent a dedicated server hosted in a datacentre that comes with a 1 gigabit uplink so that your users don’t hammer your home network to oblivion.
If you’re hosting a home server, does your internet service provider throttle you after n gigabytes downloaded? Is there a monthly data cap?
If your media server will be accessed from another continent, you will need to think about peering and locality. If your server is hosted in Germany but a lot of your users are US-based, their route to your server likely won’t be very direct, which will affect their viewing experience. Higher latency will make browsing your library slow and frustrating.
How quickly do you want to go from thinking “I want to watch this show and it’s not on Netflix, I should put it on my server!” to actually having that media available for viewing? Ideally your seedbox will have a fast network connection to download that media ASAP. If your seedbox is a separate server from your media server, you will want to transfer that media too. Is 100 megabits fast enough? Maybe spending a bit more for symmetrical gigabit is worth it to go 10x faster?
Architecture
Keep it simple, stupid
I’ve had to make several big architecture changes for my setup and each one was painful. Some things you can’t predict, but try to come up with a design that is reliable and simple. Don’t reinvent the wheel. Accept that you may have to make sacrifices on a lower budget. If you don’t have access to private torrent trackers, your choice of seedbox is limited. Conversely if you don’t want to pay for a seedbox (even a shared slot) your architecture might end up being simpler.
Let’s assume you’ve decided to go all out. You’ve set up Sonarr on your home server to automate downloading and organising TV shows. It needs to talk out to your seedbox where you have Jackett and Deluge running for finding and downloading torrents. Would it be quicker to run Sonarr on your seedbox too so that it doesn’t have to wait until the media is transferred back to your home server before it can rename that media appropriately? Maybe you don’t mind searching for torrents from your home server and only want to use the seedbox for downloading? How will you transfer the completed download from your seedbox to your home server? Will you execute a script on completion which uploads the data via SFTP, or will you mount your media library directly on your seedbox via sshfs?
Security
Let’s be honest — you want to set sail and live the pirate life. But how do you make sure your voyage is safe?
Encryption
Firstly, encrypt all remote access to your media server and any other apps with HTTPS. Let’s Encrypt is your friend and will give you signed SSL certificates for free. Using a reverse proxy like Caddy or Traefik in front of your apps gives you the ability to get certs from Let’s Encrypt and renew them automatically. It’s a no-brainer.
Terminal access to your server should be done via Secure Shell (SSH) with public key encryption. Disabling the root user from SSH and exposing SSH on a port other than port 22 is also a good idea.
Authentication
Say you’re running a whole suite of media server helper apps like Sonarr, Radarr, NZBHydra, etc. — you’re going to want to enable form-based login for all of these. Use different credentials for each one, stored safely in a password manager like BitWarden. Better yet, set up an OAuth2 proxy to require a Google login scoped only to your Google account.
Mitigate risk
Can you safely download torrents from public trackers like RARBG from your remote server? If your server host doesn’t explicitly advertise itself as a seedbox provider, then probably not. Maybe you’ve chosen to torrent from home but you live in a country with strict anti-piracy laws, so you go down the VPN route to protect yourself instead. Are you sure the VPN provider you’ve chosen is safe for torrenting? Will you get throttled eventually? Does the VPN provider log anything? Did you pay for your VPN subscription anonymously with cryptocurrency? If torrenting is too scary for you then maybe Usenet is a better choice because you don’t have to upload anything. Can you avoid using public trackers altogether and get an invite to a private tracker from a friend?
Sharing
Try to control who uses your server. Can you trust them? If they aren’t a paying customer or a friend, that likelihood decreases. Are your friends sharing their accounts with random strangers you don’t know or trust? Consider using something like Tautulli to keep an eye on what people are doing on your server. You might find it useful to generate a geographical map of where your server is being accessed from to spot any suspicious activity.
Payments
Consider paying for your seedbox and Usenet providers with cryptocurrency rather than PayPal or debit/credit card and use a fake name and address to cover your tracks. Same goes for your domain name if you choose to buy one — privacy-aware services like Njalla offer domain registration paid for via cryptocurrency.
H4ck3rz
If your server is publicly routable on the internet, all sorts of nasties will be probing you with port scans and opportunistic authentication attempts. These can bring down your server either via denial of service or, if you really messed up, successful unauthorised access to your server. Don’t open any ports that you don’t need to be open and deploy something like fail2ban to give you strong protection from intrusion attempts.
Media quality and acquisition
There are lots of ways to download TV and film. The most popular methods are torrents and Usenet, but direct downloads from one-click hosts may be an attractive option if you don’t want to risk torrenting and don’t want to pay for a Usenet provider. Maybe you like how easy it is to find torrents for niche content but you don’t want to faff with a VPN or seedbox — does a debrid service like Real-Debrid fit your use case better?
Automating your downloads
To automate the process of finding and downloading TV and film, setting up Sonarr and Radarr is a must. These apps integrate with movie and TV databases to organise your media and automatically trigger downloads for episodes/films as soon as they are available on the internet. For example, you can input IMDB’s Top 250 list and tell Radarr to download and organise every film in that list. Radarr will search your configured torrent trackers and Usenet indexers for an appropriate torrent or NZB and send it to your download client. Once the download completes, it will handle renaming that file and moving it to the correct folder.
Finding anything and everything
To cover all your bases, combining two Usenet providers from separate backbones with public and/or private torrent trackers is a good way to go. Let’s say you’ve got a few Usenet indexers and a bunch of trackers but pointing Sonarr and Radarr at all of them is messy and cumbersome. Consider using software like NZBHydra and/or Jackett to combine all your indexers and trackers into a single pane of glass, and point Sonarr/Radarr to that instead. NZBHydra gives you valuable metrics about how much your configured trackers and indexers are used, so if an indexer is underperforming or timing out a lot you can disable it, making your media automation more reliable. You might even be able to ditch an under-performing Usenet indexer and save money.
Private torrent trackers
An account on a private tracker allows you to fetch high quality encodes for new releases more quickly than public trackers, but it can be a struggle to maintain an acceptable upload/download ratio. To upload enough to earn enough buffer to download everything you want, it could be worth using software like ruTorrent with autodl-irssi to do some racing.
Conclusion
Building a media server and all the bits and pieces to make it sing is really fun. I’ve even started writing my own code to automate things even further. You can find some of my projects on GitHub:
Igglybuff/awesome-piracy-bot: Set of bots for scraping URLs from Discord, Telegram, Reddit, and IRC.
As well as my popular awesome list:
Igglybuff/awesome-piracy: A curated list of awesome warez and piracy links
The next article in this series will be a run-through of my current media server setup and the future improvements I have planned for it.
Thanks for reading!
