What Happened — Intermittent Issues to Mixlr’s Services
Updates from Mixlr
During the process of rebuilding our primary database, broadcasters and listeners had issues connecting to our services. Below are updates of what has happened to the service in that time.
As we continue to fine-tune the new system, we are back to our regular services and will continue to keep you posted should anything arise.
UPDATE 6: June 26, 2019 (6:45 am ET)
On Monday, June 24th 10:07pm ET (3:07am BST on Tuesday) we had deployed our new database server, with more capacity than required, and with a new back-up system running in the background.
This is a new infrastructure in place and we’re fine-tuning certain aspects as our database communicates with this new framework.
At 11am ET (4pm BST) on Tuesday, June 25th we noticed intermittent latency issues once more. Further investigation led us to turning off a set of queries which were responsible for generating user stats. This resolved the issue instantaneously. At 10:00pm ET (3am BST), we agreed to continue keeping these queries turned off as a temporary measure.
Since fixing this, broadcasting live and listening to live shows has been running as expected.
UPDATE 5: June 22 2019 ( 12:14pm ET)
While running on our backup server, we’re aware that connections have been unreliable for some listeners. We’ve added another server to help handle the load here, and this seems to be improving connections.
Listeners that get a 503 error when trying to listen to a broadcast are advised to try refreshing the page. Once through the livepage, streams can take longer than usual to start after hitting play, but once connected, the stream holds out the duration of the broadcast.
Meanwhile the work continues to rebuild our primary server. More updates on @MixlrStatus to follow as soon as we have more news.
As ever, the team’s very grateful for everyone’s continued patience 🙏
UPDATE 4: June 21 2019 (3:12 pm ET)
The following email was sent out to active broadcasters with news of the weekend service:
We’ve been working around the clock to resume our normal services, but so far, we’ve been unable to rebuild and move the service back to our primary database server.
Rather than rebuild the primary server entirely offline we chose to keep our services online, albeit running at a lesser capacity than is possible on our primary server.
We will continue to stay online over the weekend, with a backup server allowing an 85% success rate of connecting via the desktop app.
However, please be aware Mixlr’s service still won’t be running at 100% — the standard we know our broadcasters are used to. It may take a few attempts to access the service, which I acknowledge can be a wholly frustrating experience on its own.
I’m sincerely sorry that I can’t share a better update with you at this time.
In the nine years Mixlr has been running, we’ve never experienced a disruption to the broadcasting and listening experience such as this. Our team is still working hard to remedy this and we’re hoping we can return to our normal service by the end of the weekend.
Again, we apologise for the inconvenience this has caused for you and your listeners.
Founder and CEO of Mixlr
UPDATE 3: June 21 2019 (8:58 am ET)
We are in the process of rebuilding our primary server. This is a lengthy undertaking that will continue into the weekend.
In the meantime, our best alternative at this stage is to continue using our backup server.
This secondary server isn’t equipped with the same capacity as the primary server. Broadcasting is enabled, but some broadcasters will find they’ll have to make a few attempts to begin their shows. Once logged in, broadcasters can continue their live shows as normal. Some listeners might also have to make a few attempts to access a broadcaster’s live page.
We understand this is not an ideal experience.
Our engineers are working to reduce the latency on the secondary server while also hastening the rebuild of our primary server. And we’ll be continuing to work towards this until our services are back to the standards our users are familiar with.
UPDATE 2: June 21 2019 (6:43 am ET)
As of now, we’re still out of Read-Only mode while running on a secondary server. Broadcasting is enabled but services are slower than standard — logging in and loading live pages might take a few tries. We understand this is painful; this secondary server is a temporary means as our engineers are working to optimise the primary server.
For more on Read-Only mode, please check out Jon’s support article.
The below update from Greg, Mixlr’s Founder and CEO, was sent to active broadcasters.
UPDATE 1: June 20, 2019
Over the past few days you may have experienced issues with the Mixlr service.
The main cause for these issues was an outage to our primary database server, which affected the entire Mixlr service on Monday, June 17 from 4.30am — 10am Eastern Time.
Since Monday, the service has been running on our backup database server while our engineering team continues to work to return us to our primary database server. During this time, we have occasionally been forced to put our service into Read-Only mode, meaning new broadcasts could not be started at the following times:
· Monday, June 17 from 10am — 12.45am ET
· Wednesday, June 19 from 7.50pm — 11.13pm ET
At present, we are out of Read-Only mode but are still running on our backup database server. Our engineers are working tirelessly to move us back to our primary database and restore the service to full capacity with minimal negative impact.
Our sincere apologies for the disruption to your broadcasts, and thank you for your patience.
We will continue to update you (please also check Mixlr Status on Twitter), but if you have any questions about this outage, or anything else, please let us know.
Founder and CEO, Mixlr