How to monitor the quality of live streaming?
The importance of the chain in content delivery.
This article shares the process involved in monitoring live video streaming sessions. The goal is to show common tips and approaches that are vital in doing live broadcasting. This material can be helpful for beginners producers/streamers who are looking for ways to track the video quality.
Why should you care?
Only 15–20 percent of your viewers have patience to wait for more than 9 minutes if they experience some issues with the content.
In other words, you can lose 80% of your audience because of bad viewing experiences, 25% of your viewers leave even before 4 minutes mark.
This is the reality.
Conventional broadcasting allows for more time on the development of troubleshooting tools that can be used for rapid detection of issues in the delivery chain. This complex process involves broadcast operation centers that control broadcasts (on TV channels and networks) throughout the entire delivery process.
When something goes wrong, an operator can quickly identify at which stage the problem is occurring.
Yet, lack of such level of monitoring in real-time streaming is a bad news for broadcasters. As we know, problems may occur anywhere in the delivery chain, and the only way to isolate them is by rudimentary monitoring.
Looking at the live streaming chain, we can see that it combines a lot of elements, and it will be crucial to understand where the problem lies. We illustrate an example of a typical streamer, but exclude the path from the platform to end viewers since we cannot monitor the output from the platform side.
When we start talking about multistreaming, the situation gets even more complicated as there are more elements in the chain. Each additional platform has its own network path where the quality and stability are dependent on additional factors.
How can we identify and resolve a problem? Or just be sure that everything is performing as expected?
To solve the problem, first we need to look into the video stream itself. This is the only way to know which devices or network connections are causing problems. By doing so, we would be able to see which elements of the chain (devices or network connections) are causing the problems. In fact, it is the missing or corrupt data from the video stream that the devices in the delivery chain should report, and not their status. This type of data should also be provided in real-time rather than requested by operators, when needed. Therefore, a problem could be detected as soon as it occurs or even earlier.
“Monitor” constantly having “package” communication with the user and all end platforms gaining information and statuses to display in real time, allows us to track the flow of data to Restream.io and again from Restream to the end platforms. Simply by analyzing this information, we can find out, or predict, where a problem may occur in the chain.
What information is truly important?
We make sure you get as much critical information as possible, and that starts right from the first link of the chain. We have created a test stream to see how it works. Part of the data we collect will actually be static, as it is metadata telling Restream Monitor information about the settings that we have already provided in the encoder.
After we start our stream, we can see immediate feedback:
- Bitrate
- Keyframe interval
- FPS
- Dropped frames
- Video codec
- Audio codec and frequency
- Video resolution
Here we see some information which should reflect the same values as we have in our OBS settings. They may differ from our actual OBS settings since the indicated values are averages taken over time. If we had incorrect settings that go against the recommended values, we would see error messages and warnings.
Our bitrate is floating, are we good?!
Because of the nature of how internet traffic works, you may see slight variations in the bitrate. As long as it has no large spikes or dips, you should be just fine. Your bitrate can float from around anywhere from 300 to 500kbps depending on the content you stream.
Why is it 3660 and not 3500 that we have in our settings?!
This value is a combination of both video and audio bitrates. For example, if we have 128kbps audio bitrate and 3500kbps the average combined value should be around 3628kbps.
In this case, we have no problems as all the data has successfully reached the end platforms and our viewers are satisfied.
Experiencing issues?
Rules — don’t panic, just stay calm, stay cool — start to analyze!
Simply by looking at the graph we can see problems. Our bitrate is more than 700 kbps less than in our settings, the fps is only averaging 18, and finally we have a massive amount of dropped frames at 40%!
We need to think about the possible scenarios that may cause these kinds of problems.
Immediately, we know just by looking at this data that the problem is in the first chain: the connection from our PC to Restream.
Let’s think about the possible cause of our bit rate instability problems. There is a good reason your bitrate is shown first before all other data. The first step is to always start from the origin and work our way down the chain. In almost all cases, bitrate issues lead to problems with dropped frames, fps dips, and output lag. The encoder sometimes struggles to push enough data through a small bandwidth channel to fit your quality settings.
Bitrate — Here are some reasons that may cause bitrate issues:
- Insufficient upload bandwidth
- CPU load is too high
- Incorrect encoder settings
- Encoder bug or glitch
It’s hard to immediately identify the reason why the issues are happening at first, so we will use a process of elimination to help us:
- Check the speed test. A ping response is indicative of a good or possibly poor connection. Typically, a low millisecond (good) response time means there is a good connection and can support higher bitrates; inversely, a high millisecond (bad) response time correlates to a poor connection and may have problems with higher bitrates.
- Monitor your CPU load. If there is a lack of free memory or the CPU is pegged at high loads the encoder’s performance may suffer. Ensure your settings fit the capabilities of your hardware.
- Verify your settings and make sure that all the values are realistic. Maybe you have entered 300 frames per second.
- Make sure you have the latest updates or versions of your encoder.
The first check is the speed test to make sure that your connection is good. If the connection is bad, you need to try to find the optimal ingest server to stream. You could contact your provider and make additional tests to make sure your connection is capable of establishing the required bandwidth channel to Restream. Another solution could also be to change providers as some have been known to throttle high bandwidth users.
If the connection to Restream is good, go on and check the your encoder and PC hardware. Check the resource monitor to make sure the PC is not overloaded, which can cause the encoder to not have enough resources to process video. Consider stopping any other programs consuming high CPU or RAM resources on the PC.
If the PC handles the load well and your connection to Restream is good, the only thing left is a software problem. There are rare cases where the encoder software glitches and/or the combination of its settings are not correct, all of which depend on the content you stream. You could try changing the encoder if all else fails.
It is always important to have your encoder quality settings fit the capabilities of your PC, but also keep in mind that it changes depending on the content that you stream. For example, if you start streaming a new game that you haven’t streamed before, you should consider monitoring its resources usage and adjust your Encoder settings accordingly.
If there were no issues from the beginning of the chain (livestream to Restream), but there are still some problems, the next step is to analyze the outgoing feed from Restream to the desired platforms.
Route checking to end platforms
If you see your stream lagging on one of the platforms, or users complain about quality and lags, you need to check the graph of that specific platform.
First and foremost, make sure that your settings fit all the end platforms. The wrong settings could lead to any of those issues or all of them at once.
There is no sense in searching for a problem on Restream or the platforms, if your settings do not meet the requirements.
The main rule of thumb: If you see an outgoing stream issue, check the platform requirements and see if your settings meet them and adjust them accordingly. This is the most common problem that is easily resolved. The platforms will just deny your stream, as they are not capable of working with it.
Moving forward, we will assume everything in the chain up to Restream has no issues and your encoder settings meet the platform specific requirements.
For the sake of simplicity, we only used settings which fit all platforms.
On the graph we have the following data from Restream to the Platform:
- Bitrate
- Buffer size
- Successful stream starts
- Stream reconnects
In this case, the cause for one or more of these problems could be the same. Below we will describe some common reasons and provide potential solutions.
Reasons: Restream servers could be too far away from the end platforms.
For example, let’s say you are streaming to Restream’s US server which connects to a platform in Asia. Since the platform is located in Asia and has no ingest points in the US, there could be network instability problems rising from the lengthy connection. Sometimes the end platform’s servers are just overloaded or have problems in their own infrastructure.
You can investigate whether the platform has posted statuses on social media or has an official status page describing any issues they currently have. Also, look for official information on when an e-sport event or tournament is going to happen or is currently happening.
Most of the platforms route their streaming power to support these events which leaves only a fraction of their power for Restream’s servers to use. In some rare cases, Restream’s server is overloaded or some of the servers went down for maintenance. In such an event, you will have a notification in your Restream dashboard.
Outgoing bitrate: This value shows at what bitrate Restream is sending a stream to a particular platform. It will be roughly the same as the incoming stream that Restream receives assuming the platform doesn’t have a restriction on their incoming bitrate.
Successful starts and reconnects: Each time your stream gets to the end platform we increment a successful start. When the connection is disconnected for some reason, the system automatically attempts to reconnect to the end platform and increments the reconnect value. If it is also successful, the start value is also incremented. An ideal case is to have 1 start and 0 reconnects.
Buffer size: This is the most difficult variable to understand. If a stream cannot reach the platform in time, part of the stream is placed in a buffer to prevent data loss. When network congestion clears, Restream will flush the buffer to the platform and no data loss will occur. There’s a rough correlation between the frequency (not the amount) of buffered bytes and connectivity problems with a platform, but doesn’t necessarily mean there is a problem by itself. Use this information to make educated guesses.
In most cases, you will see poor metrics on outgoing streams all at once since they’re strongly correlated with each other (with the exception of buffered bytes). However, in rare situations you may see some undesirable values happen in only one of the metrics.
Herein lies the complicated part: deciphering a sequence of data in an attempt to resolving any issues that arise. Let’s start with some basics:
- Check if the Restream server you’re streaming to has any issues. There should be notifications on either social media and/or alerts on the dashboard about server downtime or troublesome experiences.
- Ensure the end platform’s servers has good connectivity. Try to stream directly to the platform for trouble shooting. Sometimes Restream’s servers take a different network path than directly streaming to a platform resulting in poor performance. If this is the case, we’d like to hear about it!
- Make sure you have the optimal encoder settings which are compatible with all platforms.
- Try changing the Restream server which you’re streaming towards. While a further Restream server may have a less stable connection to you, it could have a better connection to the platform.
Important note: Restream will never modify your stream quality or settings unless you use transcoding. We assume if you have enabled transcoding you know what you’re doing. We send the data the exactly the same as we get it.
Summary
While everyone is busy live streaming, there’s that slight off-chance something goes wrong and there may not be someone there to help you. In addition to providing analytics, our goal is to minimize streaming downtime and stress by providing the tools to help self-diagnose the issues at hand.
Successfully resolving issues is truly dependent on the ability to analyze the data that’s in front of you. We hope this article provides some basic techniques to help you become a better streamer and troubleshooter. After all, the shortest downtime is often mitigated by the availability of tools at hand.
Please comment below for any suggestions or improvements and we will incorporate feedback!
Happy Restreaming!
