Measuring the energy impact of hybrid peer-to-peer / CDN delivery: initial takeaways
Many questions have been raised about the environmental impact of video streaming and game downloads. While evaluation remains difficult, we at Lumen believe that it is our duty to do what we can to build efficient infrastructure and pioneer solutions for a more sustainable internet.
One recent effort involves measuring the impact of Lumen’s peer-assisted CDN delivery solution: Mesh Delivery. Mesh Delivery uses hybrid peer-to-peer technology to dynamically retrieve content from the most efficient source — either the CDN or a mesh network of user devices. It is designed to augment delivery capacity without adding physical servers.
Large climate-controlled data centers with thousands of machines and significant infrastructure consume power in Megawatts. Surely, Mesh must have a positive impact by reducing server usage — right?
It’s not that simple, because any comparison between pure CDN vs. mesh-assisted streaming and game download would need to cover the impact of the entire supply chain and all possible use cases. A study of that scale would be resource and time intensive, so we’ve simplified the model.
This article by Sanket Shettennavar and Ludovic Le Frioux details our process and some of the initial results we’ve gotten from our analyses.
Step 1: determining the variables for our model of CDN vs. hybrid peer-to-peer delivery
There are many variables to consider, but in a laboratory setup, we can control a lot of them. First, we can use the same content source. We can use the same internet connection too, but one might argue that talking to a CDN is not the same as talking to another device through WebRTC. A simple test we ran was to check the number of networks hops for a CDN request and WebRTC request. They were approximately the same, so we cautiously assumed that there is no significant difference in using the internet infrastructure for obtaining content from CDN or through a peer-to-peer network.
That left us with two important variables. On server-side, the CDN capacity required when using Mesh Delivery is reduced, so we must account for that. On the other hand, Mesh Delivery has additional client-side code, so there will be some increase in the energy consumption of the end user devices. The overall impact of Mesh Delivery can be measured as the difference between the energy consumption of the servers and user devices.
Other sources of environmental impact — such as manufacturing — were assumed to be equal for our analysis. We therefore focused on one key question: does the decrease in power consumption resulting from lower CDN provisioning outweigh the increase in power consumption for end users?
Step 2: collecting server-side CDN data — PUE, network load and capacity provisioning
We then began our data collection journey. This task was relatively simple given that Lumen has both a global CDN and Mesh Delivery technology. So we worked with our colleagues to gather power consumption data, understand how CDN servers behave under varying loads, etc.
CDNs are server-grade computers, and are not differentiated by hardware specifications. It’s not enough, however, to measure the power consumption of the machine itself; we must also consider the power consumption of auxiliary services like climate control in the datacenter. This extra consumption is calculated as a factor called Power Usage Effectiveness (PUE). PUE varies with many factors: the architecture of the datacenter, the climate in the geographical location, etc., as these affect the energy required to effectively cool the facility.
We used a PUE of 1.75 for our study as this approaches what we have seen as industry averages. If a server consumes 100W of energy to run and has a PUE of 1.75, it in fact consumes 175W of energy considering the auxiliary energy consumption.
As for the CDN capacity, it is industry practice to provision more capacity than required using two calculations. First, looking at past peaks, one can expect future peaks to be around 30% larger. This is considered the maximum expected peak. Then, to avoid CDN overload, a safety margin is applied: this “maximum expected peak” is set to equal 75% of the final CDN capacity provisioned.
Below is a CDN throughput graph of an OTT provider and the corresponding capacity that is provisioning for the traffic. We see a maximum throughput of 8.8 Tbps, whereas the maximum expected peak is over 11 Tbps, and the total CDN provisioned even higher.
Energy consumption does not vary much with network load, the difference being in the order of 10% between peak and non-peak hours. Since peak vs. non-peak consumption varies little, it would be fair to assume that power consumption is constant at the peak rate, which we can express in Watts per Gigabyte per second (Watt second per Gigabyte if you are good with fractions). If we know the average bitrate of a stream, we can then convert this into Watts per user.
Step 3: modelling the power consumption of peer-to-peer on end user devices
We now had a fair estimation of CDN power consumption for a given bandwidth. On to the fun part!
When getting power consumption from end user devices, we chose the simplest — and perhaps most reliable — option: an off-the-shelf wattmeter. So we plugged laptops into a power strip, which was in turn connected to a wattmeter, which was plugged into a standard French type E wall plug.
Isolating the power consumption of Mesh Delivery from other computer processes was difficult, but in the end was not necessary given that our goal was simply to emulate the same conditions with CDN and with hybrid Mesh Delivery.
We booted up four identical laptops (both hardware- and software-wise), then opened a test page with a basic hls.js player playing a pure CDN stream. We ran the same test but with Mesh Delivery enabled and took the difference between the Mesh-enabled session and CDN only session. This gave a rough approximation of the power consumption of Mesh Delivery on top of a CDN only session. Then, we measured the power consumption of different configurations of Mesh: varying bitrate, varying number of peers, seeding only, leeching only and so on.
Before diving into the CDN vs. Mesh comparison, here are some observations on Mesh Delivery’s power consumption.
Without surprise, we see that seeding heavily consume significantly more power than those that are mostly “leeching”. Leechers only download a chunk once to play it, but a seeder must upload the same chunk multiple times to different peers, so it will of course consume more power to process the extra data transfer.
The impact of different bitrates on extra power consumption was somewhat less intuitive, as the following graph illustrates. We see a small increase in power consumption overall, but in some cases (the 0.8 Mbps bitrate), we observe that Mesh consumes no more than CDN only. Some digging revealed that this behaviour is linked to the scaling of the user device’s CPU.
Meanwhile, the more significant increase in power consumption for very high bitrate can be explained by the fact that seeders must process large streams of data multiple times simultaneously.
With this data, we built a model which takes real traffic data and simulates the power consumption of the system using CDN only, CDN + Mesh, as well as different scenarios of Mesh Delivery activation. So have we solved the million dollar question?
Long story short, we don’t know yet. If isolating a single video or download session, Mesh Delivery generally ends up consuming more power compared to CDN only. However, we haven’t yet accounted for the fact that Mesh Delivery can help us reduce our total CDN capacity needs.
We’ve also just looked at a scenario when Mesh Delivery is running on 100% of the traffic. Could we do better with a different use of Mesh Delivery?
Step 4: simulating different delivery mix scenarios
To exhaust all possibilities, we simulated the following four scenarios.
- CDN only: pure CDN consumption; no changes in CDN capacity provisioning.
- Pure hybrid P2P/CDN: Mesh is always enabled and doing peer-to-peer transfers; there is hybrid CDN/P2P traffic for all sessions.
- P2P upload and download enabled beyond a threshold of viewers: Mesh Delivery is disabled at first. When a threshold is met, Mesh is activated. This means that some viewers do CDN only and those over the threshold do hybrid P2P/CDN.
- P2P download enabled beyond a threshold; Mesh activated in advance for seeders. This scenario differs from 3 insofar as the viewers above the threshold do essentially only P2P download (not hybrid). Having Mesh enabled before allows a portion of the CDN viewers to act as seeders to the new viewers arriving after the threshold is met.
In practice these four scenarios look something like this:
Note that the thresholds in scenarios 3 and 4 are not fixed, nor are they necessarily the same. We have established the thresholds such that power usage is minimized. As you will see in the next section, the optimal thresholds depend largely on the shape of the traffic!
Step 5: putting it all together for different traffic shapes
The final — and perhaps most important — variable we needed to look at in our analysis was the shape of the broadcaster’s traffic. CDN provisioning depends largely on the spikiness of the audiences. Or, more accurately, on the peak to average ratio — how large the traffic spikes are compared to average traffic. The larger the traffic spikes are, the more CDN must be provisioned but remains idle.
We analyzed two main types of traffic:
- Live and VOD streaming platform with daily spikes and periodic high-profile events
- Game publisher with periodic updates to install across its player base
Our results using these two traffic profiles and the activation scenarios above are summarized as follows. For this we assumed 1.5 Mbps bitrate data streams and 70% peer-to-peer offload where Mesh Delivery enabled, which is consistent with our performance in production.
For brevity, we do not include the traffic graphs of each scenario tested, but these can be found in the appendix below.
1. Live and VOD streaming platform
Following histogram summarizes the power consumption across the four scenarios for a typical live and VOD streaming platform.
We see that at a 1.5 Mbps average bitrate, overall power consumption in a full hybrid P2P/CDN scenario goes down 31% versus CDN-only delivery thanks to the decrease in CDN capacity that needs to be provisioned. However, we also see that the impact increases with the different threshold scenarios, where power consumption is halved.
In these scenarios, Mesh Delivery picks up the traffic spikes that make inefficient use of CDN servers. Using CDN for steady state traffic and Mesh for the spikes allows the platform to use both systems most efficiently: CDN provisioned and Mesh Delivery. This is illustrated if we look at scenario 3:
We therefore see that the more “spikey” the traffic (i.e. the higher gap between peak and off-peak usage), more energy savings can be achieved.
2. Game publisher with periodic updates
The second example is more extreme case. Game and software download traffic is marked by low steady state for patches, etc. and extremely high spikes for new releases and larger files. If looking at the traffic of a game publisher we see the following in our four scenarios:
As you can see, power savings up to 65% can be achieved with smart usage of Mesh Delivery for such extreme traffic shapes.
Closing remarks
While this study was exploratory, we were pleased to see that a hybrid peer-to-peer and CDN delivery mix could potentially help reduce the overall power consumption required to deliver content to screens across the world. We hope that this and future work can contribute to a growing dialog about how we can build internet infrastructure more effectively.
Through this study, we were also able to identify some inefficiencies in Mesh Delivery by observing the events which led to spikes in power consumption, and we have already identified ways to improve in this regard.
Our next step is to improve our data gathering by automating power measurement and by switching to NUCs to eliminate the variations in consumption caused by screen usage. We also plan to expand this study to our native integrations, measure the power consumption for Android devices, iPhones, iPads, TV sticks, etc., and also consider device mix ratio in our traffic simulations.
In parting, we would highly recommend doing a similar study for your software product, not just to better understand its impact, but also to help identify ways to improve your code which may not otherwise have occurred to you. This certainly helped us and we think you can benefit from it too!
For more information about Lumen® Mesh Delivery, visit our website, and if you’d like to work on similar experiments, we’re hiring!
This content is provided for informational purposes only and may require additional research and substantiation by the end user. In addition, the information is provided “as is” without any warranty or condition of any kind, either express or implied. Use of this information is at the end user’s own risk. Lumen does not warrant that the information will meet the end user’s requirements or that the implementation or usage of this information will result in the desired outcome of the end user. All third-party company and product or service names referenced in this article are for identification purposes only and do not imply endorsement or affiliation with Lumen. This document represents Lumen products and offerings as of the date of issue.
Appendix: scenario graphs
1. Live & VOD platform
2. Gaming platform
