Neon Genesis Evangelion: The control room

8 browsers in a tiny car: Energy Efficiency benchmark

I’ve recently been on a crusade to test power efficiency of most popular browsers, so you don’t have to. Chrome, Arc, Edge, Safari, Firefox, Orion, Brave, Opera in a head to head battle.

No matter how they spin it, Chrome is still the king resource hog on macOS. Edge is even worse but is at least aware of it and trying. Safari lives in a magical buggy world of its own. Brave is just Chrome. Orion is kinda startup-y. Arc is hard to work with. Opera trying to please everyone. Firefox is unprofitable but very solid. Its crazy how companies live up to their stereotypes.

Side-quest: During my time testing these, the Animal sanctuary Drazevac, ran by 2 old ladies, with its 300 furry friends was running out of food! I (ab)used using their website for tests, and posting these results on Reddit, in the hopes of people donating a bit.

Draževac

Update: The website link doesn’t work, here is Instagram. They have a Patreon and PayPal, if you wish to help out! The furry boys, girls and nbs thank you. 🐕 🐩 🦮 🐈 🐕‍🦺 🐉 🐏 🐁 🐈‍⬛

Anyway, this is the third iteration of these tests, with the previous two being posted on Reddit and discussed at length. They give me very stabile numbers and they gave me the answer I was looking for. I hope that my methodology and the command I’ve attached below will be useful to you to create your own measurements, if you’re as willing to invest a silly amount of time into it, like I was.

Please please, don’t send me proposals, comments, nor lists of all the things I did wrong, I beg you, I found my answer, yours will be different.

Why browser energy efficiency matters

I’ve gotten hooked to Arc Browser and the way it looks and works ever since the alpha. I ended up dropping it because my computer was warm — a feeling I didn’t feel since switching from Intel to M1 Pro. It got me thinking about how for better or for worse, browsers became an operating system within your operating system.

Most apps that we use for work rely on browsers, our media consumption revolves around them, research, coding, travel and even using ChatGPT happens in them. They embed them into apps and code editors. We don’t have to like it, but thats how things work nowadays.

And it is all dominated by the monopolistically spiritual successor of IE, the Blink engine, Chromium project and Google. Its not that I dislike Google, I actually judge Chrome to be the best of the bunch, but I definitely dislike that we are basically down to 1 viable browser engine for the general public. (although some are trying)

Great Teacher Onizuka (1999)

Monopoly comes with a cost — stats that aren’t important to the maintainer get left behind, and thus everyone keeps their eyes on the same stats.

Every company’s marketing team chooses what is the easiest for them to compete with. Google is the first to tout how super mega giga performant Chrome is, at every turn, and my hypothesis was that this was simply because they can’t compete in other metrics. One of which is energy efficiency.

Some people care the most about performance. That is perfectly fine. I care the most about my battery, sweaty palms and fire-thighs.

I am sick of years of warm CPUs, loud fans, and 60min battery spans. I really care only about the real world feeling. I want a whole day battery, I want to be able to forget a charger when going to work and still survive the day. Many don’t remember, but that WAS one of the promises of netbooks chromebooks and airbooks back in the day.

There are two ways of getting there. We either get bigger batteries, or use less power, or both. But much like the state of GPUs and CPUs, we seem to be only making batteries larger. Well, through the magic of using Apple’s M1, we’ve seen that a world of better efficiency does exist.

Thus, after a bit of deliberation I dropped all FPS/CPU/RAM/GPU metrics, and decided to go with the one Apple focuses on the most — POWER.

Not really a surprise, that Apple’s own browser was the best, at the metric Apple chose to market…

Sum of mW used by package during test. (lower is better)

Results

I am not concerned with performance, only energy usage, if you care about raw performance, this is not a post for you — Just go use Chrome. I’m looking for most bang for the buck.

We’ll do a deeper dive into the methodology below, first, the results.

I tried to make the test very balanced this time. In previous tests, which included a lot more tab switching, and no WebGL and media consumption, Safari was using 10x less than Chrome.

All you need to know so far is that I’ve ran 20min-tests that simulate my own real world usage, with second-precision, and I’ve done it 3x per browser to ensure data stability. Yes, I’m on vacation.

Same breakdown, but a deeper overview of the differences in various parts of the test.

Unfortunately, what we all know is true, was true. Everything is a huge hog due to Chromium. You can fit three or four Safaris into one Chrome and two Safaris into a Firefox. Anyway, here are some of my thoughts:

• Safari was amazing. Its 3 times as efficient (in some tests up to 5x), its not Blink, its privacy oriented, its stabile. That all sounds good but in fact it was too good at preserving energy. It caps animation at 60fps and it’s buggy when it comes to WebGL. I literally had to slightly change the Google Maps part of the test because of it. It was my daily driver for a long time, but once I felt how internet feels in Chrome, I’m seriously considering dropping it. But damn those results are soo good. My fan never turned on, literally. The difference was much higher in previous tests where I used more complex web applications like Discord and a lot of tab switching. If your workflow uses a lot of tabs, there is a clear, 10x winner.
  Funnily enough, I did those tests with 120fps and older WebGL, and the energy usage is not really that much different. I mean its kinda right on par with Orion. Weird that Apple would choose 5% just efficiency at that cost.
• Chrome is the king of the hill when it comes to performance. It really does feel snappier and faster. Mainly because they seem to reserve all the available resources in the world, run at max fps, preload whole of internet. Also partially because they dropped any fancy animations, which makes it feel even faster than it is. But, it is Google after all, and it drains battery like chocolate milk compared to Safari…
  Don’t let this fool you into thinking I liked it. It is super wasteful. Undoubtedly Google.
• Arc was surprisingly good. I mean, the result was almost on par with Firefox, and way below other Blink/Chromium browsers. In fact, I expected energy usage spikes while using the UI but nop, it feels very native. I hated that it was still Chromium in disguise, but I can’t deny it the crown in the Blink category. Most bang for the buck. I really wonder how they managed to do it… Reached out to them for comment a couple of times but no response.
  I can feel the warmth tho. My palms still feel its a Chromium.
• Orion is the most performant Speedometer browser in the world. It had all the good parts of Safari in it, combined with a working WebGL implementation, native PiP, multiple profiles. I was ready to just dive in and switch myself and my whole family to it. I would pay for a browser like this. On the energy front, it did great really. Almost on par with Safari. Good job folks! I do wonder if this is due to WebKit being natively efficient, or is it something they did?
  Their Low Power mode kinda disappointed me tho, as it pretty much did nothing when it comes to live usage like fps cap, autoplay etc, and only focused on background tab cleanup. Which, to be fair, they are open about. We even see a little bump in the results, as the poor thing reloaded the Notion page each time I went back to that tab. I have mixed feelings about it. They dance a thin line of being-and-not-being opinionated at the same time. Something in me decided that Safari is a safer choice, as I don’t care much for extensions.
• The good old Firefox was closer to Safari than to Chrome when it comes to energy efficiency actually. It was probably the most stabile result I’ve had. FF also felt snappy and fast while doing it. That is commendable. I don’t have anything else to add really, it did the job well, didn’t get hot or spin up fans, WebGL worked just fine, it was a very smooth experience.
  Oh, apart from all the popups for Mozilla services they’ve been trying to shove down my throat, but thats besides the point of these tests -.- It just felt sad and Brave-like, and it was clear that they were struggling for cash. The funny thing is — I’d gladly pay for a good browser.
• The Chromium gang: Brave, Edge Opera are literally Chrome with a fake moustache. Brave tries to sell you crypto, Opera is playing around with the IE-like idea of toolbars, Edge feels like a Microsoft product. But, all are unmistakably Chromium when it comes to power usage. Some of them have low power features which require tinkering, others claim its telemetry and ad-blocking that save battery, which did not really show.
  Edge’s low power mode was the only one which made any difference and brought it into Arc range. Sorry it is not on the charts but just trust me. I admire that they are aware of this issue and are trying to make it better. Still, this is my subjective crusade so my tests focus on default, out of the box settings, which kinda disqualifies it for me personally — but it might not for you, I hear its an excellent browser!
• SigmaOS — Disqualified because it feels like Vim and Notion had a baby. I understand some people like that but I’m not those people, sorry, I just uninstalled it after the first test run. The results were in Orion range.

A side by side comparison of each part of the test

Teching the Tech

I ran all the tests on the following:

• macOS Ventura 13.3.1, on MacBook Pro 14" M1 Pro with 32GB RAM
• Edge 112.0.1722.39
• Arc 0.98.1
• Brave v1.50.114
• Safari Version 16.4
• Firefox 112.0
• Opera 97.0.4719.63
• Orion v0.99.123.3

This time I used powermetrics with a per-second measurement interval, which gave me Apple’s approximation of Combined Power (or Package Power how it was previously called).

sudo powermetrics \
  -i 1000 \
  --samplers cpu_power,gpu_power \
  -a --hide-cpu-duty-cycle \
  --show-usage-summary \
  --show-extra-power-info \
| grep --line-buffered -E "Combined Power \(CPU \+ GPU \+ ANE\): [0-9]*" \
| sed -u -n 's/Combined Power (CPU + GPU + ANE): \([0-9]*\) mW/\1/p' \
| while read -r line; do printf "%s, %s\n" $(date +%s) $line; done \
> ./recordings/raw/general/brave.csv

By doing this, we eliminated the usage coming from network, brightness, fans, speakers etc. We only focus on package usage, meaning CPU, GPU and their components like RAM and video encoding engine. Which is, after all, what we care about.

I attempted doing 300ms measurements as well but it made no tangible difference in the charts, so I suspect either the values are cached to some degree ( ioreg, for example caches up to 30sec) or the difference was smaller than the variability we see between test runs anyway.

In any case, you can do this yourself. Compile a list of tests you’d like to run and just run the command above. It will output a measurement every second, and you can later chart em. I’ve done some exploration with measuring shorter tests, and they’ve held up as well.

If you need help charting the data, I’ve released a livebook with some of the sample recordings on GitHub:

GitHub - markomitranic/strom-bench: Results of a browser energy efficiency benchmark for macOS

At first I thought no way is this going to work, the energy usage will be far too unstable. But turns out they were very stabile with each run (remember, I did 3 runs for each browser) being within a very small margin of error.

One might also claim that measuring package power usage is not feasible or smart. I, however beg to differ — the actual mW (or unit) usage is not important, the only important thing is the stability of ratios between browsers, and individual tests.

In other words, I don’t care how many mW Firefox used, I only care about the fact that I can fit 2 Firefoxes into a Chrome.

P.S. I’ve also done 3 separate background noise measurements just for fun, meaning i measured the computer while doing nothing, but not sleeping, and the median usage was only 16 mW per measurement interval, in each one 🤣 That is quite impressive and not unexpected.

Methodology was very subjective

I tried replicating my own usage pattern, on my own laptop, with a second-precision each time. There was a lot of tab swapping.

This is in fact not my first time doing this, it is a third iteration of these tests, and I can freely say, the most stabile one so far. Here is the previous one, where i used ioreg and powermetrics approximation measurements, for taste.

I’ve eliminated Discord, DataDog and Slack from the tests (previously I used those) as they were hard to control with the same precision every time. Sorry. Their usage just differed wildly between runs due to unpredictable content changes on the screen.

Each test begins with a freshly powered on Mac with 100% battery and attached to power. All apps and OS are latest versions. We fully restart the computer between each test run, wait for CPU to stabilize, and do not run any other apps. Mac always kept at roughly the same temperature. The browsers don’t have any extensions (unless they come with built-in ones) and are freshly installed. All window sizes kept the same, covering 100% of the screen apart from the notch area. Whenever I open a new tab, i leave the previous one open. I swap tabs back to Notion at the end of every step.

Script is as follows:

1. Open the Notion Page with this list in it.
2. Google “speedtest” and run it.
3. Google “Speedometer” and run it.
4. Watch whole 2k HDR 60fps on YouTube (paste url) in full screen mode.
5. Google “youtube dave2d” visit his profile, then videos, scroll to the Homepod video play it in Browser PiP while doing the rest of items.
6. Search for “drazevac shelter” on Google via address bar. Open the page and slowly scroll to bottom as if you were reading it. Now scroll to the top. Open their source code in a new tab (paste url) and look at 4 files.
7. Open a local apartment searching website. Look at the map of properties and find a specific apartment every time. Go through the slideshow, read the page, copy the address. Open the floor plan PDF in a new tab.
8. Paste the apartment address into google, and click on the Google Maps widget/result. Zoom in and out. Use 3D, use StreetView. (use it exactly the same every time!)
9. Google “pinterest”, visit it, log in with Google Auth and search for “scandi coffee table”. Scroll and scroll and scroll. Open 2 items along the way.
10. Google for “Wolt”, find the same sushi place each time, look at 5 items. Scroll some more, add a 6th one.
11. Open an electronics chain homepage by searching their name on Google. Go to TVs category, and open the same TV each time.
12. Open a Danish News website, click on an article, copy URL. Open Google Translate, paste the URL and click on the result. This will open a new, translated page. Spend time slowly reading the article.