Measuring mirror throughput with bash

Measuring app network usage with a script

Measuring throughput's of mirror ranking script would make a nice dataset, I figured.

That lead me to Nethogs a tool like top, that shows total bandwidth per process:

Wow! This shows upload and download per process and device.

And a nice interface that we are used to from top.

Towards a script

For generating a dataset, I need more though:

1. A dump, and not a user interface
2. The dump must start as soon as the mirror ranking script starts
3. The dump must finish when when the mirror ranking script finishes
4. The dump must be filtered on process

Dumping nethogs

nethogs has a dump option '-t' that produces an endless stream of updates. It updates every -d seconds.

The dump looks like this:

> nethogs -tb -d 2
Adding local address: 192.168.67.145
Adding local address: 240b:c020:471:405e:5e80:b6ff:fe68:2770
Adding local address: fe80::5e80:b6ff:fe68:2770
Ethernet link detected

Refreshing:
unknown TCP/0/0 0       0

Refreshing:
unknown TCP/0/0 0       0

Refreshing:
unknown TCP/0/0 0       0

Refreshing:
/usr/lib/firefox/firefox/2853/1000      0.0347656       0.0347656
unknown TCP/0/0 0       0

sed can easily filter out the process I want to monitoring (process name is python)

nethogs -d 2 -bt | grep python

Let’s create a shell script

nethogs can be moved to the background with & so that the download script can start.

I use trap to set a callback on the EXIT signal. Once the script sends the EXIT signal, trap launches the following payload:

kill $(jobs -p)

jobs - p lists all child processes and kill ends them.

Let’s put it all together:

trap 'kill $(jobs -p)' EXIT
nethogs -d 2 -bt | grep python > rates.txt &
reflector  --score 5 > mirrors.txt

And this is resulting table of throughput:

cat rates.txt

/usr/bin/python/703838/1000	13.59	1808.46
/usr/bin/python/703838/1000	14.5596	1717.18
/usr/bin/python/703838/1000	14.6182	1694.34
/usr/bin/python/703838/1000	13.2963	1683.07

What about mirror ranking?

Reflector, measures throughput of Arch Linux mirrors and ranks them.

Here we see it’s output:

cat mirrors.txt
################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector --score 5 --sort rate
# When:       2023-11-24 08:17:19 UTC
# From:       https://archlinux.org/mirrors/status/json/
# Retrieved:  2023-11-24 08:16:59 UTC
# Last Check: 2023-11-24 08:07:33 UTC

Server = https://mirror.osbeck.com/archlinux/$repo/os/$arch
Server = http://mirror.ubrco.de/archlinux/$repo/os/$arch
Server = http://arch.jensgutermuth.de/$repo/os/$arch
Server = http://mirrors.qontinuum.space/archlinux/$repo/os/$arch
Server = http://mirror.moson.org/arch/$repo/os/$arch

There is a lot more to say about reflector and the dataset that I generated.

I will update you in a next article.