SO_REUSEPORT/ADDR (1/2) — How different about the condition of binding —
I have heard the existence of SO_REUSEADDR, PORT before but there is not many chances I have to be awear of these. But I just have to be awear of it and this time I investigated it and write up here for myself who will forget.
For the behaviour of SO_REUSEADDR, SO_REUSEPORT there are some difference between OS like Free BSD, OSX, Linux. But this post mention the behavior only for Linux.
Cap: The socket behaviour without any option
- UDP/TCP socket doesn’t allow multiple socket to bind same IP, same port
- UDP/TCP socket allow multiple socket to bind same port and different IP
- 0.0.0.0 is magic code and recognised as all IP
- Not possible that socketA bind to 0.0.0.0:80 and socketB bind to 192.168.0.1:80 because 0.0.0.0 stands for all IP.
With SO_REUSEADDR
This socket option can be used on Linux kernel 2.4 and later than that.
This bring mainly 2 features to us
- The socket not needing to call “listen” system call is allowed to bind exact same ip address and same port (TCP Client, UDP Server , UDP Client)
2. The port having TIME_WAIT status is recognised as un-used port when “bind” system call check if the port is in use or not
Let me take a look at first feature first
1. The socket not needing to call “listen” system call is allowed to bind exact same ip address and same port
We experiment it with sample python program
TCP Client Socket1 sample (python)
import socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
client_socket.bind(('127.0.0.1', 12345))
client_socket.connect(('127.0.0.1', 8080)) # any destination you want
while True:
passSO_REUSEADDR option should be set for all sockets being bound to the port you want to bind multiple socket onto, not only for new socket.
TCP Client Socket2 sample (python)
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
client_socket.bind(('127.0.0.1', 12345))
client_socket.connect(('192.168.0.1', 8080)) # differ from first prog
while True:
passAfter prepare 2 python programs to use same ip/port for tcp client socket, run these in parallel.
[vagrant@localhost ~]$ python tcp_client_1.py &
[1] 11048
[vagrant@localhost ~]$ python tcp_client_2.py &
[2] 12304
[vagrant@localhost ~]$ netstat -aln | grep 8080
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
tcp 2 0 127.0.0.1:12345 10.0.2.15:8080 ESTABLISHED
tcp 3 0 127.0.0.1:12345 127.0.0.1:8080 ESTABLISHED
tcp 0 0 127.0.0.1:8080 127.0.0.1:12345 ESTABLISHED
tcp 0 0 10.0.2.15:8080 127.0.0.1:12345 ESTABLISHEDtcp_client_2.py doesn’t raise “socket.error: [Errno 98] Address already in use” although we specified same ip/port and you can know tcp client socket bound onto exact same ip(127.0.0.1) and exact same port(12345) from the result of netstat.
One thing you have to be careful here is that we can not connect to the exact same destination ip/port from exact same source ip/port because Linux kernel will be no way to tell the difference between tcp sessions and no way to forward it to each socket correctly if source ip/port and destination ip/port are all same. When you try to do it, you would see following error “socket.error: [Errno 99] Cannot assign requested address”
Let’s see the UDP server pattern as other case although same thing is there
UDP Server Socket sample (python)
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('0.0.0.0', 12345))while True:
data = s.recv(1024)
print data
And then run UDP Server program in parallel
[vagrant@localhost ~]$ python udp_server.py &
[1] 11049
[vagrant@localhost ~]$ python udp_server.py &
[2] 11304
[vagrant@localhost ~]$ lsof -i:12345 -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 11049 vagrant 3u IPv4 78063 0t0 TCP *:12345 (LISTEN)
python 11304 vagrant 3u IPv4 78079 0t0 TCP *:12345 (LISTEN)As you can see it, we could bind 2 different sockets onto the exact same ip and same port, UDP Server socket looks LISTEN as long as we see from the result of lsof. UDP Server socket doesn’t need to call “listen” system call that’s why multiple UDP Server sockets can be bound onto the same ip/port with SO_REUSEADDR.
If you try to do same thing with TCP Server socket, you will see [Errno 98] error but SO_REUSEPORT allow you to bind multiple TCP sockets onto the same ip/port :)
2. The port having TIME_WAIT status is recognised as un-used port
First of all let me cap the TIME_WAIT status of TCP session. TIME_WAIT status is basically pause status with interval time after close TCP session and before new socket open TCP session to prevent delay TCP packet from affecting new application opening that port.
But there are many situation for App developer to think this is too kind behaviour and recognise it as just noisy, because if you develop something server program and try it and found any bugs, then fix it and try to run again but you could face “[ERRORNO 98]” although you already stopped that program because the port still exists with TIME_WAIT and socket is not allowed to bind to the port being in use. So all you can do is to just wait for TIME_WAIT to finish.
SO_REUSEADDR will resolve such a frustration for developer so that the port with TIME_WAIT status can be ignored when “bind” system call check if the ip and port combination is in use or not.
With SO_REUSEPORT
This option has been added relatively recent than SO_REUSEADDR . This can be used on laster than 3.9 kernel.
The behaviour of this option is similar concept to SO_REUSEADDR, but this option allow us to bind multiple TCP/UDP server sockets onto exact same IP and same port.
But there is 1 limitation on socket to share exact same IP and same port which is not in SO_REUSEADDR. All sockets to share the IP/port should be spawned by the process having same effective user. This can be applied for both protocol UDP/TCP, notice that binding multiple UDP server socket was possible with just SO_REUSEADDR without taking care effective user (processA executed by root open *:80 and processB executed by userA can open *:80 with SO_REUSEADDR). But SO_REUSEPORT doesn’t allow it. As for looking at just this point, we can think SO_REUSEPORT as more restricted.
TCP Server socket sample
import socket
import oss = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
s.bind((‘0.0.0.0’, 8080))
s.listen(1)
conn = swhile True:
conn, addr = s.accept()
print(‘Connected to {}’.format(os.getpid()))
data = conn.recv(1024)
conn.send(data)
Run TCP server programs in parallel with same user
[vagrant@localhost ~]$ python tcp_server.py &
[1] 1656
[vagrant@localhost ~]$ python tcp_server.py
[2] 1657
[vagrant@localhost ~]$ lsof -i:8080 -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 1656 vagrant 3u IPv4 78786 0t0 TCP *:8080 (LISTEN)
python 1657 vagrant 3u IPv4 78791 0t0 TCP *:8080 (LISTEN)
Run TCP server program with root user after run TCP server program with vagrant user
[vagrant@localhost ~]$ python tcp_server.py &
[1] 1659[vagrant@localhost ~]$ sudo python tcp_server.py
Traceback (most recent call last):
File “tcp_server.py”, line 6, in <module>
s.bind((‘0.0.0.0’, 8080))
File “/usr/lib64/python2.7/socket.py”, line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use[vagrant@localhost ~]$ python tcp_server.py &
[2] 1664[vagrant@localhost ~]$ lsof -i:8080 -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 1659 vagrant 3u IPv4 78866 0t0 TCP *:8080 (LISTEN)
python 1664 vagrant 3u IPv4 78924 0t0 TCP *:8080 (LISTEN)
We failed to run second TCP program even with root user because first TCP server program was executed with vagrant user and the effective user of second program is different from first one. That’s why we failed to bind second program onto the same IP/port of first program. As a evidence, we succeeded in binding another socket onto the same port as first program opened with vagrant user.
NB: I omit the case of UDP server socket, but UDP server socket is same bahavior as TCP server socket and notice the behaviour of binding multiple sockets onto the same ip/port with SO_REUSEADDR is different from the one with SO_REUSEPORT.
Actually the behaviour of forwarding packet across multiple sockets bound to the same ip/port was difference between SO_REUSEADDR/PORT but this will be wrote as next post.
