Linux 的 ReusePort 特性

最近在看 Django-channels, 因为自带的 daphne asgi 服务器本身没有提供多 worker 的工作方式,
而文档内推荐的部署方案是用 supervisor 做进程管理,
但考虑到本来系统上就有 systemd, 所以尝试用 systmed 来做进程管理,
用 systemd.socket 监听指定端口, 在收到连接后, 启用对应的服务, 并把 socket 传递给服务,
先贴一下单个进程的 dc.socket 和 dc.service 文件

# dc.socket
[Unit]
Description= Django Channels socket

[Socket]
ListenStream=0.0.0.0:9001

[Install]
WantedBy=sockets.target

# dc.service
[Unit]
Description=Django Channels
Requires=veops1.socket

[Service]
Type=simple
WorkingDirectory=/project_path/
NonBlocking=true
ExecStart=/project_path/virtenv/bin/daphne -e systemd:domain=INET:index=0 \
-s DC --access-log /var/log/dc/daphne_access1.log dc.asgi:application
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGQUIT
KillMode = control-group
Restart=on-failure
RestartSec=10s
User=root
Group=root


[Install]
WantedBy=multi-user.target

这样只能启用一个进程, 如果要像 supervisor 那样多个进程监听一个端口, 需要用到从 Linux kernel 3.9 开始增加的 SO_REUSEPORT 特性, 把之前的实例配置变成模板配置 dc@.socket 和 dc@.service 文件转变成模板, 像这样

# dc@.socket
[Unit]
Description= Django Channels %i socket

[Socket]
ListenStream=0.0.0.0:9001
ReusePort=true
Service=dc@%i.service

[Install]
WantedBy=sockets.target

# dc@.service
[Unit]
Description= Django Channels %i
After=dc@%i.socket

[Service]
Type=simple
WorkingDirectory=/project_path/
ExecStart=/project_path/virtenv/bin/daphne -e systemd:domain=INET:index=0 \
-s DC --access-log /var/log/dc/daphne%i.log dc.asgi:application
NonBlocking=yes
ExecReload=/bin/kill -HUP $MAINPID
KillMode = control-group
Restart=on-failure
RestartSec=10s
User=root

[Install]
WantedBy=multi-user.target

用
systemctl start dc@0
systemctl start dc@1
systemctl start dc@2
...
可以生成多个 service 实例并监听同一个端口
所有访问这个端口的连接都会被分发给各个进程

测试环境:
system-release: CentOS Linux release 7.9.2009 (Core)
uname: 3.10.0-1160.6.1.el7.x86_64
启用第二个 socket 的时候报错了

failed to listen on sockets: Address already in use

看起来 ReusePort 并没有生效
但放在 Centos8, kernel 4.19 上却可以生效,
怀疑是系统有这个特性, 但 systemd 没有启用, 搜了一番也没搜到
有在 3.10 上用 systemd 使用过这个特性的吗?
希望能告诉我一下是不是要改什么配置

tomychen

2020-12-02 17:57:51 +08:00

SO_REUSEPORT (since Linux 3.9)
Permits multiple AF_INET or AF_INET6 sockets to be bound to an identical socket address. This option must be set on each socket (including the first socket) prior to calling bind(2) on the
socket. To prevent port hijacking, all of the processes binding to the same address must have the same effective UID. This option can be employed with both TCP and UDP sockets.

For TCP sockets, this option allows accept(2) load distribution in a multi-threaded server to be improved by using a distinct listener socket for each thread. This provides improved load dis‐
tribution as compared to traditional techniques such using a single accept(2)ing thread that distributes connections, or having multiple threads that compete to accept(2) from the same socket.

For UDP sockets, the use of this option can provide better distribution of incoming datagrams to multiple processes (or threads) as compared to the traditional technique of having multiple
processes compete to receive datagrams on the same socket.

```c
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <string.h>
#include <netinet/in.h>
#include <unistd.h>

int main(void)
{
int sock = -1;
int flags = 1;
int ret;
struct sockaddr_in sa;

sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock == -1) {
perror("socket");
return 1;
}
if (setsockopt(sock, SOL_SOCKET, SO_REUSEPORT, &flags, sizeof(int)) <
0) {
perror("setsockopt");
return 1;
}

memset(&sa, 0, sizeof(sa));

sa.sin_addr.s_addr = htonl(INADDR_ANY);
sa.sin_family = AF_INET;
sa.sin_port = htons(10086);

ret = bind(sock, (struct sockaddr *)&sa, sizeof(sa));
if (ret == -1) {
perror("bind()");
return 1;
}

ret = listen(sock, 10);
if (ret == -1) {
perror("listen()");
return 1;
}
while (1) {
printf("listen\n");
sleep(2);
}
}
```
uname -a
Linux localhost.localdomain 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

代码工作正常，所以和系统无关