23 年买了一台华为 RH2288 V3 在家里当电老虎,服务器上带一个万兆光,支持 SRIOV,这样我可以分出来多个虚拟网卡来加速网络访问,创建一个给 docker,一个给 windows 虚拟机,一个给 linux 虚拟机等。

虽然系统换来换去,最终又是回到了 PVE,因为最近我想跑个飞牛,挺喜欢他家的影视的。

我写了个脚本,开机自动创建了几个网卡。

[Unit]
Description=SRIOV service

[Service]
Type=oneshot
ExecStartPre=-/usr/bin/bash -c '/usr/bin/echo 10 > /sys/class/net/enp2s0f1/device/sriov_numvfs'
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 0 mac 1e:c9:90:ab:15:0b trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 1 mac d2:1b:36:88:c3:17 trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 2 mac 6e:03:f8:19:c2:09 trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 3 mac 46:ac:a3:40:ce:46 trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 4 mac a6:57:31:92:bc:41 trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 5 mac 7e:e7:1b:e9:f5:93 trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 6 mac 86:09:25:4f:a1:4e trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 7 mac aa:4c:9c:43:03:dc trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 8 mac 7e:38:28:ee:61:64 trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set dev enp2s0f1 vf 9 mac 96:70:81:58:67:6a trust on spoofchk off
ExecStartPre=-/usr/bin/ip link set enp2s0f1v0 up
ExecStartPre=-/usr/bin/ip link set enp2s0f1v1 up
ExecStartPre=-/usr/bin/ip link set enp2s0f1v2 up
ExecStartPre=-/usr/bin/ip link set enp2s0f1v3 up
ExecStartPre=-/usr/bin/ip link set enp2s0f1v4 up
ExecStartPre=-/usr/bin/ip link set enp2s0f1v5 up
ExecStartPre=-/usr/bin/ip link set enp2s0f1v6 up
ExecStartPre=-/usr/bin/ip link set enp2s0f1v7 up
ExecStart=-/usr/bin/ip link set enp2s0f1 up
[Install]
WantedBy=default.target

使用 ip -c a 可以看到创建了许多开头是 enp2s0f1,以 v + 编号结尾的网卡。

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp2s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 70:79:90:a8:19:37 brd ff:ff:ff:ff:ff:ff
3: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 70:79:90:a8:19:38 brd ff:ff:ff:ff:ff:ff
4: enp2s0f1v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 1e:c9:90:ab:15:0b brd ff:ff:ff:ff:ff:ff
inet6 fd6c:f1fa:b46b:d747:1cc9:90ff:feab:150b/64 scope global dynamic mngtmpaddr
valid_lft 1789sec preferred_lft 1789sec
inet6 fe80::1cc9:90ff:feab:150b/64 scope link
valid_lft forever preferred_lft forever
5: enp2s0f1v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d2:1b:36:88:c3:17 brd ff:ff:ff:ff:ff:ff
inet6 fd6c:f1fa:b46b:d747:d01b:36ff:fe88:c317/64 scope global dynamic mngtmpaddr
valid_lft 1789sec preferred_lft 1789sec
inet6 fe80::d01b:36ff:fe88:c317/64 scope link
valid_lft forever preferred_lft forever

网卡的初始化已经结束了,剩下的就是 lxc 的配置了。我们可以把一个 vf 网卡分配给 lxc,这样不需要走 vmbr 那个网桥,相当于两个物理网卡互相通信,没有 cpu 参与,效率能提高不少。

创建 lxc 的过程在这里不展开了,创建完毕后先不要启动,在资源里把默认创建的网卡删除,PVE 没有提供界面分配设备,所以要修改配置文件分配一个 vf 网卡进去。

lxc 的配置文件在 /etc/pve/lxc/<id>.conf,编辑该文件,添加几行 net 配置。

lxc.net.0.type: phys
lxc.net.0.link: enp2s0f1v1
lxc.net.0.flags: up
lxc.net.0.ipv4.address: 192.168.1.30/24
lxc.net.0.ipv4.gateway: 192.168.1.1
lxc.net.0.name: eth0

其中最重要的是 type 和 link,也可以在这里直接分配 ip,以及设备名。

启动 lxc 后,可以进去看一下网卡信息。

root@k8s:~# ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
5: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether d2:1b:36:88:c3:17 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.30/24 brd 0.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fd6c:f1fa:b46b:d747:d01b:36ff:fe88:c317/64 scope global dynamic mngtmpaddr
valid_lft 1798sec preferred_lft 1798sec
inet6 fe80::d01b:36ff:fe88:c317/64 scope link
valid_lft forever preferred_lft forever

可以看到网卡设备已经被设置成 eth0 和 192.168.1.30 了。使用 iperf3 测试一下速度。

Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.30 port 44754 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.68 GBytes 14.4 Gbits/sec 616 1.18 MBytes
[ 5] 1.00-2.00 sec 1.71 GBytes 14.6 Gbits/sec 0 1.99 MBytes
[ 5] 2.00-3.00 sec 1.70 GBytes 14.6 Gbits/sec 0 2.44 MBytes
[ 5] 3.00-4.00 sec 1.68 GBytes 14.4 Gbits/sec 1552 1.54 MBytes
[ 5] 4.00-5.00 sec 1.70 GBytes 14.6 Gbits/sec 0 2.21 MBytes
[ 5] 5.00-6.00 sec 1.69 GBytes 14.5 Gbits/sec 0 2.48 MBytes
[ 5] 6.00-7.00 sec 1.68 GBytes 14.4 Gbits/sec 1733 878 KBytes
[ 5] 7.00-8.00 sec 1.70 GBytes 14.6 Gbits/sec 607 1.08 MBytes
[ 5] 8.00-9.00 sec 1.40 GBytes 12.0 Gbits/sec 0 1.33 MBytes
[ 5] 9.00-10.00 sec 1.30 GBytes 11.2 Gbits/sec 0 1.34 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 16.2 GBytes 13.9 Gbits/sec 4508 sender
[ 5] 0.00-10.00 sec 16.2 GBytes 13.9 Gbits/sec receiver

iperf Done.