在 pve 下使用 virtiofs 加速虚拟机和宿主机的文件夹共享

想用飞牛影视，但是我的显卡不支持 vGPU，所以我打算把 docker 也挪到飞牛里，刚好飞牛也有一个不错的 docker dashboard，就刚好试试吧。

第一个问题就是，怎么解决存储问题。我的服务器没有多余的 SATA 控制器，没办法把硬盘直接通给虚拟机，同时我也不想服务器直接换成飞牛。

要么使用 NFS，毕竟网卡是 SR-IOV 的，性能也足够。要么就是 virtiofs，使用共享内存直接映射到虚拟机里。

说干就干，查了一下资料，PVE 已经有人写了脚本，能较为方便的扩展挂载的目录，我就简单记录一下过程。

首先看的完整的教程是 Drallas/Mount Volumes into a Proxmox VM with virtiofs.md，该教程提供了完整的步骤和相关脚本。

创建脚本

脚本负责创建 virtiofsd 服务

sudo nano /var/lib/vz/snippets/virtiofs_hook.pl

写入以下内容：

#!/usr/bin/perl

use strict;
use warnings;

my $conf_file = '/var/lib/vz/snippets/virtiofs_hook.conf';
my %associations;

open my $cfg, '<', $conf_file or die "Failed to open virtiofs_hook.conf";
while (my $line = <$cfg>) {
    chomp $line;
    my ($vm_id, $paths_str) = split /:/, $line;
    my @path = split /,/, $paths_str;
    $associations{$vm_id} = \@path;
}

close $cfg or warn "Close virtiofs_hook.conf failed: $!";

use PVE::QemuServer;

use Template;
my $tt = Template->new;

print "GUEST HOOK: " . join(' ', @ARGV) . "\n";

my $vmid = shift;
my $conf = PVE::QemuConfig->load_config($vmid);
my $vfs_args_file = "/run/$vmid.virtfs";
my $virtiofsd_dir = "/run/virtiofsd/";
my $DEBUG = 1;
my $phase = shift;

my $unit_tpl = "[Unit]
Description=virtiofsd filesystem share at [% share %] for VM %i
StopWhenUnneeded=true

[Service]
Type=simple
RuntimeDirectory=virtiofsd
PIDFile=/run/virtiofsd/.run.virtiofsd.%i-[% share_id %].sock.pid
ExecStart=/usr/libexec/virtiofsd --log-level debug --socket-path /run/virtiofsd/%i-[% share_id %].sock --shared-dir [% share %] --cache=auto --announce-submounts --inode-file-handles=mandatory

[Install]
RequiredBy=%i.scope\n";

if ($phase eq 'pre-start') {
  print "$vmid is starting, doing preparations.\n";

  my $vfs_args = "-object memory-backend-memfd,id=mem,size=$conf->{memory}M,share=on -numa node,memdev=mem";
  my $char_id = 0;

  # Create the virtiofsd directory if it doesn't exist
  if (not -d $virtiofsd_dir) {
     print "Creating directory: $virtiofsd_dir\n";
     mkdir $virtiofsd_dir or die "Failed to create $virtiofsd_dir: $!";
    }

  # TODO: Have removal logic. Probably need to glob the systemd directory for matching files.
  for (@{$associations{$vmid}}) {
    # my $share_id = $_ =~ s/^\///r =~ s/\//_/gr;
    my $share_id = $_ =~ m/.*\/([^\/]+)/ ? $1 : '';  # only last folder from path
    my $unit_name = 'virtiofsd-' . $vmid . '-' . $share_id;
    my $unit_file = '/etc/systemd/system/' . $unit_name . '@.service';
    print "attempting to install unit $unit_name...\n";
    if (not -d $virtiofsd_dir) {
        print "ERROR: $virtiofsd_dir does not exist!\n";
    }
    else { print "DIRECTORY DOES EXIST!\n"; }

    if (not -e $unit_file) {
      $tt->process(\$unit_tpl, { share => $_, share_id => $share_id }, $unit_file)
        || die $tt->error(), "\n";
      system("/usr/bin/systemctl daemon-reload");
      system("/usr/bin/systemctl enable $unit_name\@$vmid.service");
    }
    system("/usr/bin/systemctl start $unit_name\@$vmid.service");
    $vfs_args .= " -chardev socket,id=char$char_id,path=/run/virtiofsd/$vmid-$share_id.sock";
    $vfs_args .= " -device vhost-user-fs-pci,chardev=char$char_id,tag=$vmid-$share_id";
    $char_id += 1;
  }

  open(FH, '>', $vfs_args_file) or die $!;
  print FH $vfs_args;
  close(FH);

  print $vfs_args . "\n";
  if (defined($conf->{args}) && not $conf->{args} =~ /$vfs_args/) {
    print "Appending virtiofs arguments to VM args.\n";
    $conf->{args} .= " $vfs_args";
  } else {
    print "Setting VM args to generated virtiofs arguments.\n";
    print "vfs_args: $vfs_args\n" if $DEBUG;
    $conf->{args} = " $vfs_args";
  }
  PVE::QemuConfig->write_config($vmid, $conf);
}
elsif($phase eq 'post-start') {
  print "$vmid started successfully.\n";
  my $vfs_args = do {
    local $/ = undef;
    open my $fh, "<", $vfs_args_file or die $!;
    <$fh>;
  };

  if ($conf->{args} =~ /$vfs_args/) {
    print "Removing virtiofs arguments from VM args.\n";
    print "conf->args = $conf->{args}\n" if $DEBUG;
    print "vfs_args = $vfs_args\n" if $DEBUG;
    $conf->{args} =~ s/\ *$vfs_args//g;
    print $conf->{args};
    $conf->{args} = undef if $conf->{args} =~ /^$/;
    print "conf->args = $conf->{args}\n" if $DEBUG;
    PVE::QemuConfig->write_config($vmid, $conf) if defined($conf->{args});
  }
}
elsif($phase eq 'pre-stop') {
  #print "$vmid will be stopped.\n";
}
elsif($phase eq 'post-stop') {
  #print "$vmid stopped. Doing cleanup.\n";
} else {
  die "got unknown phase '$phase'\n";
}

exit(0);

创建配置文件

sudo nano /var/lib/vz/snippets/virtiofs_hook.conf

写入配置内容

101: /mnt/pve/cephfs/<folder1>, /mnt/pve/cephfs/<folder2>
102: /mnt/pve/cephfs/<folder1>, /mnt/pve/cephfs/<folder2>, /mnt/pve/cephfs/<folder3>

这里的 101 就是虚拟机的 id，每个目录之间使用半角逗号和一个空格分开。

给虚拟机设置 HookScript

qm set <vmid> --hookscript local:snippets/virtiofs_hook.pl

启动虚拟机

qm start <vmid>

虚拟机内挂载

脚本默认会使用 <vmid>-<目录名称> 的组合创建 tag，使用该 tag 即可在虚拟机内完成挂载。

mount -t virtiofs <vmid>-<目录命令> /mnt

例如给 100 虚拟机共享了 /mnt/video 目录，则使用该命令挂载。

mount -t virtiofs 100-video /mnt

Tip

第一次启动挂载的时候会提醒 superblock 错误。

mount: /srv/cephfs-mounts/download: wrong fs type, bad option, bad superblock on mnt_pve_cephfs_multimedia, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

此时只需要重启一下虚拟机就可以了，似乎是和 pve 启动后无法更改虚拟机参数有关。