1. vmware esxi安装centos8，开启内核虚拟化

2. 安装qemu-system-arrch64

操作：

wget https://download.qemu.org/qemu-2.11.0.tar.xz 
tar xvJf qemu-2.11.0.tar.xz 
cd qemu-2.11.0 
./configure –-target-list=aarch64-softmmu 
make 
make install

报错1：Python not found. Use --python=/path/to/python

#  ./configure --target-list=aarch64-softmmu

ERROR: Python not found. Use --python=/path/to/python
 
#解决1，指定python路径，提示当前python版本过高：
# ./configure --target-list=aarch64-softmmu --python=/usr/bin/python3.6

ERROR: Cannot use '/usr/bin/python3.6', Python 2.6 or later is required.
       Note that Python 3 or later is not yet supported.
       Use --python=/path/to/python to specify a supported Python.
#解决2，下载python2，再次尝试遇到新的报错
# ./configure --target-list=aarch64-softmmu --python=/usr/bin/python2

ERROR: "cc" either does not exist or does not work

报错2：ERROR: "cc" either does not exist or does not work

下载gcc解决该报错，然后提示依然是缺包

# ./configure --target-list=aarch64-softmmu --python=/usr/bin/python2

ERROR: zlib check failed
       Make sure to have the zlib libs and headers installed.

报错3：ERROR: zlib check failed　　Make sure to have the zlib libs and headers installed.

下载zlib zlib-devel，然后再次提示缺包

# ./configure --target-list=aarch64-softmmu --python=/usr/bin/python2

ERROR: glib-2.22 gthread-2.0 is required to compile QEMU

下载glib2 glib2-devel，提示还是缺包

下载pixman pixman-devel

./config成功

make告警：缺少命令：flex bison

下载flex，bison

然后还是失败：

util/memfd.c:40:12: error: static declaration of ‘memfd_create’ follows non-static declaration
 static int memfd_create(const char *name, unsigned int flags)
            ^~~~~~~~~~~~
In file included from /usr/include/bits/mman-linux.h:117,
                 from /usr/include/bits/mman.h:49,
                 from /usr/include/sys/mman.h:41,
                 from /root/qemu-2.11.0/include/sysemu/os-posix.h:29,
                 from /root/qemu-2.11.0/include/qemu/osdep.h:104,
                 from util/memfd.c:28:
/usr/include/bits/mman-shared.h:46:5: note: previous declaration of ‘memfd_create’ was here
 int memfd_create (const char *__name, unsigned int __flags) __THROW;
     ^~~~~~~~~~~~
make: *** [/root/qemu-2.11.0/rules.mak:66: util/memfd.o] Error 1

 

看百度上各种解决办法，懒得去一个一个验证，直接换最新的包

 wget https://download.qemu.org/qemu-7.2.0.tar.xz

查看README文件，根据提示操作

 mkdir build
  cd build
  ../configure
  make

提示缺包ninjia

yum --enablerepo=powertools install ninja-build

这次提示python版本过低

Found ninja-1.8.2 at /usr/bin/ninja
Running postconf script '/usr/bin/python3 /root/qemu-7.2.0/scripts/symlink-install-tree.py'
NOTICE: You are using Python 3.6 which is EOL. Starting with v0.62.0, Meson will require Python 3.7 or newer

安装一个python3.9 然后把/usr/bin/python3指向python3.9

然后再次执行

 mkdir build
  cd build
  ../configure --target-list=aarch64-softmmu
  make

无报错

make install

缺包就装包：perl

然后make install成功

# which qemu-system-aarch64
/usr/local/bin/qemu-system-aarch64

#理论上一切OK，除非包有问题
qemu-img create -f raw -o size=30G test.img
qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios QEMU_EFI.fd -nographic -drive if=none,file=/var/lib/libvirt/images/Kylin-Server-10-SP1-Release-Build04-20200711-arm64.iso,id=cdrom,media=cdrom -device virtio-scsi-device -device scsi-cd,drive=cdrom -drive if=none,file=test.img,id=hd0 -device virtio-blk-device,drive=hd0

 果然还是报错了

#报错：EFI stub: Exiting boot services and installing virtual address map

 

查了一下，是EFI固件包的问题，重新下载UEFI固件

wget https://www.kraxel.org/repos/firmware.repo

yum install edk2.git-aarch64

然后再试一次

qemu-img create -f qcow2 KylinV10sp2.qcow2 64G
qemu-system-aarch64 -m 8192 -cpu cortex-a57 -smp 2 -M virt -bios /usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw -nographic -drive if=none,file=/var/lib/libvirt/images/Kylin-Server-10-SP2-aarch64-Release-Build09-20210524.iso,id=cdrom,media=cdrom -device virtio-scsi-device -device scsi-cd,drive=cdrom -drive if=none,file=KylinV10sp2.qcow2,id=hd0 -device virtio-blk-device,drive=hd0  -net none

这次一切正常，直到。。。这。。。直接让打电话，再试了一次，还是卡在这里下不去了。。。

[  325.623043][  1] Authorization warning: Authorization binary is corrupted, Please call 400-089-1870 for help.

既然跑流程没问题，装系统卡iso镜像，那就试试直接从现成的arm虚拟机镜像文件起给虚机试试吧

qemu-system-aarch64  -m 8192 -cpu cortex-a57 -smp 2 -M virt -bios /usr/share/edk2.git/aarch64/QEMU_EFI.fd   -nographic -device e1000e,netdev=dev0,mac='00:00:00:01:00:01' -netdev tap,ifname=tap-int,id=dev0,script=no,downscript=no,vhost=on -device virtio-net-pci,netdev=dev1,mac='00:00:00:01:00:02',vectors=32,mq=on -netdev tap,ifname=tap-0,id=dev1,script=no,downscript=no,vhost=on,queues=16  -drive format=raw,file=test-kylin10-2.raw

终于成功了，也就是说卡的是这一个iso镜像，换一家不那么"高贵"的iso应该也能行。

 

 

 

该走的流程先走完，记下后续笔记（虽然没什么用，我的virt-manager就默认了一个x64的qemu，arm64的qemu没对接上）

安装virt-manager组件，编辑/etc/libvirt/qemu.conf最后一行添加

nvram = [
   "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw"
]
 

 然后开始尝试使用virt-manager管理服务。

 

 

 其他

qemu基础

1. ttyS0串口重定向

qemu启动一个Linux Guest，如果只需要这个Guest的终端，不需要图形用户界面，可将Linux Guest的输出重定向至虚拟串口（最终是重定向到终端）。下面以Fedora Guest为例说明用法。不同发行版和版本设置方法可能不同（如grub、grub2就不一样，debian这样的发行版需要配置/etc/inittab）。
重定向虚拟串口到console，有两种用法

    

    内核不在虚拟机镜像文件里头
qemu -kernel arch/i386/boot/bzImage -hda root-2.4.20.img -append "root=/dev/hda console=ttyS0" -nographic

    参考：http://blog.csdn.net/defeattroy/article/details/5257323

    

    内核在虚拟机镜像文件里头

    先畸形启动Guest，如果是grub2，编辑文件 /etc/default/grub 

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Fedora"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 SYSFONT=True  KEYTABLE=us rd.luks=0 LANG=en_US.UTF-8 rhgb quiet console=tty0 console=ttyS0,9600n8"


GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"
 生成的/boot/grub2/grub.cfg文件应有如下配置

serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1
terminal_input serial
terminal_output serial
......
    linux    /vmlinuz-3.6.11-rt32 root=UUID=0ff1fb64-4e8d-44bf-87f7-8a2f111159d8 ro rd.md=0 rd.lvm=0 rd.dm=0 SYSFONT=True  KEYTABLE=us rd.luks=0 LANG=en_US.UTF-8 rhgb quiet console=tty0 console=ttyS0,9600n8
    

    然后用-nographic参数启动qemu

    qemu-kvm -enable-kvm -m 1024 -nographic -drive file=/mnt/sdb/vms/testfc/testfc.qcow2,if=virtio,index=0,format=qcow2

 

 

参考链接：https://wiki.ubuntu.com/UEFI/virt-install

https://www.txisfine.cn/archives/a0d5fa12

https://www.cnblogs.com/linuxxl/p/11658387.html

https://zhuanlan.zhihu.com/p/353547345

https://jgsun.github.io/2018/12/17/qemu-virt-arm64/

https://blog.csdn.net/qq_42001403/article/details/100532861

https://luomuxiaoxiao.com/?p=743

 https://wiki.qemu.org/Documentation/Networking

 

https://blog.csdn.net/richardysteven/article/details/54807927  #关于vnc，kvm与网络

--enable-kvm ，-serial telent:localhost:4321,server,nowait，-vnc :port

https://www.cnblogs.com/schips/p/15489856.html　　#关于串口重定向与自动登录

https://fadeevab.com/build-android-kernel-and-run-on-qemu-minimal-step-by-step/  #qemu虚拟化安卓

 

 

以下转载自：https://fadeevab.com/how-to-setup-qemu-output-to-console-and-automate-using-shell-script/

1. Input/output to the host terminal

-serial stdio

qemu-system-x86_64 -serial stdio wheezy.qcow2

-serial stdio redirects the virtual serial port to the host's terminal input/output. You will see a welcome string after a successful boot.

-nographic

qemu-system-x86_64 -nographic wheezy.qcow2

-nographic does the same as "-serial stdio" and also hides a QEMU's graphical window.

Cautions:

1. You will not see any early boot logs in the host's console. To get them, see Early boot messages in the host terminal below.
2. To exit the guest system without GUI, using stdio redirected to the terminal, login as a root (user: root, password: root) and shutdown the system (wait after that for a while):

   # Guest
   shutdown -h now
   

2. Early boot messages in the host terminal

console=ttyS0

If you want to see early boot logs, you should pass console=ttyS0 parameter to a Linux kernel command line:

qemu-system-x86_64 -nographic -kernel vmlinuz -hda wheezy.img -append "root=/dev/sda console=ttyS0"

or

qemu-system-x86_64 -serial stdio -kernel vmlinuz -hda wheezy.img -append "root=/dev/sda console=ttyS0"

or

qemu-system-x86_64 -serial stdio wheezy.qcow2
 # 1. Wait for a GRUB menu to show.
 # 2. Press `e`.
 # 3. Find the line starting with "linux".
 # 4. Add "console=ttyS0".

qemu-system-x86_64 -serial stdio -kernel vmlinuz -hda wheezy.img -append "root=/dev/sda console=ttyS0":

• -serial stdio or -nographic redirects input/output to the current terminal.
• -append "root=/dev/sda console=ttyS0": console=ttyS0 forces the guest kernel to send output to the first UART serial port ttyS0, which is redirected to the host by the -serial stdio option, and root=/dev/sda points the kernel to use a /dev/sda device to load the wheezy.img.

Other options:

• -kernel vmlinuz loads the kernel from the local "./vmlinuz" file.
• -hda wheezy.img is a raw image which is suitable for booting with vmlinuz binary (wheezy.qcow2 won't be recognized in the block device).

3. Input/output through a named pipe (file)

Create a named pipe

mkfifo /tmp/guest.in /tmp/guest.out

Start QEMU

qemu-system-x86_64 -serial pipe:/tmp/guest -kernel vmlinuz -hda wheezy.img -append "root=/dev/sda console=ttyS0"

-serial pipe:/tmp/guest redirects a guest's output to a /tmp/guest.out and allows to send input from host to guest via /tmp/guest.in.

Take an output from the guest

cat /tmp/guest.out

Send a command to the guest

When login screen appears, send a login string:

printf "root\n" > /tmp/guest.in

Wait until some string

Wait until SSH Daemon starts.

while read line; do
  echo "${line}"
  if [[ ${line} == *"Secure Shell server: sshd"* ]]; then
    break;
  fi
done < /tmp/quest.out

 

 

 

4. Automate QEMU guest using expect tool

Install "expect" tool

sudo apt install expect

Create an expect script

example.exp:

#!/usr/bin/expect -f

# Wait enough (forever) until a long-time boot
set timeout -1

# Start the guest VM
spawn qemu-system-x86_64 -serial stdio wheezy.qcow2

expect "login: "
send "root\n"

expect "Password: "
send "root\n"

expect "# "
send "shutdown -h now"

Original script is found there: https://stacoverflow.com/questions/314613/qemu-guest-automation, but be careful, symbol of quotes “ (which is not a ") in the original stackoverflow answer cannot be recognized by the expect utility (send "root\n").

Execute "expect" script

chmod +x example.exp
./example.exp

5. Automate QEMU guest using ssh

Set up port forwarding

qemu-system-x86_64 -netdev user,id=net0,hostfwd=tcp::10022-:22 -device e1000,netdev=net0 wheezy.qcow2

Connect via ssh

ssh root@localhost -p 10022 'uptime; ls; echo Test;'

• To apply server's public key automatically use -o "StrictHostKeyChecking no":

  ssh root@localhost -p 10022 -o "StrictHostKeyChecking no" 'uptime; ls; echo Test;'
  

Troubleshooting

1. QEMU guest has to be able to recognize a network card device (NIC, Network Interface Card): -netdev user,id=net0 -device e1000,netdev=net0.

   # Without port forwarding
   qemu-system-x86_64 -netdev user,id=net0 -device e1000,netdev=net0 wheezy.qcow2
   

2. Boot and check that the new interface has appeared on the guest system:

   # Guest
   ifconfig -a
   

   Linux kernel on the guest must support a network card emulated by QEMU. In the opposite case the guest won't get a new Ethernet interface. After booting you should find "eth0" (running broadcast device, not loopback) on the guest. It depends solely on the guest Linux kernel and on the kernel modules.
3. Check the 10022 port on the host:

   # Host
   netstat -tanp | grep 10022
      tcp  0  0 0.0.0.0:10022   0.0.0.0:*  LISTEN  16589/qemu-system-x
   

4. Check the 22 port on the guest:

   # Guest
   netstat -tanp | grep 22
       tcp  0  0 0.0.0.0:22      0.0.0.0:*  LISTEN  2430/sshd
   

5. You can forward telnet port 23 and verify the connection:

   qemu-system-x86_64 -netdev user,id=net0,hostfwd=tcp::10023-:23 -device e1000,netdev=net0 wheezy.qcow2
   

   1. Guest (server):

      # Guest
      nc -v -l -p 23
          Listening on [0.0.0.0] (family 0, port 23)
      

   2. Host (client):

      # Host
      echo asdf | nc localhost 10023
      

Establish passwordless login via ssh

1. Generate host SSH keys:

   # Host
   ssh-keygen -b 2048 -t rsa -q -N "" -f ./qemukey
   

2. Set up a public key to the guest as a trusted (authorized) key.
   1. Via ssh-copy-id
      • You need a root with password. You the guest root is passwordless, go to the guest system and set up the password:

        # Guest
        sudo passwd
        

      • Send the generated public key:

        # Host
        ssh-copy-id -p 10022 -i ~/.ssh/qemukey root@localhost
        

      • Reset the password in the guest system:

        # Guest
        sudo passwd -l root
        

   2. Manually
      • Send a public key via scp:

        # Host
        scp -P 10022 ./qemukey.pub root@localhost:/root/.ssh/
        

      • Login to the guest and set up new authorized key:

        # Guest
        cat /root/.ssh/qemukey.pub >> /root/.ssh/authorized_keys
        /etc/init.d/ssh restart
        

      • Or mount device locally, put the public key to the .ssh directory, and concatenate to authorized_keys.
3. Fix the /etc/ssh/sshd_config on the guest:

   PasswordAuthentication no
   PermitRootLogin without-password
   

4. Restart SSH daemon on the guest:

   # Guest
   /etc/init.d/ssh restart
   

5. Connect via ssh:

   # Host
   ssh root@localhost -p 10022 -i ./qemukey
   

   Viola! You don't need the password and you can automate the remote QEMU guest.

---

Binaries used in the examples

1. wheezy.qcow2 (i386): bootable Debian "Wheezy" image a QEMU copy-on-write format. Login/password: "root"/"root", and "user"/"user".

   wget https://people.debian.org/~aurel32/qemu/i386/debian_wheezy_i386_standard.qcow2 -O wheezy.qcow2
   

2. wheezy.img (i386): non-bootable Debian "Wheezy" image (without kernel) to use with own kernel (-kernel vmlinuz).

   wget https://storage.googleapis.com/syzkaller/wheezy.img
   

3. vmlinuz (i386): compressed bootable Linux kernel. Options:
   1. Build from the scratch: Build Android Kernel and Run on QEMU with Minimal Environment: Step by Step.
   2. Download from Ubuntu repository (WARNING! Port forwarding will NOT work):

      wget http://security.ubuntu.com/ubuntu/pool/main/l/linux-signed-azure/linux-image-4.15.0-1036-azure_4.15.0-1036.38~14.04.2_amd64.deb
      ar x linux-image-4.15.0-1036-azure_4.15.0-1036.38~14.04.2_amd64.deb
      tar xf data.tar.xz ./boot/vmlinuz-4.15.0-1036-azure
      cp ./boot/vmlinuz-4.15.0-1036-azure ./vmlinuz
      

   3. You can try your host's linux kernel passing one to the QEMU guest (WARNING! You could have problems either with port forwarding, or with a block device):

      sudo cp /boot/vmlinuz-$(uname -r) ./
      

   WARNING! Ubuntu's vmlinuz doesn't contain drivers for QEMU emulated network card devices (NIC). Debian's vmlinuz doesn't have prebuilt drivers to load a raw image from /dev/sda device.