New import wizard to migrate guests directly from other hypervisors

Highlights

• New import wizard to migrate guests directly from other hypervisors.
  Connect to other hypervisors using their public APIs and directly migrate guests to Proxmox VE.
  First implementation is for VMware ESXi.
  Guests can be started on Proxmox VE while their data is still being imported to the target storage in the background.
• Support for automated and unattended installation of Proxmox VE.
  Proxmox VE now ships a tool that prepares a Proxmox VE ISO for automated installation.
  The prepared ISO retrieves all required settings for automated installation from an answer file.
  The answer file can be provided directly in the ISO, on an additional disk such as a USB flash drive, or over the network.
• Backup fleecing (advanced feature).
  When creating a backup of a running VM, a slow backup target can negatively impact guest IO during the backup process.
  Fleecing can reduce this impact by using fast local storage as a buffer for data blocks.
  However, the fleecing approach requires the use of temporary local storage space.
  Backup fleecing can make sense when backing up IO-heavy guests to a remote Proxmox Backup Server with a slow network connection.
• Modernized Proxmox VE firewall implementation based on nftables (opt-in technology preview).
  The new implementation is written in Rust and intended to replace the current firewall based on iptables in the future.
  Moving to nftables will improve robustness and make it possible to fix long-standing issues with the old implementation.
  The nftables firewall is provided as a technology preview to gather feedback from the community and needs to be enabled manually.
  With a few exceptions, its feature set is nearly on par with the old firewall implementation.
• Seamless upgrade from Proxmox VE 7.4, see Upgrade from 7 to 8

Changelog Overview

• Enhancements in the web interface (GUI)
• Fix an issue where the VM creation wizard would not allow creating a VM without an ISO.
• Fix an issue that prevented users from seeing accessible replication jobs.
• Fix an issue where the node summary page would not display the version of a running foreign kernel (issue 5121).
• When editing LVM or directory storages, add an explanatory tooltip to the "Shared" checkbox.
• Expose the saferemove setting for LVM storages in the GUI.
• Add some missing packages to the "Package versions" window.
• When editing ZFS storages, display 16k as the blocksize placeholder to reflect the current ZFS defaults.
• Change "Syslog" button to "System Log" button to better reflect that the log is gathered from the systemd journal.
• Fix an issue where the GUI used an incorrect language code for Korean, and provide a clean transition for users who still have a cookie with the incorrect language code set.
• Improve documentation of the /nodes/{node}/status API call.
• Fix an issue where the date picker would choose the wrong date after changing to a different month.
• When granting permissions to a group, make the group selector searchable, similar to the user and token selectors.
• Clarify the confirmation prompt for removing a certificate without a name.
• Fix an issue where edit windows would not be correctly masked while loading.
• Fix an issue where adding a storage to a pool would break edit windows of the same browser session, and guard against similar issues in the future.
• Display the end-of-life message as a notice up until three weeks before the end-of-life date, and display it as a warning from that point on.
• Move the "Reset" button for edit windows to an icon-only button in the title bar (issue 5277).
• This reduces the risk of misclicking and accidentally resetting form data.
• Fix xterm.js not loading in certain OS+Browser constellations, for example iOS (issue 5063).
• Change the icon for editing guest notes from a gear to a more fitting pencil.
• Allow to select note contents by double-clicking.
• Previously, this was not possible because double-clicking on a note would always open the editor window.
• This behavior can now be optionally disabled by a browser-local GUI setting.
• The TFA input field now sets an autocompletion hint for improved compatibility with password managers (issue 5251).
• Fix an issue where the mobile UI did not allow to log in with TOTP.
• Display the subscription reminder in the mobile UI as well, for improved consistency with the GUI.
• The user edit window now disables the "Key IDs" field if the user has TFA configured.
• The HTTP server now supports compressing responses using deflate in addition to gzip.
• Improved translations, among others:
• French
• German
• Italian
• Japanese
• Korean
• Simplified Chinese
• Spanish
• Traditional Chinese
• Ukrainian

Virtual machines (KVM/QEMU)

• The new ESXi importer is available as a storage plugin.
  This allows to import guests from an ESXi node (preferred for performance reasons) or a vCenter instance.
  Guests need to be stopped on the VMware side before the import.
  Live import allows starting the guests on Proxmox VE immediately and migrate their disks in the background.
  Most settings are read from the source and translated to equivalent Proxmox VE configuration options.
  Overrides for configuration options can be defined manually.
• New QEMU version 8.1.5
  See the upstream changelog for details.
• Apply some upstream QEMU patches:
• Fix a potential deadlock resizing disks with IO thread enabled.
• Fix a potential deadlock when taking a snapshot or a suspend-mode backup with VirtIO block disks and IO thread.
• Fix an issue where guest IO on a VirtIO block or VirtIO SCSI disk with IO thread could get stuck on backup, storage move or hotplug.
• Increase the limit of open files for QEMU processes (issue 4507).
• This fixes issues where a VM with many network devices with multi-queue or many virtual RBD disks would exceed the limit, resulting in VM crashes or stuck guest IO.
• Fix an issue where live-importing certain VMDK images would trigger an assertion failure.
• If a backup fails, increase the chance of showing the relevant error message by ignoring irrelevant error messages.
• Add CPU models introduced in QEMU 8.1.
• Mark Windows Server 2025 as supported by the win11 OS type.
• Add support for emulated IOMMU (vIOMMU) which enables hypervisor VMs to passthrough PCI devices to nested VMs (issue 3784).
• Allow up to 256 cores per socket.
• Add Intel E1000E as a supported network device model in the GUI, as it is a popular option in VMware guests.
• Allow specifying vendor and product information for virtual SCSI disks (issue 4957). Fix a regression that temporarily prevented creation of SCSI Cloud-init disks (issue 5363).
• Make IP addresses selectable for copy & paste in the guest agent network information.
• Fail live migration early if the VNC clipboard is enabled, as this is not yet supported by QEMU.
• Prevent starting a VM with a 32-bit CPU type and a 64-bit OVMF BIOS as it is unsupported.
• Prevent CPU hotplug for non-x86_64 VMs as it is unsupported.
• Fix an issue where rebooting a VM with a passed-through mediated device would fail due to a race condition.
• Fix an issue where insecure live migration of multiple VMs in parallel could cause some migrations to fail due to a port conflict (issue 4501).
• Fix an issue where removing a snapshot of a running VM would fail if a disk was moved since VM start (issue 2258).
• Fix an issue where migration would fail to deactivate volumes on the source node in case the volume has to be renamed on the target node.
• Fix an issue where the storage for Cloud-init cicustom snippets would not be activated when starting a VM, causing the VM startup to fail (issue 4085).
• Cloning a VM now succeeds even if deactivating volumes on the source node fails (issue 1734).
• Deactivating a volume can fail if a template is cloned multiple times in parallel, and previously caused the clone task to fail.
• Increase the default timeout for drive-mirror operations to 10 minutes, to avoid timeouts when moving storage of running VMs.
• When importing a disk, emit a warning when a fallback format is used instead of the requested format.
• Allow configuring the VNC clipboard in the GUI.
• Fix an issue where VMs that were suspended in-guest would be wrongly terminated after backup.

Containers (LXC)

• New LXC version 6.0.0
• See the upstream changelog for details.
• Allow filtering firewall logs of containers, similarly to node and VM firewall logs.
• Add a keep-env option to the pct enter and pct exec commands that controls whether the current environment is passed to the container.
• For backwards compatibility, keep-env currently defaults to enabled, and will default to disabled in the next major release.
• Make host device passthrough for containers available in the GUI (issue 754).
• API and CLI already supported device passthrough since Proxmox VE 8.1.
• Avoid passing environment variables for the TLS certificate store location from the host to the container (issue 5194).
• These variables did not leak any sensitive information, but could cause errors in the container if the TLS certificate store was at a different location.
• When changing the power status of a container, lock its config asynchronously in the task worker to avoid blocking the API.
• Fix an issue that prevented hotplug of mount points when running kernel 6.5 or later (issue 5160).
• Change the vmid field type in the /nodes/{node}/lxc response from string to integer to conform with the schema.
• Fix an issue where volumes could not be hotplugged due to missing volume activation.
• Fix an issue where a device passthrough entry would be added to the container config even though the entry was invalid.
• Fix an issue where executing pct fsck would fail due to missing volume activation, and deactivate volumes afterwards even if the command failed.
• Avoid passing the deprecated noacl mount option for ext4 formatted volumes (issue 4846).

General improvements for virtual guests

• Stopping a VM or container can now overrule active shutdown tasks (issue 4474).
• Previously, stopping a guest could fail if there was a running shutdown task for that guest.
• When stopping a guest, the GUI now offers to abort running shutdown tasks.
• Overruling shutdown tasks is currently not implemented for HA guests.
• Allow moving unused disks to another storage.
• Fractional rate limits for virtual network devices can now be entered without a leading zero.

Improved management for Proxmox VE clusters

• Modernize handling of host keys for SSH connections between cluster nodes (isse 4886).
  Previously, /etc/ssh/ssh_known_hosts was a symlink to a shared file containing all node hostkeys.
  This could cause problems if conflicting hostkeys appeared in /root/.ssh/known_hosts, for example after re-joining a node to the cluster under its old name.
  Now, each node advertises its own host key over the cluster filesystem. When Proxmox VE initiates an SSH connection from one node to another, it pins the advertised host key.
  For existing clusters, pvecm updatecerts can optionally unmerge the existing /etc/ssh/ssh_known_hosts.
• Fix an issue where UTF-8-encoded files under /etc/pve would be incorrectly re-encoded as UTF-8.
• Increase key size for new SSH keys from 2048 to 4096 bit.
• Improvements and fixes to the notification system introduced in Proxmox VE 8.1:
• Fix an issue where notification recipients could not be removed from a sendmail or SMTP target (issue 5074).
• Backup job notifications include the total backup size, which was previously missing (issue 5067).
• Clarify notification fields in the backup job window.
• Deprecate notification options that were only relevant in a short transition period before the final notification system was introduced.
• As built-in targets and matchers cannot be removed, change the corresponding button labels from "Remove" to "Reset".
• Fix an issue where notification emails would contain a localized Date header that caused issues with some mail clients.
• A mail forwarded to an SMTP target by proxmox-mail-forward now contains the original message as the message body instead of a message part.
• Notification mails now contain the Auto-Submitted header.
• Support lists of values for the exact match-field mode.
• Notification events for forwarded mails now include the hostname and type.
• The Gotify notification target now also sets the X-Gotify-Key header for authentication, to improve compatibility with older versions of Gotify (issue 5274).

Backup/Restore

• Support backup fleecing as an advanced feature.
  When creating a backup of a running VM, the backup process starts sending data blocks to the backup target in the background.
  When the guest attempts to write new data to a block that is not yet backed up, the old data needs to be backed up before the guest write can proceed. This delays the guest write.
  If the backup target storage is slow, for example because it is a Proxmox Backup Server with a slow network connection, this delay can negatively impact VM performance.
  With fleecing enabled, the old data can be buffered on a fast local "fleecing storage" instead, allowing the guest write operations to proceed with a reduced delay.
  Backups with fleecing can have lower impact on VM performance, but temporarily take up additional space on the fleecing storage.
  Fleecing can be configured for a datacenter-wide backup job in the GUI, and be used through the CLI and API.
• Replace confusing output of null value when successfully removing a backup snapshot on a Proxmox Backup Server storage with a line that indicates success (issue 4971).
• Show the full error context output in operations on a Proxmox Backup Server storage.
• Fix an issue where the connection to a Proxmox Backup Server presenting a certificate signed by a CA not trusted by the Proxmox VE node would fail, even if a fingerprint is provided (issue 5248).
• Advanced options for backup jobs, such as performance settings or bandwidth limits, can now be specified in the GUI (issue 4513). Previously, these settings had to set via API or CLI.
• Fallbacks for performance-related backup settings are now computed per setting. Previously, a fallback was computed for the performance setting as a whole.
• Improve logging of failed hook scripts during backup.
• Switch to modern ntfs3g driver for the single file-restore image, since it supports more features found in the filesystems of current Windows guests (issue 5259).

Storage

• Replication jobs now fail early with an informative error in case source and target volumes both exist, but at least one of them has no snapshots.
• Fix an issue where a snapshot created by vzdump could not be deleted because it was wrongly assumed to be a replication snapshot.
• Clarify error message if an qemu-img invocation fails.
• Fix an issue where insecure migration or replication would fail if it had to wait for a lock.
• Support migration of VM templates with disks on LVM-thin storage (issue 1611).

Ceph

• Fix an issue where the ceph-crash daemon could not post crash reports, and logged recurring warnings to the journal (issue 4759).
• Since ceph-crash does not run as root, it is not allowed to read the Ceph admin keyring.
• The issue is resolved by creating a client.crash keyring with appropriate rights and permissions both for new and existing Ceph clusters.
• The new keyring is created automatically on upgrade, if necessary.
• Integrate patches that make it possible to manually install and use the upstream Ceph dashboard.
• Due to upstream changes, Proxmox VE 8.0 and 8.1 were incompatible with the Ceph dashboard.
• Fix an issue where open TCP connections to VMs could freeze or timeout while upgrading Ceph if the firewall is active (issue 5213).
• Integrate upstream patch that ensures RocksDB is built with compile-time optimizations for increased performance.
• Fix permissions on subdirectories of /var/lib/ceph to avoid errors when posting crash reports.
• Fix an issue where usage was reported as "NaN %" if no OSDs were created yet (issue 5293).
• When expanding the Ceph status in the GUI, wrap the title to make sure it is completely visible (issue 5106).
• Avoid edge case in which running pveceph purge would remove too many files.
• Fix an issue where a monitor could not be deleted due to an incorrect existence check (issue 5198).

Access control

• Keep ACL entries in the user.cfg file sorted, to ease tracking changes with tools like ansible or etckeeper (issue 5335).
• Require non-root users to enter their current password on password change.
• This is to hedge against a scenario where an attacker has local or even physical access to a computer where a user is logged in.
• Add dedicated Sys.AccessNetwork privilege to guard API calls that could be abused to gather information about the internal network, for example downloading ISOs (issue 5254).
• This new privilege can be granted to allow users to download ISOs without requiring the more powerful Sys.Modify privilege.
• Restrict the values of legacy user-keys to the ones that were allowed in the deprecated initial TFA system.
• Align the validation regular expression for the ACR value in OpenID Connect realms with the settings in Proxmox Backup Server (see related issue 5190).
• Restrict the attributes and their values collected during LDAP sync to avoid issues with unexpected LDAP entries.
• Add liberal length restrictions on user attributes (names, email addresses), to avoid bloating the user.cfg, which would needlessly restrict the number of ACL entries that can be stored.

Firewall & Software-Defined Networking

• Firewall based on nftables as an optional technology preview.
  Introduce a new modern Proxmox VE firewall implementation written in Rust using nftables.
  After installing and enabling the new implementation, firewall rules are enforced using nftables instead of iptables.
  Moving to nftables will make firewall bridges obsolete, which reduces complexity and allows fixing long-standing issues with the old implementation.
  The new implementation is very close to feature parity with the old implementation. Some minor differences, such as the handling of REJECT rules, still need to be addressed.
  The nftables firewall is provided as a technology preview to gather feedback from the community.
• IP allocation for VNets with active DHCP plugin and using both IPv4 and IPv6 are now atomic, in the sense that a failure to obtain an allocation for one family removes a successful one for the other family.
• Allow arbitrary names for bridges. Previously, bridges had to be named vmbrN to be recognized (issue 545).
• Fix guest startup when a zone has DHCP enabled, but no DHCP ranges configured for any of the subnets.
• Ensure that addresses for both IPv4 and IPv6 are requested in case that a DHCP range was added for one family after a guest has already got an allocation for the other.
• Add support for ipv6 prefix-list filtering in frr
• Adapt the generated FRR config to work around an upstream issue and update to FRR version 8.5.2.
• Use the correct ordering and add support for IPv6 routes in EVPN controller.
• Use the proxy configured for the datacenter in SDN for external API calls.
• Improve interaction with the external IPAM plugins phpIPAM and NetBox.
• Remove deprecated and now forbidden priority field from change request to PowerDNS (issue 5275).
• Correctly detect array values, such as DHCP ranges, as changed for validation.
• Handle VNets that are pending, but not yet committed, when creating the list of all VNets.
• Fix an issue that triggered Perl's Taint Mode when an API daemon attempted to delete DHCP config files.
• Improve the testing framework for the SDN stack, by refactoring the code to enable mocking interactions with the system.
• Fix an issue where firewall rules using IPs/CIDRs could not be edited (issue 4963) and changes would not be detected properly.
• Fix an issue where editing an EVPN zone would not accept an empty primary exit node.
• Provide explicit type information for the pending field in the API returns.
• Derive the BGP router-id from the interface MAC-Address in IPv6 underlay networks (issue 5364).
• Fix the syntax for IPv6 route map entries in the generated FRR config (issue 5361).
• Verify ISIS Network ID for syntactical correctness (issue 5344).
• Add support for parsing and preserving BGP community-list entries in the FRR config (issue 5319).
• Set the configured MTU of a network as DHCP option.
• Keep aliases in cluster.fw file sorted, to ease tracking changes with tools like ansible or etckeeper (issue 5335).
• Fix the interaction between ifupdown2 and packages which install hooks for network configuration (issue 5009).
• This fixes an issue where hosts could hang during network setup after a reboot if both ntpsec and ntpsec-ntpdate are installed.

Improved management of Proxmox VE nodes

• A new meta package proxmox-secure-boot-support pulls in all the needed packages to enable secure-booting Proxmox systems with the default keys. This makes it easier to switch an existing system to Secure Boot.
• The UEFI firmware update executable fwupd-amd64-signed is now signed with the Proxmox secure-boot keys, making it possible to use fwupd in Proxmox systems with secure-boot enabled.
• Proxmox repackaged the r8125-dkms driver needed for 2.5G RealTek NICs, as the driver is not merged in the upstream kernel, in order to add compatibility with the new 6.8 kernel series.
• Wake on LAN now allows configuring the bind interface and broadcast address used for sending the magic packet (issue 5255).
• Automatically rewrite the LVM config to filter out RBD disks, if possible. Previously, LVM on the host would recognize LVM Physical Volumes located on VM disks backed by KRBD.
• Order the ZFS pool specific import units created by Proxmox VE for new pools before the general zfs-import-cache and zfs-import-scan units, preventing harmless but misleading errors in the journal while booting (issue 4835).
• Fix a network configuration issue where creating a new VLAN on a bridge would overwrite the vlan-id of an existing VLAN (issue 5141).
• Clarify the description of the internal pvesubscription set-offline-key command.
• Avoid misleading error message in pvesubscription in case the offline key activation was successful.
• Avoid logging warnings on EFI systems without Secure Boot support.
• Increase the maximum size before rotating the pveam logs from 50 KiB to 256 KiB, and warn if rotation fails.
• Support for adding custom ACME enabled CA's with optional authentication through External Account Binding (EAB) is now also present in the GUI (issue 4497, issue 5093).
• Avoid spurious warnings about uninitialized values in various situations.
• When a service restart fails while upgrading pve-manager or pve-ha-manager, avoid failing the upgrade.
• Improved pvereport to provide a better status overview:
• Add packet counter to iptables-save output and format it as a table.
• Add the current kernel command line.
• List currently defined jobs.
• Include more details about recognized APT sources and packages that are held back.
• Extend the list of reported packages, by recently added library packages
• Improvements to Proxmox Offline Mirror:
• Consider the subscription status when configuring mirrors for the ceph repositories provided by Proxmox.
• Improve UX in promxox-offline-mirror-helper, when having multiple subscription keys available at the chosen mountpoint.
• Add dark mode to the documentation.
• Fix an wrong configuration setting for allowing weak RSA cryptographic parameters.
• Improve path handling with command line arguments.
• Support repositories that do not provide a Priority field (issue 5249).

Installation ISO

• Support for automated and unattended installation of Proxmox VE.
• Introduce the proxmox-auto-install-assistant tool that prepares an ISO for automated installation.
• The automated installation ISO reads all required settings from an answer file in TOML format.
• One option to provide the answer file is to directly add it to the ISO. Alternatively, the installer can retrieve it from a specifically-labeled partition or via HTTPS from a specific URL.
• If the answer file is retrieved via HTTPS, URL and fingerprint can be directly added to the ISO, or obtained via DHCP or DNS.
• See the wiki page on Automated Installation for more details.
• memtest86+ got updated to version 7.0.
• Fix indefinite hangs during country detection (issue 4872).
• The parsing of hostname options was fixed to catch invalid entries (issue 5230).
• Fix an issue where setting ZFS compression to off did not have any effect, due to a change in upstream defaults.
• Deprecated checksum options got removed in the advanced options for ZFS pools.
• Improve the layout of widgets in the GTK-based installer for consistent margins and alignment.

Notable changes

• Kernel 6.8 is reported to fix a long-standing issue where VMs running on hosts with multiple NUMA nodes could temporarily freeze for short durations. The issue was more likely to occur if KSM and/or NUMA balancing was active. See the corresponding forum thread for more information.
• Previously, the LXC power management endpoints under /api2/json/nodes/{node}/lxc/{vmid}/status/ acquired the config lock synchronously in the API handler, meaning that a successful HTTP response implied that the task worker was able to acquire the lock. Now, they acquire the config lock asynchronously in the task worker, meaning that the task worker may still fail to acquire the lock despite the successful HTTP response. This change may require adjustment of tools that use the API directly and relied on the previous behavior.
• When creating a backup and specifying custom performance settings, fallbacks for unspecified settings are now computed per performance setting. Previously, a fallback was computed for the performance setting as a whole. In a few edge cases, this may change the effective performance settings for backups. See the commit for details.

Known Issues & Breaking Changes

Kernel 6.8

The Proxmox VE 8.2 releases will install and use the 6.8 Linux kernel by default, a major kernel change can have a few, hardware specific, side effects.

You can avoid installing the 6.8 kernel by pinning the proxmox-default-kernel package version before the upgrade. The last version to depend on kernel 6.5 is 1.0.1.

To pin the package to that version, create a file in /etc/apt/preferences.d/proxmox-default-kernel with the following content. This will keep proxmox-default-kernel on the old version until that file is deleted, and a new upgrade is initiated:

• Package: proxmox-default-kernel
• Pin: version 1.0.1
• Pin-Priority: 1000

Kernel: Change in Network Interface Names

Upgrading kernels always carries the risk of network interface names changing, which can lead to invalid network configurations after a reboot. In this case, you must either update the network configuration to reflect the name changes, or pin the network interface to its name beforehand.

See the reference documentation on how to pin the interface names based on MAC Addresses.

Currently, the following models are known to be affected at higher rates:

• Models using i40e. Their names can get an additional port suffix like p0 added.

Kernel: DKMS

Since the default kernel changes to 6.8, it may happen that installed DKMS modules will not build anymore. For instance, as of April 2024 this is the case for the latest NVIDIA GRID driver 17.1.

Aside from simply removing and not using those modules, one possible workaround is to pin the proxmox-default-kernel package to an older version before the upgrade. The last version to depend on kernel 6.5 is 1.0.1.

Kernel: Split Lock Detection Slowing Down VMs

If the host CPU supports split lock detection and a VM is performing misaligned memory accesses, the host kernel might artificially slow down the VM as of kernel version 5.19 or later.

There's now a wiki article on split lock detection with more information.

Old Ceph Crash Reports

The Ceph crash service monitors other Ceph daemons for crashes and reports them as a health warning. This service did not work properly with Proxmox VE's ceph server integration due to some internal ceph restructuring. This has been fixed with the ceph packages uploaded with the Proxmox VE 8.2 release, but this also means that it will now catch up on reporting older crashes, even if they happened a few weeks or months ago.

If you see a Ceph health warning due to services being reported as crashed, please also check the date of the crash to avoid mistaking it for a recent crash caused by the upgrade.

Kernel: intel_iommu now defaults to on

The intel_iommu parameter defaults to on in the kernel 6.8 series. Enabling IOMMU can cause problems with older hardware, or systems with not up to date BIOS, due to bugs in the BIOS.

The issue can be fixed by explicitly disabling intel_iommu on the kernel commandline (intel_iommu=off) following the reference documentation.

Kernel: Broadcom Infiniband driver issue

The bnxt_re module causes issues with some Broadcom NICs, which have their Infiniband functionality enabled. As Infiniband is not used in most deployments simply preventing the module from loading mitigates the issue. Create a file /etc/modprobe.d/bnxt-re-blacklist.conf containing:

• blacklist bnxt_re
  Afterwards make sure to update the initramfs with update-initramfs -k all -u.

Alternatively you can also install Broadcom's niccli utility and the corresponding dkms module to disable the Infiniband functionality of the NIC permamently. See the relevant post in our community forum.