View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0001587channel: kernel/el9kernel-ltpublic2026-03-07 01:332026-04-23 21:28
Reporterelrepo@mx.zzux.com Assigned Totoracat  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionopen 
Platformamd64OSCentOSOS Version9
Summary0001587: Update to 6.1.166-1.el9.elrepo.x86_64 breaks IOMMU groups
DescriptionWith the kernel 6.1.166-1.el9.elrepo.x86_64 I have less IOMMU groups than with the 6.1.164 or 6.19.6-1.el9.elrepo.x86_64. (See attached file).

Grouping together, eg
0000:07:00.0 10ec:8168 [R] Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
0000:09:00.0 1002:67ef [R] VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] (rev cf)
0000:09:00.1 1002:aae0 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
is awkward to vfio-pci passthrough.
Steps To Reproducednf update

uname -r
6.1.166-1.el9.elrepo.x86_64

grub command line
linux /boot/vmlinuz-6.1.166-1.el9.elrepo.x86_64 root=UUID=443c6091-544e-4a12-8f3a-3c8d41f22014 ro net.ifnames=0 biosdevname=0 edd=off dis_ucode_ldr mitigations=off intel_iommu=on iommu=pt selinux=0 audit=0 pci-stub.ids=1002:67ef,1002:aae0,1002:675f,1002:aa90,8086:a2c5,8086:a2a1,8086:a2f0,8086:a2a3 LANG=en_US.UTF-8
TagsNo tags attached.
Attached Files
iommu_groups_list.txt (3,511 bytes)    
	*** IOMMU group 0 ***
0000:00:02.0	8086:5912 [R]	VGA compatible controller: Intel Corporation Kaby Lake-S GT2 [HD Graphics 630] (rev 04)

	*** IOMMU group 1 ***
0000:00:00.0	8086:591f	Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05)

	*** IOMMU group 2 ***
0000:00:01.0	8086:1901	PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 05)
0000:01:00.0	1002:675f [R]	VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks LE [Radeon HD 5570/6510/7510/8510]
0000:01:00.1	1002:aa90 [R]	Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Turks HDMI Audio [Radeon HD 6500/6600 / 6700M Series]

	*** IOMMU group 3 ***
0000:00:08.0	8086:1911 [R]	System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model

	*** IOMMU group 4 ***
0000:00:14.0	8086:a2af	USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller

	*** IOMMU group 5 ***
0000:00:16.0	8086:a2ba	Communication controller: Intel Corporation 200 Series PCH CSME HECI #1

	*** IOMMU group 6 ***
0000:00:17.0	8086:a282	SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]

	*** IOMMU group 7 ***
0000:00:1b.0	8086:a2e7 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)

	*** IOMMU group 8 ***
0000:00:1b.2	8086:a2e9 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #19 (rev f0)

	*** IOMMU group 9 ***
0000:00:1b.3	8086:a2ea [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #20 (rev f0)

	*** IOMMU group 10 ***
0000:00:1b.4	8086:a2eb [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)

	*** IOMMU group 11 ***
0000:00:1c.0	8086:a290 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)

	*** IOMMU group 12 ***
0000:00:1c.2	8086:a292 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #3 (rev f0)

	*** IOMMU group 13 ***
0000:00:1c.3	8086:a293 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #4 (rev f0)

	*** IOMMU group 14 ***
0000:00:1c.4	8086:a294 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)

	*** IOMMU group 15 ***
0000:00:1d.0	8086:a298 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)

	*** IOMMU group 16 ***
0000:00:1f.0	8086:a2c5	ISA bridge: Intel Corporation 200 Series PCH LPC Controller (Z270)
0000:00:1f.2	8086:a2a1	Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
0000:00:1f.3	8086:a2f0	Audio device: Intel Corporation 200 Series PCH HD Audio
0000:00:1f.4	8086:a2a3	SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller

	*** IOMMU group 17 ***
0000:07:00.0	10ec:8168 [R]	Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

	*** IOMMU group 18 ***
0000:09:00.0	1002:67ef [R]	VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] (rev cf)
0000:09:00.1	1002:aae0	Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]

	*** IOMMU group 19 ***
0000:0a:00.0	1987:5013 [R]	Non-Volatile memory controller: Phison Electronics Corporation PS5013-E13 PCIe3 NVMe Controller (DRAM-less) (rev 01)
iommu_groups_list.txt (3,511 bytes)   
iommu_groups_list_166.txt (3,286 bytes)    
	*** IOMMU group 0 ***
0000:00:02.0	8086:5912 [R]	VGA compatible controller: Intel Corporation Kaby Lake-S GT2 [HD Graphics 630] (rev 04)

	*** IOMMU group 1 ***
0000:00:00.0	8086:591f	Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05)

	*** IOMMU group 2 ***
0000:00:01.0	8086:1901	PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 05)
0000:01:00.0	1002:675f [R]	VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks LE [Radeon HD 5570/6510/7510/8510]
0000:01:00.1	1002:aa90 [R]	Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Turks HDMI Audio [Radeon HD 6500/6600 / 6700M Series]

	*** IOMMU group 3 ***
0000:00:08.0	8086:1911 [R]	System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model

	*** IOMMU group 4 ***
0000:00:14.0	8086:a2af	USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller

	*** IOMMU group 5 ***
0000:00:16.0	8086:a2ba	Communication controller: Intel Corporation 200 Series PCH CSME HECI #1

	*** IOMMU group 6 ***
0000:00:17.0	8086:a282	SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]

	*** IOMMU group 7 ***
0000:00:1b.0	8086:a2e7 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
0000:00:1b.2	8086:a2e9 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #19 (rev f0)
0000:00:1b.3	8086:a2ea [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #20 (rev f0)
0000:00:1b.4	8086:a2eb [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)

	*** IOMMU group 8 ***
0000:00:1c.0	8086:a290 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
0000:00:1c.2	8086:a292 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #3 (rev f0)
0000:00:1c.3	8086:a293 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #4 (rev f0)
0000:00:1c.4	8086:a294 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0)
0000:07:00.0	10ec:8168 [R]	Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
0000:09:00.0	1002:67ef [R]	VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460/560D / Pro 450/455/460/555/555X/560/560X] (rev cf)
0000:09:00.1	1002:aae0	Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]

	*** IOMMU group 9 ***
0000:00:1d.0	8086:a298 [R]	PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0)
0000:0a:00.0	1987:5013 [R]	Non-Volatile memory controller: Phison Electronics Corporation PS5013-E13 PCIe3 NVMe Controller (DRAM-less) (rev 01)

	*** IOMMU group 10 ***
0000:00:1f.0	8086:a2c5	ISA bridge: Intel Corporation 200 Series PCH LPC Controller (Z270)
0000:00:1f.2	8086:a2a1	Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
0000:00:1f.3	8086:a2f0	Audio device: Intel Corporation 200 Series PCH HD Audio
0000:00:1f.4	8086:a2a3	SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
iommu_groups_list_166.txt (3,286 bytes)   

Activities

toracat

2026-03-07 12:02

administrator   ~0010820

Acknowledged.

toracat

2026-03-07 21:50

administrator   ~0010821

Last edited: 2026-03-08 14:38

 As you may know, we use the kernel source code without any modifications when building our kernels. We only play with the kernel config file. As far as I can tell, the config files are identical between 6.1.164 and 6.1.166. I see some IOMMU-related patches in the 6.1.166 kernel. Whether that has to do with the difference in the IOMMU grouping, we don't know.

Your grub command line indicates you use parameter pci-stub.ids. What if you change this to vfio-pci.ids? Does this make any difference?

elrepo@mx.zzux.com

2026-03-08 05:19

reporter   ~0010822

Changing 'pci-stub.ids' to 'vfio-pci.ids' kernel parameter, as well just missing the one, has no effect, there is still reduced number of the IOMMU groups.

toracat

2026-03-09 14:05

administrator   ~0010823

Let's wait for the release of 6.1.167. If the problem persists, I suggest you file a bug upstream at bugzilla.kernel.org.

toracat

2026-03-20 14:04

administrator   ~0010827

What seems to be the same issue was reported on bugzilla.kernel.org:

https://bugzilla.kernel.org/show_bug.cgi?id=221234
" IOMMU group regression in 6.12.75"
Reported: 2026-03-14

It was filed against kernel 6.12.75 but seems to apply to 6.1.165 as well.

toracat

2026-03-20 19:56

administrator   ~0010830

@elrepo@mx.zzux.com

A patch is available that fixes the reported IOMMU issue ( https://lore.kernel.org/stable/20260320172335.29778-1-john@kernel.doghat.io/T/#u ). I have backported it to 6.1.166 and built a kernel-lt package set as kernel-lt-6.1.166-1.tcat.el9.elrepo. It is available here:

https://elrepo.org/people/akemi/testing/el9/bug1587/

Please test if you are able and see if this fixes the IOMMU problem.

elrepo@mx.zzux.com

2026-03-21 02:30

reporter   ~0010831

It seems to be ok with the 'kernel-lt-6.1.166-1.tcat.el9.elrepo.x86_64' installed,
also the last 'kernel-ml-6.19.9-1.el9.elrepo.x86_64' is ok.

toracat

2026-03-21 13:03

administrator   ~0010832

Great to hear the patch worked. Thanks for reporting back. Hopefully this makes it into the next 6.1-kernel update.

toracat

2026-03-23 13:56

administrator   ~0010833

The current issue finally got attention from kernel developers, but not in time for the release of 6.1.167. Seems like the fix will have to wait for 6.1.168.

toracat

2026-03-27 12:37

administrator   ~0010835

@elrepo@mx.zzux.com

I applied the same patch to kernel-lt-6.1.167 and made the set available to you as kernel-lt-6.1.167-1.tcat.el9.elrepo.

https://elrepo.org/people/akemi/testing/el9/bug1587/

elrepo@mx.zzux.com

2026-03-28 10:53

reporter   ~0010837

The '6.1.167-1.tcat.el9.elrepo.x86_64' & '6.19.10-1.el9.elrepo.x86_64' are free of the subj bug.

toracat

2026-03-28 13:08

administrator   ~0010838

Thanks for the confirmation.

toracat

2026-03-31 14:15

administrator   ~0010847

The patch that corrects the issue was submitted upstream today.

Subject: [PATCH] Revert "PCI: Enable ACS after configuring IOMMU for OF platforms"
Date: Tue, 31 Mar 2026 14:44:55 +0530

This reverts commit 5d57164c0ab0ac5c99eca49c577994bfbca70a2a.

The original commit attempted to enable ACS in pci_dma_configure() prior
to IOMMU group assignment in iommu_init_device() to fix the ACS enablement
issue for OF platforms. But that assumption doesn't hold true for kernel
versions prior to v6.15, because on these older kernels,
pci_dma_configure() is called *after* iommu_init_device(). So the IOMMU
groups are already created before the ACS gets enabled. This causes the
devices that should have been split into separate groups by ACS, getting
merged into one group, thereby breaking the IOMMU isolation as reported on
the AMD machines.

So revert the offending commit to restore the IOMMU group assignment on
those affected machines. It should be noted that ACS has never really
worked on kernel versions prior to v6.15, so the revert doesn't make any
difference for OF platforms.

toracat

2026-04-11 17:47

administrator   ~0010850

The linux-6.1.168 was released today but it did not have the fix. I will build a patched version of kernel-lt-6.1.168 as before.

toracat

2026-04-11 19:27

administrator   ~0010851

@elrepo@mx.zzux.com

kernel-lt-6.1.168-1.tcat.el9.elrepo is available now:

https://elrepo.org/people/akemi/testing/el9/bug1587/

toracat

2026-04-13 14:43

administrator   ~0010852

The patch has been added to linux-6.1.169-rc1. The next kernel-lt should not need any fixing.

toracat

2026-04-18 16:31

administrator   ~0010854

The patch is in today's linux update for v5.10, v5.15, v6.1 and v6.6. Unfortunately it did not make it into v6.12 due to a 'mishap'.

@elrepo@mx.zzux.com
The kernel-lt-6.1.169-1.el9.elrepo set released today has the fix. Can you confirm this?

elrepo@mx.zzux.com

2026-04-19 00:44

reporter   ~0010855

Yes, there is no wrong IOMMU groups with the 'kernel-lt-6.1.169-1.el9.elrepo.x86_64' installed. It seems to be working ok now.

toracat

2026-04-19 12:26

administrator   ~0010856

Thank you for confirming the fix. This ticket will be marked 'resolved'.

toracat

2026-04-23 21:28

administrator   ~0010861

An additional note for completeness: the patch will be added to the 6.12-stable kernel as of 6.12.83.

Issue History

Date Modified Username Field Change
2026-03-07 01:33 elrepo@mx.zzux.com New Issue
2026-03-07 01:33 elrepo@mx.zzux.com Status new => assigned
2026-03-07 01:33 elrepo@mx.zzux.com Assigned To => toracat
2026-03-07 01:33 elrepo@mx.zzux.com File Added: iommu_groups_list.txt
2026-03-07 01:33 elrepo@mx.zzux.com File Added: iommu_groups_list_166.txt
2026-03-07 12:02 toracat Note Added: 0010820
2026-03-07 21:50 toracat Note Added: 0010821
2026-03-08 05:19 elrepo@mx.zzux.com Note Added: 0010822
2026-03-08 14:38 toracat Note Edited: 0010821
2026-03-09 14:05 toracat Note Added: 0010823
2026-03-20 14:04 toracat Note Added: 0010827
2026-03-20 19:56 toracat Note Added: 0010830
2026-03-20 19:56 toracat Status assigned => feedback
2026-03-21 02:30 elrepo@mx.zzux.com Note Added: 0010831
2026-03-21 02:30 elrepo@mx.zzux.com Status feedback => assigned
2026-03-21 13:03 toracat Note Added: 0010832
2026-03-23 13:56 toracat Note Added: 0010833
2026-03-27 12:37 toracat Note Added: 0010835
2026-03-28 10:53 elrepo@mx.zzux.com Note Added: 0010837
2026-03-28 13:08 toracat Note Added: 0010838
2026-03-31 14:15 toracat Note Added: 0010847
2026-04-11 17:47 toracat Note Added: 0010850
2026-04-11 19:27 toracat Note Added: 0010851
2026-04-13 14:43 toracat Note Added: 0010852
2026-04-18 16:31 toracat Note Added: 0010854
2026-04-18 16:31 toracat Status assigned => feedback
2026-04-19 00:44 elrepo@mx.zzux.com Note Added: 0010855
2026-04-19 00:44 elrepo@mx.zzux.com Status feedback => assigned
2026-04-19 12:26 toracat Note Added: 0010856
2026-04-19 12:26 toracat Status assigned => resolved
2026-04-23 21:28 toracat Note Added: 0010861