NVidia Grace L3 cache size seems wrong #689
Comments
|
Hello Jim. I would guess it's a 1024 vs 1000 (114x1024 is close to 117x1000), somebody didn't read the specs carefully when filling hardware tables. That's what's exposed in the firmware and then by Linux, the bug wouldn't be on our side. It's somehow like people claiming they have 192GB of RAM while the BIOS actually only gives 185GB to the OS, we usually just ignore the issue. What bothers me more is how this L3 is shared. The wording in the doc makes me wonder whether it's actually 234 per dual-package superchip instead of 117 per package. Somebody once reported that their experiment results indeed point to shared per dual-package. B we couldn't get a precise answer by NVIDIA. However, this would have to be fixed in their firmware as well (or worked around in some kernel driver). |
|
Brice,
Thanks for the rapid reply.
Hello Jim. I would d guess it's a 1024 vs 1000 (1141024 is close to 1171000). That's what's exposed in the firmware and then by Linux, the bug wouldn't be on our side. It's somehow like people claiming they have 192GB of RAM while the BIOS actually only gives 185GB to the OS, we usually just ignore the issue.
I did consider that this could be a MB vs MiB difference, but since the ratio is 1.024*1.024 ~= 1.049, 117Mb == 111.6 MiB, so it did't seem like it.
The 117MiB also seemed more likely since that can at least be split semi-reasonably by 72 (giving 1.625 (== 13/8) MiB/cache-slice), whereas 114 (2*3*19) doesn’t manage that, ending up with 1.633333… (== 19/12MiB). Of course, it may be that the number of cache-slices enabled is not the same as the number of cores enabled (since there are 80 available cache slices, and 76 cores to allow for manufacturing defects, so you might be able to enable a few more cache-slices than cores).
NV also seem to be fairly consistently using MiB (even if they label them as MB) on the diagram, at least for the other caches.
I can easily believe that the BIOS is wrong!
I may be able to run a micro-benchmark to see what shows up, though I’m not sure I have exactly that one written :-) (I’ll let you know if I do find anything out).
What bothers me more is how this L3 is shared. The wording in the doc makes me wonder whether it's actually 234 per dual-package superchip instead of 117 per package. Somebody once reported that their experiment results indeed point to shared per dual-package. B we couldn't get a precise answer by NVIDIA. However, this would have to be fixed in their firmware as well (or worked around in some kernel driver).
I’d expect that you have the topology right. Given their diagram, the L3$ is clearly on die, so I’d expect not to be going off chip, which woudl significantly increase the latency (I see a ~5x slowdown to go to an L1 or L2 cache on another die).
So, I agree there’s probably nothing you can do at the hwloc level. It seemed worth mentioning it, though!
… On 11 Sep 2024, at 13:09, Brice Goglin ***@***.***> wrote:
Hello Jim. I would d guess it's a 1024 vs 1000 (1141024 is close to 1171000). That's what's exposed in the firmware and then by Linux, the bug wouldn't be on our side. It's somehow like people claiming they have 192GB of RAM while the BIOS actually only gives 185GB to the OS, we usually just ignore the issue.
What bothers me more is how this L3 is shared. The wording in the doc makes me wonder whether it's actually 234 per dual-package superchip instead of 117 per package. Somebody once reported that their experiment results indeed point to shared per dual-package. B we couldn't get a precise answer by NVIDIA. However, this would have to be fixed in their firmware as well (or worked around in some kernel driver).
—
Reply to this email directly, view it on GitHub <#689 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALY7TVFO2TE4LJDV64Q6SSDZWAXJFAVCNFSM6AAAAABOAWHLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBTGQ4TGNJTGU>.
You are receiving this because you authored the thread.
-- Jim
James Cownie ***@***.***>
Mob: +44 780 637 7146
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of hwloc are you using?
2.10.0
lstopo says
% lstopo --version
lstopo 2.9.0
Which operating system and hardware are you running on?
% uname -a
Linux nid001040 5.14.21-150500.55.31_13.0.53-cray_shasta_c_64k #1 SMP Mon Dec 4 22:56:47 UTC 2023 (03d3f83) aarch64 aarch64 aarch64 GNU/Linux
% lstopo -
Machine (477GB total)
Package L#0
NUMANode L#0 (P#0 117GB)
L3 L#0 (114MB)
L2 L#0 (1024KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + PU L#0 (P#0)
L2 L#1 (1024KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + PU L#1 (P#1)
L2 L#2 (1024KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + PU L#2 (P#2)
L2 L#3 (1024KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + PU L#3 (P#3)
L2 L#4 (1024KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + PU L#4 (P#4)
L2 L#5 (1024KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + PU L#5 (P#5)
L2 L#6 (1024KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + PU L#6 (P#6)
L2 L#7 (1024KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + PU L#7 (P#7)
L2 L#8 (1024KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + PU L#8 (P#8)
L2 L#9 (1024KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + PU L#9 (P#9)
L2 L#10 (1024KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core L#10 + PU L#10 (P#10)
L2 L#11 (1024KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core L#11 + PU L#11 (P#11)
L2 L#12 (1024KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core L#12 + PU L#12 (P#12)
L2 L#13 (1024KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core L#13 + PU L#13 (P#13)
L2 L#14 (1024KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core L#14 + PU L#14 (P#14)
L2 L#15 (1024KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core L#15 + PU L#15 (P#15)
L2 L#16 (1024KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core L#16 + PU L#16 (P#16)
L2 L#17 (1024KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core L#17 + PU L#17 (P#17)
L2 L#18 (1024KB) + L1d L#18 (64KB) + L1i L#18 (64KB) + Core L#18 + PU L#18 (P#18)
L2 L#19 (1024KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core L#19 + PU L#19 (P#19)
L2 L#20 (1024KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core L#20 + PU L#20 (P#20)
L2 L#21 (1024KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core L#21 + PU L#21 (P#21)
L2 L#22 (1024KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core L#22 + PU L#22 (P#22)
L2 L#23 (1024KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core L#23 + PU L#23 (P#23)
L2 L#24 (1024KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core L#24 + PU L#24 (P#24)
L2 L#25 (1024KB) + L1d L#25 (64KB) + L1i L#25 (64KB) + Core L#25 + PU L#25 (P#25)
L2 L#26 (1024KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core L#26 + PU L#26 (P#26)
L2 L#27 (1024KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core L#27 + PU L#27 (P#27)
L2 L#28 (1024KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core L#28 + PU L#28 (P#28)
L2 L#29 (1024KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core L#29 + PU L#29 (P#29)
L2 L#30 (1024KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core L#30 + PU L#30 (P#30)
L2 L#31 (1024KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core L#31 + PU L#31 (P#31)
L2 L#32 (1024KB) + L1d L#32 (64KB) + L1i L#32 (64KB) + Core L#32 + PU L#32 (P#32)
L2 L#33 (1024KB) + L1d L#33 (64KB) + L1i L#33 (64KB) + Core L#33 + PU L#33 (P#33)
L2 L#34 (1024KB) + L1d L#34 (64KB) + L1i L#34 (64KB) + Core L#34 + PU L#34 (P#34)
L2 L#35 (1024KB) + L1d L#35 (64KB) + L1i L#35 (64KB) + Core L#35 + PU L#35 (P#35)
L2 L#36 (1024KB) + L1d L#36 (64KB) + L1i L#36 (64KB) + Core L#36 + PU L#36 (P#36)
L2 L#37 (1024KB) + L1d L#37 (64KB) + L1i L#37 (64KB) + Core L#37 + PU L#37 (P#37)
L2 L#38 (1024KB) + L1d L#38 (64KB) + L1i L#38 (64KB) + Core L#38 + PU L#38 (P#38)
L2 L#39 (1024KB) + L1d L#39 (64KB) + L1i L#39 (64KB) + Core L#39 + PU L#39 (P#39)
L2 L#40 (1024KB) + L1d L#40 (64KB) + L1i L#40 (64KB) + Core L#40 + PU L#40 (P#40)
L2 L#41 (1024KB) + L1d L#41 (64KB) + L1i L#41 (64KB) + Core L#41 + PU L#41 (P#41)
L2 L#42 (1024KB) + L1d L#42 (64KB) + L1i L#42 (64KB) + Core L#42 + PU L#42 (P#42)
L2 L#43 (1024KB) + L1d L#43 (64KB) + L1i L#43 (64KB) + Core L#43 + PU L#43 (P#43)
L2 L#44 (1024KB) + L1d L#44 (64KB) + L1i L#44 (64KB) + Core L#44 + PU L#44 (P#44)
L2 L#45 (1024KB) + L1d L#45 (64KB) + L1i L#45 (64KB) + Core L#45 + PU L#45 (P#45)
L2 L#46 (1024KB) + L1d L#46 (64KB) + L1i L#46 (64KB) + Core L#46 + PU L#46 (P#46)
L2 L#47 (1024KB) + L1d L#47 (64KB) + L1i L#47 (64KB) + Core L#47 + PU L#47 (P#47)
L2 L#48 (1024KB) + L1d L#48 (64KB) + L1i L#48 (64KB) + Core L#48 + PU L#48 (P#48)
L2 L#49 (1024KB) + L1d L#49 (64KB) + L1i L#49 (64KB) + Core L#49 + PU L#49 (P#49)
L2 L#50 (1024KB) + L1d L#50 (64KB) + L1i L#50 (64KB) + Core L#50 + PU L#50 (P#50)
L2 L#51 (1024KB) + L1d L#51 (64KB) + L1i L#51 (64KB) + Core L#51 + PU L#51 (P#51)
L2 L#52 (1024KB) + L1d L#52 (64KB) + L1i L#52 (64KB) + Core L#52 + PU L#52 (P#52)
L2 L#53 (1024KB) + L1d L#53 (64KB) + L1i L#53 (64KB) + Core L#53 + PU L#53 (P#53)
L2 L#54 (1024KB) + L1d L#54 (64KB) + L1i L#54 (64KB) + Core L#54 + PU L#54 (P#54)
L2 L#55 (1024KB) + L1d L#55 (64KB) + L1i L#55 (64KB) + Core L#55 + PU L#55 (P#55)
L2 L#56 (1024KB) + L1d L#56 (64KB) + L1i L#56 (64KB) + Core L#56 + PU L#56 (P#56)
L2 L#57 (1024KB) + L1d L#57 (64KB) + L1i L#57 (64KB) + Core L#57 + PU L#57 (P#57)
L2 L#58 (1024KB) + L1d L#58 (64KB) + L1i L#58 (64KB) + Core L#58 + PU L#58 (P#58)
L2 L#59 (1024KB) + L1d L#59 (64KB) + L1i L#59 (64KB) + Core L#59 + PU L#59 (P#59)
L2 L#60 (1024KB) + L1d L#60 (64KB) + L1i L#60 (64KB) + Core L#60 + PU L#60 (P#60)
L2 L#61 (1024KB) + L1d L#61 (64KB) + L1i L#61 (64KB) + Core L#61 + PU L#61 (P#61)
L2 L#62 (1024KB) + L1d L#62 (64KB) + L1i L#62 (64KB) + Core L#62 + PU L#62 (P#62)
L2 L#63 (1024KB) + L1d L#63 (64KB) + L1i L#63 (64KB) + Core L#63 + PU L#63 (P#63)
L2 L#64 (1024KB) + L1d L#64 (64KB) + L1i L#64 (64KB) + Core L#64 + PU L#64 (P#64)
L2 L#65 (1024KB) + L1d L#65 (64KB) + L1i L#65 (64KB) + Core L#65 + PU L#65 (P#65)
L2 L#66 (1024KB) + L1d L#66 (64KB) + L1i L#66 (64KB) + Core L#66 + PU L#66 (P#66)
L2 L#67 (1024KB) + L1d L#67 (64KB) + L1i L#67 (64KB) + Core L#67 + PU L#67 (P#67)
L2 L#68 (1024KB) + L1d L#68 (64KB) + L1i L#68 (64KB) + Core L#68 + PU L#68 (P#68)
L2 L#69 (1024KB) + L1d L#69 (64KB) + L1i L#69 (64KB) + Core L#69 + PU L#69 (P#69)
L2 L#70 (1024KB) + L1d L#70 (64KB) + L1i L#70 (64KB) + Core L#70 + PU L#70 (P#70)
L2 L#71 (1024KB) + L1d L#71 (64KB) + L1i L#71 (64KB) + Core L#71 + PU L#71 (P#71)
HostBridge
PCIBridge
PCI 0002:01:00.0 (Ethernet)
Net "hsn0"
HostBridge
PCIBridge
PCI 0005:01:00.0 (Ethernet)
Net "nmn0"
HostBridge
PCIBridge
PCI 0007:01:00.0 (NVMExp)
Block(Disk) "nvme0n1"
HostBridge
PCIBridge
PCI 0009:01:00.0 (3D)
Package L#1
NUMANode L#1 (P#1 120GB)
L3 L#1 (114MB)
L2 L#72 (1024KB) + L1d L#72 (64KB) + L1i L#72 (64KB) + Core L#72 + PU L#72 (P#72)
L2 L#73 (1024KB) + L1d L#73 (64KB) + L1i L#73 (64KB) + Core L#73 + PU L#73 (P#73)
L2 L#74 (1024KB) + L1d L#74 (64KB) + L1i L#74 (64KB) + Core L#74 + PU L#74 (P#74)
L2 L#75 (1024KB) + L1d L#75 (64KB) + L1i L#75 (64KB) + Core L#75 + PU L#75 (P#75)
L2 L#76 (1024KB) + L1d L#76 (64KB) + L1i L#76 (64KB) + Core L#76 + PU L#76 (P#76)
L2 L#77 (1024KB) + L1d L#77 (64KB) + L1i L#77 (64KB) + Core L#77 + PU L#77 (P#77)
L2 L#78 (1024KB) + L1d L#78 (64KB) + L1i L#78 (64KB) + Core L#78 + PU L#78 (P#78)
L2 L#79 (1024KB) + L1d L#79 (64KB) + L1i L#79 (64KB) + Core L#79 + PU L#79 (P#79)
L2 L#80 (1024KB) + L1d L#80 (64KB) + L1i L#80 (64KB) + Core L#80 + PU L#80 (P#80)
L2 L#81 (1024KB) + L1d L#81 (64KB) + L1i L#81 (64KB) + Core L#81 + PU L#81 (P#81)
L2 L#82 (1024KB) + L1d L#82 (64KB) + L1i L#82 (64KB) + Core L#82 + PU L#82 (P#82)
L2 L#83 (1024KB) + L1d L#83 (64KB) + L1i L#83 (64KB) + Core L#83 + PU L#83 (P#83)
L2 L#84 (1024KB) + L1d L#84 (64KB) + L1i L#84 (64KB) + Core L#84 + PU L#84 (P#84)
L2 L#85 (1024KB) + L1d L#85 (64KB) + L1i L#85 (64KB) + Core L#85 + PU L#85 (P#85)
L2 L#86 (1024KB) + L1d L#86 (64KB) + L1i L#86 (64KB) + Core L#86 + PU L#86 (P#86)
L2 L#87 (1024KB) + L1d L#87 (64KB) + L1i L#87 (64KB) + Core L#87 + PU L#87 (P#87)
L2 L#88 (1024KB) + L1d L#88 (64KB) + L1i L#88 (64KB) + Core L#88 + PU L#88 (P#88)
L2 L#89 (1024KB) + L1d L#89 (64KB) + L1i L#89 (64KB) + Core L#89 + PU L#89 (P#89)
L2 L#90 (1024KB) + L1d L#90 (64KB) + L1i L#90 (64KB) + Core L#90 + PU L#90 (P#90)
L2 L#91 (1024KB) + L1d L#91 (64KB) + L1i L#91 (64KB) + Core L#91 + PU L#91 (P#91)
L2 L#92 (1024KB) + L1d L#92 (64KB) + L1i L#92 (64KB) + Core L#92 + PU L#92 (P#92)
L2 L#93 (1024KB) + L1d L#93 (64KB) + L1i L#93 (64KB) + Core L#93 + PU L#93 (P#93)
L2 L#94 (1024KB) + L1d L#94 (64KB) + L1i L#94 (64KB) + Core L#94 + PU L#94 (P#94)
L2 L#95 (1024KB) + L1d L#95 (64KB) + L1i L#95 (64KB) + Core L#95 + PU L#95 (P#95)
L2 L#96 (1024KB) + L1d L#96 (64KB) + L1i L#96 (64KB) + Core L#96 + PU L#96 (P#96)
L2 L#97 (1024KB) + L1d L#97 (64KB) + L1i L#97 (64KB) + Core L#97 + PU L#97 (P#97)
L2 L#98 (1024KB) + L1d L#98 (64KB) + L1i L#98 (64KB) + Core L#98 + PU L#98 (P#98)
L2 L#99 (1024KB) + L1d L#99 (64KB) + L1i L#99 (64KB) + Core L#99 + PU L#99 (P#99)
L2 L#100 (1024KB) + L1d L#100 (64KB) + L1i L#100 (64KB) + Core L#100 + PU L#100 (P#100)
L2 L#101 (1024KB) + L1d L#101 (64KB) + L1i L#101 (64KB) + Core L#101 + PU L#101 (P#101)
L2 L#102 (1024KB) + L1d L#102 (64KB) + L1i L#102 (64KB) + Core L#102 + PU L#102 (P#102)
L2 L#103 (1024KB) + L1d L#103 (64KB) + L1i L#103 (64KB) + Core L#103 + PU L#103 (P#103)
L2 L#104 (1024KB) + L1d L#104 (64KB) + L1i L#104 (64KB) + Core L#104 + PU L#104 (P#104)
L2 L#105 (1024KB) + L1d L#105 (64KB) + L1i L#105 (64KB) + Core L#105 + PU L#105 (P#105)
L2 L#106 (1024KB) + L1d L#106 (64KB) + L1i L#106 (64KB) + Core L#106 + PU L#106 (P#106)
L2 L#107 (1024KB) + L1d L#107 (64KB) + L1i L#107 (64KB) + Core L#107 + PU L#107 (P#107)
L2 L#108 (1024KB) + L1d L#108 (64KB) + L1i L#108 (64KB) + Core L#108 + PU L#108 (P#108)
L2 L#109 (1024KB) + L1d L#109 (64KB) + L1i L#109 (64KB) + Core L#109 + PU L#109 (P#109)
L2 L#110 (1024KB) + L1d L#110 (64KB) + L1i L#110 (64KB) + Core L#110 + PU L#110 (P#110)
L2 L#111 (1024KB) + L1d L#111 (64KB) + L1i L#111 (64KB) + Core L#111 + PU L#111 (P#111)
L2 L#112 (1024KB) + L1d L#112 (64KB) + L1i L#112 (64KB) + Core L#112 + PU L#112 (P#112)
L2 L#113 (1024KB) + L1d L#113 (64KB) + L1i L#113 (64KB) + Core L#113 + PU L#113 (P#113)
L2 L#114 (1024KB) + L1d L#114 (64KB) + L1i L#114 (64KB) + Core L#114 + PU L#114 (P#114)
L2 L#115 (1024KB) + L1d L#115 (64KB) + L1i L#115 (64KB) + Core L#115 + PU L#115 (P#115)
L2 L#116 (1024KB) + L1d L#116 (64KB) + L1i L#116 (64KB) + Core L#116 + PU L#116 (P#116)
L2 L#117 (1024KB) + L1d L#117 (64KB) + L1i L#117 (64KB) + Core L#117 + PU L#117 (P#117)
L2 L#118 (1024KB) + L1d L#118 (64KB) + L1i L#118 (64KB) + Core L#118 + PU L#118 (P#118)
L2 L#119 (1024KB) + L1d L#119 (64KB) + L1i L#119 (64KB) + Core L#119 + PU L#119 (P#119)
L2 L#120 (1024KB) + L1d L#120 (64KB) + L1i L#120 (64KB) + Core L#120 + PU L#120 (P#120)
L2 L#121 (1024KB) + L1d L#121 (64KB) + L1i L#121 (64KB) + Core L#121 + PU L#121 (P#121)
L2 L#122 (1024KB) + L1d L#122 (64KB) + L1i L#122 (64KB) + Core L#122 + PU L#122 (P#122)
L2 L#123 (1024KB) + L1d L#123 (64KB) + L1i L#123 (64KB) + Core L#123 + PU L#123 (P#123)
L2 L#124 (1024KB) + L1d L#124 (64KB) + L1i L#124 (64KB) + Core L#124 + PU L#124 (P#124)
L2 L#125 (1024KB) + L1d L#125 (64KB) + L1i L#125 (64KB) + Core L#125 + PU L#125 (P#125)
L2 L#126 (1024KB) + L1d L#126 (64KB) + L1i L#126 (64KB) + Core L#126 + PU L#126 (P#126)
L2 L#127 (1024KB) + L1d L#127 (64KB) + L1i L#127 (64KB) + Core L#127 + PU L#127 (P#127)
L2 L#128 (1024KB) + L1d L#128 (64KB) + L1i L#128 (64KB) + Core L#128 + PU L#128 (P#128)
L2 L#129 (1024KB) + L1d L#129 (64KB) + L1i L#129 (64KB) + Core L#129 + PU L#129 (P#129)
L2 L#130 (1024KB) + L1d L#130 (64KB) + L1i L#130 (64KB) + Core L#130 + PU L#130 (P#130)
L2 L#131 (1024KB) + L1d L#131 (64KB) + L1i L#131 (64KB) + Core L#131 + PU L#131 (P#131)
L2 L#132 (1024KB) + L1d L#132 (64KB) + L1i L#132 (64KB) + Core L#132 + PU L#132 (P#132)
L2 L#133 (1024KB) + L1d L#133 (64KB) + L1i L#133 (64KB) + Core L#133 + PU L#133 (P#133)
L2 L#134 (1024KB) + L1d L#134 (64KB) + L1i L#134 (64KB) + Core L#134 + PU L#134 (P#134)
L2 L#135 (1024KB) + L1d L#135 (64KB) + L1i L#135 (64KB) + Core L#135 + PU L#135 (P#135)
L2 L#136 (1024KB) + L1d L#136 (64KB) + L1i L#136 (64KB) + Core L#136 + PU L#136 (P#136)
L2 L#137 (1024KB) + L1d L#137 (64KB) + L1i L#137 (64KB) + Core L#137 + PU L#137 (P#137)
L2 L#138 (1024KB) + L1d L#138 (64KB) + L1i L#138 (64KB) + Core L#138 + PU L#138 (P#138)
L2 L#139 (1024KB) + L1d L#139 (64KB) + L1i L#139 (64KB) + Core L#139 + PU L#139 (P#139)
L2 L#140 (1024KB) + L1d L#140 (64KB) + L1i L#140 (64KB) + Core L#140 + PU L#140 (P#140)
L2 L#141 (1024KB) + L1d L#141 (64KB) + L1i L#141 (64KB) + Core L#141 + PU L#141 (P#141)
L2 L#142 (1024KB) + L1d L#142 (64KB) + L1i L#142 (64KB) + Core L#142 + PU L#142 (P#142)
L2 L#143 (1024KB) + L1d L#143 (64KB) + L1i L#143 (64KB) + Core L#143 + PU L#143 (P#143)
HostBridge
PCIBridge
PCI 0010:01:00.0 (Ethernet)
Net "hsn2"
HostBridge
PCIBridge
PCI 0019:01:00.0 (3D)
Package L#2
NUMANode L#2 (P#2 120GB)
L3 L#2 (114MB)
L2 L#144 (1024KB) + L1d L#144 (64KB) + L1i L#144 (64KB) + Core L#144 + PU L#144 (P#144)
L2 L#145 (1024KB) + L1d L#145 (64KB) + L1i L#145 (64KB) + Core L#145 + PU L#145 (P#145)
L2 L#146 (1024KB) + L1d L#146 (64KB) + L1i L#146 (64KB) + Core L#146 + PU L#146 (P#146)
L2 L#147 (1024KB) + L1d L#147 (64KB) + L1i L#147 (64KB) + Core L#147 + PU L#147 (P#147)
L2 L#148 (1024KB) + L1d L#148 (64KB) + L1i L#148 (64KB) + Core L#148 + PU L#148 (P#148)
L2 L#149 (1024KB) + L1d L#149 (64KB) + L1i L#149 (64KB) + Core L#149 + PU L#149 (P#149)
L2 L#150 (1024KB) + L1d L#150 (64KB) + L1i L#150 (64KB) + Core L#150 + PU L#150 (P#150)
L2 L#151 (1024KB) + L1d L#151 (64KB) + L1i L#151 (64KB) + Core L#151 + PU L#151 (P#151)
L2 L#152 (1024KB) + L1d L#152 (64KB) + L1i L#152 (64KB) + Core L#152 + PU L#152 (P#152)
L2 L#153 (1024KB) + L1d L#153 (64KB) + L1i L#153 (64KB) + Core L#153 + PU L#153 (P#153)
L2 L#154 (1024KB) + L1d L#154 (64KB) + L1i L#154 (64KB) + Core L#154 + PU L#154 (P#154)
L2 L#155 (1024KB) + L1d L#155 (64KB) + L1i L#155 (64KB) + Core L#155 + PU L#155 (P#155)
L2 L#156 (1024KB) + L1d L#156 (64KB) + L1i L#156 (64KB) + Core L#156 + PU L#156 (P#156)
L2 L#157 (1024KB) + L1d L#157 (64KB) + L1i L#157 (64KB) + Core L#157 + PU L#157 (P#157)
L2 L#158 (1024KB) + L1d L#158 (64KB) + L1i L#158 (64KB) + Core L#158 + PU L#158 (P#158)
L2 L#159 (1024KB) + L1d L#159 (64KB) + L1i L#159 (64KB) + Core L#159 + PU L#159 (P#159)
L2 L#160 (1024KB) + L1d L#160 (64KB) + L1i L#160 (64KB) + Core L#160 + PU L#160 (P#160)
L2 L#161 (1024KB) + L1d L#161 (64KB) + L1i L#161 (64KB) + Core L#161 + PU L#161 (P#161)
L2 L#162 (1024KB) + L1d L#162 (64KB) + L1i L#162 (64KB) + Core L#162 + PU L#162 (P#162)
L2 L#163 (1024KB) + L1d L#163 (64KB) + L1i L#163 (64KB) + Core L#163 + PU L#163 (P#163)
L2 L#164 (1024KB) + L1d L#164 (64KB) + L1i L#164 (64KB) + Core L#164 + PU L#164 (P#164)
L2 L#165 (1024KB) + L1d L#165 (64KB) + L1i L#165 (64KB) + Core L#165 + PU L#165 (P#165)
L2 L#166 (1024KB) + L1d L#166 (64KB) + L1i L#166 (64KB) + Core L#166 + PU L#166 (P#166)
L2 L#167 (1024KB) + L1d L#167 (64KB) + L1i L#167 (64KB) + Core L#167 + PU L#167 (P#167)
L2 L#168 (1024KB) + L1d L#168 (64KB) + L1i L#168 (64KB) + Core L#168 + PU L#168 (P#168)
L2 L#169 (1024KB) + L1d L#169 (64KB) + L1i L#169 (64KB) + Core L#169 + PU L#169 (P#169)
L2 L#170 (1024KB) + L1d L#170 (64KB) + L1i L#170 (64KB) + Core L#170 + PU L#170 (P#170)
L2 L#171 (1024KB) + L1d L#171 (64KB) + L1i L#171 (64KB) + Core L#171 + PU L#171 (P#171)
L2 L#172 (1024KB) + L1d L#172 (64KB) + L1i L#172 (64KB) + Core L#172 + PU L#172 (P#172)
L2 L#173 (1024KB) + L1d L#173 (64KB) + L1i L#173 (64KB) + Core L#173 + PU L#173 (P#173)
L2 L#174 (1024KB) + L1d L#174 (64KB) + L1i L#174 (64KB) + Core L#174 + PU L#174 (P#174)
L2 L#175 (1024KB) + L1d L#175 (64KB) + L1i L#175 (64KB) + Core L#175 + PU L#175 (P#175)
L2 L#176 (1024KB) + L1d L#176 (64KB) + L1i L#176 (64KB) + Core L#176 + PU L#176 (P#176)
L2 L#177 (1024KB) + L1d L#177 (64KB) + L1i L#177 (64KB) + Core L#177 + PU L#177 (P#177)
L2 L#178 (1024KB) + L1d L#178 (64KB) + L1i L#178 (64KB) + Core L#178 + PU L#178 (P#178)
L2 L#179 (1024KB) + L1d L#179 (64KB) + L1i L#179 (64KB) + Core L#179 + PU L#179 (P#179)
L2 L#180 (1024KB) + L1d L#180 (64KB) + L1i L#180 (64KB) + Core L#180 + PU L#180 (P#180)
L2 L#181 (1024KB) + L1d L#181 (64KB) + L1i L#181 (64KB) + Core L#181 + PU L#181 (P#181)
L2 L#182 (1024KB) + L1d L#182 (64KB) + L1i L#182 (64KB) + Core L#182 + PU L#182 (P#182)
L2 L#183 (1024KB) + L1d L#183 (64KB) + L1i L#183 (64KB) + Core L#183 + PU L#183 (P#183)
L2 L#184 (1024KB) + L1d L#184 (64KB) + L1i L#184 (64KB) + Core L#184 + PU L#184 (P#184)
L2 L#185 (1024KB) + L1d L#185 (64KB) + L1i L#185 (64KB) + Core L#185 + PU L#185 (P#185)
L2 L#186 (1024KB) + L1d L#186 (64KB) + L1i L#186 (64KB) + Core L#186 + PU L#186 (P#186)
L2 L#187 (1024KB) + L1d L#187 (64KB) + L1i L#187 (64KB) + Core L#187 + PU L#187 (P#187)
L2 L#188 (1024KB) + L1d L#188 (64KB) + L1i L#188 (64KB) + Core L#188 + PU L#188 (P#188)
L2 L#189 (1024KB) + L1d L#189 (64KB) + L1i L#189 (64KB) + Core L#189 + PU L#189 (P#189)
L2 L#190 (1024KB) + L1d L#190 (64KB) + L1i L#190 (64KB) + Core L#190 + PU L#190 (P#190)
L2 L#191 (1024KB) + L1d L#191 (64KB) + L1i L#191 (64KB) + Core L#191 + PU L#191 (P#191)
L2 L#192 (1024KB) + L1d L#192 (64KB) + L1i L#192 (64KB) + Core L#192 + PU L#192 (P#192)
L2 L#193 (1024KB) + L1d L#193 (64KB) + L1i L#193 (64KB) + Core L#193 + PU L#193 (P#193)
L2 L#194 (1024KB) + L1d L#194 (64KB) + L1i L#194 (64KB) + Core L#194 + PU L#194 (P#194)
L2 L#195 (1024KB) + L1d L#195 (64KB) + L1i L#195 (64KB) + Core L#195 + PU L#195 (P#195)
L2 L#196 (1024KB) + L1d L#196 (64KB) + L1i L#196 (64KB) + Core L#196 + PU L#196 (P#196)
L2 L#197 (1024KB) + L1d L#197 (64KB) + L1i L#197 (64KB) + Core L#197 + PU L#197 (P#197)
L2 L#198 (1024KB) + L1d L#198 (64KB) + L1i L#198 (64KB) + Core L#198 + PU L#198 (P#198)
L2 L#199 (1024KB) + L1d L#199 (64KB) + L1i L#199 (64KB) + Core L#199 + PU L#199 (P#199)
L2 L#200 (1024KB) + L1d L#200 (64KB) + L1i L#200 (64KB) + Core L#200 + PU L#200 (P#200)
L2 L#201 (1024KB) + L1d L#201 (64KB) + L1i L#201 (64KB) + Core L#201 + PU L#201 (P#201)
L2 L#202 (1024KB) + L1d L#202 (64KB) + L1i L#202 (64KB) + Core L#202 + PU L#202 (P#202)
L2 L#203 (1024KB) + L1d L#203 (64KB) + L1i L#203 (64KB) + Core L#203 + PU L#203 (P#203)
L2 L#204 (1024KB) + L1d L#204 (64KB) + L1i L#204 (64KB) + Core L#204 + PU L#204 (P#204)
L2 L#205 (1024KB) + L1d L#205 (64KB) + L1i L#205 (64KB) + Core L#205 + PU L#205 (P#205)
L2 L#206 (1024KB) + L1d L#206 (64KB) + L1i L#206 (64KB) + Core L#206 + PU L#206 (P#206)
L2 L#207 (1024KB) + L1d L#207 (64KB) + L1i L#207 (64KB) + Core L#207 + PU L#207 (P#207)
L2 L#208 (1024KB) + L1d L#208 (64KB) + L1i L#208 (64KB) + Core L#208 + PU L#208 (P#208)
L2 L#209 (1024KB) + L1d L#209 (64KB) + L1i L#209 (64KB) + Core L#209 + PU L#209 (P#209)
L2 L#210 (1024KB) + L1d L#210 (64KB) + L1i L#210 (64KB) + Core L#210 + PU L#210 (P#210)
L2 L#211 (1024KB) + L1d L#211 (64KB) + L1i L#211 (64KB) + Core L#211 + PU L#211 (P#211)
L2 L#212 (1024KB) + L1d L#212 (64KB) + L1i L#212 (64KB) + Core L#212 + PU L#212 (P#212)
L2 L#213 (1024KB) + L1d L#213 (64KB) + L1i L#213 (64KB) + Core L#213 + PU L#213 (P#213)
L2 L#214 (1024KB) + L1d L#214 (64KB) + L1i L#214 (64KB) + Core L#214 + PU L#214 (P#214)
L2 L#215 (1024KB) + L1d L#215 (64KB) + L1i L#215 (64KB) + Core L#215 + PU L#215 (P#215)
HostBridge
PCIBridge
PCI 0020:01:00.0 (Ethernet)
Net "hsn3"
HostBridge
PCIBridge
PCI 0029:01:00.0 (3D)
Package L#3
NUMANode L#3 (P#3 120GB)
L3 L#3 (114MB)
L2 L#216 (1024KB) + L1d L#216 (64KB) + L1i L#216 (64KB) + Core L#216 + PU L#216 (P#216)
L2 L#217 (1024KB) + L1d L#217 (64KB) + L1i L#217 (64KB) + Core L#217 + PU L#217 (P#217)
L2 L#218 (1024KB) + L1d L#218 (64KB) + L1i L#218 (64KB) + Core L#218 + PU L#218 (P#218)
L2 L#219 (1024KB) + L1d L#219 (64KB) + L1i L#219 (64KB) + Core L#219 + PU L#219 (P#219)
L2 L#220 (1024KB) + L1d L#220 (64KB) + L1i L#220 (64KB) + Core L#220 + PU L#220 (P#220)
L2 L#221 (1024KB) + L1d L#221 (64KB) + L1i L#221 (64KB) + Core L#221 + PU L#221 (P#221)
L2 L#222 (1024KB) + L1d L#222 (64KB) + L1i L#222 (64KB) + Core L#222 + PU L#222 (P#222)
L2 L#223 (1024KB) + L1d L#223 (64KB) + L1i L#223 (64KB) + Core L#223 + PU L#223 (P#223)
L2 L#224 (1024KB) + L1d L#224 (64KB) + L1i L#224 (64KB) + Core L#224 + PU L#224 (P#224)
L2 L#225 (1024KB) + L1d L#225 (64KB) + L1i L#225 (64KB) + Core L#225 + PU L#225 (P#225)
L2 L#226 (1024KB) + L1d L#226 (64KB) + L1i L#226 (64KB) + Core L#226 + PU L#226 (P#226)
L2 L#227 (1024KB) + L1d L#227 (64KB) + L1i L#227 (64KB) + Core L#227 + PU L#227 (P#227)
L2 L#228 (1024KB) + L1d L#228 (64KB) + L1i L#228 (64KB) + Core L#228 + PU L#228 (P#228)
L2 L#229 (1024KB) + L1d L#229 (64KB) + L1i L#229 (64KB) + Core L#229 + PU L#229 (P#229)
L2 L#230 (1024KB) + L1d L#230 (64KB) + L1i L#230 (64KB) + Core L#230 + PU L#230 (P#230)
L2 L#231 (1024KB) + L1d L#231 (64KB) + L1i L#231 (64KB) + Core L#231 + PU L#231 (P#231)
L2 L#232 (1024KB) + L1d L#232 (64KB) + L1i L#232 (64KB) + Core L#232 + PU L#232 (P#232)
L2 L#233 (1024KB) + L1d L#233 (64KB) + L1i L#233 (64KB) + Core L#233 + PU L#233 (P#233)
L2 L#234 (1024KB) + L1d L#234 (64KB) + L1i L#234 (64KB) + Core L#234 + PU L#234 (P#234)
L2 L#235 (1024KB) + L1d L#235 (64KB) + L1i L#235 (64KB) + Core L#235 + PU L#235 (P#235)
L2 L#236 (1024KB) + L1d L#236 (64KB) + L1i L#236 (64KB) + Core L#236 + PU L#236 (P#236)
L2 L#237 (1024KB) + L1d L#237 (64KB) + L1i L#237 (64KB) + Core L#237 + PU L#237 (P#237)
L2 L#238 (1024KB) + L1d L#238 (64KB) + L1i L#238 (64KB) + Core L#238 + PU L#238 (P#238)
L2 L#239 (1024KB) + L1d L#239 (64KB) + L1i L#239 (64KB) + Core L#239 + PU L#239 (P#239)
L2 L#240 (1024KB) + L1d L#240 (64KB) + L1i L#240 (64KB) + Core L#240 + PU L#240 (P#240)
L2 L#241 (1024KB) + L1d L#241 (64KB) + L1i L#241 (64KB) + Core L#241 + PU L#241 (P#241)
L2 L#242 (1024KB) + L1d L#242 (64KB) + L1i L#242 (64KB) + Core L#242 + PU L#242 (P#242)
L2 L#243 (1024KB) + L1d L#243 (64KB) + L1i L#243 (64KB) + Core L#243 + PU L#243 (P#243)
L2 L#244 (1024KB) + L1d L#244 (64KB) + L1i L#244 (64KB) + Core L#244 + PU L#244 (P#244)
L2 L#245 (1024KB) + L1d L#245 (64KB) + L1i L#245 (64KB) + Core L#245 + PU L#245 (P#245)
L2 L#246 (1024KB) + L1d L#246 (64KB) + L1i L#246 (64KB) + Core L#246 + PU L#246 (P#246)
L2 L#247 (1024KB) + L1d L#247 (64KB) + L1i L#247 (64KB) + Core L#247 + PU L#247 (P#247)
L2 L#248 (1024KB) + L1d L#248 (64KB) + L1i L#248 (64KB) + Core L#248 + PU L#248 (P#248)
L2 L#249 (1024KB) + L1d L#249 (64KB) + L1i L#249 (64KB) + Core L#249 + PU L#249 (P#249)
L2 L#250 (1024KB) + L1d L#250 (64KB) + L1i L#250 (64KB) + Core L#250 + PU L#250 (P#250)
L2 L#251 (1024KB) + L1d L#251 (64KB) + L1i L#251 (64KB) + Core L#251 + PU L#251 (P#251)
L2 L#252 (1024KB) + L1d L#252 (64KB) + L1i L#252 (64KB) + Core L#252 + PU L#252 (P#252)
L2 L#253 (1024KB) + L1d L#253 (64KB) + L1i L#253 (64KB) + Core L#253 + PU L#253 (P#253)
L2 L#254 (1024KB) + L1d L#254 (64KB) + L1i L#254 (64KB) + Core L#254 + PU L#254 (P#254)
L2 L#255 (1024KB) + L1d L#255 (64KB) + L1i L#255 (64KB) + Core L#255 + PU L#255 (P#255)
L2 L#256 (1024KB) + L1d L#256 (64KB) + L1i L#256 (64KB) + Core L#256 + PU L#256 (P#256)
L2 L#257 (1024KB) + L1d L#257 (64KB) + L1i L#257 (64KB) + Core L#257 + PU L#257 (P#257)
L2 L#258 (1024KB) + L1d L#258 (64KB) + L1i L#258 (64KB) + Core L#258 + PU L#258 (P#258)
L2 L#259 (1024KB) + L1d L#259 (64KB) + L1i L#259 (64KB) + Core L#259 + PU L#259 (P#259)
L2 L#260 (1024KB) + L1d L#260 (64KB) + L1i L#260 (64KB) + Core L#260 + PU L#260 (P#260)
L2 L#261 (1024KB) + L1d L#261 (64KB) + L1i L#261 (64KB) + Core L#261 + PU L#261 (P#261)
L2 L#262 (1024KB) + L1d L#262 (64KB) + L1i L#262 (64KB) + Core L#262 + PU L#262 (P#262)
L2 L#263 (1024KB) + L1d L#263 (64KB) + L1i L#263 (64KB) + Core L#263 + PU L#263 (P#263)
L2 L#264 (1024KB) + L1d L#264 (64KB) + L1i L#264 (64KB) + Core L#264 + PU L#264 (P#264)
L2 L#265 (1024KB) + L1d L#265 (64KB) + L1i L#265 (64KB) + Core L#265 + PU L#265 (P#265)
L2 L#266 (1024KB) + L1d L#266 (64KB) + L1i L#266 (64KB) + Core L#266 + PU L#266 (P#266)
L2 L#267 (1024KB) + L1d L#267 (64KB) + L1i L#267 (64KB) + Core L#267 + PU L#267 (P#267)
L2 L#268 (1024KB) + L1d L#268 (64KB) + L1i L#268 (64KB) + Core L#268 + PU L#268 (P#268)
L2 L#269 (1024KB) + L1d L#269 (64KB) + L1i L#269 (64KB) + Core L#269 + PU L#269 (P#269)
L2 L#270 (1024KB) + L1d L#270 (64KB) + L1i L#270 (64KB) + Core L#270 + PU L#270 (P#270)
L2 L#271 (1024KB) + L1d L#271 (64KB) + L1i L#271 (64KB) + Core L#271 + PU L#271 (P#271)
L2 L#272 (1024KB) + L1d L#272 (64KB) + L1i L#272 (64KB) + Core L#272 + PU L#272 (P#272)
L2 L#273 (1024KB) + L1d L#273 (64KB) + L1i L#273 (64KB) + Core L#273 + PU L#273 (P#273)
L2 L#274 (1024KB) + L1d L#274 (64KB) + L1i L#274 (64KB) + Core L#274 + PU L#274 (P#274)
L2 L#275 (1024KB) + L1d L#275 (64KB) + L1i L#275 (64KB) + Core L#275 + PU L#275 (P#275)
L2 L#276 (1024KB) + L1d L#276 (64KB) + L1i L#276 (64KB) + Core L#276 + PU L#276 (P#276)
L2 L#277 (1024KB) + L1d L#277 (64KB) + L1i L#277 (64KB) + Core L#277 + PU L#277 (P#277)
L2 L#278 (1024KB) + L1d L#278 (64KB) + L1i L#278 (64KB) + Core L#278 + PU L#278 (P#278)
L2 L#279 (1024KB) + L1d L#279 (64KB) + L1i L#279 (64KB) + Core L#279 + PU L#279 (P#279)
L2 L#280 (1024KB) + L1d L#280 (64KB) + L1i L#280 (64KB) + Core L#280 + PU L#280 (P#280)
L2 L#281 (1024KB) + L1d L#281 (64KB) + L1i L#281 (64KB) + Core L#281 + PU L#281 (P#281)
L2 L#282 (1024KB) + L1d L#282 (64KB) + L1i L#282 (64KB) + Core L#282 + PU L#282 (P#282)
L2 L#283 (1024KB) + L1d L#283 (64KB) + L1i L#283 (64KB) + Core L#283 + PU L#283 (P#283)
L2 L#284 (1024KB) + L1d L#284 (64KB) + L1i L#284 (64KB) + Core L#284 + PU L#284 (P#284)
L2 L#285 (1024KB) + L1d L#285 (64KB) + L1i L#285 (64KB) + Core L#285 + PU L#285 (P#285)
L2 L#286 (1024KB) + L1d L#286 (64KB) + L1i L#286 (64KB) + Core L#286 + PU L#286 (P#286)
L2 L#287 (1024KB) + L1d L#287 (64KB) + L1i L#287 (64KB) + Core L#287 + PU L#287 (P#287)
HostBridge
PCIBridge
PCI 0030:01:00.0 (Ethernet)
Net "hsn1"
HostBridge
PCIBridge
PCI 0039:01:00.0 (3D)
%
This is an Nvidia "Grace"/"Hopper" machine (Isambard-AI)
Details of the problem
lstopo shows the L3$ as being 114MiB/package, whereas NVidia's documentation (here) says that there is 117MiB/package.
Additional information
If you need me to run experiments on the machine, as you may not have access to one, please let me know. That should be possible.
The text was updated successfully, but these errors were encountered: