Off topic, but is it safe to share what I’m assuming is a stack trace/debug info QR code? Does it have any potentially sensitive data?

Screen freezes should also leave traces in your syslog, if they’re caused by any panic or GPU driver issue. You might want to check if your system is still accessible via SSH, if only the screen froze, and try killing X from there, if switching to text VTs doesn’t work. SysRq might become helpful, too.

screen froze, and I was forced to reboot the PC by pressing the power button for 3s

seems like some data was saved, while other files were discarded

I would not worry too much about a somehow “forgetful” file system immediately after a hard power cycle. This is exactly what happens if data could not be flushed to disk. Thanks to journaling, your FS does not get corrupted, but data lingering in caches is still lost and discarded on fsck, to retain a consistent fs. I would recommend to repeat the installations you did before the crash, and maybe shove a manual sync behind it, to make sure you don’t encounter totally weird “bugs” with man later, when you don’t remember this as a cause anymore. Your bash history is saved to file on clean shell exit only, and is generally a bit non-intuitive, especially with multiple interactive shells in parallel, so I would personally disregard the old .bash_history file as “not a fault, only confusing” and let that rest, too.

Starting a long SMART self-test and a keen eye on the drive’s error logs (smartctl -l error <drive>), or better yet, all available SMART info (-x), to see if anything seems fishy with your drive is a good idea, anyway. Keep in mind that your mainboard / drive controller or its connection may just as well be (intermittently) faulty. In ye olden times, a defective disk cable or socket was messing up my system once or twice. You will see particular faults in your syslog, though - this is not invisible. You don’t only get a kernel panic without some sprinkling of I/O errors as well. If your drive is SMART-OK, but you clearly get disk I/O errors, time to inspect and clean the SSD socket and contacts and re-seat once more. If you never saw any disk I/O errors, and your disk’s logs are clean, I’d consider the SSD as not an issue.

If you encouter random kernel panics, random as in “in different and unrelated call stacks that do not make sense in any other way”, I agree that RAM is a likely culprit, or an electrical fault somewhere on the mainboard. It’s rare, but it happens. If you can, replace (only) the mainboard, or better yet, take a working PC with compatible parts, and replace the working MBO with your suspected broken one to see if the previously working machine now faults. “Carrying the fault with you” is easier/quicker than proving an intermittent fault gone.

Unless you get different kernel panics, my money’s still on your c-states handling. I’d prefer the lowest level you can find to inhibit your CPUs from going to sleep, i. e. BIOS > kernel boot args > sysctl > cpupower, to keep the stack thin. If that is finnicky somehow, you could alternatively boot with a single CPU and leave the rest disabled (bootarg nosmp). The point is just to find out where to focus your attention, not to keep this as a long-term workaround.

To keep N CPUs running, I usually just background N infinite loops in bash:

$ cpus=4; for i in $(seq 1 $cpus); do { while true; do true; done; } & done 
[1] 7185
[2] 7186
[3] 7187
[4] 7188

In your case you might change that to:

cpus=4; for i in $(seq 0 $((cpus - 1))); do { taskset -c $i bash -c 'while true; do sleep 1; done'; } & done

To just kick each CPU every second, it does not have to be stressed. The taskset will bind each loop to one CPU, to prevent the system from cleverly distributing the tiny load. This could also become a terrible, terrible workaround to keep running if all else fails. :)

the new linux BSOD is looking cool. i’ve never seen it before.

Looking at the call trace:

[ 1641.073507] RIP: 0010:rb_erase+0x199/0x3b0
...
[ 1641.073601] Call Trace:
[ 1641.073608]  <TASK>
[ 1641.073615]  timerqueue_del+0x2e/0x50
[ 1641.073632]  tmigr_update_events+0x1b5/0x340
[ 1641.073650]  tmigr_inactive_up+0x84/0x120
[ 1641.073663]  tmigr_cpu_deactivate+0xc2/0x190
[ 1641.073680]  __get_next_timer_interrupt+0x1c2/0x2e0
[ 1641.073698]  tick_nohz_stop_tick+0x5f/0x230
[ 1641.073714]  tick_nohz_idle_stop_tick+0x70/0xd0
[ 1641.073728]  do_idle+0x19f/0x210
[ 1641.073745]  cpu_startup_entry+0x29/0x30
[ 1641.073757]  start_secondary+0x11e/0x140
[ 1641.073768]  common_startup_64+0x13e/0x141
[ 1641.073794]  </TASK>

What’s happening here leading up to the panic is start_secondary followed by cpu_startup_entry, eventually ending up in CPU idle time management (tmigr), giving a context of “waking up/sleeping an idle CPU”. I’ve had a few systems in my life where somewhat aggressive power-saving settings in the BIOS were not cleanly communicated to Linux, so to say, causing such issues.

This area is notorious for being subtly borked, but you can test this hypothesis easily by either disabling a setting akin to “Global C States” in your BIOS, which effectively disables power-saving for your CPUs, or try an equivalent setting of the kernel arguments processor.max_cstate=1 intel_idle.max_cstate=0, or even a cpuidle.off=1.

This is obviously losing your power-saving capability of the CPUs, but if your system runs stable that way, you’re likely in the right ballpark and find a specific solution for that issue, possibly in a BIOS/Fimware update. Here’s a not too shabby gist roughly explaining what c-states are. Don’t read too many of the comments, they’re more confusing than enlightening.

The kernel docs I linked to above are comprehensive, and utterly indecipherable for a layperson. Instead of fumbling about in sysfs, try the cpupower tool/package to visualize the CPU idle settings, and try increasing enabled idle states until your system crashes again, to find out if a specific (deep) sleep state triggers your issue, and disable exactly that if you cannot find a bugfix/BIOS update.

If this is your problem, to reproduce the panic, try leaving your system as idle as possible after bootup. If a panic happens regularly that way, try starting processes exercising all your CPUs - if the hypothesis holds, this should not panic at any time, as no CPU is ever idle.

What troubleshooting steps did you take so far? I would try these:

• different OS, maybe a live usb running fedora or ubuntu if it is possible to emulate the workload where this appears
• bios reset to defaults, no OC not even XMP
• memtest, either the memtest86+ boot iso or the runtime memtester can detect obvious errors
• long smart self test on OS drive and an fsck or scrub based on FS

Also the logs show a very old nvidia gpu which is not supported by the new driver. I don’t know if this can cause crashes, haven’t used one in ages, maybe someone else has more insight.

I had issues with reboots on a old server and it turned out to be the memory even though I didnt find anything in the memtests, maybe pull one stick out and try, and if it happens swap the other stick and try

Issues with nvidia driver, maybe 🤔

Have you been keeping an eye on the CPU temps? What’s the usual workload of this machine at any given time?

Also, any metrics over time would useful. Maybe run Prometheus and Grafana.

Not an expert, but looked at a few libxul.so searches and it was a mixed bag, but one of them said it could be ram problems. Might be worth trying memtest86+.

It’s of course also possible that this is a completely seperate issue, but I would still like some tips. Please help me because I am about to buy an entirely new PC because this has been driving me nuts!!