Linux IO + Memory + CPU Contention

Ritesh Raj Sarraf's picture

I very recently met someone, and we had a good productive discussion on the features and (long standing) bugs of the Linux kernel. No doubt, Linux is the most featureful kernel in the market. Is also a lot appealing given its breadth of platform support.

Of that discussion we had, it led about Linux's behavior in tighter stressed scenarios where there is a lot of contention among the core subsystems. From the conversation, I got the feedback that perhaps the issue is no more valid. My conclusion was I must have missed out on the fix because I haven't really spent any Engineering Lab time in recent past.

But a picture says a thousand words. And no, the issue is NOT fixed. And it is still simple to reproduce the bug. And it is just that we have faster advancements in underlying hardware, that we may eventually find it much harder to reproduce these bugs.

 

The kernel spitting the screen is Linux 4.4 Stock Debian.

Keywords: 

Categories: 

Comments

Ritesh Raj Sarraf's picture

The linked bug report has

The linked bug report has multiple use cases that can help reproduce the problem.

For me, the following has worked reliably.

  1. Assume you have a slow media underneath (like rotational disk, or a disk that you access over slow transport)
  2. Spawn multiple processes that pump buffered I/O. For example, five separate instances of dd
  3. Of Step 2, you also need to ensure that the total of those processes should pump I/O greater than the system memory. This is IMPORTANT. You need to consume all your system RAM.
  4. Typically, this is more than enough. But if it still gets hard, trigger a simple CPU intensive task.

By this time, your overall system should be doomed. In theory, your machine should be able to recover if left untouched. But for how long is uncertain.

This bug will disguise itself in multiple forms, but if root caused, the bug will be the same. Here's the latest example

Add new comment