Predicting Soft Errors in PCG with ML

Created on June 27, 2024

2024   ·   machine-learning   hpc   ·   research

Introduction

During my time as a student researcher at the Research Group Theory and Applications of Algorithms at the University of Vienna, I delved into an interesting problem at the intersection of high-performance computing (HPC) and machine learning: developing machine learning methods to predict the impact of soft errors on the convergence of the Preconditioned Conjugate Gradient (PCG) method.

Now you might be wondering: Why is this problem relevant? What are soft errors and why do we care about them? How can machine learning help us address them?

Let’s dive right in to unravel these questions one by one.

The Preconditioned Conjugate Gradient method

First, let’s briefly introduce the Preconditioned Conjugate Gradient (PCG) method. PCG is an iterative algorithm for solving symmetric and positive definite linear systems of the form \(Ax = b\). PCG is typically used for solving large sparse systems (i.e. \(A\) has mostly zero-entries) where using traditional direct methods is wasteful and too computationally expensive.

If you’re familiar with the Conjugate Gradient (CG) method, PCG is very similar. The key difference is that PCG uses preconditioning, which is essentially a mathematical trick where we multiply both sides of our equation with a non-singular matrix \(M^{-1}\) such that \(M^{-1}A\) is better conditioned than \(A\). We call \(M\) the preconditioner. The equation then becomes:

\[M^{-1}Ax = M^{-1}b\]

The goal of preconditioning is to accelerate convergence of the algorithm.

Fig. 1. Pseudocode of PCG. (Source: Agullo et al. [1])

Where is PCG used?

PCG is used in many areas of science. Common applications include simulating heat distribution in complex structures and calculating airflow around aircraft wings. These problems are described by partial differential equations (PDEs), which ultimately boil down to solving enormous sparse linear systems. This is where using PCG becomes very efficient, and therefore it is a crucial algorithm for many scientific applications.

Soft errors in PCG

Now that we have an overview of PCG and its applications, we can move onto the actual problem at hand: the impact of soft errors on PCG convergence. Soft errors, such as bit-flips, are particularly tricky because they don’t cause immediate program failure. Instead, they silently corrupt data, thereby making it unreliable. With the increasing scale of supercomputers, developing resilience strategies against soft errors has become crucial in high-performance computing.

The simplest solution to this problem would be to duplicate all computations to be able to detect these errors. However, this is not computationally affordable because of the expensive matrix-vector multiplication (line 3 in Fig. 1) we do in every iteration. Therefore, protecting all elements is infeasible. On the other hand, not protecting any computations leaves us vulnerable to errors that can severely impact both accuracy and performance.

The question that naturally arises now is: Can we selectively protect only the most important elements in the matrix vector multiplication while minimizing overhead? This is exactly where machine learning comes into play.

ML prediction of soft error impacts

Machine learning can help us predict which elements, when affected by soft errors, would significantly impact PCG’s performance. Chen et al. [2] approached this as a classification problem, training various models (including random forest, SVM and KNN) to predict whether a soft error in a specific element of the p vector (line 3 of Fig. 1) would significantly affect PCG convergence.

What does “significant” impact mean? Since the goal is to reduce overhead compared to the full-protection scheme, the authors define a significant impact as one that at least doubles the number of iterations needed for convergence compared to the error-free case.

The model is trained using two features (the reason for using these features is described in [2]):

  1. The iteration number when the fault occurs
  2. The row 2-norm of the matrix row corresponding to the affected element

Based on these features, the model makes binary decisions: should an element be protected because soft errors would significantly slow down convergence, or can it remain unprotected since errors would only have little impact? This allows for a dynamic protection scheme that can be used for an individual matrix.

The authors demonstrate the effectiveness of their approach by comparing the proposed dynamic protection scheme against zero- and full-protection strategies, showing experiments using two matrices. [2]

However, there are several limitations that remain unsolved:

  1. The approach needs to be individually tailored for each matrix and lacks generalization across different matrix types.
  2. Numerical properties that work well as features for one matrix type might not be informative for others.
  3. In practice, most soft errors manifest as single or double-bit flips, and their impact heavily depends on which specific bits are affected.

So while this approach represents a promising initial exploration, significant limitations prevent its immediate generalization to broader scenarios.

Conclusion

In this post, we explored how machine learning can be leveraged to tackle a specific challenge in scientific computing: protecting elements in the Preconditioned Conjugate Gradient from soft errors. While this problem may seem abstract, it has practical significance in ensuring the reliability of numerical simulations and computations. Although interesting work has been done in this field in recent years, there are still many open challenges which present opportunities for future research. By addressing these challenges, we can make scientific computing more robust and efficient.

References

[1] E. Agullo, S. Cools, Emrullah Fatih-Yetkin, L. Giraud, N. Schenkels, and Wim Vanroose, "On Soft Errors in the Conjugate Gradient Method: Sensitivity and Robust Numerical Detection," SIAM Journal on Scientific Computing, vol. 42, no. 6, pp. C335–C358, Jan 2020. https://doi.org/10.1137/18m122858x

[2] Z. Chen, T. Verrecchia, H. Sun, J. Booth, and P. Raghavan, "Dynamic Selective Protection of Sparse Iterative Solvers via ML Prediction of Soft Error Impacts," Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, pp. 488–491, Nov 2023. https://doi.org/10.1145/3624062.3624117