Unleashing GPU Power:
Accelerating Linear Algebra, 
Simulation and More

Charles Pooh, Wolfram Research

#WolframTechConf

The Computational Challenge

The growing demand for performance in data 
and compute intensive tasks

The Rise of the GPU

From a specialized graphics chip to a 
general-purpose computing powerhouse

Presentation Agenda

GPU vs. CPU: A Paradigm Shift

Foundation of GPU Computing

Accelerated Linear Algebra

Simulation and More

GPU vs. CPU
 A Paradigm Shift

Architectural Differences

CPU architecture — few cores, sequential processing
GPU architecture —thousands of cores, parallel processing.

When to Use a GPU

Massive Data Parallelism

High Computational Intensity

Iterative Algorithms

Memory Bandwidth Requirements

Scalable Workloads

The Heterogeneous Approach

Modern systems combine the strengths of both 
CPU and GPU for optimal performance

Foundations of 
GPU Computing

Parallel Programming Models

NVidia CUDA

Apple Metal

High-Level Libraries

Domain-Specific Frameworks

CUDALink

transpose[matrix_] := Module[{dInputMatrix, dOutputMatrix, hOutputMatrix, blockDim, transposeFun},
   
   dInputMatrix = CUDAMemoryLoad[matrix];
   dOutputMatrix = CUDAMemoryAllocate[Real, Reverse@ Dimensions[matrix]];
   
   blockDim = {16, 16};
   transposeFun = CUDAFunctionLoad[{FileNameJoin[{$CUDALinkPath, "SupportFiles", "transpose.cu"}]}, 
                    "transpose_kernel", {{_Real, "Input"}, {_Real, "Output"}, "Integer32", "Integer32"}, blockDim];
   
   transposeFun[dInputMatrix, dOutputMatrix, Sequence @@ Reverse[Dimensions[matrix]]];
   hOutputMatrix = CUDAMemoryGet[dOutputMatrix];
   
   CUDAMemoryUnload[dInputMatrix];
   CUDAMemoryUnload[dOutputMatrix];
   Return[hOutputMatrix];
]

CUDA Compiler Tools

Integration with the Wolfram Compiler

Access to NVIDIA Libraries

GPU Data Management

Automatic Memory Management

transpose = FunctionCompile[
	Function[Typed[matrix, "NumericArray"["Real32", 2]], 
	  Module[{res, vec},
	    vec = CreateCUDAMatrix[matrix];
	    res = CUDATranspose[vec];
	    FromCUDAMatrix[vec]
	  ]
	]
]

GPUArray

Transpose[GPUArray[matrix]]

An Example: Root Mean Square Estimator

The root mean square error (RMSE) measures the average difference between a statistical model’s predicted values and the actual values

Basic Implementation

RSME[a_,b_]:=Sqrt[Total[(a-b)^2]/Length[a]]

n=10^8;
a=RandomReal[n,n];
b=RandomReal[n,n];

RSME[a,b]

GPU Implementation

ag=GPUArray[a];
bg=GPUArray[b];

RSME[ag,bg]

RootMeanSquare[ag-bg]

GPU Compiler Implementation

rms= FunctionCompile[
Function[{
Typed[a,"PackedArrayArray"["MachineReal",1]],Typed[b,"PackedArrayArray"["MachineReal",1]]},
Module[{ga,gb},
ga=GPUArray[a];
gb=GPUArray[b];
Sqrt[Total[(ga-gb)^2]/Length[ga]]
]
]
];

rms[a,b]

Performance Comparison

Accelerating 
Linear Algebra

Linear Algebra on the GPU

Linear algebra operations are at the heart of many scientific and engineering applications

Massive Parallelism

High Memory Bandwidth

Optimized Libraries

Vector and Matrix operations

GPUArray — represent an array in GPU accessible memory
ArrayQ ▫ VectorQ ▫ MatrixQ ▫ ArrayDepth ▫ Dimensions ▫ Length ▫ Part ▫ First ▫ Last ▫ ByteCount

Plus, Times — arithmetic operations
Subtract ▫ Divide ▫ Total ▫ Mean ▫ Accumulate ▫ Differences ▫ Ratios ▫ ...

Dot — matrix operations
Transpose ▫ ConjugateTranspose ▫ Diagonal ▫ Tr ▫ ...

LinearSolve — solve linear systems

Inverse ▫ PseudoInverse ▫ LeastSquares ▫ ...

Eigenvalues — solve eigenvalues problems

Eigenvectors ▫ Eigensystem ▫ SingularValueList ▫ ...

SingularValueDecomposition — matrix decompositions

QRDecomposition ▫ LUDecomposition ▫ CholeskyDecomposition ▫ ...

The Heterogeneous Approach

Combine the strengths of both CPU and GPU for optimal performance

(Legended)

Simulation and More

GPU Optimized Mandelbrot Set

Faster High-Resolution Images for Mandelbrot Set

Simulating Complex Dynamics

Idealizations of coupled nonlinear oscillators

Wolfram Language Example Repository

Discrete Heat Equation

Discrete Reaction diffusion

Logistic Map Bifurcation Diagram

Cellular Automata Activity

Conclusion

Key Takeaways

Tools to Solve High-Level Tasks

High-Level Abstraction

Heterogeneous Computing

CPU-GPU-Agnostic Design

Direct Integration with Functions

Optimal Performance

Resources

GPU Computing

GPU Computing with NVidia GPUs

GPU Computing with Apple GPUs

CUDA Compiler Tools

Introducing GPU-Accelerated Arrays

Wolfram R&D YouTube Channel

Wolfram U