I have a large table of data, in the range of hundreds of millions of rows/events, each which has around 50 numerical columns, call them c1 through c50. For each event, say I want to perform matrix-vector multiplication where I define a matrix via, e.g.

M = ( (c1, c2, c3), (c4, c5, c6), (c7, c8, c9) )

and a vector via, e.g.

v = (c48, c49, c50)

Now, I could do this via numpy, but due to the size of my dataset, a single operation exceeds RAM, an increasingly important consideration these days. The obvious solution is to split the job into batches and manually adjust the size of each. However, it's not a fun task, and I have discovered the Polars library can do the job by using the streaming engine, and saving the following outputs:

v1 = c1 * c48 + c2 * c49 + c3 * c50

and similarly for the other two outputs vector components. Notably, using the streaming engine means I don't have to thinking about partitioning the dataset into batches. Thankfully, all these operations being only per-row means all the multiplication and addition are available in the streaming engine. However, if I have to chain matrix multiplications together, or the matrices get larger, you can imagine just how cumbersome this gets.

Thus, I'm trying to think through a system or wrapper to automate the expansion of all the linear algebra operations into multiplication and addition, but I haven't had much luck, though it probably shouldn't be a very hard problem. I know polars might not have been designed for this specifically, but its streaming engine makes this direction very appealing compared to numpy.

Thanks for the help.