I'm working on a fairly large project that need to handle around 10 GB of real time data per hour with computation and also long text (locally process 100 million tokens per hour), but also sends to the LLM through the OpenAI SDK's chatcompletion API or response API.

Originally I thought about to use c++ because tools such as llama.cpp can allow for some local inferences and also AI support. However, I realized that most of the agentic system somehow preferred Rust, such as codex claude CLI etc. I wonder if it was because the ownership of the memory that made it more attractive to use Rust for repeated text handling and garbage collection. Because if the garbage collection became messy with c++ then obviously rust is more convenient to handle the text operations. (Since both c++ and rust are statically typed What is the difference between statically typed and dynamically typed languages?, the string operation were more complicated than python.) The OpenAI sdk also has a community maintained version for rust but not in c++, that's a huge factor as well.

Does anyone has any experience to choose between rust or c++ for such program that required fast compute and OpenAI sdk integration?