We are currently building Neural Network using keras and tensorflow for evaluating chess positions. And we have encountered the problem with speed of prediction on a single sample, which is used in our search tree. The usage in the search tree is to check for legal moves in given position, evaluate each position and get the best move according to the best evaluation.

For higher depths the prediction speed is what makes it slow. It is worth to mention that our Neural network is kind of shallow - 3CNN layers and 2 dense layers. We've trained the model on CPU and we're using CPU in process of predicting aswell. We've assumed that, on this particular (shallow) network it won't affect the perfomance. While there is no parallelism in our use case, therefore there is no need for GPU computing.

Versions:

Python 3.12.4

Tensorflow 2.16.1

Keras 3.3.3

We evaluate a single sample at the time using either model.predict() or model(x). We've discovered that when used as predict_on_batch() is roughly as fast as predict on a single sample. Our goal is to get the prediction as fast as possible while keeping the prediction used on single sample at the time.

We've tried to convert the model to TFLite as it was suggested to get slighlty better performance. But we couldn't convert the model due to newest versions incompatibility, while downgrading didn't work out aswell.

We were observing speeds of each predict function on various batch sizes.

model = keras.models.load_model('firstModel.keras')

print("Durations using model __call__() on small batch")
for i in range(5):
    start = time.time()
    prediction = model(bitboard)
    end = time.time()
    print(end - start)

print("Durations using model.predict_on_batch() on small batch")
for i in range(5):
    start = time.time()
    prediction = model.predict_on_batch(bitboard)
    end = time.time()
    print(end - start)

print("Durations using model.predict() on small batch")
for i in range(5):
    start = time.time()
    prediction = model.predict(bitboard, batch_size=1, verbose=0)
    end = time.time()
    print(end - start)

print("Durations using model.__call__() on larger batch (100 samples)")
for i in range(5):
    start = time.time()
    prediction = model(bitboards)
    end = time.time()
    print(end - start)

print("Durations using model.predict_on_batch() on larger batch (100 samples)")
for i in range(5):
    start = time.time()
    prediction = model.predict_on_batch(bitboards)
    end = time.time()
    print(end - start)

print("Durations using model.predict() on larger batch (100 samples)")
for i in range(5):
    start = time.time()
    prediction = model.predict(bitboards, batch_size=1, verbose=0)
    end = time.time()
    print(end - start)

And the speeds were as follows:

Durations using model __call__() on small batch
0.055520057678222656
0.007033586502075195
0.006206035614013672
0.007121562957763672
0.005555391311645508
Durations using model.predict_on_batch() on small batch
0.06325101852416992
0.0020132064819335938
0.0010013580322265625
0.0009975433349609375
0.0025305747985839844
Durations using model.predict() on small batch
0.1571955680847168
0.05691671371459961
0.05576348304748535
0.05414080619812012
0.05917525291442871
Durations using model.__call__() on larger batch (100 samples)
0.01164698600769043
0.00638890266418457
0.007528543472290039
0.006807804107666016
0.00751185417175293
Durations using model.predict_on_batch() on larger batch (100 samples)
0.04664158821105957
0.0025255680084228516
0.0010013580322265625
0.0020008087158203125
0.0025064945220947266
Durations using model.predict() on larger batch (100 samples)
0.05106091499328613
0.04923701286315918
0.06421136856079102
0.0651085376739502
0.055069923400878906

What troubles us and yet we don't understand how is possible to get prediction on larger batch size in lower execution time, than predicting on a single samples. We've assumed it could be due to wrong keras/tensorflow usage.

Main questions:

Any suggestions how to speed the prediction up?

Is there any possibility, that running the code with GPU would increase the performance?

Would you recommend any other approach to the problem or different use case that would fit for our problem?

EDIT:

We tried converting model to .onnx format. That really improved the time performance of the evaluation function and that's great, but we encountered another performance problem. The problem is that our tree search (based on alpha beta pruning) is also very slow. We are now hardly reaching depth 6 without interference of the model(searching about 2 500 000 nodes in one minute, with interference even less), which is terrible.

So far we haven't added anything like transposition table or move ordering, simply it is just plain tree search. We are now considering translating it to C/C++. We have very little experience with performance tuning, so our question is, how much will it help and how many nodes per second can we get?

We are also wondering if it is a good idea to use our models trained with Python in our future C programs?