> As the drawing routine is the slowest, a good solution is to run a prediction server in parallel to the drawing commands.

Your drawing routine is unreasonably slow. 250–270 ms is way too much for blitting an image and drawing few rectangles with text over it. Your typical web browser can draw more complex stuff at 60+ fps. Proper solution will be to fix it first (prior to doing parallelization).