Adaptive Optics (AO) is an essential technique for obtaining high-quality images in large telescopes, particularly for the planned extremely large telescopes. AO focuses on correcting the aberrations introduced by the atmosphere to the incoming wavefront, requiring accurate and timely wavefront measurements and estimations of the aberrations in real-time. Deep learning has shown to assist and improve the estimation accuracy provided by traditional wavefront sensors. However, its usage in astronomical AO is limited by the inference speed which largely depends on the size of the neural network and the hardware capabilities. We are developing a real-time controller for AO systems that rely on deep learning wavefront sensors. We are using state-of-the-art GPU hardware that allow to receive image data from wavefront sensors directly, bypassing CPU memory. The modal or zonal estimates can be directed to the deformable mirror driver at reasonable speeds, approaching a real-time performance of nearly 1 KHz when using very complex transformer neural network architectures designed to estimate more than 200 modes for non-modulated pyramid wavefront sensors.