The Pyramid Wavefront Sensor (PWFS) is a popular choice for measuring wavefront aberrations in demanding adaptive optics driven astronomy. However, it suffers from low linearity, which either limits its operational range or requires complex optical modulation. We developed an End-To-End trainable system inspired by artificial intelligence, consisting of a modified PWFS propagation model with added programmable diffractive layers, plus the incorporation of a deep neural network for wavefront reconstruction. We developed several training strategies for the optical and digital layers under various atmospheric conditions. Our approach dramatically improved the linearity of the PWFS without requiring any optical modulation. In our simulations and current experimental demonstration, we are able to close the adaptive optics loop using our modified unmodulated PWFS for varying atmospheric conditions.