In this study, we employ a Transformers-based architecture [1] to predict time-dependent variations within the reconstructed parameters, and states, accompanied by their respective confidence intervals. The available times series data are driven by the implementation of Ensemble-based Simultaneous Input and State Filtering [2] which is a powerful Data Assimilation technique for the concurrently probabilistic estimation of joint unknown input and state in the presence of process noise and measurement noise for heat conduction problems. Radial Basis Functions have been integrated to reduce the computational cost. The transformer network has been proven to be one of the most optimal time series predictors in deriving coherent structures that exist in the reconstructed heat flux and temperature. By integrating Ensemble-based Simultaneous Input and State Filtering with the transformer network, we establish the capability of our approach to reduce the computational costs associated with conventional ensemble methods significantly.