While deep learning has achieved remarkable success in the data-driven modeling of dynamical systems, standard black-box neural networks often struggle with physical consistency, data efficiency, and robustness against noise. When applied to complex physical processes, these models can produce predictions that violate fundamental physical laws. To address these limitations, we introduce a thermodynamics-informed machine learning framework that strictly embeds the GENERIC (General Equation for the Non-Equilibrium Reversible-Irreversible Coupling) formalism directly into the neural network architecture. In this talk, we trace the development of this framework across several key advancements. We first introduce GENERIC Formalism Informed Neural Networks (GFINNs), which provide a rigorous structural prior that guarantees the preservation of energy conservation and entropy dissipation for both deterministic and stochastic systems. To address the practical challenge of learning from corrupted or noisy measurement data, we then present Weak GFINNs (WGFINNs), which leverage weak formulations to significantly enhance the robustness of data-driven system identification. Finally, we scale these principles to high-dimensional parametric problems using Thermodynamics-informed Latent Space Dynamics Identification (tLaSDI). By extracting a low-dimensional, thermodynamically valid representation of complex systems, we will demonstrate that embedding these physical inductive biases ensures consistency and fundamentally improves model robustness, generalizability, and accuracy in scientific applications.