CRUNCH Seminars
July 26, 2024:
Presentation #1: Multiscale particle simulation of nonequilibrium gas flows and data-driven discovery of governing equations, Dr. Zhang Jun, Beihang University
Link: N/A
Abstract: The simulation of non-equilibrium gas flows has garnered significant interest in modern engineering problems, notably in micro-electro-mechanical systems and aerospace engineering. The direct simulation Monte Carlo (DSMC) method has been very successful for the simulation of rarefied gas flows. However, due to the limitation of cell sizes and time steps, DSMC requires extraordinary computational resources for the simulation of near-continuum flows. Dr. Zhang Jun presents a novel method called the unified stochastic particle (USP) method, which can be implemented using much larger time steps and cell sizes by coupling the effects of molecular movements and collisions. Various applications have demonstrated that the USP method can improve computational efficiency by several orders of magnitude compared to DSMC. On the other hand, extending the application of macroscopic equations to nonequilibrium gas flows is also intriguing. It is known that in strong nonequilibrium flows, linear constitutive relations break down, and thus, the Navier-Stokes-Fourier equations are no longer applicable. Dr. Zhang Jun presents their recent work on data-driven discovery of governing equations by combining multiscale particle simulations and two types of machine learning methods: sparse regression and gene expression programming (GEP). Specifically, Dr. Jun proposes a novel dimensional homogeneity constrained gene expression programming (DHC-GEP) method. In the shock wave structure, the derived constitutive relations using DHC-GEP are more accurate than conventional equations over a wide range of Knudsen numbers and Mach numbers.
Presentation #2: One Factor to Bind the Cross-Section of Returns, Dr. Nicola Borri & Dr. Aleh Tsyvinski, LUISS University
Link: N/A
Abstract: Dr. Nicola Borri & Dr. Aleh Tsyvinski propose a new non-linear single-factor asset pricing model. Despite its parsimony, this model represents exactly any non-linear model with an arbitrary number of factors and loadings – a consequence of the Kolmogorov-Arnold representation theorem. It features only one pricing component, comprising a nonparametric link function of the time-dependent factor and factor loading that Dr. Borri and Dr. Tsyvisnki jointly estimate with sieve-based estimators. Using 171 assets across major classes, Dr. Borri’s and Dr. Tsyvisnki’s model delivers superior cross-sectional performance with a low-dimensional approximation of the link function. Most known finance and macro factors become insignificant controlling for their single-factor.
July 19, 2024:
Presentation #1: Toward Efficient Neuromorphic Computing, Sen Lu, Penn State University
Link: N/A
Abstract: Spiking Neural Networks (SNNs) are considered to be the third generation of artificial neural networks due to their unique temporal, event-driven characteristics. By leveraging bio-plausible spike-based computing between neurons in tandem with sparse on-demand computation, SNNs can demonstrate orders of magnitude power efficiency on neuromorphic hardware in contrast to traditional Machine Learning (ML) methods. This seminar reviews some of Sen Lu’s recent proposals in the domain of neuromorphic SNN algorithms from an overarching system science perspective with an end-to-end co-design focus from algorithms to hardware and applications. Sen Lu will specifically discuss SNN designs in the extreme quantization regime, neuroevolutionary optimized SNNs along with scaling deep unsupervised learning in SNN models. Leveraging the sparse, event-driven operation of SNNs, Sen Lu demonstrates significant energy savings of SNNs in applications that match its computing style like event-driven sensors, cybersecurity attack detection, among others. The talk outlines opportunities at designing hybrid neuromorphic platforms where leveraging benefits of both traditional ML methods and neuroscience concepts in the training and architecture design choice can actualize SNNs to their fullest potential.
July 12, 2024:
Presentation #1: Physics-informed neural network for simulation of problems in dynamic linear elasticity, Venkatesh Gopinath and Vijay Kag, Bosch Research, India
Link: https://www.youtube.com/watch?v=6dCc7OYPjFo
Abstract: This work presents the physics-informed neural network (PINN) model applied particularly to dynamic problems in solid mechanics. It focuses on forward and inverse problems. Particularly, showing how a PINN model can be used efficiently for material identification in a dynamic setting. In this work, it is assumed linear continuum elasticity. This shows results for two-dimensional (2D) plane strain problem and then we proceed to apply the same techniques for a three-dimensional (3D) problem. As for the training data used the solution based on the finite element method. This rigorously shows that PINN models are accurate, robust and computationally efficient, especially as a surrogate model for material identification problems. Also, by employing state-of-the-art techniques from the PINN literature which are an improvement to the vanilla implementation of PINN. Based on these results, it is believed that the framework has developed can be readily adapted to computational platforms for solving multiple dynamic problems in solid mechanics.
Presentation #2: Geometric deep learning and 3D field predictions using Deep Operator Network, Jimmy He, Ansys Inc.
Link: https://www.youtube.com/watch?v=6dCc7OYPjFo
Abstract: Data-driven deep learning models have been widely used as surrogate models for traditional numerical simulations. Besides material and geometric nonlinearities, one of the biggest challenges in creating surrogate models for engineering simulations is the varying geometries of the problem domains. The shape of an engineering design affects the result field distribution, and accurate, generalizable encoding of the geometries plays a vital role in a successful surrogate model. Geometric deep learning, which focuses on capturing different input geometries, has been studied intensively in the literature, with methods like graph neural networks and implicit neural representations being developed. This work enhances the Deep Operator Network (DeepONet) architecture with key elements from geometric deep learning, such as the signed distance function and the sinusoidal activation (SIREN), to further enhance the network’s spatial awareness towards varying geometries. Intermediate data fusion is introduced between the branch and trunk networks, which improves the model prediction accuracy. This novel architecture, called the Geom-DeepONet, is benchmarked against the classical PointNet and the vanilla DeepONet models. Geom-DeepONet shows a much smaller GPU memory usage footprint compared to PointNet and has the highest accuracy over the three models. Unlike PointNet, once trained, Geom-DeepONet can generate predictions on geometries discretized by arbitrary numbers of nodes and elements. Compared to finite element simulations, the predictions can be 10^5 times faster. Geom-DeepONet also demonstrates superior generalizability towards the vanilla DeepONet on dissimilar shapes, which makes it a viable candidate to be used as a surrogate model for rapid preliminary design screening.
July 5, 2024:
Presentation #1: On the use of “conventional” unconstrained minimization solvers for training regression problems in Scientific Machine Learning, Stefano Zampini, KAUST
Link: https://www.youtube.com/watch?v=taEnrJIpl1g
Abstract: In recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis, by means of deep-learning techniques, of data produced by computational science and engineering applications. At the core of these methods is the supervised training algorithm to learn the neural network realization, a highly non-convex optimization problem that is usually solved using stochastic gradient methods. However, distinct from deep-learning practice, scientific machine-learning training problems feature a much larger volume of smooth data and better characterizations of the empirical risk functions, which make them suited for conventional solvers for unconstrained optimization. In this talk, we introduce PETScML, a lightweight software framework built on top of the Portable and Extensible Toolkit for Scientific computation (PETSc) to bridge the gap between deep-learning software and conventional solvers for unconstrained minimization. Using PETScML, we empirically demonstrate the superior efficacy of a trust region method based on the Gauss-Newton approximation of the Hessian in improving the generalization errors arising from regression tasks when learning surrogate models for a wide range of scientific machine-learning techniques and test cases. All the conventional solvers tested, including L-BFGS and inexact Newton with line-search, compare favorably, either in terms of cost or accuracy, with the adaptive first-order methods used to validate the surrogate models.
Presentation #2: On Sampling Tasks with Langevin Dynamics, Haoyang Zheng, Purdue University
Link: https://www.youtube.com/watch?v=taEnrJIpl1g
Abstract: Langevin dynamics, driven by Brownian motion, are a class of stochastic processes widely utilized in various machine learning sampling tasks. This discussion will explore the topics of sampling from gradient Langevin dynamics using Markov Chain Monte Carlo (MCMC), variant algorithms such as underdamped Langevin dynamics (ULD) and replica exchange stochastic gradient Langevin dynamics (reSGLD), as well as their applications in reinforcement learning (also called Thompson sampling) and constrained sampling. First introduced is accelerated approximate Thompson Sampling algorithm based on ULD. Under smooth and convex conditions, theoretically and empirically demonstrate that our algorithm reduces sample complexity from O(d) to O(√d) and derive O(log(N)) regrets, where d is the number of model parameters and N is the number of times to select actions. reSGLD is an effective sampler for non-convex learning in large-scale datasets. However, it may stagnate when the high-temperature chain explores the distribution tails too deeply. To address this, we propose reflected reSGLD (r2SGLD), which incorporates reflection steps within a bounded domain to enhance constrained non-convex exploration. Both theoretical and empirical evidence underscores its significance in improving simulation efficiency.
June 28, 2024:
Presentation #1: Simulation-Calibrated Scientific Machine Learning, Yiping Lu, Courant Institute of Mathematical Sciences, New York University
Link: https://www.youtube.com/watch?v=pnulf2VBeVs
Abstract: Machine learning (ML) has achieved great success in a variety of applications suggesting a new way to build flexible, universal, and efficient approximators for complex high-dimensional data. These successes have inspired many researchers to apply ML to other scientific applications such as industrial engineering, scientific computing, and operational research, where similar challenges often occur. However, the luminous success of ML is overshadowed by persistent concerns that the mathematical theory of large-scale machine learning, especially deep learning, is still lacking and the trained ML predictor is always biased. This seminar introduces a novel framework of (S)imulation-(Ca)librated (S)cientific (M)achine (L)earning (SCaSML), which can leverage the structure of physical models to achieve the following goals: 1) make unbiased predictions even based on biased machine learning predictors; 2) beat the curse of dimensionality with an estimator suffers from it. The SCASML paradigm combines a (possibly) biased machine learning algorithm with a de-biasing step design using rigorous numerical analysis and stochastic simulation. Theoretically, trying to understand whether the SCaSML algorithms are optimal and what factors (e.g., smoothness, dimension, and boundness) determine the improvement of the convergence rate. Empirically introducing different estimators that enable unbiased and trustworthy estimation for physical quantities with a biased machine learning estimator. Applications include but are not limited to estimating the moment of a function, simulating high-dimensional stochastic processes, uncertainty quantification using bootstrap methods, and randomized linear algebra.
Presentation #2: HPINNs: Gradient is not enough! You need curvature., Mostafa Abbaszadeh , Amirkabir University of Technology in Tehran, Iran
Link: https://www.youtube.com/watch?v=pnulf2VBeVs
Abstract: Deep learning has proven to be an effective tool for solving partial differential equations (PDEs) through Physics-Informed Neural Networks (PINNs). PINNs embed the PDE residual into the neural network’s loss function and have been successfully used to solve various forward and inverse PDE problems. However, the first generation of PINNs often suffers from limited accuracy, necessitating the use of extensive training points. Prior work “Gradient-Enhanced PINNs”, suggested that the gradient of the residual should be zero because the residual itself should be zero. This work proposes an enhanced method for improving the accuracy and training efficiency of PINNs. By creating a smooth, flat landscape for residual losses and ensuring zero residual curvature, the approach improves the network’s ability to learn from residuals more effectively. Employing Hutchinson Trace Estimation to calculate the curvature, further refining the loss function. Extensive experiments demonstrate that the method significantly outperforms existing approaches, including Gradient-Enhanced PINNs (gPINNs). The results show improved accuracy and efficiency in solving PDEs, highlighting the effectiveness of the approach.
June 21, 2024:
Presentation #1: FastVPINNs: Tensor-Driven Acceleration of VPINNs for Complex Geometries, Dr. Sashikumaar Ganesan, Divij Tirthhankar Ghose, & Thivin Anandh, Department of Computational and Data Sciences, IISc Bangalore
Link: https://youtu.be/YAxf4gOdehQ?feature=shared
Abstract: Variational Physics-Informed Neural Networks (VPINNs) solve partial differential equations (PDEs) using a variational loss function, similar to Finite Element Methods. While hp-VPINNs are generally more effective than PINNs, they are computationally intensive and do not scale well with increasing element counts. This work introduces FastVPINNs, a tensor-based framework that significantly reduces training time and handles complex geometries. Optimized tensor operations in FastVPINNs achieve up to a 100-fold reduction in median training time per epoch compared to traditional hp-VPINNs. With the right hyperparameters, FastVPINNs can outperform conventional PINNs in both speed and accuracy, particularly for problems with high-frequency solutions. The proposed method will be demonstrated with scalar and vector problems, showcasing its versatility and effectiveness in various applications.
June 14, 2024:
Presentation #1: Score-based Diffusion Models in Hilbert Spaces, Dr. Sungbin Lim, Korea University
Link: https://youtu.be/HmcjUq9DNO4?feature=shared
Abstract: Diffusion models have recently gained significant attention in probabilistic machine learning due to their theoretical properties and impressive applications in generative AI, including Stable Diffusion and DALL-E. This talk will provide a brief introduction to the theory of score-based diffusion models in Euclidean space. It will also present recent findings on score-based generative modeling in infinite-dimensional spaces, based on the time reversal theory of diffusion processes in Hilbert space.
June 7, 2024:
Presentation #1: On the Mathematical Foundations of Deep Learning Methods for Solving Partial Differential Equations, Dr. Aras Bacho, Ludwig-Maximillians University of Munich
Link: https://youtu.be/XkZ_IX_0y7Q?feature=shared
Abstract: Partial Differential Equations are essential for modeling phenomena across various domains, including physics, engineering, and finance. However, despite centuries of theoretical evolution, solving PDEs remains a challenge, both from theoretical and numerical perspectives. Traditional approaches, such as Finite Element Methods, Finite Difference Methods, and Spectral Methods, often reach their limits when faced with problems in high dimensions and with significant nonlinearity. The advent of high computational power and the availability of large datasets have made Machine Learning methods, particularly Deep Learning, a hope for practically overcoming these obstacles. Innovations such as Physics-Informed Neural Networks, Operator Networks, Neural Operators, not the Deep Ritz Method, among others, offer new pathways. Yet, the theoretical foundation of these methods is still in its infancy. In this presentation, Dr. Aras Bacho will present some recently obtained theoretical results underpinning such methods.
May 31, 2024:
Presentation #1: From Optimization to Generalization Analysis for Deep Information Bottleneck, Dr. Shujian Yu, Vrije Universiteit Amsterdam
Link: https://youtu.be/YoRQb3-veMs?feature=shared
Abstract: The information bottleneck (IB) approach is popular to improve the generalization of deep neural networks (DNNs). Essentially, it aims to find a minimum sufficient representation t from input variable x that is relevant for predicting a desirable response variable y, by striking a trade-off between a compression term I(x;t) and a prediction term I(y;t), where I refers to the mutual information (MI). However, optimizing IB remains a challenging problem. In this talk, Dr. Shujian Yu first discusses the IB principle for the regression problem and develop a new way to parameterize IB with DNNs, by replacing the Kullback-Leibler (KL) divergence with the Cauchy-Schwarz (CS) divergence. By doing so, Dr. Yu moves away from the mean squared error (MSE) loss-based regression and eases estimation of MI terms by avoiding variational approximations or distributional assumptions. Dr. Yu observes the improved generalization ability of his proposed CS-IB in benchmark datasets. Dr. Yu then delves deeper to demonstrate the benefits of the IB method by relating the compression term I(x;t) to generalization errors using a recently developed generalization error bound. Finally, Dr. Yu discusses enhancing this bound by substituting I(x;t) with loss entropy, which not only offers computational tractability but also provides quantitatively tighter estimates, particularly for large neural networks.
Presentation #2: Exploring the applicability and the optimization process of Physics Informed Neural Networks, Jorge Urbán Gutiérrez, University of Alicante & University of Valencia
Link: https://youtu.be/YoRQb3-veMs?feature=shared
Abstract: Recent advancements in Physics-Informed Neural Networks (PINNs) have positioned them as serious contenders in the domain of computational physics, challenging the longstanding monopoly held by classical numerical methods. Their disruptive potential stems from their innate ability to integrate domain-specific physics principles with the powerful learning capabilities of neural networks. Jorge Urbán Gutiérrez studies the applicability of PINNs for diverse scenarios, such as simultaneous solution of partial differential equations under varied boundary conditions and source terms, or problems where solving the differential equations are difficult to implement in finite differences. Furthermore, by introducing minor but mathematically motivated changes into the optimization process, Jorge Urbán Gutiérrez substantially improves the accuracy of PINNs for a variety of physical problems, suggesting ample room for advancement in this field.
May 24, 2024:
Presentation #1: Physics-enhanced deep surrogate models for partial differential equations, Raphael Pestourie, Georgia Tech
Link: https://youtu.be/4PP6074RO1M?feature=shared
Abstract: Surrogate models leverage data to efficiently predict a property of a partial differential equation. By accelerating the evaluation of a target property, they enable the discovery of new engineering solutions. However, in the context of supervised learning, the benefit of surrogate models is hampered by their training costs. Often dominated by the cost of the data generation, the curse of dimensionality makes the training costs prohibitive as the number of input parameters increases. Dr. Pestourie will present physics- enhanced deep surrogate models (PEDS) which combine a neural network generator and a low-fidelity solver for partial differential equations. Trained end-to-end to match high-fidelity data, the neural network learns to generate the input that will make the low- fidelity solver accurate for the target property. The geometries that are generated by the neural network can be inspected and interpreted because they are the inputs of a physical simulation. The low-fidelity solver introduces a physical bias by computing the low-fidelity solution of the governing partial differential equation. In low-data regimes, Dr. Pestourie shows on several examples that PEDS reduces the data need by at least two orders of magnitude compared to a supervised neural network. The low-fidelity solver makes PEDS slower than a neural network. However, Dr. Pestourie reports for multiple examples that PEDS is 100 to 10’000 times faster than the high-fidelity solvers. Many questions remain open regarding this methodology. Dr. Pestourie will present some insights on why it works and discuss challenges and future opportunities.
Presentation #2: From Theory to Therapy: Leveraging Universal Physics-Informed Neural Networks for Model Discovery in Quantitative Systems Pharmacology, Mohammad Kohandel, University of Waterloo
Link: https://youtu.be/4PP6074RO1M?feature=shared
Abstract: Physics-Informed Neural Networks (PINNs) have demonstrated remarkable capabilities in reconstructing solutions for differential equations and performing parameter estimations. This talk introduces Universal Physics-Informed Neural Networks (UPINNs), an advanced variant of PINNs that includes an additional neural network designed to identify unknown, hidden terms within differential equations. UPINNs are particularly effective at uncovering these hidden terms from sparse and noisy data. Furthermore, UPINNs can be integrated with symbolic regression to derive closed-form expressions for these terms. The presentation will explore how UPINNs are applied to model the dynamics of chemotherapy drugs, an area primarily addressed by Quantitative Systems Pharmacology (QSP). QSP often requires extensive manual analysis and relies on simplifying assumptions. By utilizing UPINNs, we identify the unknown components in the differential equations that dictate chemotherapy pharmacodynamics, enhancing model accuracy with both synthetic and real experimental data
May 17, 2024:
Presentation #1: Hyperdimensional Computing for Efficient, Robust, and Interpretable Cognitive Learning, Dr. Mohsen Imani, University of California Irvine
Link: N/A
Abstract: There are several challenges with today’s AI systems, including lack of interpretability, being extremely data-hungry, and inefficiency in performing learning tasks. In this talk, Dr. Mohsen Imani will present a new brain-inspired computing system that supports various learning and cognitive tasks while offering transparency and significantly higher computational efficiency and robustness than existing platforms. Dr Imani’s platform utilizes HyperDimensional Computing (HDC), an alternative computation method that implements principles of brain functionality for high-efficiency and noise-tolerant computation. HDC is motivated by the observation that the human brain operates on high-dimensional data representations. It mimics important functionalities of the human memory model with vector operations, which are computationally tractable and mathematically rigorous in describing human cognition. A key advantage of HDC is its training capability in one or a few shots, where data are learned from a few examples in a single pass over the training data, instead of requiring many iterations. These features make HDC a promising solution for today’s embedded devices with limited resources and for future computing systems facing high noise and variability issues. Dr. Imani will demonstrate how our hyperdimensional cognitive framework can detect complex scenarios, such as shoplifting, that are challenging for today’s AI systems to generalize.
May 10, 2024:
Presentation #1: Spatiotemporal Learning of High-dimensional Cell Fate, Dr. Qing Nie, University of California
Link: https://youtu.be/qwlVYnsxb9E?feature=shared
Abstract: Cells make fate decisions in response to dynamic environments, and multicellular structures emerge from multiscale interplays among cells and genes in space and time. The recent single-cell genomics technology provides an unprecedented opportunity to profile cells for all their genes. While those measurements provide high-dimensional gene expression profiles for all cells, the experimental techniques often lead to a loss of critical spatiotemporal information for individual cells. Is it possible to infer temporal relationships among cells from single or multiple snapshots? How to recover spatial
interactions among cells, for example, cell-cell communication? In this talk Qing Nie will give a short overview on our newly developed tools based on dynamical models and machine-learning methods, with a focus on inference and analysis of transitional properties of cells and cell-cell communication using both high-dimensional single-cell and spatial transcriptomics. After the overview, Dr. Yutong Sha will present details for a method called TIGON that is designed to connect a small number of snapshot datasets. This method combines high-dimensional PDEs, Optimal Transport, and machine learning approaches to reconstruct continuous temporal trajectories of high-dimensional cell fate.
Presentation #2: Discovering slow manifolds arising from fast-slow systems via Physics- Informed Neural Networks, Dr. Dimitrios Patsatzis, National Technical University of Athens
Link: https://youtu.be/qwlVYnsxb9E?feature=shared
Abstract: Slow Invariant Manifolds (SIMs) are low-dim. topological spaces parameterizing the long-term behavior of complex dynamical systems characterized
by the action of multiple timescales. The framework of Geometric Singular Perturbation Theory (GSPT) has been traditionally used for computing SIM
approximations, tackling either stiff systems where the timescale splitting was explicitly known (singularly perturbed systems), or more generally, fast-slow systems, where this information is not available. In this seminar, I will present a Physics-Informed Neural Network (PINN) approach for discovering SIM approximations in the context of GSPT for both the above classes of dynamical systems. The resulting SIM functionals are of explicit form and thus, facilitate the construction and numerical integration of reduced order models (ROMs). In comparison to classic model reduction techniques, such as QSSA, PEA and CSP, the PINN approach provides SIM approximations of equivalent or even higher approximation accuracy. Most importantly, I will demonstrate that the accuracy of the PINN approach is not affected by the magnitude of the perturbation parameter ε, or by the distance from the boundaries of the underlying SIM; to factors that critically affect the accuracy of the traditional methodologies.
May 3, 2024:
Presentation #1: Integrating PDE operators into neural network architecture in a multi-resolution manner for spatiotemporal prediction, Xin-Yang Liu, University of Notre Dame
Link: https://youtu.be/5kXiuq_sCK4?feature=shared
Abstract: Traditional data-driven deep learning models often struggle with high training costs, error accumulation, and poor generalizability for learning complex physical processes. Physics-informed deep learning (PiDL) addresses these challenges by incorporating physical principles into the model. Most PiDL approaches regularize training by embedding governing equations into the loss function, yet this process heavily depends on extensive hyperparameter tuning to balance each loss term. As an alternative strategy, Xin-Yang Liu proposes leveraging physics prior knowledge by ‘baking’ the discretized governing equations into the neural network architecture. This is achieved through the connection between the partial differential equations (PDE) operators and network structures, resulting in a neural differentiable modeling framework using differentiable programming. Embedding discretized PDEs through convolutional residual networks in a multi-resolution setting significantly improves the generalizability and long-term prediction accuracy, outperforming conventional black-box models. In this talk, Xin-Yang Liu will introduce our original multi-resolution PDE-integrated neural network architecture and its extension that is inspired by finite volume methods. This extension leverages the conservative property of finite volumes on the global scale and the strong learnability of neural operators on the local scale. Xin-Yang Liu demonstrates the effectiveness of both methods on several spatiotemporal systems governed by PDEs, including the diffusion equation, Burger’s equation, Kuramoto–Sivashinsky equations, and Navier-Stokes equations. These approaches achieve superior performance in predicting spatiotemporal dynamics, surpassing purely black-box deep learning counterparts and offering a promising avenue for emulating complex dynamic systems with improved accuracy and efficiency.
April 26, 2024:
Presentation #1: Structure-conforming Operator Learning via Transformers, Shuhao Cao, Purdue University
Link: https://youtu.be/h6d7ayfMSww?feature=shared
Abstract: GPT, Stable Diffusion, AlphaFold 2, etc., all these state-of-the-art deep learning models use a neural architecture called “Transformer”. Since the emergence of “Attention Is All You Need” paper by Google, Transformer is now the ubiquitous architecture in deep learning. At Transformer’s heart and soul is the “attention mechanism”. In this talk, we shall dissect the “attention mechanism” through the lens of traditional numerical methods, such as Galerkin
methods, and hierarchical matrix decomposition. We will report some numerical results on designing attention-based neural networks according to the structure of a problem in traditional scientific computing, such as inverse problems for Neumann-to-Dirichlet operator (EIT) or multiscale elliptic problems. Progress within different communities will be briefed to answer some open problems on the mathematical properties of the attention mechanism in
Transformers, as well as design new neural operators for a scientific computing problem.
Presentation #2: Exploring the Intersection of Diffusion Models and (Partial) Differential Equation Solving, Chieh-Hsin Lai, Sony AI
Link: https://youtu.be/h6d7ayfMSww?feature=shared
Abstract: Diffusion models, pioneers in Generative AI, have significantly propelled the creation of synthetic images, audio, 3D objects/scenes, and proteins. Beyond their role in generation, these models have found practical applications in tasks like media content editing/restoration, as well as in diverse domains such as robotics learning. In this talk, Dr. Chieh-Hsin Lai explores the origins of diffusion models and their role in solving differential equations (DE), as discussed by Song et al. in ICLR 2020. Dr. Lai introduces FP-Diffusion (Lai et al. ICML 2023), which enhances the model by aligning it with its underlying mathematical structure, the Fokker-Planck (FP) equation. Additionally, he will discuss limitations related to slow sampling speeds in thousand-step generation, motivating the introduction of the Consistency Trajectory Model (CTM) (Kim & Lai et al. ICLR 2024). The goal is to inspire mathematical research into diffusion models and deep learning methods for solving (partial) differential equations.
April 19, 2024:
Presentation #1: PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks, Sifan Wang, University of Pennsylvania
Link: https://youtu.be/Rvgn_-DFpUE?feature=shared
Abstract: While physics-informed neural networks (PINNs) have become a popular deep learning framework for tackling forward and inverse problems governed by partial differential equations (PDEs), their performance is known to degrade when larger and deeper neural network architectures are employed. Dr. Sifan Wang study identifies that the root of this counter-intuitive behavior lies in the use of multi-layer perceptron (MLP) architectures with non-suitable initialization schemes, which result in poor trainability for the network derivatives, and ultimately lead to an unstable minimization of the PDE residual loss. To address this, Dr. Wang introduces Physics-informed Residual Adaptive Networks (PirateNets), a novel architecture that is designed to facilitate stable and efficient training of deep PINN models. PirateNets leverage a novel adaptive residual connection, which allows the networks to be initialized as shallow networks that progressively deepen during training. Dr. Wang also shows that the proposed initialization scheme allows us to encode appropriate inductive biases corresponding to a given PDE system into the network architecture. Dr. Wang provides comprehensive empirical evidence showing that PirateNets are easier to optimize and can gain accuracy from considerably increased depth, ultimately achieving state-of-the-art results across various benchmarks.
Presentation #2: Tackling the Curse of Dimensionality with Physics-Informed Neural Networks, Zheyuan Hu, National University of Singapore
Link: https://youtu.be/Rvgn_-DFpUE?feature=shared
Abstract: The curse-of-dimensionality taxes computational resources heavily with exponentially increasing computational cost as the dimension increases. This poses great challenges in solving high-dimensional partial differential equations (PDEs), as Richard E. Bellman first pointed out over 60 years ago. While there has been some recent success in solving numerical PDEs in high dimensions, such computations are prohibitively expensive, and true scaling of general nonlinear PDEs to high dimensions has never been achieved. Zheyuan Hu developed new methods of scaling up physics-informed neural networks (PINNs) to solve arbitrary high-dimensional and high-order PDEs. The first new method, called Stochastic Dimension Gradient Descent (SDGD), decomposes a gradient of PDEs’ and PINNs’ residual into pieces corresponding to different dimensions and randomly samples a subset of these dimensional pieces in each iteration of training PINNs. Furthermore, inspired by the Hessian trace operator in second-order PDEs, Zheyuan introduces Hutchinson Trace Estimation (HTE) to accelerate and scale up PINN. Zheyuan demonstrates how SDGD and HTE can be unified and their difference. Lastly, with the recently developed high-dimensional PDE solvers, Zheyuan conducts extensive experiments on Hamilton-Jacobi-Bellman, Fokker-Planck, and other nonlinear PDEs. He demonstrates respective algorithms for various PDEs and scale up PINNs to 100,000 dimensions whose training can be done in a few hours or even minutes.
April 12, 2024:
Presentation #1: Stochastic Thermodynamics of Learning Parametric Probabilistic Models, Shervin Parsi, City University of New York
Link: https://youtu.be/9H2jVWWKFGM?feature=shared
Abstract: Dr. Shervin Parsi has formulated a family of machine learning problems as the time evolution of parametric probabilistic models (PPMs), inherently rendering a thermodynamic process. His primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. Dr. Parsi first introduces two information-theoretic metrics, memorized information (M-info) and learned information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and the parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.
Presentation #2: Resolution invariant deep operator network for PDEs with complex geometries, Yue Qiu, College of Mathematics and Statistics of Chongqing University
Link: https://youtu.be/9H2jVWWKFGM?feature=shared
Abstract: Neural operators (NO) are discretization invariant deep learning methods with functional output and can approximate any continuous operators. NO has demonstrated the superiority of solving partial differential equations (PDEs) over other deep learning methods. However, the domain of its input function needs to be identical to its output, which limits its applicability. For instance, the widely used Fourier neural operator (FNO) fails to approximate the operator that maps the boundary condition to the PDE solution. To address this issue, Dr. Yue Qiu proposes a novel framework called resolution-invariant deep operator (RDO) that decouples the spatial domain of the input and output. RDO is motivated by the Deep operator network (DeepONet) and it does not require retraining the network when the input/output is changed compared with DeepONet. RDO takes functional input and its output is also functional so that it keeps the resolution invariant property of NO. It can also resolve PDEs with complex geometries whereas NO fails. Various numerical experiments demonstrate the advantage of our method over DeepONet and FNO.
April 5, 2024:
Presentation #1: U-Net-PINN for 3D lithographic simulations and nano-optical design, Vlad Medvedev, Fraunhofer IISB
Link: https://youtu.be/Q82Es_qA1Os?feature=shared
Abstract: The increasing demands on computational lithography and imaging in the design and optimization of lithography processes necessitate rigorous modeling of EUV light diffracted from the mask. Traditional numerical solvers are inefficient for large-scale technology problems, while deep neural networks rely on a huge amount of expensive rigorously simulated or measured data. To overcome these constraints, Dr. Medvedev explore the potential of physics-informed neural networks (PINN) as a promising solution for addressing complex optical problems in EUV lithography and accurate modeling of light diffraction from reflective EUV masks. The coupling of the predicted diffraction spectrum with image simulations enables the evaluation of PINN performance in terms of relevant lithographic metrics. The capabilities of the established PINNs approach to simulate typical 3D mask effects including non-telecentricities, shifts of the best focus position, and image blur will be demonstrated. Dr. Medvedev’s study proves a real benefit of PINN: differently from numerical solvers, once trained, generalized PINN can simulate light scattering in several milliseconds without re-training and independently of problem complexity.
Presentation #2: Exploring the Frontiers of Computational Medicine, Yixiang Deng, Ragon Institute of Mass General, MIT, and Harvard University
Link: https://youtu.be/Q82Es_qA1Os?feature=shared
Abstract: Computational models have greatly improved how we understand complex biological systems. Yet, the variety of these systems prohibits a one-size-fits-all solution. Hence, to effectively tackle the specific challenges posed by varying contexts within computational medicine, we must tailor our computational strategies whether they be data-driven, knowledge-driven, or a hybrid approach integrating the two. In this talk, Dr. Deng will dissect the unique strengths and situational superiority of each modeling paradigm in computational medicine. First, I will show how to provide accurate predictions and distill novel biological knowledge using data-driven models. Next, Dr. Deng will demonstrate how to validate observed disease-mediated changes in blood rheology via knowledge-driven models. Additionally, Dr. Deng will also discuss patient-specific decision-making enabled by a hybrid model. The doctor will conclude this discussion by focusing on the crucial factors, such as age and sex, that are essential to tailoring treatments in precision medicine, and how to synergistically integrate data-driven, knowledge-driven, and hybrid models to tackle these challenges.”
March 29, 2024:
Presentation #1: Modeling Fracture using Physics-Informed Deep Learning, Manav Manav, ETH Zurich
Link: https://youtu.be/mB1lWmecbro?feature=shared
Abstract: Phase-field modeling of fracture, a promising approach to model fracture, recasts the problem of fracture as a variational problem which completely determines the fracture process including crack nucleation, propagation, bifurcation, and coalescence, and obviates the need for ad-hoc conditions. In this approach, a phase field is introduced which regularizes a crack. It is, however, a nonlocal model which introduces a small length scale. Resolving this length scale in computation is expensive. Hence, uncertainty quantification, design optimization, material parameter identification, among others, using this approach become prohibitively expensive. Deep learning offers a potential pathway to address this challenge.
As an initial step in this direction, we explore the application of physics-informed deep learning to phase-field fracture modeling with the aim to capture various fracture processes [1]. Nonconvexity of the variational energy, and initiation and evolution of the fields with sharp gradients governed by this energy are the two key challenges to learning the solution field. Dr. Manav uses deep Ritz method (DRM) in which training of the network representing the solution field proceeds by directly minimizing the variational energy of the system. Guided by the challenges, Dr. Manav constructs a network and select an optimization scheme to learn the solution accurately. Dr. Manav also elucidates the challenges in learning the solution field with the same level of domain discretization as needed in finite element analysis and suggest ways to overcome it. Finally, Dr. Manav solves some benchmark problems in phase-field fracture literature, exhibiting the capability of the approach to capture crack nucleation, propagation, kinking, branching, and coalescence. The details of the model and the challenges in obtaining the correct solution will be discussed.
References:
[1] Manav, M., Molinaro, R., Mishra, S., & De Lorenzis, L. “Phase-field modeling of complex fracture processes using physics-informed deep learning,” In preparation.
March 22, 2024:
Presentation #1: Domain decomposition for physics-informed neural networks, Alexander Heinlein, Delft University of Technology
Link: https://youtu.be/087Y9pLFNqI?feature=shared
Abstract: Physics-informed neural networks (PINNs) are a class of methods for solving differential equation-based problems using a neural network as the discretization. They have been introduced by Raissi et al. in [6] and combine the pioneering collocation approach for neural network functions introduced by Lagaris et al. in [4] with the incorporation of data via an additional loss term. PINNs are very versatile as they do not require an explicit mesh, allow for the solution of parameter identification problems, and are well-suited for high-dimensional problems. However, the training of a PINN model is generally not very robust and may require a lot of hyper parameter tuning. In particular, due to the so-called spectral bias, the training of PINN models is notoriously difficult when scaling up to large computational domains as well as for multiscale problems. In this talk, overlapping domain decomposition-based techniques for PINNs are being discussed. Compared with other domain decomposition techniques for PINNs, in the finite basis physics-informed neural networks (FBPINNs) approach [5], the coupling is done implicitly via the overlapping regions and does not require additional loss terms. Using the classical Schwarz domain decomposition framework, a very general framework, that also allows for mult-level extensions, can be introduced [1]. The method outperforms classical PINNs on several types of problems, including multiscale problems, both in terms of accuracy and efficiency. Furthermore, the combination of the multi-level domain decomposition strategy with multifidelity stacking PINNs [3], as introduced in [2] for time-dependent problems, will be discussed. It can be observed that the combination of multifidelity stacking PINNs with a domain decomposition in time clearly improves the reference results without a domain decomposition.
References: Dolean, Victorita, et al. “Multilevel domain decomposition-based architectures for physics-informed neural networks.” arXiv preprint arXiv:2306.05486 (2023). Heinlein, Alexander, et al. “Multifidelity domain decomposition-based physics-informed neural networks for time-dependent problems.” arXiv preprint arXiv:2401.07888 (2024). Howard, Amanda A., et al. “Stacked networks improve physics-informed training: applications to neural networks and deep operator networks.” arXiv preprint arXiv:2311.06483 (2023). Lagaris, Isaac E., Aristidis Likas, and Dimitrios I. Fotiadis. “Artificial neural networks for solving ordinary and partial differential equations.” IEEE transactions on neural networks 9.5 (1998): 987-1000. Moseley, Ben, Andrew Markham, and Tarje Nissen-Meyer. “Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations.” Advances in Computational Mathematics 49.4 (2023): 62. Raissi, Maziar, Paris Perdikaris, and George E. Karniadakis. “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.” Journal of Computational physics 378 (2019): 686-707.
Presentation #2: Physics-based and data-driven methods for precision medicine in computational cardiology, Matteo Salvador, Stanford University
Link: https://youtu.be/087Y9pLFNqI?feature=shared
Abstract: In recent years, blending physics-based modeling with data-driven methods has had a major impact on computational medicine. Several frameworks have been proposed to create certified digital replicas of the cardiovascular system. These computational pipelines include multiscale and multiphysics mathematical models based on rigorous differential equations, scientific machine learning methods to build accurate and efficient surrogate models, sensitivity analysis, and robust parameter estimation with uncertainty quantification. In this seminar, we will use cardiac mathematical models for electrophysiology, active and passive mechanics, and hemodynamics, combined with various artificial intelligence-based methods, such as Latent Neural Ordinary Differential Equations, Branched Latent Neural Maps, and Latent Dynamics Networks, to learn complex time and space-time physical processes underlying these systems
of ordinary and partial differential equations. Dr. Salvador will use these reduced-order models to infer physics-based parameters from cell to organ scale with uncertainty quantification in a Bayesian framework, while fitting clinical data such as 12-lead electrocardiograms and pressure-volume loops for human hearts. These computational tools define important contributions for digital twinning in computational cardiology.
March 15, 2024:
Presentation #1: A Python module for easily and efficiently solving problems with the Theory of Functional Connections, Carl Leake, Texas A&M University
Link: https://youtu.be/qDB66Vt1JH4?feature=shared
Abstract: Theory of Functional Connections (TFC) is a functional interpolation framework that can be used to solve a wide variety of problems, e.g., boundary value problems. The tfc Python module, the focus of this talk, is designed to help its users solve problems with TFC easily and efficiently: easily here refers to the time it takes the user to write a Python script to solve their problem and efficiently refers to the computational efficiency of said script. The tfc module leverages the automatic differentiation and just-in-time compilation capabilities of the JAX library to do this. In addition, the module provides other convenience, quality-of-life, and sanity-checking capabilities that reduce/alleviate the most common errors users make when numerically solving problems with TFC.
March 8, 2024:
Presentation #1: Can Physics-Informed Neural Networks beat the Finite Element Method?, Jonas Latz, University of Manchester
Link: https://youtu.be/bgsqCTgF24w?feature=shared
Abstract: Partial differential equations play a fundamental role in the mathematical modelling of many processes and systems in physical, biological and other sciences. To simulate such processes and systems, the solutions of PDEs often need to be approximated numerically. The finite element method, for instance, is a usual standard methodology to do so. The recent success of deep neural networks at various approximation tasks has motivated their use in the numerical solution of PDEs. These so-called physics-informed neural networks and their variants have shown to be able to successfully approximate a large range of partial differential equations. So far, physics-informed neural networks and the finite element method have mainly been studied in isolation of each other. In this work, Dr. Latz compares the methodologies in a systematic computational study. Dr. Latz employed both methods to numerically solve various linear and nonlinear partial differential equations: Poisson in 1D, 2D, and 3D, Allen-Cahn in 1D, semilinear Schrödinger in 1D and 2D. He then compared computational costs and approximation accuracies. In terms of solution time and accuracy, physics-informed neural networks have not been able to outperform the finite element method in our study. In some experiments, they were faster at evaluating the solved PDE.
Presentation #2: On flows and diffusions: from many-body Fokker-Planck to stochastic interpolants, Nicholas Boffi, Courant Institute of Mathematical Sciences
Link: https://youtu.be/bgsqCTgF24w?feature=shared
Abstract: Given a stochastic differential equation, its corresponding Fokker-Planck equation is generically intractable to solve, because its high dimensionality prohibits the application of standard numerical techniques. In this talk, Dr. Boffi will exploit an analogy between the Fokker-Planck equation and modern generative models from machine learning to develop an algorithm for its solution in high dimension. The method enables the computation of previously intractable quantities of interest, such as the entropy production rate of active matter systems, which quantifies the magnitude of nonequilibrium effects. Dr. Boffi will then highlight how insight from the Fokker-Planck equation facilitates the development of a new class of generative models known as stochastic interpolants, which generalize state-of-the-art diffusion models in several key ways that can be leveraged to improve practical performance.
March 1, 2024:
Presentation #1: Lax pairs informed neural networks solving integrable systems, Chen Yong, East China Normal University
Link: https://youtu.be/rKvekSv8j0Q?feature=shared
Abstract: Lax pairs are one of the most important features of integrable system. In this talk, Dr. Yong proposes the Lax pairs informed neural networks (LPINNs) tailored for integrable systems with Lax pairs by designing novel network architectures and loss functions, comprising LPINN-v1 and LPINN-v2. The most noteworthy advantage of LPINN-v1 is that it can transform the solving of complex integrable systems into the solving of a simpler Lax pairs to simplify the study of integrable systems, and it not only efficiently solves data-driven localized wave solutions, but also obtains spectral parameters and corresponding spectral functions in Lax pairs. On the basis of LPINN-v1, Dr. Yong additionally incorporates the compatibility condition/zero curvature equation of Lax pairs in LPINN-v2, its major advantage is the ability to solve and explore high-accuracy data-driven localized wave solutions and associated spectral problems for all integrable systems with Lax pairs. The numerical experiments in this work involve several important and classic low-dimensional and high-dimensional integrable systems, abundant localized wave solutions and their Lax pairs , including the soliton of the Korteweg-de Vries (KdV) equation and modified KdV equation, rogue wave solution of the nonlinear Schrodinger equation, kink solution of the sine-Gordon equation, non-smooth peakon solution of the Camassa-Holm equation and pulse solution of the short pulse equation, as well as the line-soliton solution Kadomtsev-Petviashvili equation and lump solution of high-dimensional KdV equation. The innovation of this work lies in the pioneering integration of Lax pairs informed of integrable systems into deep neural networks, thereby presenting a fresh methodology and pathway for investigating data-driven localized wave solutions and spectral problems of Lax pairs.
February 23, 2024:
Presentation #1: Density physics-informed neural networks reveal sources of cell heterogeneity in signal transduction, Jae Kyoung Kim, KAIST
Link: https://youtu.be/dq_-iUrMhiY?feature=shared
Abstract: In this talk, Dr. Jae Kyoung Kim introduces Density-Physics Informed Neural Networks (Density-PINNs) for inferring probability distributions from timeseries data. Density-PINNs leverage Rayleigh distributions as kernel and a variational autoencoder for noise filtering. Dr. Kim demonstrates the power of Density-PINNs by analyzing single-cell gene expression data from sixteen promoters regulated by unknown pathways during antibiotic stress response. By inferring the probability distributions of gene expression patterns, Density-PINNs successfully identify key signaling pathways crucial for consistent cellular responses, offering a valuable strategy for treatment optimization.
February 16, 2024:
Presentation #1: DeepOnet Based Preconditioning Strategies For Solving Parametric Linear Systems of Equations, Alena Kopanicakova, Brown University
Link: https://youtu.be/_ziSqwA8NzM?feature=shared
Abstract: Dr. Kopanicakova introduces a new class of hybrid preconditioners for solving parametric linear systems of equations. The proposed preconditioners are constructed by hybridizing the deep operator network, namely DeepONet, with standard iterative methods. Exploiting the spectral bias, DeepONet-based components are harnessed to address low-frequency error components, while conventional iterative methods are employed to mitigate high-frequency error components. Dr. Kopanicakova’s preconditioning framework comprises two distinct hybridization approaches: direct preconditioning (DP) and trunk basis (TB) approaches. In the DP approach, DeepONet is used to approximate an action of an inverse operator to a vector during each preconditioning step. In contrast, the TB approach extracts basis functions from the trained DeepONet to construct a map to a smaller subspace, in which the low-frequency component of the error can be effectively eliminated. Dr. Kopanicakova’s numerical results demonstrate that utilizing the TB approach enhances the convergence of Krylov methods by a large margin compared to standard non-hybrid preconditioning strategies. Moreover, the proposed hybrid preconditioners exhibit robustness across a wide range of model parameters and problem resolutions.
February 9, 2024:
Presentation #1: Neural oscillators for generalization of physics-informed machine learning, Taniya Kapoor, TU Delft
Link: https://youtu.be/zJExHI-MYvE?feature=shared
Abstract: A primary challenge of physics-informed machine learning (PIML) is its generalization beyond the training domain, especially when dealing with complex physical problems represented by partial differential equations (PDEs). This paper aims to enhance the generalization capabilities of PIML, facilitating practical, real-world applications where accurate predictions in unexplored regions are crucial. Taniya Kapoor leverages the inherent causality and temporal sequential characteristics of PDE solutions to fuse PIML models with recurrent neural architectures based on systems of ordinary differential equations, referred to as neural oscillators. Through effectively capturing long-time dependencies and mitigating the exploding and vanishing gradient problem, neural oscillators foster improved generalization in PIML tasks. Extensive experimentation involving time-dependent nonlinear PDEs and biharmonic beam equations demonstrates the efficacy of the proposed approach. Incorporating neural oscillators outperforms existing state-of-the-art methods on benchmark problems across various metrics. Consequently, the proposed method improves the generalization capabilities of PIML, providing accurate solutions for extrapolation and prediction beyond the training data.
February 2, 2024:
Presentation #1: Efficient and Physically Consistent Surrogate Modeling of Chemical Kinetics Using Deep Operator Networks, Anuj Kumar, North Carolina State University
Link: https://youtu.be/UYzU7q37tPk?feature=shared
Abstract: In the talk, Anuj Kumar explores a new combustion chemistry acceleration scheme he has developed for reacting flow simulations, utilizing deep operator networks (DeepONets). The scheme, implemented on a subset of thermochemical scalars crucial for chemical system’s evolution, advances the current solution vector by adaptive time steps. In addition, the original DeepONet architecture is modified to incorporate the parametric dependence of these stiff ODEs associated with chemical kinetics. Unlike previous DeepONet training approaches, his training is conducted over short time windows, using intermediate solutions as initial states. An additional framework of latent-space kinetics identification with modified DeepONet is proposed, which enhances the computational efficiency and widens the applicability of the proposed scheme. The scheme is demonstrated on the “simple” chemical kinetics of hydrogen oxidation and the more complex chemical kinetics of n-dodecane high-and low-temperatures. The proposed framework accurately learns the chemical kinetics and efficiently reproduces species and temperature temporal profiles. Moreover, a very large speed-up with a good extrapolation capability is also observed with the proposed scheme. Additional framework of incorporating physical constraints such as total mass and elemental conservation, into the training of DeepONet for subset of thermochemical scalars of complex reaction mechanisms is proposed. Levering the strong correlation between full and subset of scalars, the framework establishes an accurate and physically consistent mapping. The framework is demonstrated on the chemical kinetics of CH4 oxidation.
Presentation #2: SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training, Kazem Meidani, Carnegie Mellon University
Link: https://youtu.be/UYzU7q37tPk?feature=shared
Abstract: In an era where symbolic mathematical equations are indispensable for modeling complex natural phenomena, scientific inquiry often involves collecting observations and translating them into mathematical expressions. Recently, deep learning has emerged as a powerful tool for extracting insights from data. However, existing models typically specialize in either numeric or symbolic domains and are usually trained in a supervised manner tailored to specific tasks. This approach neglects the substantial benefits that could arise from a task-agnostic unified understanding between symbolic equations and their numeric counterparts. To bridge the gap, we introduce SNIP, a Symbolic-Numeric Integrated Pre-training, which employs joint contrastive learning between symbolic and numeric domains, enhancing their mutual similarities in the pre-trained embeddings. By performing latent space analysis, Dr. Meidani observes that SNIP provides cross-domain insights into the representations, revealing that symbolic supervision enhances the embeddings of numeric data and vice versa. Kazem evaluates SNIP across diverse tasks, including symbolic-to-numeric mathematical property prediction and numeric-to-symbolic equation discovery, commonly known as symbolic regression. Results show that SNIP effectively transfers to various tasks, consistently outperforming fully supervised baselines and competing strongly with established task-specific methods, especially in few-shot learning scenarios where available data is limited.
January 26, 2024:
Presentation #1: Physics-informed neural networks for quantum control, Dr. Ariel Norambuena, Pontifical Catholic University
Link: https://youtu.be/Ci85LdBM_J0?feature=shared
Abstract: In this talk, Dr. Norambuena will introduce a computational method for optimal quantum control problems using physics-informed neural networks (PINNs). Motivated by recent advances in open quantum systems and quantum computing, he will discuss the relevance of PINNs for finding realistic and robust control fields. Through this talk, we will learn about the flexibility and universality of PINNs to solve different quantum control problems, showing the main advantages of PINNs compared to standard control techniques.
January 19, 2024:
Presentation #1: U-DeepONet: U-Net Enhanced Deep Operator Network for Geologic Carbon Sequestration, Waleed Diab, Khalifa University
Link: https://youtu.be/AUPou43OuYo?feature=shared
Abstract: Fourier Neural Operator (FNO) and Deep Operator Network (DeepONet) are by far the most popular neural operator learning algorithms. FNO seems to enjoy an edge in popularity due to its ease of use, especially with high dimensional data. However, a lesser-acknowledged feature of DeepONet is its modularity. This feature allows the user the flexibility of choosing the kind of neural network to be used in the trunk and/or branch of the DeepONet. This is beneficial because it has been shown many times that different types of problems require different kinds of network architectures for effective learning. In this work, Waleed Diab will take advantage of this feature by carefully designing a more efficient neural operator based on the DeepONet architecture. Waleed will introduce U-Net enhanced DeepONet (U-DeepONet) for learning the solution operator of highly complex CO2-water two-phase flow in heterogeneous porous media. The U-DeepONet is more accurate in predicting gas saturation and pressure buildup than the state-of-the-art U-Net based Fourier Neural Operator (U-FNO) and the Fourier-enhanced Multiple-Input Operator (Fourier-MIONet) trained on the same dataset. In addition, the proposed U-DeepONet is significantly more efficient in training times than both the U-FNO (more than 18 times faster) and the Fourier-MIONet (more than 5 times faster), while consuming less computational resources. Waleed also shows that the U-DeepONet is more data efficient and better at generalization than both the U-FNO and the Fourier-MIONet.
January 12, 2024:
Presentation #1: PPDONet: Deep Operator Networks for forward and inverse problems in astronomy, Shunyuan Mao, University of Victoria
Link: https://youtu.be/_IhB9R33zCk?feature=shared
Abstract: This talk presents Shunyuan Mao’s research on applying Deep Operator Networks (DeepONets) to fluid dynamics in astronomy. The focus is specifically on protoplanetary disks — the gaseous disks surrounding young stars, which are the birthplaces of planets. The physical processes in these disks are governed by Navier-Stokes (NS) equations. Traditional numerical methods for solving these equations are computationally expensive, especially when modeling multiple systems for tasks such as exploring parameter spaces or inferring parameters from observations. Shunyuan Mao addresses this issue by using DeepONets to rapidly map PDE parameters to their solutions. His development, Protoplanetary Disk Operator Network (PPDONet), significantly reduces computational cost, predicting field solutions within seconds— a task that would typically require hundreds of CPU hours. The utility of this tool is demonstrated in two key applications: 1) Its swift solution predictions facilitate the exploration of relationships between PDE parameters and observables extracted from field solutions. 2) When integrated with the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), DeepONets effectively address inverse problems by efficiently inferring PDE parameters from unseen solutions.
Presentation #2: Physics-informed neural networks for solving phonon Boltzmann transport equations, Dr. Tengfei Luo, University of Notre Dame
Link: https://youtu.be/_IhB9R33zCk?feature=shared
Abstract: The phonon Boltzmann transport equation (pBTE) has been proven to be capable of precisely predicting heat conduction in sub-micron electronic devices. However, numerically solving pBTE is extremely computationally costly due to its high dimensionality, especially when phonon dispersion and time evolution are considered. In this study, we use physics-informed neural networks (PINNs) to solve pBTE for multiscale non-equilibrium thermal transport problems both efficiently and accurately. In particular, a PINN framework is devised to predict phonon energy distribution by minimizing the residuals of governing equations, boundary conditions, and initial conditions without the need for any labeled training data. With phonon energy distribution predicted by the PINN, temperature and heat flux can be obtained thereby. In addition, geometric parameters, such as characteristic length scale, are also considered as a part of the input to PINN, which makes our model capable of predicting heat distribution in different length scales. Besides pBTE, Dr. Tengfei Luo also extended the applicability of the PINN framework for modeling coupled electron-phonon (e-ph) transport. e-ph coupling and transport are ubiquitous in modern electronic devices. The coupled electron and phonon Boltzmann transport equations (BTEs) hold great potential for the simulation of thermal transport in metal and semiconductor systems.
January 5, 2024:
Presentation #1: Neural Operator Learning Enhanced Physics-informed Neural Networks for solving differential equations with sharp solutions, Professor Mao, Xiamen University
Link: https://youtu.be/7NNyjWxp2zQ?feature=shared
Abstract: In the talk, Professor Mao shall present some numerical results for the forward and inverse problems of PDEs with sharp solutions by using deep neural network-based methods. In particular, he developed a deep operator learning enhanced PINN for PDEs with sharp solutions, which can be asymptotically approached by using problems with smooth solutions. Firstly, Mao solves the smooth problems by using deep operator learning, and adopts the framework of DeepONet. Then Professor Mao combines the pre-trained DeepONet and PINN to solve the sharp problem. Professor Mao demonstrates the effectiveness of the present method by testing several equations, including viscous Burger equation, Cavity flow as well Navier-stokes equation. Furthermore, we solve the ill-posed problems that with insufficient boundary conditions by using the present method.
Presentation #2: Physics-Informed Parallel Neural Networks with Self-Adaptive Loss
Weighting for the Identification of Structural Systems, Rui Zhang, Pennsylvania State University
Link: https://youtu.be/7NNyjWxp2zQ?feature=shared
Abstract: Rui Zhang has developed a physics-informed parallel neural networks (PIPNNs) framework for the identification of continuous structural systems described by a system of partial differential equations. PIPNNs integrate the physics of the system into the loss function of the NNs, enabling the simultaneous updating of both unknown structural and NN parameters during the process of minimizing the loss function. The PIPNNs framework accommodates structural discontinuities by dividing the computational domain into subdomains, each uniquely represented through a parallelized and interconnected NN architecture. Furthermore, the PIPNNs framework is incorporated a self-adaptive weighted loss function based on Neural Tangent Kernel (NTK) theory. The self-adaptive weights, determined based upon the eigenvalues of the NTK matrix of the PIPNNs, dynamically adjust the convergence rates of each loss term to achieve a balanced convergence, while requiring less training data. This advancement is particularly beneficial for inverse problem-solving and structural identification, as the NTK matrix reflects the training progress of both unknown structural and NN parameters. The PIPNNs framework is verified, and its accuracy is assessed through the application of numerical examples of several continuous structural systems, including bars, beams, and plates.