580 California St., Suite 400
San Francisco, CA, 94104
Key research themes
1. How do architectural innovations in recurrent neural networks improve modeling of long-term dependencies and multidimensional data?
This research area focuses on developing novel RNN architectures designed to effectively model long-term temporal dependencies and multidimensional inputs, overcoming challenges such as vanishing/exploding gradients and efficient gradient flow through time and layers. Advances include extensions of LSTM cells into multi-dimensional grids, novel recurrent units inspired by dynamical systems theory ensuring stability, and architectural designs leveraging gating mechanisms to improve learning dynamics. These innovations are crucial for complex real-world applications including natural language processing, algorithmic tasks, and image-related sequence modeling, where capturing extended temporal and spatial relationships is essential.
Key finding: Introduces Grid LSTM, extending standard LSTM cells into multidimensional grids, enabling simultaneous communication along depth and spatio-temporal dimensions; demonstrates that incorporating LSTM cells along the depth... Read more
Key finding: Proposes AntisymmetricRNN, a recurrent architecture derived by discretizing stable ordinary differential equations with antisymmetric weight matrices, ensuring well-behaved, predictable dynamics; the model effectively... Read more
Key finding: Identifies saturation-induced learning difficulties in gating mechanisms of recurrent networks and proposes two synergistic modifications — uniform gate initialization and an auxiliary refine gate — that improve gate... Read more
2. How can training methodologies and optimization algorithms be designed to better align RNNs with sequence-level objectives and improve prediction robustness?
This line of research addresses the mismatch between traditional word-level training losses and sequence-level evaluation metrics in RNN-based sequential models. Researchers devise training algorithms that incorporate reinforcement learning ideas or spectral decomposition to directly optimize metrics like BLEU or ROUGE, mitigating issues such as exposure bias. Such approaches provide theoretical guarantees and demonstrate empirical advances in generating coherent sequences across NLP tasks, while also proposing spectral algorithms for guaranteed parameter recovery in input-output RNNs, thus advancing the reliability and interpretability of RNN training.
Key finding: Proposes the MIXER algorithm combining incremental cross-entropy and REINFORCE methods to directly optimize sequence-level metrics (e.g., BLEU) for RNNs, addressing exposure bias by incorporating model predictions during... Read more
Key finding: Develops a spectral decompositional approach for consistent and efficient learning of input-output RNN parameters based on the decomposition of cross-moment tensors involving score functions, under mild assumptions including... Read more
Key finding: Demonstrates that optimizing a Minimum Description Length (MDL) objective, balancing model complexity and accuracy via a genetic algorithm, yields small, transparent RNNs capable of perfect generalization on formal languages... Read more
Key finding: Provides analytic and empirical evidence that gradient-based training of NARX recurrent neural networks, which include explicit memory of multiple delayed inputs, is more effective at learning long-term dependencies compared... Read more
3. How can unsupervised and semi-supervised recurrent latent variable models improve representation learning for sequential data?
This theme investigates models that embed RNN architectures within variational or probabilistic frameworks to learn meaningful latent representations of sequential data. By combining the temporal dynamic modeling capabilities of RNNs with variational auto-encoders, these methods enable unsupervised learning on sequence datasets and provide pretrained initializations improving supervised learning efficiency. Their generative nature allows sampling from latent spaces for sequence generation, facilitating improved modeling of complex temporal dependencies beyond deterministic RNN approaches.
Key finding: Introduces Variational Recurrent Auto-Encoder (VRAE) models that integrate RNNs with Stochastic Gradient Variational Bayes, enabling efficient unsupervised learning of latent representations for time series data; demonstrates... Read more
Key finding: By analyzing RNN hidden states as discrete-time dynamical systems, identifies slow points approximating steady states as the loci of memory formation, establishing a correlation between their stability and a network’s ability... Read more