Qiuxia Lai

Followers

Following

Public Views

Interests

Uploads

Papers by Qiuxia Lai

DeepSAT: An EDA-Driven Learning Framework for SAT

We present DeepSAT, a novel end-to-end learning framework for the Boolean satisfiability (SAT) pr... more We present DeepSAT, a novel end-to-end learning framework for the Boolean satisfiability (SAT) problem. Unlike existing solutions trained on random SAT instances with relatively weak supervisions, we propose applying the knowledge of the well-developed electronic design automation (EDA) field for SAT solving. Specifically, we first resort to advanced logic synthesis algorithms to pre-process SAT instances into optimized and-inverter graphs (AIGs). By doing so, our training and test sets have a unified distribution, thus the learned model can generalize well to test sets of various sources of SAT instances. Next, we regard the distribution of SAT solutions being a product of conditional Bernoulli distributions. Based on this observation, we approximate the SAT solving procedure with a conditional generative model, leveraging a directed acyclic graph neural network with two polarity prototypes for conditional SAT modeling. To effectively train the generative model, with the help of logic simulation tools, we obtain the probabilities of nodes in the AIG being logic '1' as rich supervisions. We conduct extensive experiments on various SAT instances. DeepSAT achieves significant accuracy improvements over state-of-the-art learning-based SAT solutions, especially when generalized to SAT instances that are large or with diverse distributions. Preprint. Under review.

Download

T-WaveNet: Tree-Structured Wavelet Neural Network for Sensor-Based Time Series Analysis

ArXiv, 2020

Sensor-based time series analysis is an essential task for applications such as activity recognit... more Sensor-based time series analysis is an essential task for applications such as activity recognition and brain-computer interface. Recently, features extracted with deep neural networks (DNNs) are shown to be more effective than conventional hand-crafted ones. However, most of these solutions rely solely on the network to extract application-specific information carried in the sensor data. Motivated by the fact that usually a small subset of the frequency components carries the primary information for sensor data, we propose a novel tree-structured wavelet neural network for sensor data analysis, namely \emph{T-WaveNet}. To be specific, with T-WaveNet, we first conduct a power spectrum analysis for the sensor data and decompose the input signal into various frequency subbands accordingly. Then, we construct a tree-structured network, and each node on the tree (corresponding to a frequency subband) is built with an invertible neural network (INN) based wavelet transform. By doing so,...

Download

TestRank: Bringing Order into Unlabeled Test Instances for Deep Learning Tasks

ArXiv, 2021

Deep Learning (DL) models have achieved unprecedented success in various tasks and are pervasivel... more Deep Learning (DL) models have achieved unprecedented success in various tasks and are pervasively deployed in real-world applications. It is critical to guarantee their correctness by testing their behaviors. However, DL systems are notoriously difficult to test and debug due to the lack of explainability and the huge test input space to cover. Generally speaking, it is relatively easy to collect a massive amount of test data, but the labeling cost can be quite high. Consequently, it is essential to conduct test selection and label only those selected ‘high quality’ bug-revealing test inputs for test cost reduction. In this paper, we propose a novel test prioritization technique that brings order into the unlabeled test instances according to their bug-revealing capabilities, namely TestRank. Different from existing solutions, TestRank leverages both intrinsic attributes and contextual attributes of test instances when prioritizing them. To be specific, we first build a similarity ...

Download

Human vs Machine Attention in Neural Networks: A Comparative Study

ArXiv, 2019

Human visual system can selectively attend to parts of a scene for quick perception, a biological... more Human visual system can selectively attend to parts of a scene for quick perception, a biological mechanism known as Human attention. Inspired by this, recent deep learning models encode attention mechanisms to focus on the most task-relevant parts of the input signal for further processing, which is called Machine/Neural/Artificial attention. Understanding the relation between human and machine attention is important for interpreting and designing neural networks. Many works claim that the attention mechanism offers an extra dimension of interpretability by explaining where the neural networks look. However, recent studies demonstrate that artificial attention maps do not always coincide with common intuition. In view of these conflicting evidence, here we make a systematic study on using artificial attention and human attention in neural network design. With three example computer vision tasks, diverse representative backbones, and famous architectures, corresponding real human ga...

Download

Information Bottleneck Approach to Spatial Attention Learning

The selective visual attention mechanism in the human visual system (HVS) restricts the amount of... more The selective visual attention mechanism in the human visual system (HVS) restricts the amount of information to reach visual awareness for perceiving natural scenes, allowing near real-time information processing with limited computational capacity [Koch and Ullman, 1987]. This kind of selectivity acts as an ‘Information Bottleneck (IB)’, which seeks a trade-off between information compression and predictive accuracy. However, such information constraints are rarely explored in the attention mechanism for deep neural networks (DNNs). In this paper, we propose an IB-inspired spatial attention module for DNN structures built for visual recognition. The module takes as input an intermediate representation of the input image, and outputs a variational 2D attention map that minimizes the mutual information (MI) between the attention-modulated representation and the input, while maximizing the MI between the attentionmodulated representation and the task label. To further restrict the in...

Download

Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction

ArXiv, 2021

Time series is a special type of sequence data, a set of observations collected at even time inte... more Time series is a special type of sequence data, a set of observations collected at even time intervals and ordered chronologically. Existing deep learning techniques use generic sequence models (e.g., recurrent neural network, Transformer model, or temporal convolutional network) for time series analysis, which ignore some of its unique properties. In particular, three components characterize time series: trend, seasonality, and irregular components, and the former two components enable us to perform forecasting with reasonable accuracy. Other types of sequence data do not have such characteristics. Motivated by the above, in this paper, we propose a novel neural network architecture that conducts sample convolution and interaction for temporal modeling and apply it for the time series forecasting problem, namely SCINet. Compared to conventional dilated causal convolution architectures, the proposed downsample-convolve-interact architecture enables multi-resolution analysis besides ...

Download

Multi-video object synopsis integrating optimal view switching

SIGGRAPH Asia 2017 Technical Briefs, 2017

In this paper, we propose to integrate multi-video object synopsis with the display method of opt... more In this paper, we propose to integrate multi-video object synopsis with the display method of optimal multi-view switching. We group occurrences of the same object in different cameras together and operate them in the same way, based on which we propose a unified framework that optimizes object-based synopsis variables and multi-view switching variables simultaneously. The unified formulation encodes activity, occlusion, chronological order, and view switching costs. We develop an alternative optimization scheme composed of graph cuts and dynamic programming to minimize the combined costs efficiently. This integrated framework provides a more user convenient browsing approach for object-based synopsis results, and allows to further reduce occlusion artifacts between synopsized objects. Experiments demonstrate the effectiveness and user convenience of our multi-video synopsis method.

Temporal Coherent Video Super-resolution via Pre-frame-constrained Sparse Reconstruction

In this paper, we extend the sparse representation based image super-resolution method to process... more In this paper, we extend the sparse representation based image super-resolution method to process videos, mainly aiming at obtaining temporally consistent consecutive high-resolution (HR) video frames. In our formulation, the previous estimated HR frame is used to guide the sparse reconstruction of current low-resolution (LR) frame, which is able to obtain more consistent representations. We show that such guidance is robust and effective by incorporating with a non-rigid dense correspondence based motion compensation schema. We also propose a dictionary updating strategy which regularly updates the dictionaries that are critical for the sparse representation procedure using the newly reconstructed HR frames. To further preserve sharp edges and remove reconstruction errors, once a HR image is recovered, we refine it with a L0-norm based optimization that constrains the final HR output with relatively sparse gradients. Experimental results on natural videos demonstrated the effective...

Understanding More About Human and Machine Attention in Deep Neural Networks

IEEE Transactions on Multimedia, 2021

Human visual system can selectively attend to parts of a scene for quick perception, a biological... more Human visual system can selectively attend to parts of a scene for quick perception, a biological mechanism known as Human attention. Inspired by this, recent deep learning models encode attention mechanisms to focus on the most task-relevant parts of the input signal for further processing, which is called Machine/Neural/Artificial attention. Understanding the relation between human and machine attention is important for interpreting and designing neural networks. Many works claim that the attention mechanism offers an extra dimension of interpretability by explaining where the neural networks look. However, recent studies demonstrate that artificial attention maps do not always coincide with common intuition. In view of these conflicting evidence, here we make a systematic study on using artificial attention and human attention in neural network design. With three example computer vision tasks (i.e., salient object segmentation, video action recognition, and fine-grained image cla...

MixDefense: A Defense-in-Depth Framework for Adversarial Example Detection Based on Statistical and Semantic Analysis

ArXiv, 2021

Machine learning with deep neural networks (DNNs) has become one of the foundation techniques in ... more Machine learning with deep neural networks (DNNs) has become one of the foundation techniques in many safety-critical systems, such as autonomous vehicles and medical diagnosis systems. DNN-based systems, however, are known to be vulnerable to adversarial examples (AEs) that are maliciously perturbed variants of legitimate inputs. While there has been a vast body of research to defend against AE attacks in the literature, the performances of existing defense techniques are still far from satisfactory, especially for adaptive attacks, wherein attackers are knowledgeable about the defense mechanisms and craft AEs accordingly. In this work, we propose a multilayer defense-in-depth framework for AE detection, namely MixDefense. For the first layer, we focus on those AEs with large perturbations. We propose to leverage the `noise' features extracted from the inputs to discover the statistical difference between natural images and tampered ones for AE detection. For AEs with small perturbations, the inference result of such inputs would largely deviate from their semantic information. Consequently, we propose a novel learning-based solution to model such contradictions for AE detection. Both layers are resilient to adaptive attacks because there do not exist gradient propagation paths for AE generation. Experimental results with various AE attack methods on image classification datasets show that the proposed MixDefense solution outperforms the existing AE detection techniques by a considerable margin.

Download

DeepFuse: An IMU-Aware Network for Real-Time 3D Human Pose Estimation from Multi-View Image

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

In this paper, we propose a two-stage fully 3D network, namely DeepFuse, to estimate human pose i... more In this paper, we propose a two-stage fully 3D network, namely DeepFuse, to estimate human pose in 3D space by fusing body-worn Inertial Measurement Unit (IMU) data and multi-view images deeply. The first stage is designed for pure vision estimation. To preserve data primitiveness of multi-view inputs, the vision stage uses multi-channel volume as data representation and 3D soft-argmax as activation layer. The second one is the IMU refinement stage which introduces an IMU-bone layer to fuse the IMU and vision data earlier at data level. without requiring a given skeleton model a priori, we can achieve a mean joint error of 28.9mm on TotalCapture dataset and 13.4mm on Hu-man3.6M dataset under protocol 1, improving the SOTA result by a large margin. Finally, we discuss the effectiveness of a fully 3D network for 3D pose estimation experimentally which may benefit future research.

Download

Salient Object Detection in the Deep Learning Era: An In-depth Survey

IEEE Transactions on Pattern Analysis and Machine Intelligence

As an important problem in computer vision, salient object detection (SOD) from images has been a... more As an important problem in computer vision, salient object detection (SOD) from images has been attracting an increasing amount of research effort over the years. Recent advances in SOD, not surprisingly, are dominantly led by deep learning-based solutions (named deep SOD) and reflected by hundreds of published papers. To facilitate the in-depth understanding of deep SODs, in this paper we provide a comprehensive survey covering various aspects ranging from algorithm taxonomy to unsolved open issues. In particular, we first review deep SOD algorithms from different perspectives including network architecture, level of supervision, learning paradigm and object/instance level detection. Following that, we summarize existing SOD evaluation datasets and metrics. Then, we carefully compile a thorough benchmark results of SOD methods based on previous work, and provide detailed analysis of the comparison results. Moreover, we study the performance of SOD algorithms under different attributes, which have been barely explored previously, by constructing a novel SOD dataset with rich attribute annotations. We further analyze, for the first time in the field, the robustness and transferability of deep SOD models w.r.t. adversarial attacks. We also look into the influence of input perturbations, and the generalization and hardness of existing SOD datasets. Finally, we discuss several open issues and challenges of SOD, and point out possible research directions in future. All the saliency prediction maps, our constructed dataset with annotations, and codes for evaluation are made publicly available at https://github.com/wenguanwang/SODsurvey.

Download

Video Super-Resolution via Pre-Frame Constrained and Deep-Feature Enhanced Sparse Reconstruction

Pattern Recognition

Abstract This paper presents a new video super-resolution (SR) method that can generate high-qual... more Abstract This paper presents a new video super-resolution (SR) method that can generate high-quality and temporally coherent high-resolution (HR) videos. Starting from the traditional sparse reconstruction framework that works well for image SR, we improve it significantly from the following aspects to obtain an effect video SR method. Firstly, to enhance the temporal coherence between adjacent HR frames, once a HR frame is estimated, we use it to guide the sparse reconstruction of the next low-resolution frame. Secondly, instead of using just hand-craft features, we further incorporate deep features generated by VGG16 into our sparse reconstruction based video SR method. Thirdly, we constantly update the dictionary, which is the core of the sparse reconstruction, by making use of the previously estimated HR frame. Finally, after the HR video is reconstructed, we use a joint bilateral filter to post-process it to remove artifacts and transfer image details. Experiments demonstrate that the proposed four strategies effectively improve our final results. In most of the experiments of this paper, our results are better than those produced by the latest deep learning based approaches.

Multi-View Video Synopsis via Simultaneous Object-Shifting and View-Switching Optimization

IEEE Transactions on Image Processing

We present a method for synopsizing multiple videos captured by a set of surveillance cameras wit... more We present a method for synopsizing multiple videos captured by a set of surveillance cameras with some overlapped field-of-views. Currently, object-based approaches that directly shift objects along the time axis are already able to compute compact synopsis results for multiple surveillance videos. The challenge is how to present the multiple synopsis results in a more compact and understandable way. Previous approaches show them side by side on the screen, which however is difficult for user to comprehend. In this paper, we solve the problem by joint object-shifting and camera view-switching. Firstly, we synchronize the input videos, and group the same object in different videos together. Then we shift the groups of objects along the time axis to obtain multiple synopsis videos. Instead of showing them simultaneously, we just show one of them at each time, and allow to switch among the views of different synopsis videos. In this view switching way, we obtain just a single synopsis results consisting of content from all the input videos, which is much easier for user to follow and understand. To obtain the best synopsis result, we construct a simultaneous object-shifting and view-switching optimization framework instead of solving them separately. We also present an alternative optimization strategy composed of graph cuts and dynamic programming to solve the unified optimization. Experiments demonstrate that our single synopsis video generated from multiple input videos is compact, complete, and easy to understand.

A Framework for Neural Prosthetic Sensory Feedback Using Electrical Stimulation with Time-variant Parameters

Providing a neural prosthesis with sensory feedback is essential for refine manipulation. Here, w... more Providing a neural prosthesis with sensory feedback is essential for refine manipulation. Here, we present a novel approach to provide neural prosthesis with sensory feedback. The prosthesis is constituted by a SCHUNK robot that can grasp different objects and an electrical stimulator that can generate time-variant stimulation patterns to offer sensory feedback to amputees. We convert the output of tactile array sensors on the prosthesis into certain patterns of electrical stimulation of the nerve. Especially, the stimulator is fully implantable, the stimulation parameters and power can percutaneously transfer through an inductively coupled link. The sensory feedback experiment and the simulation pattern generation experiment are conducted separately, indicating that sensory information is available and time-variant stimulation is realizable. In future, based on the novel prosthesis system, we will conduct more experiments by advanced organisms (such as macaque or amputee) to develop Human-machine interfaces.

Download

Video Saliency Prediction using Spatiotemporal Residual Attentive Networks

IEEE Transactions on Image Processing

This paper proposes a novel residual attentive learning network architecture for predicting dynam... more This paper proposes a novel residual attentive learning network architecture for predicting dynamic eye-fixation maps. The proposed model emphasizes two essential issues, i.e., effective spatiotemporal feature integration and multi-scale saliency learning. For the first problem, appearance and motion streams are tightly coupled via dense residual cross connections, which integrate appearance information with multi-layer, comprehensive motion features in a residual and dense way. Beyond traditional two-stream models learning appearance and motion features separately, such design allows early, multi-path information exchange between different domains, leading to a unified and powerful spatiotemporal learning architecture. For the second one, we propose a composite attention mechanism that learns multi-scale local attentions and global attention priors end-to-end. It is used for enhancing the fused spatiotemporal features via emphasizing important features in multi-scales. A lightweight convolutional Gated Recurrent Unit (convGRU), which is flexible for small training data situation, is used for long-term temporal characteristics modeling. Extensive experiments over four benchmark datasets clearly demonstrate the advantage of the proposed video saliency model over other competitors and the effectiveness of each component of our network. Our code and all the results will be available at https://github.com/ashleylqx/STRA-Net.

Qiuxia Lai

Uploads

Papers by Qiuxia Lai

Log In