Fully Convolutional Network

Historically, weather forecasting was unreliable and imprecise, relying on intuition and local knowledge. Inaccurate weather forecasts can cause severe impacts on agriculture, construction, and daily life. Existing methods struggle with rural and urban weather prediction, requiring faster and more accurate solutions. This research proposes a deep learning system using realtime images to address this challenge. This research employs a deep learning model fully convolutional network-long short-term memory (FCN-LSTM) to analyze images and predict weather conditions. In this case, the model forecasts a sunny and cloudy environment, which facilitates defining the ideal conditions for every given climatic zone in the weather classification model. The model is trained on a dataset of weather images obtained from Kaggle. The performance of the proposed model FCN-LSTM achieves an accuracy of 96.88% and a validation accuracy of 91.22%. Also, the mean squared error (MSE) is 7.11, which is significantly lower and supports efficient enhancement in weather forecasting. This significant improvement demonstrates the potential of deep learning for real-time weather forecasting. The model provides efficient weather classification, enabling informed decision-making across various sectors. This research lays the foundation for automated weather analysis using deep learning, eliminating human bias and improving accuracy.

Early diagnosis and proper grouping of tumors in the brain are critical for successful therapy and positive outcomes for patients. This work proposes a complete technique for identifying brain tumors that employ sophisticated artificial intelligence methodologies and achieve an accuracy rate of 97.18%. The work makes use of the brain tumor magnetic resonance imaging (MRI) collection in Kaggle, which has 723 MRI scans classified as glioma, meningioma, pituitary tumor, and no tumor. These images are initially preprocessed, which includes scaling to a homogeneous size normalizing, and removal of noise to ensure uniformity and clarity. To improve the information set, generative adversarial networks (GANs) are used to perform data augmentation, producing artificial pictures that improve the database variety and resilience. To achieve exact cancer localization, the U-Net construction, recognized for its encoder-decoder design and skip links, is used to divide up tumor areas across images generated by MRI. The image segments are then input into gated recurrent units (GRUs), to analyze a collection of features to capture periods and differences between segments. The last classification is accomplished using an entirely linked layer and then a softmax stimulation, which provides the tumors classes. This method helps for medical experiments and clinical methods.

Fully Convolutional Networks have been achieving remarkable results in image semantic segmentation, while being efficient. Such efficiency results from the capability of segmenting several voxels in a single forward pass. So, there is a direct spatial correspondence between a unit in a feature map and the voxel in the same location. In a convolutional layer, the kernel spans over all channels and extracts information from them. We observe that linear recombination of feature maps by increasing the number of channels followed by compression may enhance their discriminative power. Moreover, not all feature maps have the same relevance for the classes being predicted. In order to learn the inter-channel relationships and recalibrate the channels to suppress the less relevant ones, Squeeze and Excitation blocks were proposed in the context of image classification with Convolutional Neural Networks. However, this is not well adapted for segmentation with Fully Convolutional Networks since they segment several objects simultaneously, hence a feature map may contain relevant information only in some locations. In this paper, we propose recombination of features and a spatially adaptive recalibration block that is adapted for semantic segmentation with Fully Convolutional Networks-the SegSE block. Feature maps are recalibrated by considering the cross-channel information together with spatial relevance. Experimental results indicate that Recombination and Recalibration improve the results of a competitive baseline, and generalize across three different problems: brain tumor segmentation, stroke penumbra estimation, and ischemic stroke lesion outcome prediction. The obtained results are competitive or outperform the state of the art in the three applications.

Facial landmark detection has been studied over decades. Numerous neural network (NN)-based approaches have been proposed for detecting landmarks, especially the convolutional neural network (CNN)-based approaches. In general, CNN-based approaches can be divided into regression and heatmap approaches. However, no research systematically studies the characteristics of different approaches. In this paper, we investigate both CNN-based approaches, generalize their advantages and disadvantages, and introduce a variation of the heatmap approach, a pixel-wise classification (PWC) model. To the best of our knowledge, using the PWC model to detect facial landmarks have not been comprehensively studied. We further design a hybrid loss function and a discrimination network for strengthening the landmarks' interrelationship implied in the PWC model to improve the detection accuracy without modifying the original model architecture. Six common facial landmark datasets, AFW, Helen, LFPW, 300-W, IBUG, and COFW are adopted to train or evaluate our model. A comprehensive evaluation is conducted and the result shows that the proposed model outperforms other models in all tested datasets.

Fully Convolutional Networks have been achieving remarkable results in image semantic segmentation, while being efficient. Such efficiency results from the capability of segmenting several voxels in a single forward pass. So, there is a direct spatial correspondence between a unit in a feature map and the voxel in the same location. In a convolutional layer, the kernel spans over all channels and extracts information from them. We observe that linear recombination of feature maps by increasing the number of channels followed by compression may enhance their discriminative power. Moreover, not all feature maps have the same relevance for the classes being predicted. In order to learn the inter-channel relationships and recalibrate the channels to suppress the less relevant ones, Squeeze and Excitation blocks were proposed in the context of image classification with Convolutional Neural Networks. However, this is not well adapted for segmentation with Fully Convolutional Networks since they segment several objects simultaneously, hence a feature map may contain relevant information only in some locations. In this paper, we propose recombination of features and a spatially adaptive recalibration block that is adapted for semantic segmentation with Fully Convolutional Networks-the SegSE block. Feature maps are recalibrated by considering the cross-channel information together with spatial relevance. Experimental results indicate that Recombination and Recalibration improve the results of a competitive baseline, and generalize across three different problems: brain tumor segmentation, stroke penumbra estimation, and ischemic stroke lesion outcome prediction. The obtained results are competitive or outperform the state of the art in the three applications.

Convolutional Neural Network (CNN) has become one of the most popular techniques in image classification. Usually CNN models are trained on a large amount of data, but in this paper, it is discussed CNN usage on data shortage and class imbalance issues. The study is conducted on a small dataset of vine leaves images on a classification task with five classes using two different approaches. In the first approach, a simple CNN model is used, while in the second approach, the Visual Geometry Group (VGG) model with transfer learning is used. It is shown that using different deep learning techniques such as transfer learning, stratified sampling, data augmentation, and the state of arts CNN models such as VGG gives a relatively very good model performance with up to 87% accuracy.

Ship plate recognition is challenging due to variations of plate locations and text types. This paper proposes an effcient Fully Convolutional Network based Plate Recognition approach FCNPR, which uses a CNN (Convolutional Neural Network) to locate ships, then detects plate text lines with the fully convolutional network (FCN). The recognition accuracy is improved with integrating the AIS (Automatic Identification System) information. The actual FCNPR deployment demonstrates that it can work reliably with a high accuracy for satisfying practical usages.

The world around us may be viewed as a network of entities interconnected via their social, economic, and political interactions. These entities and their interactions form a social network. A social network is often modeled as a graph whose nodes represent entities, and edges represent interactions between these entities. These networks are characterized by the collective latent behavior that does not follow trivially from the behaviors of the individual entities in the network. One such behavior is the existence of hierarchy in the network structure, the sub-networks being popularly known as communities. Discovery of the community structure in a social network is a key problem in social network analysis as it refines our understanding of the social fabric. Not surprisingly, the problem of detecting communities in social networks has received substantial attention from the researchers. In this paper, we propose parallel implementations of recently proposed community detection algorithms that employ variants of the well-known quantum-inspired evolutionary algorithm (QIEA). Like any other evolutionary algorithm, a quantum-inspired evolutionary algorithm is also characterized by the representation of the individual, the evaluation function, and the population dynamics. However, individual bits called qubits, are in a superposition of states. As chromosomes evolve individually, the quantum-inspired evolutionary algorithms (QIEAs) are intrinsically suitable for parallelization. In recent years, programmable graphics processing units-GPUs, have evolved into massively parallel environments with tremendous computational power. NVIDIA R compute unified device architecture (CUDA R) technology, one of the leading general-purpose parallel computing architectures with hundreds of cores, can concurrently run thousands of computing threads. The paper proposes novel parallel implementations of quantum-inspired evolutionary algorithms in the field of community detection on CUDA-enabled GPUs. The proposed implementations employ a single-population fine-grained approach that is suited for massively parallel computations. In the proposed approach, each element of a chromosome is assigned to a separate thread. It is observed that the proposed algorithms perform significantly better than the benchmark algorithms. Further, the proposed parallel implementations achieve significant speedup over the serial versions. Due to the highly parallel nature of the proposed algorithms, an increase in the number of multiprocessors and GPU devices may lead to a further speedup.

Brain tumors present a significant medical concern, posing challenges in both diagnosis and treatment. Deep learning has emerged as an evolving technique for automating the diagnostic process for brain tumors. This research paper introduces a novel deep-learning framework designed explicitly for brain tumor diagnosis. The framework encompasses various tasks: tumor detection, classification, segmentation, and survival rate prediction. The framework was applied to the BraTS dataset, an extensive collection of brain tumor images, to evaluate its effectiveness. The proposed workflow initiates with data acquisition, followed by an enhancement of this data using a Convolutional Normalized Mean Filter (CNMF) during pre-processing. This prepares the data for the multi-class classification performed using the novel DBT-CNN classifier model. The RU-Net2+ model is employed for precise tumor demarcation, yielding segmented regions from which features are subsequently extracted utilizing the Cox model. These extracted features play a pivotal role in the final step, where the survival rate of patients is predicted using a logistic regression model. The experimental results showcased the exceptional performance of the proposed framework, surpassing current benchmarks in classification accuracy, tumor segmentation precision, and survival rate prediction. For high-grade glioma (HGG) tumors, the framework achieved an impressive classification accuracy of 99.51%, while for low-grade glioma (LGG) tumors, the accuracy reached 99.28%. The accuracy of tumor segmentation stood at 98.39% for HGG tumors and 99.1% for LGG tumors. The RU-Net2+ algorithm accurately predicts patient survival rates: 85.71% long-term, 72.72% medium-term, and 61.54% short-term, with corresponding Mean Squared Errors of 0.13, 0.21, and 0.31. These results provide valuable insights for medical professionals making brain tumor treatment decisions. Additionally, the framework shows promise for automating brain tumor diagnosis and enhancing patient care. INDEX TERMS Brain tumor, MRI images, deep learning, machine learning, CNMF, RU-Net2+, DBT-CNN, BraTs. I. INTRODUCTION Brain tumors pose a significant health concern and can cause severe patient consequences. It is crucial to promptly and precisely diagnose brain tumors to facilitate effective treatment strategies [1]. The conventional approaches to segmenting, classifying, and predicting the risks associated with brain tumors have encountered limitations in accuracy and efficiency. Deep learning-based models have recently emerged as The associate editor coordinating the review of this manuscript and approving it for publication was Zhan-Li Sun. powerful tools in medical imaging analysis [2]. These models can significantly improve the accuracy and efficiency of brain tumor diagnosis. However, significant challenges hinder their effective deployment in clinical settings. These challenges include data quality and availability, computational complexity, inter-modality variations, model generalization, overfitting, interpretability, temporal dynamics, annotation, labeling issues, integration into clinical workflows, and ethical considerations, including data privacy and biases [3]. In this environment, there's a pressing requirement for an advanced deep learning model that can adeptly and precisely

Non-intrusive, real-time analysis of the dynamics of the eye region allows us to monitor humans’ visual attention allocation and estimate their mental state during the performance of real-world tasks, which can potentially benefit a wide range of human-computer interaction (HCI) applications. While commercial eye-tracking devices have been frequently employed, the difficulty of customizing these devices places unnecessary constraints on the exploration of more efficient, end-to-end models of eye dynamics. In this work, we propose CLERA, a unified model for Cognitive Load and Eye Region Analysis, which achieves precise keypoint detection and spatiotemporal tracking in a joint-learning framework. Our method demonstrates significant efficiency and outperforms prior work on tasks including cognitive load estimation, eye landmark detection, and blink estimation. We also introduce a large-scale dataset of 30 k human faces with joint pupil, eye-openness, and landmark annotation, which aims...

The analysis of complex structured data like video has been a long-standing challenge for computer vision algorithms. Innovative deep learning architectures like Convolutional Neural Networks (CNNs), however are demonstrating remarkable performance in challenging image and video understanding tasks. In this is work we propose a architecture for the automated detection of scored points during tennis matches. We explore two approaches based on CNNs for the analysis of video streams of broadcasted tennis games. We first explore the two-stream approach, which involves extracting features related either pixel intensity values via the analysis of grayscale frames or the encoding of motion related information via optical flow. However, we explore the case of using higher order 3D CNN for simultaneously encoding both spatial and temporal correlations. Furthermore, we explore the late fusion of the individual stream in order to extract and encode both structural and motion spatio-temporal dynamics. We validate the merits of the proposed scheme using a novel manually annotated dataset created from publically available videos.

Rapid and uncontrolled cellular proliferation is what distinguishes a brain tumor. Unfortunately, brain tumors cannot be prevented other than via surgery. As predicted, deep learning may help diagnose and cure brain cancers. The segmentation approach has been widely studied for brain tumor removal. This uses the segmentation approach, one of the most advanced methods for object detection and categorization. To efficiently assess the tumor's size, an accurate and automated brain tumor segmentation approach is needed. We present a fully automated brain tumor separation method for imaging investigations. The approach has been developed with convolutional neural networks. The Multimodal Brain Tumor Image Segmentation (BRATS) datasets tested our strategy. This result suggests that DL should investigate heterogeneous MRI image segmentation to improve brain tumor segmentation accuracy and efficacy. This study may lead to more accurate medical diagnoses and treatments. Researchers in th...

Pupil localization extracts pupil center coordinates from images and videos of the human eye along with the pupillary boundary. Pupil localization essentially plays a major role in identity verification, disease recognition, visual focus of attention (VFOA) tracking, dementia cognitive assessment, and human fatigue detection. However, the process of pupil localization still remains challenging due to various factors, such as poor-quality images, eye makeup, contact lenses, eyelashes, hair strips, eyebrows, closed eyes, and eye saccades. The pupil localization strategies are essentially divided into learning-based and non-learningbased approaches and discussed in detail with the relevant techniques used. This article aims to deliver the essence of current trends in pupil localization and critically discuss the advantages and disadvantages of each method. Hence, this article can be useful to a broad spectrum of readers as a guide to analyzing the latest trends in pupil localization. INDEX TERMS Eye and face databases, eye tracking, gaze tracking, iris recognition, learning-based methods, non-learning-based methods, pupil localization.

Classification of images has been a widely regarded challenge for the past decade, but a new type of object recognition problem which deals with pixellevel segmentation is posing a more complex task for both computer vision enthusiasts and researcher alike. The convolutional neural network has become a staple for any recognition task, but a new type of ConvNet which is Fully convolutional in architecture has yielded more fine features and proponents. We propose a neural net where we take VGG19 [20], a well-known classification CNN, make it fully convolutional for extracting deeper features and lastly use skip-architectures[15] for getting finer output. This yields better result than the pre-existing FCN segmentation architecture [15, 25, 6]. Training was done on augmented VOC12 [4] with SBD [6]training data and validation set was used from reduced VOC12 validation dataset. The model scored mIOU of 68.1 percent in PASCAL VOC 2012 Segmentation challenge.

Deep learning has shown state-of-art classification performance on datasets such as ImageNet, which contain a single object in each image. However, multi-object classification is far more challenging. We present a unified framework which leverages the strengths of multiple machine learning methods, viz deep learning, probabilistic models and kernel methods to obtain state-of-art performance on Microsoft COCO, consisting of non-iconic images. We incorporate contextual information in natural images through a conditional latent tree probabilistic model (CLTM), where the object co-occurrences are conditioned on the extracted fc7 features from pre-trained Imagenet CNN as input. We learn the CLTM tree structure using conditional pairwise probabilities for object co-occurrences, estimated through kernel methods, and we learn its node and edge potentials by training a new 3-layer neural network, which takes fc7 features as input. Object classification is carried out via inference on the learnt conditional tree model, and we obtain significant gain in precision-recall and F-measures on MS-COCO, especially for difficult object categories. Moreover, the latent variables in the CLTM capture scene information: the images with top activations for a latent node have common themes such as being a grasslands or a food scene, and on on. In addition, we show that a simple k-means clustering of the inferred latent nodes alone significantly improves scene classification performance on the MIT-Indoor dataset, without the need for any retraining, and without using scene labels during training. Thus, we present a unified framework for multi-object classification and unsupervised scene understanding.

This project focuses on solving the inpainting problem using both Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) approaches. Popular methods of inpainting include Adobe Photoshop’s Content Aware Fill, which do not use neural networks. The goal of this project was to explore successful architectures and use them as a foundation in creating our own. The best architectures that we came up with are a CNN with Sigmoid Euclidean Loss and a simplified PixelRNN.

Deep learning is a fast-growing machine learning approach to perceive and understand large amounts of data. In this paper, general information about the deep learning approach which is attracted much attention in the field of machine learning is given in recent years and an application about semantic image segmentation is carried out in order to help autonomous driving of autonomous vehicles. This application is implemented with Fully Convolutional Network (FCN) architectures obtained by modifying the Convolutional Neural Network (CNN) architectures based on deep learning. Experimental studies for the application are utilized 4 different FCN architectures named FCN-AlexNet, FCN-8s, FCN-16s and FCN-32s. For the experimental studies, FCNs are first trained separately and validation accuracies of these trained network models on the used dataset is compared. In addition, image segmentation inferences are visualized to take account of how precisely FCN architectures can segment objects.

New ongoing rural construction has resulted in an extensive mixture of new settlements with old ones in the rural areas of China. Understanding the spatial characteristic of these rural settlements is of crucial importance as it provides essential information for land management and decision-making. Despite a great advance in High Spatial Resolution (HSR) satellite images and deep learning techniques, it remains a challenging task for mapping rural settlements accurately because of their irregular morphology and distribution pattern. In this study, we proposed a novel framework to map rural settlements by leveraging the merits of Gaofen-2 HSR images and representation learning of deep learning. We combined a dilated residual convolutional network (Dilated-ResNet) and a multi-scale context subnetwork into an end-to-end architecture in order to learn high resolution feature representations from HSR images and to aggregate and refine the multi-scale features extracted by the aforementi...

Accurate face landmark localization is an essential part of face recognition, reconstruction and morphing. To accurately localize face landmarks, we present our heatmap regression approach. Each model consists of a MobileNetV2 backbone followed by several upscaling layers, with different tricks to optimize both performance and inference cost. We use five naïve face landmarks from a publicly available face detector to position and align the face instead of using the bounding box like traditional methods. Moreover, we show by adding random rotation, displacement and scaling—after alignment—that the model is more sensitive to the face position than orientation. We also show that it is possible to reduce the upscaling complexity by using a mixture of deconvolution and pixel-shuffle layers without impeding localization performance. We present our state-of-the-art face landmark localization model (ranking second on The 2nd Grand Challenge of 106-Point Facial Landmark Localization validati...

Accurate automatic detection of measurement points in ultrasound video sequences is challenging due to noise, shadows, anatomical differences, and scan plane variation. This paper proposes to address these challenges by a Fully Convolutional Neural Network (FCN) trained to regress the point locations. The series of convolutional and pooling layers is followed by a collection of upsampling and convolutional layers with feature forwarding from the earlier layers. The final location estimates are produced by computing the center of mass of the regression maps in the last layer. The temporal consistency of the estimates is achieved by a Long Short-Term memory cells which processes several previous frames in order to refine the estimate in the current frame. The results on automatic measurement of left ventricle in parasternal long axis view of the heart show detection errors below 5% of the measurement line which is within inter-observer variability.

Recently proposed methods for weakly-supervised semantic segmentation have achieved impressive performance in predicting pixel classes despite being trained with only image labels which lack positional information. Because image annotations are cheaper and quicker to generate, weak supervision is more practical than full supervision for training segmentation algorithms. These methods have been predominantly developed to solve the background separation and partial segmentation problems presented by natural scene images and it is unclear whether they can be simply transferred to other domains with different characteristics, such as histopathology and satellite images, and still perform well. This paper evaluates state-of-the-art weakly-supervised semantic segmentation methods on natural scene, histopathology, and satellite image datasets and analyzes how to determine which method is most suitable for a given dataset. Our experiments indicate that histopathology and satellite images present a different set of problems for weakly-supervised semantic segmentation than natural scene images, such as ambiguous boundaries and class co-occurrence. Methods perform well for datasets they were developed on, but tend to perform poorly on other datasets. We present some practical techniques for these methods on unseen datasets and argue that more work is needed for a generalizable approach to weakly-supervised semantic segmentation. Our full code implementation is available on GitHub: https://github. com/lyndonchan/wsss-analysis.

Tese de mestrado integrado, Engenharia Biomédica e Biofísica (Engenharia Clínica e Instrumentação Médica) Universidade de Lisboa, Faculdade de Ciências, 2021In collective behaviour studies, the use of multi-animal tracking systems is extremely valuable. To be able to identify and track each individual in a group helps in the study and understanding of their behaviour in the collective. For this, researchers can use tracking systems, which can use sensors to detect the individuals; or they can use image-based tracking, with or without the need to mark the individuals. idtracker.ai is a state-of-the-art multi-animal image-based tracking system that uses convolutional neural networks to identify each of the individuals in a video. In videos with a higher density of individuals, idtracker.ai cannot extract enough frames of the single individuals (few frames ≈ 30) and the training of the identification network is slower. With the idea of decreasing this training time, here we propose to ...

In this paper, we give a new double twist to the robot localization problem. We solve the problem for the case of prior maps which are semantically annotated perhaps even sketched by hand. Data association is achieved not through the detection of visual features but the detection of object classes used in the annotation of the prior maps. To avoid the caveats of general object recognition, we propose a new representation of the query images that consists of a vector of the detection scores for each object class. Given such soft object detections we are able to create hypotheses about pose and to refine them through particle filtering. As opposed to small confined office and kitchen spaces, our experiment takes place in a large open urban rail station with multiple semantically ambiguous places. The success of our approach shows that our new representation is a robust way to exploit the plethora of existing prior maps for GPS-denied environments avoiding the data association problems when matching point clouds or visual features.

The segmented brain tissues from magnetic resonance images (MRI) always pose substantive challenges to the clinical researcher community, especially while making precise estimation of such tissues. In the recent years, advancements in deep learning techniques, more specifically in fully convolution neural networks (FCN) have yielded path breaking results in segmenting brain tumour tissues with pin-point accuracy and precision, much to the relief of clinical physicians and researchers alike. A new hybrid deep learning architecture combining SegNet and U-Net techniques to segment brain tissue is proposed here. Here, a skip connection of the concerned U-Net network was suitably explored. The results indicated optimal multi-scale information generated from the SegNet, which was further exploited to obtain precise tissue boundaries from the brain images. Further, in order to ensure that the segmentation method performed better in conjunction with precisely delineated contours, the output...

In this paper, we present a new iris ROI segmentation algorithm using a deep convolutional neural network (NN) to achieve the state-of-the-art segmentation performance on well-known iris image data sets. The authors' model surpasses the performance of state-of-the-art Iris DenseNet framework by applying several strategies, including multi-scale/ multi-orientation training, model training from scratch, and proper hyper-parameterisation of crucial parameters. The proposed PixISegNet consists of an autoencoder which primarily uses long and short skip connections and a stacked hourglass network between encoder and decoder. There is a continuous scale up-down in stacked hourglass networks, which helps in extracting features at multiple scales and robustly segments the iris even in an occluded environment. Furthermore, cross-entropy loss and content loss optimise the proposed model. The content loss considers the high-level features, thus operating at a different scale of abstraction, which compliments the cross-entropy loss, which considers pixel-to-pixel classification loss. Additionally, they have checked the robustness of the proposed network by rotating images to certain degrees with a change in the aspect ratio along with blurring and a change in contrast. Experimental results on the various iris characteristics demonstrate the superiority of the proposed method over state-of-the-art iris segmentation methods considered in this study. In order to demonstrate the network generalisation, they deploy a very stringent TOTA (i.e. train-once-test-all) strategy. Their proposed method achieves E 1 scores of 0.00672, 0.00916 and 0.00117 on UBIRIS-V2, IIT-D and CASIA V3.0 Interval data sets, respectively. Moreover, such a deep convolutional NN for segmentation when included in an end-to-end iris recognition system with a siamese based matching network will augment the performance of the siamese network.

Many computer vision applications Computer Aided Diagnosis require an accurate and efficient eye detector. We represent, in this work, an efficient approach for determining the position of the eye in images presenting faces. First, a series of candidate regions are proposed; next, a set of cascaded U-net is used to detect the eye regions; then edge detection methods are used to detect eyes lids and iris boundary and thus helping determining the gaze direction of person having ASD. Our proposed approach achieved a precision of detection that is better than the current most recent methods in trials utilizing GI4E, BioID, and columbiaGaze datasets. The proposed approach is robust to picture alterations, such as changes in external illumination, facial occlusion, and image modality.

Structured real world data can be represented with graphs whose structure encodes independence assumptions within the data. Due to statistical advantages over generative graphical models, Conditional Random Fields (CRFs) are used in a wide range of classification tasks on structured data sets. CRFs can be learned from both, fully or partially supervised data, and may be used to infer fully unlabeled or partially labelled data. However, performing inference in CRFs with an arbitrary graphical structure on a large amount of data is computational expensive and nearly intractable on a reseacher’s workstation. Hence, we take advantage of recent developments in computer hardware, namely generalpurpose Graphics Processing Units (GPUs). We not merely run given algorithms on GPUs, but present a novel framework of parallel algorithms at several levels for training general CRFs on very large data sets. We evaluate their performance in terms of runtime and F1-Score.

This paper presents a novel Transformer-based facial landmark localization network named Localization Transformer (LOTR). The proposed framework is a direct coordinate regression approach leveraging a Transformer network to better utilize the spatial information in a feature map. An LOTR model consists of three main modules: 1) a visual backbone that converts an input image into a feature map, 2) a Transformer module that improves the feature representation from the visual backbone, and 3) a landmark prediction head that directly predicts coordinates from the Transformer's representation. Given cropped-and-aligned face images, the proposed LOTR can be trained end-to-end without requiring any postprocessing steps. This paper also introduces a loss function named smooth-Wing loss, which addresses the gradient discontinuity of the Wing loss, leading to better convergence than standard loss functions such as L1, L2, and Wing loss. Experimental results on the JD landmark dataset provided by the First Grand Challenge of 106-Point Facial Landmark Localization indicate the superiority of LOTR over the existing methods on the leaderboard and two recent heatmap-based approaches. On the WFLW dataset, the proposed LOTR framework demonstrates promising results compared with several state-of-the-art methods. Additionally, we report an improvement in the performance of state-of-the-art face recognition systems when using our proposed LOTRs for face alignment.

Accurate face landmark localization is an essential part of face recognition, reconstruction and morphing. To accurately localize face landmarks, we present our heatmap regression approach. Each model consists of a MobileNetV2 backbone followed by several upscaling layers, with different tricks to optimize both performance and inference cost. We use five naïve face landmarks from a publicly available face detector to position and align the face instead of using the bounding box like traditional methods. Moreover, we show by adding random rotation, displacement and scaling—after alignment—that the model is more sensitive to the face position than orientation. We also show that it is possible to reduce the upscaling complexity by using a mixture of deconvolution and pixel-shuffle layers without impeding localization performance. We present our state-of-the-art face landmark localization model (ranking second on The 2nd Grand Challenge of 106-Point Facial Landmark Localization validati...

Remote eye tracking technology has suffered an increasing growth in recent years due to its applicability in many research areas. In this paper, a video-oculography method based on convolutional neural networks (CNNs) for pupil center detection over webcam images is proposed. As the first contribution of this work and in order to train the model, a pupil center manual labeling procedure of a facial landmark dataset has been performed. The model has been tested over both real and synthetic databases and outperforms state-of-the-art methods, achieving pupil center estimation errors below the size of a constricted pupil in more than 95% of the images, while reducing computing time by a 8 factor. Results show the importance of use high quality training data and well-known architectures to achieve an outstanding performance.

The segmented brain tissues from magnetic resonance images (MRI) always pose substantive challenges to the clinical researcher community, especially while making precise estimation of such tissues. In the recent years, advancements in deep learning techniques, more specifically in fully convolution neural networks (FCN) have yielded path breaking results in segmenting brain tumour tissues with pin-point accuracy and precision, much to the relief of clinical physicians and researchers alike. A new hybrid deep learning architecture combining SegNet and U-Net techniques to segment brain tissue is proposed here. Here, a skip connection of the concerned U-Net network was suitably explored. The results indicated optimal multi-scale information generated from the SegNet, which was further exploited to obtain precise tissue boundaries from the brain images. Further, in order to ensure that the segmentation method performed better in conjunction with precisely delineated contours, the output...

Eye detection algorithms are being used in many fields such as camera applications for entertainment and commercial purposes, gaze detection applications, computer-human interaction applications, and eye recognition applications for security, etc. Successful and fast eye detection is an essential step for all these applications in order to achieve good results. There are many eye detection methods in the literature, and most of them rely on the Viola-Jones method to detect the face before localizing eyes. In this paper, a straightforward approach to detect eyes from images which contain a frontal face is proposed. The approach can be used for real-time eye detection using cheap web cameras or other cameras. First, face landmarks are detected from the image, and by utilizing these landmarks; the eye region is determined. The eye radius is estimated by utilizing eye corners. Then, reduced input images are tested with a tailored matching algorithm which does not need image reduction to determine where the eye is.

Deep learning has shown state-of-art classification perfor mance on datasets such as ImageNet, which contain a single object in each image. How ever, multi-object classification is far more challenging. We present a unified f ramework which leverages the strengths of multiple machine learning metho ds, viz deep learning, probabilistic models and kernel methods to obtain stat e-of-art performance on Microsoft COCO, consisting of non-iconic images. We inco rporate contextual information in natural images through a conditional la tent tree probabilistic model (CLTM), where the object co-occurrences are conditio ned on the extracted fc7 features from pre-trained Imagenet CNN as input. We lear n the CLTM tree structure using conditional pairwise probabilities for ob ject co-occurrences, estimated through kernel methods, and we learn its node and edg e potentials by training a new 3-layer neural network, which takes fc7 featu res as input. Object classification is carried out via inferenc...

Deep learning has shown state-of-art classification performance on datasets such as ImageNet, which contain a single object in each image. However, multi-object classification is far more challenging. We present a unified framework which leverages the strengths of multiple machine learning methods, viz deep learning, probabilistic models and kernel methods to obtain state-of-art performance on Microsoft COCO, consisting of non-iconic images. We incorporate contextual information in natural images through a conditional latent tree probabilistic model (CLTM), where the object co-occurrences are conditioned on the extracted fc7 features from pre-trained Imagenet CNN as input. We learn the CLTM tree structure using conditional pairwise probabilities for object co-occurrences, estimated through kernel methods, and we learn its node and edge potentials by training a new 3-layer neural network, which takes fc7 features as input. Object classification is carried out via inference on the learnt conditional tree model, and we obtain significant gain in precision-recall and F-measures on MS-COCO, especially for difficult object categories. Moreover, the latent variables in the CLTM capture scene information: the images with top activations for a latent node have common themes such as being a grasslands or a food scene, and on on. In addition, we show that a simple k-means clustering of the inferred latent nodes alone significantly improves scene classification performance on the MIT-Indoor dataset, without the need for any retraining, and without using scene labels during training. Thus, we present a unified framework for multi-object classification and unsupervised scene understanding.

For the challenging semantic image segmentation task the most efficient models have traditionally combined the structured modelling capabilities of Conditional Random Fields (CRFs) with the feature extraction power of CNNs. In more recent works however, CRF post-processing has fallen out of favour. We argue that this is mainly due to the slow training and inference speeds of CRFs, as well as the difficulty of learning the internal CRF parameters. To overcome both issues we propose to add the assumption of conditional independence to the framework of fully-connected CRFs. This allows us to reformulate the inference in terms of convolutions, which can be implemented highly efficiently on GPUs. Doing so speeds up inference and training by two orders of magnitude. All parameters of the convolutional CRFs can easily be optimized using backpropagation. To facilitating further CRF research we make our implementation publicly available.

This paper deals with a mobile phone application that allows the user to analyse their driving habits and therefore recognize whether an electric vehicle would suit the users' requirements. The application analyses the daily trips and charging possibilities, recorded as per user input. The resulting demands are compared to available electric vehicles as well as extended range electric vehicles and plug-in hybrid electric vehicles. The vehicles that suit the requirements are presented to the user, taking not only the ideal range into account but also the realistic range in mild and cold weather conditions. Accordingly, the user receives reliable results, and therefore might utilize an electric vehicle sooner than originally envisaged.

In recent years new trends such as industry 4.0 boosted the research and development in the field of autonomous systems and robotics. Robots collaborate and even take over complete tasks of humans. But the high degree of automation requires high reliability even in complex and changing environments. Those challenging conditions make it hard to rely on static models of the real world. In addition to adaptable maps, mobile robots require a local and current understanding of the scene. The Bosch Start-Up Company is developing robots for intra-logistic systems, which could highly benefit from such a detailed scene understanding. The aim of this work is to research and develop such a system for warehouse environments. While the possible field of application is in general very broad, this work will focus on the detection and localization of warehouse specific objects such as palettes. In order to provide a meaningful perception of the surrounding a RGB-D camera is used. A pre-trained conv...

Recently, many works have been inspired by the success of deep learning in computer vision for plant diseases classification. Unfortunately, these end-to-end deep classifiers lack transparency which can limit their adoption in practice. In this paper, we propose a new trainable visualization method for plant diseases classification based on a Convolutional Neural Network (CNN) architecture composed of two deep classifiers. The first one is named Teacher and the second one Student. This architecture leverages the multitask learning to train the Teac1her and the Student jointly. Then, the communicated representation between the Teacher and the Student is used as a proxy to visualize the most important image regions for classification. This new architecture produces sharper visualization than the existing methods in plant diseases context. All experiments are achieved on PlantVillage dataset that contains 54306 plant images.

The segmented brain tissues from magnetic resonance images (MRI) always pose substantive challenges to the clinical researcher community, especially while making precise estimation of such tissues. In the recent years, advancements in deep learning techniques, more specifically in fully convolution neural networks (FCN) have yielded path breaking results in segmenting brain tumour tissues with pin-point accuracy and precision, much to the relief of clinical physicians and researchers alike. A new hybrid deep learning architecture combining SegNet and U-Net techniques to segment brain tissue is proposed here. Here, a skip connection of the concerned U-Net network was suitably explored. The results indicated optimal multi-scale information generated from the SegNet, which was further exploited to obtain precise tissue boundaries from the brain images. Further, in order to ensure that the segmentation method performed better in conjunction with precisely delineated contours, the output...

Accurate face landmark localization is an essential part of face recognition, reconstruction and morphing. To accurately localize face landmarks, we present our heatmap regression approach. Each model consists of a MobileNetV2 backbone followed by several upscaling layers, with different tricks to optimize both performance and inference cost. We use five naïve face landmarks from a publicly available face detector to position and align the face instead of using the bounding box like traditional methods. Moreover, we show by adding random rotation, displacement and scaling—after alignment—that the model is more sensitive to the face position than orientation. We also show that it is possible to reduce the upscaling complexity by using a mixture of deconvolution and pixel-shuffle layers without impeding localization performance. We present our state-of-the-art face landmark localization model (ranking second on The 2nd Grand Challenge of 106-Point Facial Landmark Localization validati...

In the medical field, landmark detection in MRI plays an important role in reducing medical technician efforts in tasks like scan planning, image registration, etc. First, 88 landmarks spread across the brain anatomy in the three respective viewssagittal, coronal, and axial are manually annotated, later guidelines from the expert clinical technicians are taken subanatomy-wise, for better localization of the existing landmarks, in order to identify and locate the important atlas landmarks even in oblique scans. To overcome limited data availability, we implement realistic data augmentation to generate synthetic 3D volumetric data. We use a modified HighRes3DNet model for solving brain MRI volumetric landmark detection problem. In order to visually explain our trained model on unseen data, and discern a stronger model from a weaker model, we implement Gradient-weighted Class Activation Mapping (GradCAM) which produces a coarse localization map highlighting the regions the model is foc...

Robust real-time tracking of the human body is crucial to applications that benefit from live visualizations performed on the underlying body. Such applications could fall in the category of Augmented Reality for Human Bodies, finding great usage in the broader fields of Medicine and Apparel. Specifically, robust real time tracking of the female torso is a crucial component in the pre-visualization of cosmetic breast surgeries. In order to track a torso from monocular RGB input, landmarks that describe the pose and shape of the torso have to be detected. Existing state of the art in algorithms for human pose estimation are dominated by deep neural networks and rely on the availability of large databases with high quality annotations. However, for the requirement of pre-visualizing cosmetic breast surgeries, existing databases fall short as they contain no or very few landmarks that can reliably help estimate the shape of the female torso. Therefore, by building on top of openly available databases of human character models, we create a pipeline for generating synthetic female torsos in both naked and clothed scenarios. We show that deep landmark detectors trained using such a synthetic dataset are capable of generalizing well to unconstrained real world images.

This paper presents a novel Transformer-based facial landmark localization network named Localization Transformer (LOTR). The proposed framework is a direct coordinate regression approach leveraging a Transformer network to better utilize the spatial information in the feature map. An LOTR model consists of three main modules: 1) a visual backbone that converts an input image into a feature map, 2) a Transformer module that improves the feature representation from the visual backbone, and 3) a landmark prediction head that directly predicts the landmark coordinates from the Transformer’s representation. Given cropped-and-aligned face images, the proposed LOTR can be trained end-to-end without requiring any post-processing steps. This paper also introduces the smooth-Wing loss function, which addresses the gradient discontinuity of the Wing loss, leading to better convergence than standard loss functions such as L1, L2, and Wing loss. Experimental results on the JD landmark dataset p...

A key step to driver safety is to observe the driver's activities with the face being a key step in this process to extracting information such as head pose, blink rate, yawns, talking to passenger which can then help derive higher level information such as distraction, drowsiness, intent, and where they are looking. In the context of driving safety, it is important for the system perform robust estimation under harsh lighting and occlusion but also be able to detect when the occlusion occurs so that information predicted from occluded parts of the face can be taken into account properly. This paper introduces the Occluded Stacked Hourglass, based on the work of original Stacked Hourglass network for body pose joint estimation, which is retrained to process a detected face window and output 68 occlusion heat maps, each corresponding to a facial landmark. Landmark location, occlusion levels and a refined face detection score, to reject false positives, are extracted from these he...

This paper presents a novel Transformer-based facial landmark localization network named Localization Transformer (LOTR). The proposed framework is a direct coordinate regression approach leveraging a Transformer network to better utilize the spatial information in the feature map. An LOTR model consists of three main modules: 1) a visual backbone that converts an input image into a feature map, 2) a Transformer module that improves the feature representation from the visual backbone, and 3) a landmark prediction head that directly predicts the landmark coordinates from the Transformer’s representation. Given cropped-and-aligned face images, the proposed LOTR can be trained end-to-end without requiring any post-processing steps. This paper also introduces the smooth-Wing loss function, which addresses the gradient discontinuity of the Wing loss, leading to better convergence than standard loss functions such as L1, L2, and Wing loss. Experimental results on the JD landmark dataset p...

Accurate eye segmentation can improve eye-gaze estimation and support interactive computing based on visual attention; however, existing eye segmentation methods suffer from issues such as person-dependent accuracy, lack of robustness, and an inability to be run in real-time. Here, we present the RITnet model, which is a deep neural network that combines U-Net and DenseNet. RITnet is under 1 MB and achieves 95.3% accuracy on the 2019 OpenEDS Semantic Segmentation challenge. Using a GeForce GTX 1080 Ti, RITnet tracks at > 300Hz, enabling real-time gaze tracking applications. Pre-trained models and source code are available 1 .

Accurate face landmark localization is an essential part of face recognition, reconstruction and morphing. To accurately localize face landmarks, we present our heatmap regression approach. Each model consists of a MobileNetV2 backbone followed by several upscaling layers, with different tricks to optimize both performance and inference cost. We use five naïve face landmarks from a publicly available face detector to position and align the face instead of using the bounding box like traditional methods. Moreover, we show by adding random rotation, displacement and scaling—after alignment—that the model is more sensitive to the face position than orientation. We also show that it is possible to reduce the upscaling complexity by using a mixture of deconvolution and pixel-shuffle layers without impeding localization performance. We present our state-of-the-art face landmark localization model (ranking second on The 2nd Grand Challenge of 106-Point Facial Landmark Localization validati...