Electromagnetic Articulography

A beszédhangok többsége jól felismerhető a nekik tulajdonított szakasz közepéről, de ez nem mondható el a zárhangokról. Az alábbiakban a zárhangok kétféle szegmentálását hasonlítjuk össze. Az egyik megőrzi a felismerési mechanizmus egységességét, de az anyanyelvi beszélők intuíciójától eltérően a zárhangok zárrészét és felpattanását két különöző szegmentumként elemzi. A másik bonyolultabb felismerőmechanizmust igényel, de jobban modellálja a beszélők intuícióját, és a fel nem pattanó zárhangok felismerését is lehetővé teszi

In this paper, an electromagnetic articulography (EMA) study is conducted on four standard Korean speakers to examine their articulation of Korean /l/. Their articulation of /l/ is compared to another sonorant, /n/, in the context of the vowels /a/, /i/, and /u/. Lateralization is quantified by calculating the lateralization angles of both the left-and right-side parasagittal sensors, and retroflexion is examined by observing the angle of elevation seen in the tongue-tip sensor. Considerable intra-and inter-speaker variance is observed in the articulation of /l/, with a general tendency for /l/ to be asymmetrically lateralized in coda position, suggesting a lateral approximant articulation. Lateralization appears to be optional in onset position, but the tongue tip is generally raised, indicating a flap, retroflex lateral, or lateral flap articulation. Coda /l/ is also observed to have a tendency to have a raised tongue tip in the context of /a/, suggesting it is a retroflex lateral approximant in this environment.

The article describes the structure and functionality of the articulographic database for storing data from articulographic research using an electromagnetic articulograph, an acoustic camera and 3 video cameras. The database enables selective extraction of various types of data for scientific research and interoperates with programs that carry out experiments. Structure and construction of the database is described. Potential future application in statistical analysis and experiments on speech inversion using dynamic Bayesian networks (DBN) was also presented.

The electromagnetic articulography (EMA) is a relatively exact and efficient method used in study on speech production physiology. It allows to precisely estimate movement trajectories of speech articulators like tongue, lips, jaw etc. by tracking position of sensors fixed to the articulators. This paper presents results of EMA research on Polish oral and nasalised vowels in orthography represented by the graphemes , , <ę>, <ą>. Inter-individual variability of tongue and lips position in X-axis direction during realization of the same phoneme has been estimated. Differences between oral and nasalised vowels in terms of movement of articulators in X-axis direction have been assessed too.

Electromagnetic articulography (EMA) is one of the instrumental phonetic research methods used for recording and assessing articulatory movements. Usually, articulographic data are analysed together with standard audio recordings. This paper, however, demonstrates how coupling the articulograph with devices providing other types of information may be used in more advanced speech research. A novel measurement system is presented that consists of the AG 500 electromagnetic articulograph, a 16-channel microphone array with a dedicated audio recorder and a video module consisting of 3 high-speed cameras. It is argued that synchronization of all these devices allows for comparative analyses of results obtained with the three components. To complement the description of the system, the article presents innovative data analysis techniques developed by the authors as well as preliminary results of the system’s accuracy.

This study analyses the prosodic variation of pitch accents in broad and narrow-contrastive focus conditions in sentence-initial position as produced in L2 English by NorthEast Italian speakers, and compares them with similar productions in native Italian and native English. Our aim is to understand how the phonetic properties of accent (alignment, pitch scaling, duration) are modulated to mark differences in focus in the two native languages, and to compare the use of these properties in the productions in non-native English by Italian speakers. Preliminary results show that the most remarkable difference between native Italian and native English is in the use of the pitch height: native Italian speakers use lower pitch range and pitch span in contrastive focus, while English speakers do not. Italian speakers producing English show a strong influence of the native system and a systematic lowering of peaks in initial accents of contrastive focus sentences.

In this paper, an electromagnetic articulography (EMA) study is conducted on four standard Korean speakers to examine their articulation of Korean /l/. Their articulation of /l/ is compared to another sonorant, /n/, in the context of the vowels /a/, /i/, and /u/. Lateralization is quantified by calculating the lateralization angles of both the left- and right-side parasagittal sensors, and retroflexion is examined by observing the angle of elevation seen in the tongue-tip sensor. Considerable intra- and inter-speaker variance is observed in the articulation of /l/, with a general ten- dency for /l/ to be asymmetrically lateralized in coda position, suggesting a lateral approximant articulation. Lateralization appears to be optional in onset position, but the tongue tip is generally raised, indicating a flap, retroflex lateral, or lateral flap articulation. Coda /l/ is also observed to have a tendency to have a raised tongue tip in the context of /a/, suggesting it is a retroflex lateral approximant in this environment.

Parkinson's Disease dysarthria affects the speech motor control, causing alterations at the suprasegmental level of speech. In previous researches, vowel percentage (%V) and the mean interval between two consecutive vowel onset points (VtoV) were effectively used in the synchronic description of the rhythmic variations of Italian PD speech, compared to healthy speech, even at a very early stage of the disease. This study aims at verifying the early alteration of PD speech rhythm using a diachronic approach. To reach this goal, a corpus of read speech produced by a single PD subject (female, 66 years old) has been collected, consisting of 15 radiophonic speech samples (about 100 s each) on the same topic, recorded between 2001 and 2021. The speech samples were manually segmented in consonantal and vocalic intervals by means of Praat, allowing the calculation of %V and VtoV. The results show an alteration of %V values since 2018, two years before the diagnosis and the insurgence of motor symptoms. Moreover, first results of the application of the automatic segmentation performed by SPPAS on a selection of PD speech samples will also be presented.

Articulography and functional neuroimaging are two major tools for studying the neurobiology of speech production. Until now, however, it has generally not been possible to use both in the same experimental setup because of technical incompatibilities between the two methodologies. Here we describe results from a novel articulography system dubbed Magneto-articulography for the Assessment of Speech Kinematics (MASK), used for the first time to obtain kinematic profiles of oro-facial movements during speech together with concurrent magnetoencephalographic (MEG) measurements of neuromotor brain activity. MASK was used to characterise speech kinematics in a healthy adult, and the results were compared to measurements from the same participant with a conventional electromagnetic articulography (EMA) setup. We also characterised speech movement kinematics with MASK in a group of ten typically developing children, aged 8-12 years. Analyses targeted the gestural landmarks of the utterances...

There is a significant need for more comprehensive electromagnetic articulography (EMA) datasets that can provide matched acoustics and articulatory kinematic data with good spatial and temporal resolution. The Marquette University Electromagnetic Articulography Mandarin Accented English (EMA-MAE) corpus provides kinematic and acoustic data from 40 gender and dialect balanced speakers representing 20 Midwestern standard American English L1 speakers and 20 Mandarin Accented English (MAE) L2 speakers, half Beijing region dialect and half are Shanghai region dialect. Three dimensional EMA data were collected at a 400 Hz sampling rate using the NDI Wave system, with articulatory sensors on the midsagittal lips, lower incisors, tongue blade and dorsum, plus lateral lip corner and tongue body. Sensors provide three-dimensional position data as well as two-dimensional orientation data representing the orientation of the sensor plane. Data have been corrected for head movement relative to a fixed reference sensor and also adjusted using a biteplate calibration system to place the data in an articulatory working space relative to each subject's individual midsagittal and maxillary occlusal planes. Speech materials include isolated words chosen to focus on specific contrasts between the English and Mandarin languages, as well as sentences and paragraphs for continuous speech, totaling approximately 45 minutes of data per subject. A beta version of the EMA-MAE corpus is now available, and the full corpus is in preparation for public release to help advance research in areas such as pronunciation modeling, acoustic-articulatory inversion, L1-L2 comparisons, pronunciation error detection, and accent modification training.

Articulography and functional neuroimaging are two major tools for studying the neurobiology of speech production. Until now, however, it has generally not been possible to use both in the same experimental setup because of technical incompatibilities between the two methodologies. Here we describe results from a novel articulography system dubbed Magneto-articulography for the Assessment of Speech Kinematics (MASK), used for the first time to obtain kinematic profiles of orofacial movements during speech together with concurrent magnetoencephalographic (MEG) measurements of neuromotor brain activity. MASK was used to characterise speech kinematics in a healthy adult, and the results were compared to measurements from the same participant with a conventional electromagnetic articulography (EMA) setup. We also characterised speech movement kinematics with MASK in a group of ten typically developing children, aged 8-12 years. Analyses targeted the gestural landmarks of the utterances /ida/, /ila/ and reiterated productions of /pataka/. These results demonstrate that the MASK technique can be used to reliably characterise movement profiles and kinematic parameters that reflect development of speech motor control, together with MEG measurements of brain responses from speech sensorimotor cortex. This new capability sets the stage for cross-disciplinary efforts to understand the developmental neurobiology of human speech production.

Articulography and functional neuroimaging are two major tools for studying the neurobiology of speech production. Until now, however, it has generally not been possible to use both in the same experimental setup because of technical incompatibilities between the two methodologies. Here we describe results from a novel articulography system dubbed Magneto-articulography for the Assessment of Speech Kinematics (MASK), used for the first time to obtain kinematic profiles of oro-facial movements during speech together with concurrent magnetoencephalographic (MEG) measurements of neuromotor brain activity. MASK was used to characterise speech kinematics in a healthy adult, and the results were compared to measurements from the same participant with a conventional electromagnetic articulography (EMA) setup. We also characterised speech movement kinematics with MASK in a group of ten typically developing children, aged 8-12 years. Analyses targeted the gestural landmarks of the utterances...

The phonemic inventory of Arabic includes a plainemphatic contrast in a number of coronal stops and fricatives. The emphatic members in these contrastive pairs are articulated with a secondary posterior constriction in the velopharyngeal region of the vocal tract. This secondary constriction is absent in the plain counterparts. The primary constriction is also believed to differ in the plainemphatic pairs. This study examines the differences in the primary articulation of the plain-emphatic voiceless alveolar fricatives /s/-/sʕ/ in Lebanese Arabic as reflected in the configuration of the tongue blade. Findings suggest that the tongue blade is lower during /sʕ/ than during /s/. There is also evidence for tongue blade concavity during /sʕ/, a configuration not assumed during /s/.

Speech and music prosody deals with the presence of rhythm. This work is the analysis of the presence of the chaotic nature in the duration patterns of speech segments. A chaotic system is one whose characteristics are apparently random with some amount of predictability. This study is carried out both quantitatively and qualitatively. The qualitative analysis is carried out using recurrence plots and phase space plots, whereas quantitative analysis is carried out using statistical measurements such as Hurst Exponent and Recurrence period density entropy (RPDE). Results obtained in the chaotic analysis are useful in electronic music synthesis and speech synthesis in order to improve the naturality of speech generated.

The object of our research is to identify the main features of acoustic phonetics from the point of view of the English language, and in particular, in intonation, rhythm, volume, tempo, pause and sound phonetic processes in order to improve phonemic hearing in English learners.Modern society is distinguished by a certain amount of paradoxicality. With a general, widespread desire for globalization, unification, simplification of speech formulas and averaging, for equalizing pronunciation, regional and local accents are gaining more and more popularity in the world, which indicates an active process of ethnic consolidation. Thus, it is very premature to say that the population of the Earth is united into one huge monoculture. We consider the changes inherent in the phonetic level using the example of the British and American languages.

We introduced a measurement procedure for the involuntary response of voice fundamental-frequency to frequency modulated auditory stimulation. This involuntary response plays an essential role in voice fundamental frequency control while less investigated due to technical difficulties. This article introduces an interactive and real-time tool for investigating this response and supporting tools adopting our new measurement method. The method enables simultaneous measurement of multiple system properties based on a novel set of extended time-stretched pulses combined with orthogonalization. We made MATLAB implementation of these tools available as an open-source repository. This article also provides the detailed measurement procedure using the interactive tool followed by offline measurement tools for conducting subjective experiments and statistical analyses. It also provides technical descriptions of constituent signal processing subsystems as appendices. This application serves as an example for adopting our method to biological system analysis.

Movements of the head and speech articulators have been observed in tandem during an alternating word pair production task driven by an accelerating rate metronome. Word pairs contrasted either onset or coda dissimilarity with same word controls. Results show that as production effort increased, so did speaker head nodding, and that nodding increased abruptly following errors. More errors occurred under faster production rates, and in coda rather than onset alternations. The greatest entrainment between head and articulators was observed at the fastest rate under coda alternation. Neither jaw coupling nor imposed prosodic stress was observed to be a primary driver of head movement. In alternating pairs, nodding frequency tracked the slower alternation rate rather than the syllable rate, interpreted as recruitment of additional degrees of freedom to stabilize the alternation pattern under increasing production rate pressure.

Pauses are acoustically silent intervals where speakers plan subsequent utterances on different levels (Krivokapic 2014). Planning on the phonetic level involves the articulatory movements towards the first segments. That means, some preparatory vocal tract activities, such as various tongue postures and lips position occur during speech preparation and they seem to be temporally organized with each other (Gick et al. 2004; Ramanarayanan et al. 2009, 2013; Rasskazova et al. 2018). Respiration has not been considered in this process so far, although it cannot be separated from articulation and so involved in the speech planning process. Although respiration is slower than oral articulation, both should be well timed with each other, particularly at the onset of speech. The aim of present study is to investigate the coordination of respiratory, acoustic and articulation events prior to the utterance begin. Furthermore, we explore if there is a coordination between exhalation onset and a specific phase of the articulatory gesture of an upcoming segment. For this purpose, 12 native speakers of German have been recorded with Electromagnetic Articulography and Inductance Plethysmography reading sentences that were controlled for length, stress of the first word and the initial segment. The results for six speakers so far indicate that the initiation of the first post-pausal segments starts during the final phase of the inhalation. The onset of expiration seems to be tightly coupled with the acoustic and the articulatory onset, particularly with nucleus onset phase of the alveolar articulator of the initial segment. The manner of articulation of the initial segment seems to affect the temporal organization of preparatory events. These results can be interpreted as evidence for a close coupling between respiration and oral action for gestural organization.

Movements of the head and speech articulators have been observed in tandem during an alternating word pair production task driven by an accelerating rate metronome. Word pairs contrasted either onset or coda dissimilarity with same word controls. Results show that as production effort increased, so did speaker head nodding, and that nodding increased abruptly following errors. More errors occurred under faster production rates, and in coda rather than onset alternations. The greatest entrainment between head and articulators was observed at the fastest rate under coda alternation. Neither jaw coupling nor imposed prosodic stress was observed to be a primary driver of head movement. In alternating pairs, nodding frequency tracked the slower alternation rate rather than the syllable rate, interpreted as recruitment of additional degrees of freedom to stabilize the alternation pattern under increasing production rate pressure.

We address the problem of audiovisual speech inversion, namely recovering the vocal tract's geometry from auditory and visual speech cues. We approach the problem in a statistical framework, combining ideas from multistream Hidden Markov Models and canonical correlation analysis, and demonstrate effective estimation of the trajectories followed by certain points of interest in the speech production system. Our experiments show that exploiting both audio and visual modalities clearly improves performance relative to either audio-only or visual-only estimation. We report experiments on the QSMT database which contains audio, video, and electromagnetic articulography data recorded in parallel.

We are interested in recovering aspects of vocal tract's geometry and dynamics from speech, a problem referred to as speech inversion. Traditional audio-only speech inversion techniques are inherently ill-posed since the same speech acoustics can be produced by multiple articulatory configurations. To alleviate the ill-posedness of the audio-only inversion process, we propose an inversion scheme which also exploits visual information from the speaker's face. The complex audiovisual-to-articulatory mapping is approximated by an adaptive piecewise linear model. Model switching is governed by a Markovian discrete process which captures articulatory dynamic information. Each constituent linear mapping is effectively estimated via canonical correlation analysis. In the described multimodal context, we investigate alternative fusion schemes which allow interaction between the audio and visual modalities at various synchronization levels. For facial analysis, we employ active appearance models (AAMs) and demonstrate fully automatic face tracking and visual feature extraction. Using the AAM features in conjunction with audio features such as Mel frequency cepstral coefficients (MFCCs) or line spectral frequencies (LSFs) leads to effective estimation of the trajectories followed by certain points of interest in the speech production system. We report experiments on the QSMT and MOCHA databases which contain audio, video, and electromagnetic articulography data recorded in parallel. The results show that exploiting both audio and visual modalities in a multistream hidden Markov model based scheme clearly improves performance relative to either audio or visual-only estimation. Index Terms-Active appearance models (AAMs) , audiovisual-to-articulatory speech inversion, canonical correlation analysis (CCA), multimodal fusion. I. INTRODUCTION T REATING speech as essentially a multimodal process has led to interesting advances in speech technologies the recent years. For example, by properly exploiting visual cues from the speaker's face, speech recognition systems have gained robustness in noise [1]. The introduction of speaking faces or avatars in speech synthesis systems improves their naturalness

We aim to identify visual cues resulting from facial movements made during Mandarin tone production and examine how they are associated with each of the four tones. We use signal processing and computer vision techniques to analyze audio-video recordings of 21 native Mandarin speakers uttering the vowel /ɜ/ with each tone. Four facial interest points were automatically detected and tracked in the video frames: medial point of left-eyebrow, nose tip (proxy for head movement), and midpoints of the upper and lower lips. Spatiotemporal features were extracted from the positional profiles of each tracked point. These features included distance, velocity, and acceleration of local facial movements with respect to the resting face of each speaker. Analysis of variance and feature importance analysis based on random decision forest were performed to examine the significance of each feature for representing each tone and how well these features can individually and collectively characterize ...

We investigate the relationship between vowel quantity and the utilization of formant space in Slovak, and how prosodic variation in speech rate and lexical stress marking affects this relationship. Slovak presents a common five-vowel system with full phonemic quantity contrast for all vowels in all positions. We found that 1) phonemic quantity contrast in Slovak is salient and minimally affected by lexical stress and speech rate, and 2) shortening due to phonemic contrast and destressing but not due to speech rate, are accompanied by vowel space contraction. We compare the results to the geographically neighboring languages Czech and Hungarian that display similar prosodic characteristics to Slovak.

This paper presents the TYPALOC corpus of French Dysarthric and Healthy speech and the rationale underlying its constitution. The objective is to compare phonetic variation in the speech of dysarthric vs. healthy speakers in different speech conditions (read and unprepared speech). More precisely, we aim to compare the extent, types and location of phonetic variation within these different populations and speech conditions. The TYPALOC corpus is constituted of a selection of 28 dysarthric patients (three different pathologies) and of 12 healthy control speakers recorded while reading the same text and in a more natural continuous speech condition. Each audio signal has been segmented into Inter-Pausal Units. Then, the corpus has been manually transcribed and automatically aligned. The alignment has been corrected by an expert phonetician. Moreover, the corpus benefits from an automatic syllabification and an Automatic Detection of Acoustic Phone-Based Anomalies. Finally, in order to...

Comparatively little is known about the role that the speed of different articulatory movements plays in speech production. Using 3D Electromagnetic Articulography, the present experiment analyzes articulatory data from Moroccan Arabic for independent influences of oral articulator and manner on stiffness, an important property of articulator movement. Tongue back movements were found to have lower stiffness than those of the

An articulatory model of the tongue composed of linear components was determined by a statistical analysis of tongue shapes and jaw-opening measured on lateral x-ray motion pictures. The sum of the four components describes adequately the tongue shapes observed during utterances of 12 French vowels in sentences and of pV1CV2 disyllables, where V1 V2 = [i,a,u] and C = [b,d,g]. These components represent the effect of the jaw articulator, of the tongue-body articulator, of the dorsal articulator, and of the apical articulator upon the tongue shape, respectively. The vowels plotted on the articulatory space, where the x axis corresponds to the tongue-body component and the y axis to the jaw component, form the vowel triangle. The model seems to explain, at least in part, the origin of articulatory “front-back” and “high-low” dimensions. We speculate that such model should be suitable for the description of the dynamics of the tongue, since that of the individual articulators may be gov...

Total removal of the larynx may be required to treat laryngeal cancer: speech is lost. This article shows that it may be possible to restore speech by sensing movement of the remaining speech articulators and use machine learning algorithms to derive a transformation to convert this sensor data into an acoustic signal. The resulting "silent speech," which may be delivered in real time, is intelligible and sounds natural. The identity of the speaker is recognisable. The sensing technique involves attaching small, unobtrusive magnets to the lips and tongue and monitoring changes in the magnetic field induced by their movement.

This study aims to explore the effects of healthy aging and Parkinson’s disease on speech motor performance. One area of speech production which requires fine speech motor control is prominence marking. Therefore, strategies of prominence marking of three speaker groups with four speakers each were investigated: younger speakers, older speakers, and speakers with Parkinson’s disease (PD). Acoustic and articulatory data were collected. Speech data were analyzed focusing on prominence-related adjustments of vowel production and tongue body movements in the temporal and spatial domain. Longer durations, varying initiation of the tongue movements and smaller vowel sizes in older speakers and in speakers with PD were found compared to younger speakers. The data indicate further that all speaker groups mark prominence by changing relevant parameters in the vowel articulation; however, strategies seem to differ between the groups: (i) in the temporal domain, articulatory movement durations...

Electromagnetic articulography (EMA) is one of the instrumental phonetic research methods used for recording and assessing articulatory movements. Usually, articulographic data are analysed together with standard audio recordings. This paper, however, demonstrates how coupling the articulograph with devices providing other types of information may be used in more advanced speech research. A novel measurement system is presented that consists of the AG 500 electromagnetic articulograph, a 16-channel microphone array with a dedicated audio recorder and a video module consisting of 3 high-speed cameras. It is argued that synchronization of all these devices allows for comparative analyses of results obtained with the three components. To complement the description of the system, the article presents innovative data analysis techniques developed by the authors as well as preliminary results of the system’s accuracy.

Acoustic analysis of laterality in speech sounds poses numerous obstacles to researchers. Spectral characteristics of such segments vary depending on their phonetic context and shaping of the vocal tract [1]. There are no unambiguous acoustic parameters indicating that a sound is produced laterally. The main alternatives to spectrographic analysis in studying laterality are costly devices, such as electropalatographs (EPG) and magnetic resonance imaging (MRI) scanners. This article demonstrates how a much more affordable device, the multi-channel recorder, may be used in detecting and assessing laterality in speech. The system records multi-channel audio and calculates spatial coordinates of sound propagation sources, allowing the researcher to establish if the release is central, unilateral or bilateral. This method allows also for evaluating the dominance of the release on the left or right side. It may be used in assessing nasality, too.

Parkinson's Disease dysarthria affects the speech motor control, causing alterations at the suprasegmental level of speech. In previous researches, vowel percentage (%V) and the mean interval between two consecutive vowel onset points (VtoV) were effectively used in the synchronic description of the rhythmic variations of Italian PD speech, compared to healthy speech, even at a very early stage of the disease. This study aims at verifying the early alteration of PD speech rhythm using a diachronic approach. To reach this goal, a corpus of read speech produced by a single PD subject (female, 66 years old) has been collected, consisting of 15 radiophonic speech samples (about 100 s each) on the same topic, recorded between 2001 and 2021. The speech samples were manually segmented in consonantal and vocalic intervals by means of Praat, allowing the calculation of %V and VtoV. The results show an alteration of %V values since 2018, two years before the diagnosis and the insurgence of motor symptoms. Moreover, first results of the application of the automatic segmentation performed by SPPAS on a selection of PD speech samples will also be presented.

The production of speech requires the interplay of a number of cognitive and motoric activities, which make it an interesting object of study from both a linguistic and a medical point of view. In this paper, we discuss, first, the features and domain of application of the most used technologies in linguistic research on speech production, focusing on those that have been applied to medicine. Second, we offer an insight into the main results that have been obtained so far in studying dysarthria in Italian Parkinson's Disease, as an example of the interdisciplinary, experimental research at the border between linguistics and medicine.

The goal of this study is to qualitatively and quantitatively assess the reliability of the AG501 Electromagnetic Articulograph (Carstens Medizinelektronik GmbH) and to compare it with the previous model, the AG500, which is still widely used in the 3D recording of articulatory movements. To explore and test the spatial accuracy of the articulographs in various areas of the recording volume, controlled sensor positions at fixed locations and along circular trajectories as well as movements of articulators of the speaker"s vocal tract during speech production tasks were tracked and analysed. It is well known that the AG500 trajectories are affected by perturbations which: (i) depend on the position of the sensors/subject within the recording volume and (ii) randomly occur along repetitions of the same sound in the same recording area. This study has shown that these issues do not affect the newer AG501, which not only performs according to the manufacturer"s claim of 0.3 mm dynamical accuracy within a 20-cm-wide spherical region inside the recording volume, but also performs well outside. Furthermore, while the AG500 shows perturbed trajectories in some instances, the AG501 consistently shows accurate results in reproducing the displacements of consonantal and vocalic gestures for the tested speech tasks. Our findings reveal that the AG501 is more stable and significantly more accurate than the previous model, the AG500, which, in turn, performs reasonably well only in specific limited conditions.

Mirko Grimaldi 1 , Barbara Gili Fivela 1 , Francesco Sigona 1 , Michele Tavella 2 , Paul Fitzpatrick 3 , ... Laila Craighero 3 , Luciano Fadiga 3 , Giulio Sandini 2 , Giorgio Metta 2 ... 1 Centro di Ricerca Interdisciplinare sul Linguaggio, University of Salento, Lecce, Italy 2 Laboratory for Integrated Advanced Robotics, University of Genoa, Italy 3 Dep. SBTA, Section of Human Physiology, University of Ferrara, Italy ... {mirko.grimaldi, barbara.gili, francesco.sigona}@ateneo.unile.it; {michele, paulfitz, pasa}@liralab.it; {fdl, crh}@unife.it; sandini@dist.unige.it

Neurophysiological changes in the brain associated with early dementia can disrupt articulatory timing and precision in speech production. Motivated by this observation, we address the hypothesis that speaking rate and articulatory coordination, as manifested through formant frequency tracks, can predict performance on an animal fluency task administerd to the elderly. Specifically, using phoneme-based measures of speaking rate and articulatory coordination derived from formant cross-correlation measures, we investigate the capability of speech features, estimated from paragraph-recall and naturalistic free speech, to predict animal fluency assessment scores. Using a database consisting of audio from elderly subjects over a 4 year period, we develop least-squares regression models of our cognitive performance measures. The best performing model combined speaking rate and formant features, resulting in a correlation (R) of 0.61 and a root mean squared error (RMSE) of 5.07 with respect to a 9-34 score range. Vocal features thus provide a reduction by about 30% in MSE from a baseline (mean score) in predicting cognitive performance derived from the animal fluency assessment.

Recently it was shown that within the Silent Speech Interface (SSI) field, the prediction of F0 is possible from Ultrasound Tongue Images (UTI) as the articulatory input, using Deep Neural Networks for articulatory-to-acoustic mapping. Moreover, text-to-speech synthesizers were shown to produce higher quality speech when using a continuous pitch estimate, which takes non-zero pitch values even when voicing is not present. Therefore, in this paper on UTI-based SSI, we use a simple continuous F0 tracker which does not apply a strict voiced / unvoiced decision. Continuous vocoder parameters (ContF0, Maximum Voiced Frequency and Mel-Generalized Cepstrum) are predicted using a convolutional neural network, with UTI as input. The results demonstrate that during the articulatory-toacoustic mapping experiments, the continuous F0 is predicted with lower error, and the continuous vocoder produces slightly more natural synthesized speech than the baseline vocoder using standard discontinuous F0.

Because the lips are external organs, they can be easily observed by means that are non-invasive, including simple video recording. The current paper introduces a free-of-charge, highly portable, automatic solution for extracting oral posture from digital video, based on an existing facetracking utility. We describe how this solution might benefit various lines of laboratory phonology research, including analysis of articulatory coordination and audiovisual speech, as well as phonetic fieldwork. We then provide a tutorial, framed as a simple experiment in metronomic speech. The tutorial describes how to configure the software to work with one's digital camera of choice, how to extract relevant articulatory parameters from the face tracker, and how to approach statistical analysis. We have supplied pre-written Python scripts to aid the reader in following along with the experiment. The portability, ease, and affordability of the described solution have important implications for the accessibility of articulatory measurement, both in the lab and out in the field.