Table 1: Numbers adopted to represent some Arabic let- ters when Latin grapheme are used to write Arabic gerian sentences are given. 3 Building a parallel corpus It is well known that parallel corpora are the foun- dation stone of several natural language processing tasks, particularly cross-language applications such as machine translation, bilingual lexicon extraction and multilingual information retrieval. Building this kind of resources is a challenging task espe- cially when it deals with under-resourced languages (Skadina et al., 2010). The problem is much deeper with the Arabic dialects which are used by a huge number of people only in daily oral communication. Moreover, they are not taught in schools and are absent from formal written communications. This makes building dialectal resources automatically al- most impossible. The few available texts in social networks, usually produced by less educated Arab people are not homogeneous and suffer from the va- rieties in orthography, due to the absence of writing rules. In addition, some Arabic dialects are writ- ten by using Latin graphemes by a slice of Ara- bic societies. The reason is that Arabic language was not widely supported by devices. Consequently, Arab people have been used to this situation by us- ing Latin graphemes. TT . _44 «Ll. $f. ...... 4°... yf .._._.._. _ a, Le: se Table 2: Different ways of writing dialects on Facebook. was produced by 20 native speakers. Each one was responsible of translating almost 320 sentences from MSA to TUN. Speakers have very slight differences in their spoken languages. All of them are from the south of Tunisia where people tend to use Arabic words rather than French words as it is the case in the north of the country. In fact, the dialect used in the south is closer to MSA than that used in the north of Tunisia. Syrian and Palestinian corpora were created in the same way by respectively two native speak- ers. Finally, we get a multi-dialectal parallel cor- pus PADIC composed of the five aforementioned di- alects, in addition to MSA. PADIC is made of 6400 parallel sentences, for which we present some statis- tics in Table 3. Table 3: PADIC statistics The MSA part contains 40906 words including 9131 different words. PADIC includes an average of 37500 words for one a dialect with a vocabulary which does not exceed 10250 words. The average number of words in a dialectal sentence is of 6 while it is of 7 for MSA. The shortest sentence in the cor- pus is composed of 4 words and the longest one con- tains 25 words. ‘hla wexcn: Sew Takhla A weornelac «wh wnenTlal cantawnnea Table 5: BLEU score of Machine Translation on different pairs of languages using two smoothing techniques Table 6: TER score (in %) of Machine Translation on different pairs of languages using two smoothing techniques Table 7: METEOR score of Machine Translation on different pairs of languages using two smoothing techniques Table 8: Statistics on machine translation with small training corpus and by varying the smoothing techniques of the language model. of machine translation by varying the smoothing
