Academia.eduAcademia.edu

Figure 18 – uploaded by John Mylopoulos

See full PDFdownloadDownload figure
Observing the calculations vs. human assisted opinions, we note high performance of the tool: both precision and recall measures show very high scores, reaching 100% for some items; fallout is very low for all items, thus demonstrating that the tool is not misled with the large quantity of irrelevant information; error rate is also fairly low; and accuracy reaches almost 100% rate.

Table 18 Observing the calculations vs. human assisted opinions, we note high performance of the tool: both precision and recall measures show very high scores, reaching 100% for some items; fallout is very low for all items, thus demonstrating that the tool is not misled with the large quantity of irrelevant information; error rate is also fairly low; and accuracy reaches almost 100% rate.

Related Figures (64)

International Doctorate School in Information and Communication Technologies DIT - University of Trento TOOL SUPPORT FOR SEMANTIC ANNOTATION Nadzeya Kiyavitskaya
International Doctorate School in Information and Communication Technologies DIT - University of Trento TOOL SUPPORT FOR SEMANTIC ANNOTATION Nadzeya Kiyavitskaya
Table 1-1. Approximate processing time on a single Pentium 4 - 2.5GHz machine for 1 Tb text corpus The obtained estimates lead us to conclusion that, given the state of the art for NLP systems, heavyweight tools are incapable of supporting large-scale semantic annotation. Instead, it might be a rational solution to adopt a lightweight approach that can be tai- lored to particular analysis tasks and scale up. Moreover, such
Table 1-1. Approximate processing time on a single Pentium 4 - 2.5GHz machine for 1 Tb text corpus The obtained estimates lead us to conclusion that, given the state of the art for NLP systems, heavyweight tools are incapable of supporting large-scale semantic annotation. Instead, it might be a rational solution to adopt a lightweight approach that can be tai- lored to particular analysis tasks and scale up. Moreover, such
Figure 4-1. The TXL paradigm
Figure 4-1. The TXL paradigm
Figure 4-2. The structure of a TXL program
Figure 4-2. The structure of a TXL program
TXL engine automatically tokenizes and parses input according to the specified grammar resulting in a parse tree represented in- ternally, as in the example shown in Figure 4-6. TXL has two kinds of transformation rules, called respectively functions and rules. Functions transform one occurrence of their pattern, whereas rules transform many occurrences of their pat- tern. Each transformation rule specifies a target type to the trans- formed, a pattern — an example-like form of the particular in- stance of the type we are interested in replacing — and a replacement — an example-like form of the result we would like when we replace such an instance. The nonterminal type of the pattern is given at the beginning of the rule, and the replacement is implicitly constrained to be of the same type. Replacement re- sults in a new scope tree. See an explanatory example in Figure 4-7.
TXL engine automatically tokenizes and parses input according to the specified grammar resulting in a parse tree represented in- ternally, as in the example shown in Figure 4-6. TXL has two kinds of transformation rules, called respectively functions and rules. Functions transform one occurrence of their pattern, whereas rules transform many occurrences of their pat- tern. Each transformation rule specifies a target type to the trans- formed, a pattern — an example-like form of the particular in- stance of the type we are interested in replacing — and a replacement — an example-like form of the result we would like when we replace such an instance. The nonterminal type of the pattern is given at the beginning of the rule, and the replacement is implicitly constrained to be of the same type. Replacement re- sults in a new scope tree. See an explanatory example in Figure 4-7.
The execution of transformation rule is performed in the fol- lowing way: the scope of the rule matching the pattern tree is found and substituted by a copy of the replacement tree. The process is repeated on the result until no new matches can be found.
The execution of transformation rule is performed in the fol- lowing way: the scope of the rule matching the pattern tree is found and substituted by a copy of the replacement tree. The process is repeated on the result until no new matches can be found.
Figure 4-8. Architecture of the semantic annotation process Figure 4-8 demonstrates the framework of the entire process.
Figure 4-8. Architecture of the semantic annotation process Figure 4-8 demonstrates the framework of the entire process.
Figure 4-10. A part of the parse tree produced on the first stage The input is parsed into a sequence of ads separated by empty lines (“newline”), each ad consists of sentences, each sentences is a sequence of words and spaces. Word can be an identifier (“id”), number or object (“phone”).
Figure 4-10. A part of the parse tree produced on the first stage The input is parsed into a sequence of ads separated by empty lines (“newline”), each ad consists of sentences, each sentences is a sequence of words and spaces. Word can be an identifier (“id”), number or object (“phone”).
Figure 4-14. Database template schema for accommodation ads Figure 4-14. The schema is derived from the semantic model for he target domain. Sentences and phrases with multiple markups are cloned using TXL source transformation to appear as multiple copies, one for each different markup, before populating the data- base. In this way, we do not prejudice one interpretation as being preferred; rather we assume that a single sentence or phrase may in fact be a reasonable answer for all of the semantic categories it is marked as. The products of the process are both the XML marked-up text as shown earlier in Figure 4-13, and the populated database — see Figure 4-15. The populated database can be then queried by < standard SQL database engine. In the present work, we used Mi- crosoft Office Access 2003 for storing and querying annotated text fragments.
Figure 4-14. Database template schema for accommodation ads Figure 4-14. The schema is derived from the semantic model for he target domain. Sentences and phrases with multiple markups are cloned using TXL source transformation to appear as multiple copies, one for each different markup, before populating the data- base. In this way, we do not prejudice one interpretation as being preferred; rather we assume that a single sentence or phrase may in fact be a reasonable answer for all of the semantic categories it is marked as. The products of the process are both the XML marked-up text as shown earlier in Figure 4-13, and the populated database — see Figure 4-15. The populated database can be then queried by < standard SQL database engine. In the present work, we used Mi- crosoft Office Access 2003 for storing and querying annotated text fragments.
Figure 5-2. Conceptual model for accommodation ads To annotate these ads, we designed a conceptual model which represents the information needs of a tourist, provided in Figure 5-2.
Figure 5-2. Conceptual model for accommodation ads To annotate these ads, we designed a conceptual model which represents the information needs of a tourist, provided in Figure 5-2.
Figure 5-4. Semantic category definition for Tourist websites To adapt the annotation framework to this specific task, we had to replace the domain-de pose, the initial d ism experts was was constructed using ogy editor’ and s mode conce mantic relations. The fragmen RDFGravity” too pendix C. The picture s placement in the in Ap pts in total, omain-speci transformed ored in RDF arranged in , is depicted hierarchy, re he means of . The model c o a hierarchy of this mode in Figure 5-5. hows semantic information, such as he Protégé 3 See the RD ationships, at information, i.e., keywords or object patterns, associa conce pts. pendent components. For this pur- fic knowledge provided by the tour- into a rich semantic model. The .0 ontol- onsisted of about 130 and connected by se- , visualized using the F source ributes, and syntactic ed with
Figure 5-4. Semantic category definition for Tourist websites To adapt the annotation framework to this specific task, we had to replace the domain-de pose, the initial d ism experts was was constructed using ogy editor’ and s mode conce mantic relations. The fragmen RDFGravity” too pendix C. The picture s placement in the in Ap pts in total, omain-speci transformed ored in RDF arranged in , is depicted hierarchy, re he means of . The model c o a hierarchy of this mode in Figure 5-5. hows semantic information, such as he Protégé 3 See the RD ationships, at information, i.e., keywords or object patterns, associa conce pts. pendent components. For this pur- fic knowledge provided by the tour- into a rich semantic model. The .0 ontol- onsisted of about 130 and connected by se- , visualized using the F source ributes, and syntactic ed with
Figure 5-9. A fragment of the program realizing refinement rules Then, the TXL program realizing the whole set of the refine- ment rules was implemented. Its fragment is shown in Figure 5-9. In this step, the analysis is carried out as follows: first, the pro- gram detects a text fragment marked with one of the category la- bels previously; then for the given category the program searches for the refinement patterns related to this category. If the match is found, the word combination is marked with the appropriate label.
Figure 5-9. A fragment of the program realizing refinement rules Then, the TXL program realizing the whole set of the refine- ment rules was implemented. Its fragment is shown in Figure 5-9. In this step, the analysis is carried out as follows: first, the pro- gram detects a text fragment marked with one of the category la- bels previously; then for the given category the program searches for the refinement patterns related to this category. If the match is found, the word combination is marked with the appropriate label.
Figure 6-3. The evaluation tool in testing and validation process evaluation of output produced by any semantic annotation system. The tool can be exploited not only to access and compare the quality of existing annotation tools, but also when devising new tools in order to check the quality of results in different stages of development, as shown in Figure 6-3.
Figure 6-3. The evaluation tool in testing and validation process evaluation of output produced by any semantic annotation system. The tool can be exploited not only to access and compare the quality of existing annotation tools, but also when devising new tools in order to check the quality of results in different stages of development, as shown in Figure 6-3.
Figure 6-4. The interface of the evaluation tool The screenshot shows the tags used for a case study not included in this dis- sertation, as this new application of Cerno is a part of an other PhD thesis [Zeni 2006]
Figure 6-4. The interface of the evaluation tool The screenshot shows the tags used for a case study not included in this dis- sertation, as this new application of Cerno is a part of an other PhD thesis [Zeni 2006]
Figure 6-5. An example of the output table
Figure 6-5. An example of the output table
Table 6-2. Evaluating system annotation vs. human Annotator 2 Let’s analyze in detail these two tables. As we could expect, detection of Contact entities is very good. This concept is associ- ated with structured patterns: e-mail, webaddress, and phone number, which easily allow identifying related phrases. More- over, meaning of this particular concept is unambiguous for hu- man annotators, this way reducing the probability of disagreement to minimum. The results of evaluating system performance against assisted human opinions, shown in Table 6-7, confirm that the tool performs well on this entity.
Table 6-2. Evaluating system annotation vs. human Annotator 2 Let’s analyze in detail these two tables. As we could expect, detection of Contact entities is very good. This concept is associ- ated with structured patterns: e-mail, webaddress, and phone number, which easily allow identifying related phrases. More- over, meaning of this particular concept is unambiguous for hu- man annotators, this way reducing the probability of disagreement to minimum. The results of evaluating system performance against assisted human opinions, shown in Table 6-7, confirm that the tool performs well on this entity.
Table 6-1. Evaluating system annotation vs. human Annotator 1!
Table 6-1. Evaluating system annotation vs. human Annotator 1!
Table 6-3. Comparing total system results vs. human (10 ads) These estimates show that compared to each of the human an- notators the tool demonstrated higher recall, but lower precision. Thus, automatic annotation by Cerno is greedy. In more detail, by comparison with these two human annotators, the system exhib- ited a high level of recall: about 92% compared to either human, higher than either human compared to the other, but a lower level of precision: about 74 and 76% compared to each human, whereas they each exhibit about 88-91% compared to the other. However, the system was able to show a 90-92% accuracy rating compared to either human, extremely high for a first prototype.
Table 6-3. Comparing total system results vs. human (10 ads) These estimates show that compared to each of the human an- notators the tool demonstrated higher recall, but lower precision. Thus, automatic annotation by Cerno is greedy. In more detail, by comparison with these two human annotators, the system exhib- ited a high level of recall: about 92% compared to either human, higher than either human compared to the other, but a lower level of precision: about 74 and 76% compared to each human, whereas they each exhibit about 88-91% compared to the other. However, the system was able to show a 90-92% accuracy rating compared to either human, extremely high for a first prototype.
Table 6-4. Evaluating total system performance vs. assisted human opinions Concerning the ads from Rome, we can observe that the tool demonstrated similar performance on the new set of 10 ads com- pared to the initial set used for tuning, and maintained good qual- ity rates for the entire set of 100 ads. In the new set from Venice, he total accuracy was measured as over 95% with similar recall o the Rome test set and lower precision. The same tendency, with even a bigger difference in recall, is seen in the set of Paris. A drop in precision is the result indicative of locality effects from he original training set.
Table 6-4. Evaluating total system performance vs. assisted human opinions Concerning the ads from Rome, we can observe that the tool demonstrated similar performance on the new set of 10 ads com- pared to the initial set used for tuning, and maintained good qual- ity rates for the entire set of 100 ads. In the new set from Venice, he total accuracy was measured as over 95% with similar recall o the Rome test set and lower precision. The same tendency, with even a bigger difference in recall, is seen in the set of Paris. A drop in precision is the result indicative of locality effects from he original training set.
Table 6-5. Evaluating tool annotation (Paris, 10 ads)
Table 6-5. Evaluating tool annotation (Paris, 10 ads)
Table 6-6. Evaluating tool annotation (Venice, 10 ads)
Table 6-6. Evaluating tool annotation (Venice, 10 ads)
Table 6-7. Evaluating system performance vs. assisted human opinions for Rome, 100 ads, Test Set 1
Table 6-7. Evaluating system performance vs. assisted human opinions for Rome, 100 ads, Test Set 1
Table 6-8. Evaluating system annotation on the annotation on which both humans agreed (10 ads)
Table 6-8. Evaluating system annotation on the annotation on which both humans agreed (10 ads)
Table 6-10. Evaluating system annotation vs. human Annotator 2 Table 6-11. Summary of the evaluation results
Table 6-10. Evaluating system annotation vs. human Annotator 2 Table 6-11. Summary of the evaluation results
Table 6-9. Evaluating system annotation vs. human Annotator |
Table 6-9. Evaluating system annotation vs. human Annotator |
As we can observe from these results, for the given annotation schema the task turned out to be difficult both for the system and for the humans. One reason for this shortcoming is the absence of structural patterns for all of the semantic categories. Another rea- son is that ambiguities occur frequently in such tourism docu- ments, as they contain arbitrary information from a broad spec- trum of topics, which are not always independent. For example, text about local food may be associated with either or both of the Local Products category and the Food and Refreshment category,
As we can observe from these results, for the given annotation schema the task turned out to be difficult both for the system and for the humans. One reason for this shortcoming is the absence of structural patterns for all of the semantic categories. Another rea- son is that ambiguities occur frequently in such tourism docu- ments, as they contain arbitrary information from a broad spec- trum of topics, which are not always independent. For example, text about local food may be associated with either or both of the Local Products category and the Food and Refreshment category,
Table 6-12. Comparing system results vs. human annotators, total scores By comparison with these two human annotators, the system exhibited a 65-76% level of recall whereas they each exhibit about 55-76% compared to the other, and a 55-90% level of pre- cision yet higher than either human compared to the other. Still, even the absolute recall and precision rates are higher than those superior limits reported by MUC, as described in Section 6.2.
Table 6-12. Comparing system results vs. human annotators, total scores By comparison with these two human annotators, the system exhibited a 65-76% level of recall whereas they each exhibit about 55-76% compared to the other, and a 55-90% level of pre- cision yet higher than either human compared to the other. Still, even the absolute recall and precision rates are higher than those superior limits reported by MUC, as described in Section 6.2.
Table 6-14. Evaluating system annotation vs. human Annotator 2 as as- sisted by the tool Table 6-15. Summary of the evaluation results for human assisted opin- ions
Table 6-14. Evaluating system annotation vs. human Annotator 2 as as- sisted by the tool Table 6-15. Summary of the evaluation results for human assisted opin- ions
Table 6-13. Evaluating system annotation vs. human Annotator | as as- sisted by the tool
Table 6-13. Evaluating system annotation vs. human Annotator | as as- sisted by the tool
List of indicators for the categories in the first experimental study involving the accommodation ads documents.
List of indicators for the categories in the first experimental study involving the accommodation ads documents.
List of the relevant semantic categories for the second experimen- tal case study in the domain of Tourist Boards websites and their descriptions.
List of the relevant semantic categories for the second experimen- tal case study in the domain of Tourist Boards websites and their descriptions.
The RDF(S) source of the conceptual model fragment for th Sport category.
The RDF(S) source of the conceptual model fragment for th Sport category.
List of indicators for the categories in the second experimental study involving the Tourist Board websites.
List of indicators for the categories in the second experimental study involving the Tourist Board websites.
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved