Academia.eduAcademia.edu

Figure 8 – uploaded by Monireh Ebrahimi

See full PDFdownloadDownload figure
Towards Bridging the Neuro-Symbolic Gap: Deep Deductive Reasoners

Table 8 Towards Bridging the Neuro-Symbolic Gap: Deep Deductive Reasoners

Related Figures (33)

Table 2.1: Overview of published deep deductive reasoning work. See the main text for details on the columns. no* indicates that the paper claims that transfer is possible in principle, but it was not demonstrated or evaluated to what extent transfer really happens. first-order predicate logic (FOL) [9].
Table 2.1: Overview of published deep deductive reasoning work. See the main text for details on the columns. no* indicates that the paper claims that transfer is possible in principle, but it was not demonstrated or evaluated to what extent transfer really happens. first-order predicate logic (FOL) [9].
Figure 3.1: E£L* Completion Rule (1): ”(a) Sequence-to-Sequence - An RNN (green) pro- cesses the input sequence to create a code vector that is used to generate the output sequence (purple) using the probability chain rule and another RNN. (b) Ptr-Net - An encoding RNN converts the input sequence to a code (green) that is fed to the generating network (purple). At each step, the generating network produces a vector that modulates a content-based atten- tion mechanism over inputs. The output of the attention mechanism is a softmasx distribution with dictionary size equal to the length of the input.” lem as an input-output mapping task: Given some logic £, for each theory T’ over L, the set
Figure 3.1: E£L* Completion Rule (1): ”(a) Sequence-to-Sequence - An RNN (green) pro- cesses the input sequence to create a code vector that is used to generate the output sequence (purple) using the probability chain rule and another RNN. (b) Ptr-Net - An encoding RNN converts the input sequence to a code (green) that is fed to the generating network (purple). At each step, the generating network produces a vector that modulates a content-based atten- tion mechanism over inputs. The output of the attention mechanism is a softmasx distribution with dictionary size equal to the length of the input.” lem as an input-output mapping task: Given some logic £, for each theory T’ over L, the set
RDF Reasoning using Memory Networks: We wish to train a neural model that will learn to reason over one set of theories, and can then transfer that learning to new theories represent training and test data. To use standard neural approaches, symbols will have to be
RDF Reasoning using Memory Networks: We wish to train a neural model that will learn to reason over one set of theories, and can then transfer that learning to new theories represent training and test data. To use standard neural approaches, symbols will have to be
Table 1 Overview of published deep deductive reasoning work. See the main text for details on the columns. no* indicates that the paper claims that transfer is possible in principle, but it was not demonstrated or evaluated to what extent transfer really happens. On the Capabilities of Pointer Networks for Deep Deductive Reasoning
Table 1 Overview of published deep deductive reasoning work. See the main text for details on the columns. no* indicates that the paper claims that transfer is possible in principle, but it was not demonstrated or evaluated to what extent transfer really happens. On the Capabilities of Pointer Networks for Deep Deductive Reasoning
Fig. 1 €£+ Completion Rule (1): ”(a) Sequence-to-Sequence - An RNN (green) processes the input sequence to create a code vector that is used to generate the output sequence (purple) using the probability chain rule and another RNN. (b) Ptr-Net - An encoding RNN converts the input sequence to a code (green) that is fed to the generating network (purple). At each step, the generating network produces a vector that modulates a content- based attention mechanism over inputs. The output of the attention mechanism is a softmax distribution with dictionary size equal to the length of the input.”
Fig. 1 €£+ Completion Rule (1): ”(a) Sequence-to-Sequence - An RNN (green) processes the input sequence to create a code vector that is used to generate the output sequence (purple) using the probability chain rule and another RNN. (b) Ptr-Net - An encoding RNN converts the input sequence to a code (green) that is fed to the generating network (purple). At each step, the generating network produces a vector that modulates a content- based attention mechanism over inputs. The output of the attention mechanism is a softmax distribution with dictionary size equal to the length of the input.”
Table 3 €L+ Completion Rules
Table 3 €L+ Completion Rules
Table 4 €£L+ Knowledge Graph & Inference Knowledge Graph
Table 4 €£L+ Knowledge Graph & Inference Knowledge Graph
On the Capabilities of Pointer Networks for Deep Deductive Reasoning 4.5 Correctness
On the Capabilities of Pointer Networks for Deep Deductive Reasoning 4.5 Correctness
Table 6 Exact Match Accuracy Results for Transfer Learning/Representation: SubWord- Text Tokenization Encoding Table 7 Exact Match Accuracy Results for Transfer Learning/ Representation: Whitespace Tokenization Encoding
Table 6 Exact Match Accuracy Results for Transfer Learning/Representation: SubWord- Text Tokenization Encoding Table 7 Exact Match Accuracy Results for Transfer Learning/ Representation: Whitespace Tokenization Encoding
Fig. 1 Our work explores the capabilities and limits of subsymbolic systems (DNNs) to per- form transferrable deductive reasoning. Experiments are carried out along three investigative dimensions: type of logic, embeddings of logic to a quantitative space, and DNN architectures.
Fig. 1 Our work explores the capabilities and limits of subsymbolic systems (DNNs) to per- form transferrable deductive reasoning. Experiments are carried out along three investigative dimensions: type of logic, embeddings of logic to a quantitative space, and DNN architectures.
The semantics of EL+ is defined by means of interpretations J = (A’," ‘F) which map N7, Nc, Nr to elements, sets, and relations in the domain of interpretation Al. For an interpretation I and C(i), R(i) € &, the function -/ is shown in Table 1. Both RDFS and €L* can be expressed in first-order predicate logic.
The semantics of EL+ is defined by means of interpretations J = (A’," ‘F) which map N7, Nc, Nr to elements, sets, and relations in the domain of interpretation Al. For an interpretation I and C(i), R(i) € &, the function -/ is shown in Table 1. Both RDFS and €L* can be expressed in first-order predicate logic.
Table 3 Experimental results of proposed model
Table 3 Experimental results of proposed model
Fig. 3 PCA projection of embeddings for the vocabulary Towards Bridging the Neuro-Symbolic Gap: Deep Deductive Reasoner
Fig. 3 PCA projection of embeddings for the vocabulary Towards Bridging the Neuro-Symbolic Gap: Deep Deductive Reasoner
Table 6 Ancestor completion task with different number of axioms. Value out of the paren. theses are computed over the complete ancestor knowledge base, KB, while those withir the parenthesis are computed only on those axiom that require require multi-hop inferences KBe tered:
Table 6 Ancestor completion task with different number of axioms. Value out of the paren. theses are computed over the complete ancestor knowledge base, KB, while those withir the parenthesis are computed only on those axiom that require require multi-hop inferences KBe tered:
Table 8 The truth values of novel axioms. We additionally extended this experiment by considering KG triples from DB- pedia of the following types: nationality(Person, Country), bornIn(Person, City, and locatedIn(City, Country). In total, we collected 200 training examples and we defined some simple axioms like
Table 8 The truth values of novel axioms. We additionally extended this experiment by considering KG triples from DB- pedia of the following types: nationality(Person, Country), bornIn(Person, City, and locatedIn(City, Country). In total, we collected 200 training examples and we defined some simple axioms like
Also, we introduce k different constants that will be the domain of the Vpredn(). In our setting k and n will take the following values [4, 8, 12, 20, 30]. This means that in the setting with k = 4 constants and n = 8, for predicates of arity 3 we in- troduce 4 constants (a, b, c, d) in the model and 8 predicates (pred1, pred2,..., predg) and each predicate is universally quantified (e.g., Vz, y, z : predi(x,y, z)). We run the model for 5000 epochs with an embedding size equal to 10. Figures 5, 6, 7 show the seconds needed to complete the learning phase for each setting. These results clearly show that what impacts the most in the models is the arity of the predicate: this requires the model to create multiple combinations of the inputs to pass to the network, slowing the entire training procedure.
Also, we introduce k different constants that will be the domain of the Vpredn(). In our setting k and n will take the following values [4, 8, 12, 20, 30]. This means that in the setting with k = 4 constants and n = 8, for predicates of arity 3 we in- troduce 4 constants (a, b, c, d) in the model and 8 predicates (pred1, pred2,..., predg) and each predicate is universally quantified (e.g., Vz, y, z : predi(x,y, z)). We run the model for 5000 epochs with an embedding size equal to 10. Figures 5, 6, 7 show the seconds needed to complete the learning phase for each setting. These results clearly show that what impacts the most in the models is the arity of the predicate: this requires the model to create multiple combinations of the inputs to pass to the network, slowing the entire training procedure.
Fig. 8 First Iteration of Sequence in an Example
Fig. 8 First Iteration of Sequence in an Example
Table 12 Example SNOMED Outputs For our evaluations we use three unique edit-distance measurements. Edit dis- tance is used because it captures the degree to which each predicted statement misses what it should have been better than a simple accuracy number. We have a naive “Character” Levenshtein distance function that takes two unaltered knowl- edge base statement strings and computes their edit distance [80]. However, be- cause some names in the namespace are one digit numbers and other names are two digit numbers, we include a modified version of this function, called “Atomic”, that uniformly substitutes all two digit numbers in the strings with symbols that do not occur in either. Since there cannot be more than eight unique numbers in two decoded strings there are no issues with finding enough new symbols. By doing the substitutions we can see the impact that the number digits were having on the edits from the Atomic Levenshtein distance. Finally we devise a distance function that is based on our encoding scheme. The “Predicate” Distance method
Table 12 Example SNOMED Outputs For our evaluations we use three unique edit-distance measurements. Edit dis- tance is used because it captures the degree to which each predicted statement misses what it should have been better than a simple accuracy number. We have a naive “Character” Levenshtein distance function that takes two unaltered knowl- edge base statement strings and computes their edit distance [80]. However, be- cause some names in the namespace are one digit numbers and other names are two digit numbers, we include a modified version of this function, called “Atomic”, that uniformly substitutes all two digit numbers in the strings with symbols that do not occur in either. Since there cannot be more than eight unique numbers in two decoded strings there are no issues with finding enough new symbols. By doing the substitutions we can see the impact that the number digits were having on the edits from the Atomic Levenshtein distance. Finally we devise a distance function that is based on our encoding scheme. The “Predicate” Distance method
Table 14 Average Precision Recall and F1-score For each Distance Evaluation
Table 14 Average Precision Recall and F1-score For each Distance Evaluation
Table 13 Average Statement Edit Distances with Reasoner
Table 13 Average Statement Edit Distances with Reasoner
Fig. 10 SNOMED Training Fig. 9 Synthetic Training
Fig. 10 SNOMED Training Fig. 9 Synthetic Training
It is interesting to note that by comparing Table 14 with Table 13 we can see that on the much harder SNOMED data the deep system appears to have a better result because of the higher F1 score, but the average edit distance, which is our preferred alternative measure for evaluation, is not obviously correlated with the Fl-score. This is reflective of the shift in focus from purely accuracy optimizing systems and a more semantic type of structural learning. The “best” result will depend on which criteria is preferred.
It is interesting to note that by comparing Table 14 with Table 13 we can see that on the much harder SNOMED data the deep system appears to have a better result because of the higher F1 score, but the average edit distance, which is our preferred alternative measure for evaluation, is not obviously correlated with the Fl-score. This is reflective of the shift in focus from purely accuracy optimizing systems and a more semantic type of structural learning. The “best” result will depend on which criteria is preferred.
Figure 1: Diagram of the proposed model, for K=1
Figure 1: Diagram of the proposed model, for K=1
4 More Tricky Nos & Balanced Dataset» Completely Different Domain. Table 1
4 More Tricky Nos & Balanced Dataset» Completely Different Domain. Table 1
Table 2 F-measure and Data Distribution over each reasoning hop
Table 2 F-measure and Data Distribution over each reasoning hop
Figure 2: PCA projection of embeddings for the vocabulary
Figure 2: PCA projection of embeddings for the vocabulary
describes the computation within a single hop. The inter- nal state of the Aner vector updates for the next hop as uk+1 — y* + oF. The process repeats K times where IX is the number of computational hops. The output of the A“ hop is used to predict the label @ by passing o% and u through a weight matrix of size V x d and a softmax: ae Sey ER ee pew Feapy ke | by, 3. Model Architecture We consider a model architecture that adapts the MemN2N with fundamental alterations necessary for abstract reason- ing. A high-level view of our model is shown in Figure 1. It takes a discrete set G of normalized RDFS statements (called triples) ty, ..., tn that are stored in memory, a query q, and outputs a “yes” or “no” answer to determine if q is entailed by G. Each of the normalized t; and q contains sym- bols coming from a general dictionary with V normalized words shared among all of the normalized RDFS theories in both training and test sets. The model writes all triples to the memory and then calculates a continuous embedding for G and q. Through multiple hop attention over those continu- ous representations, the model then classifies the query. The model is trained by back-propagation of error from output to the input through multiple memory accesses. More Specifi- cally, the model is augmented with an external memory that stores the embeddings of the normalized triples in our KB. This memory is defined as an n x d tensor where n denotes the number of triples in the KB and d is the dimensionality of the embeddings. The KB is stored in the memory vectors from two continuous representations of m,; and c; obtained from two input and output embedding matrices of A and C with size d x V where V is the size of vocabulary. Similarly, the query g is embedded via a matrix B to obtain an internal state u. In each reasoning step, those memory slots useful for finding the correct answers should have their contents retrieved. To enable this, we use an attention mechanism for q over memory input representations by taking an internal product followed by a softmax:
describes the computation within a single hop. The inter- nal state of the Aner vector updates for the next hop as uk+1 — y* + oF. The process repeats K times where IX is the number of computational hops. The output of the A“ hop is used to predict the label @ by passing o% and u through a weight matrix of size V x d and a softmax: ae Sey ER ee pew Feapy ke | by, 3. Model Architecture We consider a model architecture that adapts the MemN2N with fundamental alterations necessary for abstract reason- ing. A high-level view of our model is shown in Figure 1. It takes a discrete set G of normalized RDFS statements (called triples) ty, ..., tn that are stored in memory, a query q, and outputs a “yes” or “no” answer to determine if q is entailed by G. Each of the normalized t; and q contains sym- bols coming from a general dictionary with V normalized words shared among all of the normalized RDFS theories in both training and test sets. The model writes all triples to the memory and then calculates a continuous embedding for G and q. Through multiple hop attention over those continu- ous representations, the model then classifies the query. The model is trained by back-propagation of error from output to the input through multiple memory accesses. More Specifi- cally, the model is augmented with an external memory that stores the embeddings of the normalized triples in our KB. This memory is defined as an n x d tensor where n denotes the number of triples in the KB and d is the dimensionality of the embeddings. The KB is stored in the memory vectors from two continuous representations of m,; and c; obtained from two input and output embedding matrices of A and C with size d x V where V is the size of vocabulary. Similarly, the query g is embedded via a matrix B to obtain an internal state u. In each reasoning step, those memory slots useful for finding the correct answers should have their contents retrieved. To enable this, we use an attention mechanism for q over memory input representations by taking an internal product followed by a softmax:
On the Generalization Capability of Memory Networks for Reasoning
On the Generalization Capability of Memory Networks for Reasoning
Related topics:
Memory Networks
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved