Academia.eduAcademia.edu

Figure 1 – uploaded by Yair Weiss

See full PDFdownloadDownload figure
Fig. 1. A small factor graph representing the joint probability distribution p(#1,%2,%3,24) = $fa(w1, 2) fB(w2,03, 04) fo (wa).

Figure 1 A small factor graph representing the joint probability distribution p(#1,%2,%3,24) = $fa(w1, 2) fB(w2,03, 04) fo (wa).

Related Figures (21)

Fig. 2. Anillustration of the definition of a region. Regions are sets of variable and factor nodes in a factor graph such that all variable nodes connected to any included factor nodes are included. Thus, the sets of nodes {1,2} and {B,C, 2,3, 4} could be regions, but {B, 3} could not be a region (since factor node B was included, variable nodes 2 and 4 must also be included.) has been described in the physics literature, and will therefore refer to such approximations as region-based approximations. We refer to the sub-class of approximations specifically gener- ated using the cluster variation method as Kikuchi approxima- tions. We begin by assuming that p(x) has the factor graph form of equation (1). We define a region R of a factor graph to be a set Vr of variable nodes and a set Fr of factor nodes, such thatif a factor node a belongs to Fr, all the variable nodes neighboring a are in Vy. We give examples of sets of nodes that could or could not be considered regions in figure 2. Note that the set FR may be empty, and that a factor a need not be included in Fp even if all its neighboring variable nodes are in Vp.
Fig. 2. Anillustration of the definition of a region. Regions are sets of variable and factor nodes in a factor graph such that all variable nodes connected to any included factor nodes are included. Thus, the sets of nodes {1,2} and {B,C, 2,3, 4} could be regions, but {B, 3} could not be a region (since factor node B was included, variable nodes 2 and 4 must also be included.) has been described in the physics literature, and will therefore refer to such approximations as region-based approximations. We refer to the sub-class of approximations specifically gener- ated using the cluster variation method as Kikuchi approxima- tions. We begin by assuming that p(x) has the factor graph form of equation (1). We define a region R of a factor graph to be a set Vr of variable nodes and a set Fr of factor nodes, such thatif a factor node a belongs to Fr, all the variable nodes neighboring a are in Vy. We give examples of sets of nodes that could or could not be considered regions in figure 2. Note that the set FR may be empty, and that a factor a need not be included in Fp even if all its neighboring variable nodes are in Vp.
Fig. 3. A factor graph which we use to illustrate a variety of region-based free energy approximations. The Bethe method is a special case of the much more gen- eral region graph method. In appendices A and B, we discuss two other important methods that are also special cases of the region graph method: the junction graph method and the clus- ter variation method. In appendix C, we discuss in detail the relationship between the different methods.
Fig. 3. A factor graph which we use to illustrate a variety of region-based free energy approximations. The Bethe method is a special case of the much more gen- eral region graph method. In appendices A and B, we discuss two other important methods that are also special cases of the region graph method: the junction graph method and the clus- ter variation method. In appendix C, we discuss in detail the relationship between the different methods.
Setting the derivative of the resulting Lagrangian Lpethe with respect to the beliefs b;(;) and b,(x,) equal to zero gives: Proof: We want to minimize the Bethe free energy, while in- sisting that all the beliefs b;(a;) and 6,(xq) are consistent. To this end, we add Lagrange multipliers \, («;) which enforce the constraint that Yixa\z; 2 a(Xa) = b;(x;) for every factor node a and all its neighboring variable nodes 7. We also need to add Lagrange multipliers to normalize the beliefs, but we do not clutter our equations with them, as their effects are automati- cally taken into account if we simply normalize our beliefs. ALYY 9 The fact that Leethe is bounded below implies that the BP lations always possess a fixed point (obtained at the global nimum of Lyethe). To our knowledge, this is the first proof the existence of BP fixed points for a general graph with ar- rary potentials. Of course, the existence of a fixed point does ‘imply that the BP algorithm will converge starting from ar- rary initial conditions.
Setting the derivative of the resulting Lagrangian Lpethe with respect to the beliefs b;(;) and b,(x,) equal to zero gives: Proof: We want to minimize the Bethe free energy, while in- sisting that all the beliefs b;(a;) and 6,(xq) are consistent. To this end, we add Lagrange multipliers \, («;) which enforce the constraint that Yixa\z; 2 a(Xa) = b;(x;) for every factor node a and all its neighboring variable nodes 7. We also need to add Lagrange multipliers to normalize the beliefs, but we do not clutter our equations with them, as their effects are automati- cally taken into account if we simply normalize our beliefs. ALYY 9 The fact that Leethe is bounded below implies that the BP lations always possess a fixed point (obtained at the global nimum of Lyethe). To our knowledge, this is the first proof the existence of BP fixed points for a general graph with ar- rary potentials. Of course, the existence of a fixed point does ‘imply that the BP algorithm will converge starting from ar- rary initial conditions.
Fig. 4. An example of a region graph. We have listed the counting number cr next to each region. Having defined region graphs, it is almost trivial to define a corresponding method for generating valid region-based free energy approximations. We simply create a region graph such that the vertices correspond to regions, with labels correspond- ing to the factor and variable nodes in a region, and we re- quire that every factor and variable node be contained in at least one region. We associate the counting numbers cr for regions directly with the counting numbers c, for the region graph, and the region graph free energy F'rg will be given by Fre = >i p¢rFr, where Fp is the free energy of the region R.
Fig. 4. An example of a region graph. We have listed the counting number cr next to each region. Having defined region graphs, it is almost trivial to define a corresponding method for generating valid region-based free energy approximations. We simply create a region graph such that the vertices correspond to regions, with labels correspond- ing to the factor and variable nodes in a region, and we re- quire that every factor and variable node be contained in at least one region. We associate the counting numbers cr for regions directly with the counting numbers c, for the region graph, and the region graph free energy F'rg will be given by Fre = >i p¢rFr, where Fp is the free energy of the region R.
Fig. 5. An example of a graph of regions that is not a region graph because the sum of the counting numbers of regions containing variable node 5 is not one. the following reasons. First, to reiterate the comments we made about valid region-based free energy approximations, the con- dition that every factor node in the factor graph is counted once when we do the weighted sum over all regions ensures that the region graph average energy is exact if the region beliefs are ex- act; and the condition that every variable node is counted once ensures that the region graph entropy is a reasonable approxi- mation. The condition that the regions containing a particular variable node form a connected sub-graph will ensure that the marginal probability at any node is consistent irrespective of which region’s beliefs one uses to compute it. Empirically, we have found in limited experiments that if one attempts to run a GBP algorithm (as described below) on graphs that do not satisfy all the region graph conditions, the results will be poor. Anexample of a “false region graph” or graph of regions that does not satisfy the region graph conditions is shown in figure 5. The problem with this plausible-looking construction is that the sum of the counting numbers of the regions containing variable node 5 is zero, rather than one. We could modify this false region graph in a variety of ways to obtain a real region graph. For example, we could simply remove node 5 from the region {2,5}. The resulting region graph would be an example of a junction graph; see appendix A. Alternatively, we could add a region {5} which just contained variable node 5, and connect the regions {2,5}, {C,4,5}, {D,5,6}, and {5, 8} to it (the result of using the cluster variation method; see appendix B).
Fig. 5. An example of a graph of regions that is not a region graph because the sum of the counting numbers of regions containing variable node 5 is not one. the following reasons. First, to reiterate the comments we made about valid region-based free energy approximations, the con- dition that every factor node in the factor graph is counted once when we do the weighted sum over all regions ensures that the region graph average energy is exact if the region beliefs are ex- act; and the condition that every variable node is counted once ensures that the region graph entropy is a reasonable approxi- mation. The condition that the regions containing a particular variable node form a connected sub-graph will ensure that the marginal probability at any node is consistent irrespective of which region’s beliefs one uses to compute it. Empirically, we have found in limited experiments that if one attempts to run a GBP algorithm (as described below) on graphs that do not satisfy all the region graph conditions, the results will be poor. Anexample of a “false region graph” or graph of regions that does not satisfy the region graph conditions is shown in figure 5. The problem with this plausible-looking construction is that the sum of the counting numbers of the regions containing variable node 5 is zero, rather than one. We could modify this false region graph in a variety of ways to obtain a real region graph. For example, we could simply remove node 5 from the region {2,5}. The resulting region graph would be an example of a junction graph; see appendix A. Alternatively, we could add a region {5} which just contained variable node 5, and connect the regions {2,5}, {C,4,5}, {D,5,6}, and {5, 8} to it (the result of using the cluster variation method; see appendix B).
Fig. 7. A region graph with no cycles that has a corresponding region graph free energy approximation which is exact. Fig. 6. A factor graph that has a tree region graph shown in figure 7.
Fig. 7. A region graph with no cycles that has a corresponding region graph free energy approximation which is exact. Fig. 6. A factor graph that has a tree region graph shown in figure 7.
Here P(£) is the set of regions that are parents to region R, D(R) is the set of all regions that are descendants of region R, E(R) = RUD(R) is the set of all regions that are descendants of R and also region RF itself, and P(D)\E(R) is the set of all regions that are parents of region D except those that are also descendants of region R or region R. so ee To be more specific, in the parent-to-child algorithm, we only have one kind of message mp_;r(xR) from a parent region to a child region. Each region R has a belief bg(xp) given by
Here P(£) is the set of regions that are parents to region R, D(R) is the set of all regions that are descendants of region R, E(R) = RUD(R) is the set of all regions that are descendants of R and also region RF itself, and P(D)\E(R) is the set of all regions that are parents of region D except those that are also descendants of region R or region R. so ee To be more specific, in the parent-to-child algorithm, we only have one kind of message mp_;r(xR) from a parent region to a child region. Each region R has a belief bg(xp) given by
Fig. 8. A region graph used to illustrate the parent-to-child GBP algorithm. Note that we do not explicitly give the variable and factor node labels for each region, as for our purposes, we are only interested in the topology of the region graph.
Fig. 8. A region graph used to illustrate the parent-to-child GBP algorithm. Note that we do not explicitly give the variable and factor node labels for each region, as for our purposes, we are only interested in the topology of the region graph.
Fig. 9. A factor graph that we will use for our detailed example of how to construct a GBP algorithm.
Fig. 9. A factor graph that we will use for our detailed example of how to construct a GBP algorithm.
In words, this equation says that the belief at each region is a product of the local factors in that region, the messages from parents, and the messages into descendant regions from other parents who are not also descendants. Recall that in the parent; to-child algorithm, we only have one kind of message mp_,r(xr) from a parent region to a child region. Each region R has a belief be(xr) given by equation (44) which we re-write here:
In words, this equation says that the belief at each region is a product of the local factors in that region, the messages from parents, and the messages into descendant regions from other parents who are not also descendants. Recall that in the parent; to-child algorithm, we only have one kind of message mp_,r(xr) from a parent region to a child region. Each region R has a belief be(xr) given by equation (44) which we re-write here:
For the decoding problem, the functions f;(x;) represent the likelihoods of the possible states of the bits, in light of the re- ceived block from the channel and the assumed channel model. Fig. 10. A region graph obtained for the factor graph of figure 9 using the cluster variation method.
For the decoding problem, the functions f;(x;) represent the likelihoods of the possible states of the bits, in light of the re- ceived block from the channel and the assumed channel model. Fig. 10. A region graph obtained for the factor graph of figure 9 using the cluster variation method.
In practice, it often helps convergence to only step the mes- sages part-way to their newly computed values. This simple heuristic can eliminate “over-shooting” problems.
In practice, it often helps convergence to only step the mes- sages part-way to their newly computed values. This simple heuristic can eliminate “over-shooting” problems.
Fig. 12. A junction graph for the factor graph shown in figure 3 generated using the Bethe method. Note the isomorphism between this junction graph and the original factor graph.
Fig. 12. A junction graph for the factor graph shown in figure 3 generated using the Bethe method. Note the isomorphism between this junction graph and the original factor graph.
First, we show that the belief equations (E-4) hold. We have so that indeed bg (xp) is product of local potentials and incom- ing messages. Of course, we have one m message and one n message for every Lagrange multiplier A, so for these definitions to hold, we also need to have constraints relating the m’s and n’s. The constraints will be given by the definitions of the pseudo- messages and the relations between the messages and the pseudo-messages that we defined above. We want to show that these relations, as well as the two-way GBP belief equations previously defined, must hold. Tuming to the constraints, we have from the definition of n%_, p(Xp), that
First, we show that the belief equations (E-4) hold. We have so that indeed bg (xp) is product of local potentials and incom- ing messages. Of course, we have one m message and one n message for every Lagrange multiplier A, so for these definitions to hold, we also need to have constraints relating the m’s and n’s. The constraints will be given by the definitions of the pseudo- messages and the relations between the messages and the pseudo-messages that we defined above. We want to show that these relations, as well as the two-way GBP belief equations previously defined, must hold. Tuming to the constraints, we have from the definition of n%_, p(Xp), that
Fig. 16. Anillustration of how one can take a region graph with some regions that have counting number zero, and obtain another region graph with no such regions but with an identical free energy. One first removes regions with a counting number of zero, and then directly connects any ancestor descendant pairs that have become disconnected. In this example, we form new direct connections between regions R and H and between regions B and H.
Fig. 16. Anillustration of how one can take a region graph with some regions that have counting number zero, and obtain another region graph with no such regions but with an identical free energy. One first removes regions with a counting number of zero, and then directly connects any ancestor descendant pairs that have become disconnected. In this example, we form new direct connections between regions R and H and between regions B and H.
Fig. 17. A small illustrative region graph (see text). Note that region F' has counting number cy = 0. equal to zero, and when one implements it, one finds that the results at its fixed points are identical to those obtained when one surgically removes the cp = 0 regions. The reason that the algorithm still gives proper results, even though the above proof breaks down, is that the constraints that cannot be de- rived from the » constraints are actually not necessary- they all involve cr = 0 regions that do not contribute to the free energy in any case.
Fig. 17. A small illustrative region graph (see text). Note that region F' has counting number cy = 0. equal to zero, and when one implements it, one finds that the results at its fixed points are identical to those obtained when one surgically removes the cp = 0 regions. The reason that the algorithm still gives proper results, even though the above proof breaks down, is that the constraints that cannot be de- rived from the » constraints are actually not necessary- they all involve cr = 0 regions that do not contribute to the free energy in any case.
Related topics:
Artificial IntelligenceComputer VisionApproximation AlgorithmsStatistical PhysicsProbabilityFixed Point TheoryBelief PropagationSum Product AlgorithmElectrical and Electronic EngineeringError Correction Code
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved