unnormalized models

Author: rcoreilly

citedrefs.json

@@ -12,7 +12,7 @@ $$
 
 This is also known as the **posterior** probability of the hypothesis. This "pure" form of the expression is in terms of the **joint probabilities** $P(h_i,d)$, which is the probability that the given hypothesis is true and the given data is present. In this form, it is clear that the conditional probability is just a form of _normalization_ where you consider one specific hypothesis $h_j$ relative to all other possible hypotheses $h_i$, and this normalization ensures that the resulting probability sums to 1. If the data are continuous, then you can replace the sum with an _integral_.
 
-The fundamental problem in dealing with probabilities is this pesky requirement that they sum to 1. This ultimately requires some way of accounting for the space of _all possible combinations of hypotheses and data_ (i.e., the normalizing denominator in [[#eq_phd]]), which entails the [[curse of dimensionality]] as the size of these spaces gets larger. Thus, all but very small-scale applications of true probability-based computations are computationally _intractable_ ([[@ChaterTenenbaumYuille06]]; [[@vanRooijWrightWareham12]]). We return to this issue below ([[#approximate solutions]]), where we see that the powerful gradient descent process described in [[search]] can be used to overcome this difficulty.
+The fundamental problem in dealing with probabilities is this pesky requirement that they sum to 1. This ultimately requires some way of accounting for the space of _all possible combinations of hypotheses and data_ (i.e., the normalizing denominator in [[#eq_phd]], also known as the **partition function**), which entails the [[curse of dimensionality]] as the size of these spaces gets larger. Thus, all but very small-scale applications of true probability-based computations are computationally _intractable_ ([[@ChaterTenenbaumYuille06]]; [[@vanRooijWrightWareham12]]). We return to this issue below ([[#approximate solutions]]), where we see that the powerful gradient descent process described in [[search]] can be used to overcome this difficulty.
 
 Bayes' theorem is, mechanically, just a simple bit of algebra to re-express the posterior probability in terms of the **likelihood** conditional probability, $P(d | h_j)$, which tells you how likely the given observed data is, assuming the hypothesis $h_j$ is true. The key advantage of a likelihood is that it is theoretically _independent_ of all the other hypotheses, and therefore could be computed directly from the given hypothesis itself (we'll see that this actually involves a bit of wishful thinking, as you can see in [[#eq_phd]] that some kind of accounting for the other hypotheses must enter somewhere).
 
@@ -32,7 +32,7 @@ $$
 P(d) = \sum_i P(h_i,d)
 $$
 
-The key idea here is that the overall probability of any given data value can be computed by adding up its probability of occuring under each of the different hypotheses. The term _marginal_ comes from the computation of these sums in the margins of a table listing all possible values of $h$ in one axis and all possible values of $d$ in the other axis (see [WikiPedia](https://en.wikipedia.org/wiki/Marginal_distribution)).
+The key idea here is that the overall probability of any given data value can be computed by adding up its probability of occuring under each of the different hypotheses. The term _marginal_ comes from the computation of these sums in the margins of a table listing all possible values of $h$ in one axis and all possible values of $d$ in the other axis (see [Wikipedia](https://en.wikipedia.org/wiki/Marginal_distribution)).
 
 Thus, we can re-write the conditional probability in a more commonly-used form:
 
@@ -75,7 +75,7 @@ However, you still end up with this pesky normalization factor of $P(d)$ in the
 
 You can get rid of the denominator by computing **odds ratios**, where the shared denominator cancels out:
 
-{id="eq_odds"}
+{id="eq_odds" title="Odds ratio"}
 $$
 O_d(h_i|h_j) = \frac{P(h_i|d)}{P(h_j|d)} = \frac{P(d|h_i) P(h_i)}{P(d|h_j) P(h_j)}
 $$
@@ -126,4 +126,5 @@ The key point here is that both of these terms are computationally tractable --
 
 Interestingly, this variational inference framework and its connection to the gradient descent computation performed in error backpropagation has also allowed the derivation of a number of different learning algorithms starting from a more abstract, generalized probabilistic framework ([[@KhanRue23]]; [[@VastolaGershmanRajan26]]).
 
+There are also other techniques for performing inference in unnormalized probabilistic models, for example by estimating the normalization denominator, and using contrastive functions that subtract away the shared normalization function ([[@GutmannHyvarinen13]]; [[@MatsudaHyvarinen19]]; [[@Hinton02]]).
 

content/bayes.md

@@ -726,6 +726,8 @@
 
 <p id="GutfreundYaromSegev95">Gutfreund, Y., Yarom, Y., & Segev, I. (1995). Subthreshold oscillations and resonant frequency in guinea-pig cortical neurons: physiology and modelling. <i>The Journal of Physiology, 483</i>, 621–640. <a href="https://onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.1995.sp020611">https://onlinelibrary.wiley.com/doi/abs/10.1113/jphysiol.1995.sp020611</a><a href="http://doi.org/10.1113/jphysiol.1995.sp020611"> http://doi.org/10.1113/jphysiol.1995.sp020611</a></p>
 
+<p id="GutmannHyvarinen13">Gutmann, M., & Hyvärinen, A. (2013). <i>Estimation of unnormalized statistical models without numerical integration. </i>In Proc. Workshop on Information Theoretic Methods in Science and Engineering (WITMSE2013) <a href="https://www.research.ed.ac.uk/en/publications/estimation-of-unnormalized-statistical-models-without-numerical-i/">https://www.research.ed.ac.uk/en/publications/estimation-of-unnormalized-statistical-models-without-numerical-i/</a></p>
+
 <p id="Haber03">Haber, S.N. (2003). The primate basal ganglia: parallel and integrative networks. <i>Journal of Chemical Neuroanatomy, 26</i>, 317–330. <a href="http://www.ncbi.nlm.nih.gov/pubmed/14729134">http://www.ncbi.nlm.nih.gov/pubmed/14729134</a></p>
 
 <p id="HaberFudgeMcFarland00">Haber, S.N., Fudge, J.L., & McFarland, N.R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. <i>The Journal of Neuroscience, 20</i>, 2369–2382. <a href="http://www.ncbi.nlm.nih.gov/pubmed/10704511">http://www.ncbi.nlm.nih.gov/pubmed/10704511</a></p>
@@ -796,6 +798,8 @@
 
 <p id="HinnekensBarbu-RothDoEtAl23">Hinnekens, E., Barbu-Roth, M., Do, M., Berret, B., & Teulier, C. (2023). Generating variability from motor primitives during infant locomotor development. <i>eLife, 12</i>, e87463. <a href="https://doi.org/10.7554/eLife.87463">https://doi.org/10.7554/eLife.87463</a><a href="http://doi.org/10.7554/eLife.87463"> http://doi.org/10.7554/eLife.87463</a></p>
 
+<p id="Hinton02">Hinton, G.E. (2002). Training products of experts by minimizing contrastive divergence. <i>Neural Computation, 14</i>, 1771–1800. <a href="http://www.ncbi.nlm.nih.gov/pubmed/12180402">http://www.ncbi.nlm.nih.gov/pubmed/12180402</a></p>
+
 <p id="HintonMcClelland88">Hinton, G.E., & McClelland, J.L. (1988). Learning representations by recirculation. In D.Z. Anderson (Ed.), <i>Neural Information Processing Systems (NIPS 1987</i> (pp. 358–366)) American Institute of Physics. <a href="http://papers.nips.cc/paper/78-learning-representations-by-recirculation.pdf">http://papers.nips.cc/paper/78-learning-representations-by-recirculation.pdf</a></p>
 
 <p id="HintonSejnowski83">Hinton, G.E., & Sejnowski, T.J. (1983). <i>Optimal perceptual inference. </i>In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. </p>
@@ -1158,6 +1162,8 @@
 
 <p id="MathyHoDavieEtAl09">Mathy, A., Ho, S.S.N., Davie, J.T., Duguid, I.C., Clark, B.A., & Häusser, M. (2009). Encoding of Oscillations by Axonal Bursts in Inferior Olive Neurons. <i>Neuron, 62</i>, 388–399. <a href="https://www.cell.com/neuron/abstract/S0896-6273(09)00248-7">https://www.cell.com/neuron/abstract/S0896-6273(09)00248-7</a><a href="http://doi.org/10.1016/j.neuron.2009.03.023"> http://doi.org/10.1016/j.neuron.2009.03.023</a></p>
 
+<p id="MatsudaHyvarinen19">Matsuda, T., & Hyvärinen, A. (2019). <i>Estimation of Non-Normalized Mixture Models. </i>In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics (pp. 2555–2563). PMLR. <a href="https://proceedings.mlr.press/v89/matsuda19a.html">https://proceedings.mlr.press/v89/matsuda19a.html</a></p>
+
 <p id="MatsumotoMinamimotoGraybielEtAl01">Matsumoto, N., Minamimoto, T., Graybiel, A.M., & Kimura, M. (2001). Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events. <i>Journal of neurophysiology, 85</i>, 960–976. <a href="http://www.ncbi.nlm.nih.gov/pubmed/11160526">http://www.ncbi.nlm.nih.gov/pubmed/11160526</a></p>
 
 <p id="MatsunoKudohWatakabeEtAl16">Matsuno, H., Kudoh, M., Watakabe, A., Yamamori, T., Shigemoto, R., & Nagao, S. (2016). Distribution and Structure of Synapses on Medial Vestibular Nuclear Neurons Targeted by Cerebellar Flocculus Purkinje Cells and Vestibular Nerve in Mice: Light and Electron Microscopy Studies. <i>PLOS ONE, 11</i>, e0164037. <a href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0164037">https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0164037</a><a href="http://doi.org/10.1371/journal.pone.0164037"> http://doi.org/10.1371/journal.pone.0164037</a></p>