Understanding the Limitations of Variational Mutual Information Estimators
How to read this guide
We go through the paper in its own order, a sentence or paragraph at a time. Each excerpt of the original appears in a yellow 📄 Paper block, and the explanation follows right after it. Four other kinds of boxes are used:
💡 Intuition
The idea in plain words, with pictures. Read these even if you skip the math.🧮 Math unpacked
Derivations written step by step. Every step says why it holds.🔢 Worked example
Real numbers you can check with a calculator.⚠️ Careful
Pitfalls, hidden assumptions and typos in the paper.This part covers the Primer, Abstract and Sections 1–3. It includes an interactive widget with a slider (Widget 1) that runs a small simulation in your browser.
1. Mutual information (MI) measures how much two variables tell you about each other. It is often estimated by training a neural network, which is called a variational estimator.
2. The authors show that nearly all these estimators work the same way: guess a density ratio , normalize it, then average .
3. The normalizing step needs an average of over "unpaired" samples. The variance of that average grows like , so MINE and NWJ need exponentially large batches (and CPC is capped at ).
4. They propose SMILE, which clips to inside that average. This trades a little bias for a big drop in variance.
5. They also propose self-consistency tests (independence, data processing, additivity). No estimator passes all of them on images, which suggests "maximizing MI" in deep learning may not really be about MI.
0 · Primer: the concepts you need
The paper assumes you are comfortable with probability, information theory and some statistics. If you are, skip to the Abstract. If not, these eight short sections cover everything that comes up later.
0.1 Distributions, densities, expectations and sample averages
A random variable is a quantity whose value is uncertain, such as the pixels of a randomly chosen image. Its distribution says how likely each value is. When is continuous we describe with a density : values where is high are more likely.
The expectation of a function under is the "average value of if you drew infinitely many samples":
In practice we never have infinitely many samples. We have of them, , and we replace the expectation by a sample average. The paper writes this as , where is the empirical distribution, which puts weight on each observed sample:
Every estimator in this paper starts from an exact formula with expectations and turns it into something computable by swapping for . Most of the paper is about what goes wrong when you make that swap.
0.2 Entropy and KL divergence
The entropy measures uncertainty. The paper uses natural logarithms, so information is measured in nats (1 nat ≈ 1.44 bits).
The Kullback–Leibler (KL) divergence measures how different a distribution is from another distribution :
Three facts are used over and over:
- , with equality only when . This is Gibbs' inequality.
- KL is not symmetric: in general .
- , the Radon–Nikodym derivative, is the measure-theory name for the ratio of densities . Wherever you see , read "".
0.3 Mutual information
The mutual information answers the question: how much does knowing reduce my uncertainty about ?
The KL form is the one the paper uses. Compare two ways of generating a pair :
- Joint : draw a real pair, for example an image together with its own caption.
- Product of marginals : draw and independently, for example an image with the caption of a random other image.
If and are independent, the two procedures give the same distribution and the MI is 0. The more the real pairs differ from random pairings, the larger the MI.
Coin copy. is a fair coin and . Knowing removes all uncertainty about , so nats, which is exactly 1 bit.
Correlated Gaussians. If are standard normal with correlation , then . For this is nats, and for it is nats. With independent such coordinate pairs the MI adds up: . This is Eq. (55) of the paper, which the experiments use.
0.4 The density ratio: the central object of the paper
Define
Then, directly from the definition of KL, . Two properties of do all the work in this paper:
- It averages to 1 under : . Section 3 is built on this.
- Under it is usually tiny but occasionally huge. Most random pairings are implausible (). Once in a while a random pairing happens to look like a real pair, and then is enormous. Those rare huge values are what make the average equal 1. A quantity like this is called heavy-tailed, and its sample averages are very noisy. Section 4 is built on this. You can see it in Widget 1: compare "max r under Q" with "mean r under Q".
0.5 Estimators: bias, variance and MSE
An estimator is a recipe that turns random samples into a number. Because the samples are random, is random too. We judge it by:
- Bias
- : is it right on average?
- Variance
- : how much does it jump around from batch to batch?
- MSE
- : the overall error.
One fact about averages matters a lot here. If are i.i.d. (independent and identically distributed) with variance , then
To keep the error of an average fixed, the batch size must grow in proportion to . So if grows like , the batch has to grow like too. That is Theorem 2 in one line.
0.6 Jensen's inequality
For a concave function such as , "the log of the average ≥ the average of the logs":
MINE computes . By Jensen, , so the estimate is biased even when the sample average itself is unbiased. Sections 2.2 and 3.1 come back to this.
0.7 What "variational" means here
A variational bound rewrites a hard quantity as the best value of an optimization problem:
Two consequences follow. (a) Any particular gives a lower bound. (b) A neural network trained by gradient ascent on the objective pushes that lower bound up toward the true value. "Variational MI estimators" are exactly this: a neural network (the critic) is trained to maximize a lower bound on MI.
0.8 "Partition function"
The term comes from statistical physics. When you build a distribution as , the constant that makes it sum to 1 is called the partition function (normalizer). Every time the paper says "partition function", read "the average of the unnormalized density ratio under ". Estimating it from a mini-batch is the bottleneck the paper studies.
Abstract
Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables.
Classical MI estimators (histograms, nearest neighbours, kernel density estimates) break down in high dimensions. Imagine estimating a density over 784-pixel MNIST images with a histogram. The modern alternative trains a neural network to maximize a lower bound on MI (see 0.7). These are the "variational approaches".
However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs.
Some estimators systematically under- or over-estimate (bias). Others give wildly different answers from one mini-batch to the next (variance). Before this paper there was no clear theory explaining why.
We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI.
First contribution (Theorem 2). If the true MI is nats, the variance of MINE's normalization term is at least for a batch of size . Going from 5 to 10 nats multiplies this lower bound by about .
We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence.
Second contribution (Section 4.2 / 6.2). The true MI of real images is unknown, so we cannot check estimators against it. We can check whether they obey rules that true MI must obey, for example "two independent copies of a problem should have twice the MI". The tested estimators break these rules.
Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction.
Third contribution (Section 3 and Section 5). The authors show that all these estimators are instances of one template (Theorem 1). The template points to the exact term responsible for the variance, the partition function. They then build an estimator, SMILE, that tames that term.
Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.
On Gaussian toy problems, where the true MI is known, SMILE has a much lower mean-squared error than CPC or NWJ (Section 6.1, Table 2).
1 · Introduction
Mutual information (MI) estimation and optimization are crucial to many important problems in machine learning, such as representation learning (Chen et al., 2016; Zhao et al., 2018b; Tishby & Zaslavsky, 2015; Higgins et al., 2018) and reinforcement learning (Pathak et al., 2017; van den Oord et al., 2018).
Why should ML practitioners care?
- Representation learning: a good representation of an input should keep a lot of information about it, so you maximize . InfoGAN (Chen et al.) maximizes MI between latent codes and generated images. The Information Bottleneck (Tishby) trades off against .
- Reinforcement learning: curiosity-driven exploration (Pathak) and contrastive predictive coding (van den Oord) use information-based signals as rewards or training objectives.
Note that "estimation" (getting the number) and "optimization" (pushing it up or down during training) are different goals. The paper returns to this distinction in Section 4.2.
However, estimating mutual information from samples is challenging (McAllester & Statos, 2018) and traditional parametric and non-parametric approaches (Nemenman et al., 2004; Gao et al., 2015; 2017) struggle to scale up to modern machine learning problems, such as estimating the MI between images and learned representations.
McAllester & Stratos (the paper misspells it "Statos") proved a sobering result: any distribution-free estimator that gives a high-confidence lower bound on MI from samples cannot certify more than about nats. Keep this in mind: Section 4.1 rediscovers the same "exponential" barrier from a different angle. "Parametric" means assuming a specific distribution family (e.g. Gaussian). "Non-parametric" means methods like k-nearest neighbours. Both scale badly to images.
Recently, there has been a surge of interest in MI estimation with variational approaches (Barber & Agakov, 2003; Nguyen et al., 2010; Donsker & Varadhan, 1975), which can be naturally combined with deep learning methods (Alemi et al., 2016; van den Oord et al., 2018; Poole et al., 2019).
The three cited classics give the three basic bounds (BA, NWJ, DV) explained in Section 2.2. The modern papers plug neural networks into them. Poole et al. (2019), "On variational bounds of MI", is the closest predecessor of this paper and worth reading alongside it.
Despite their empirical effectiveness in downstream tasks such as representation learning (Hjelm et al., 2018; Velickovi´c et al., 2018), their effectiveness for MI estimation remains unclear. In particular, higher estimated MI between observations and learned representations do not seem to indicate improved predictive performance when the representations are used for downstream supervised learning tasks (Tschannen et al., 2019).
Deep InfoMax (Hjelm) and Deep Graph Infomax (Veličković) learn good representations by maximizing MI estimates. But Tschannen et al. found that a higher MI estimate does not mean better features. That is a red flag: either the estimators do not measure MI well, or MI is not what makes the features good, or both. This paper investigates the first possibility.
In this paper, we discuss two limitations of variational approaches to MI estimation. First, we theoretically demonstrate that the variance of certain estimators, such as MINE (Belghazi et al., 2018), could grow exponentially with the ground truth MI, leading to poor bias-variance trade-offs.
Limitation 1 is theoretical: variance . "Ground truth MI" means the actual, unknown value. "Poor bias-variance trade-offs" means you can reduce the variance only by accepting a lot of bias, and vice versa.
Second, we propose a set of self-consistency tests over basic properties of MI, and empirically demonstrate that all considered variational estimators fail to satisfy critical properties of MI, such as data processing and additivity under independence. These limitations challenge the effectiveness of these methods for estimating or optimizing MI.
Limitation 2 is empirical: the estimators violate rules any MI must obey. If an estimator does not behave like MI, then maximizing it is not really maximizing MI.
To mitigate these issues, we propose a unified perspective over variational estimators treating variational MI estimation as an optimization problem over (valid) density ratios. This view highlights the role of partition functions estimation, which is the culprit of high variance issues in MINE.
"Valid density ratio" means a non-negative function that averages to 1 under (defined precisely in 2.1). The authors show that every estimator is "pick a valid ratio and average its log under ". To make a ratio valid you must divide by its average under , the partition function, and estimating that from a mini-batch is what blows up.
To address this issue, we propose to improve MI estimation via variance reduction techniques for partition function estimation. Empirical results demonstrate that our estimators have much better bias-variance trade-off compared to existing methods on standard benchmark tasks.
The variance reduction technique is clipping (Section 5). Clipping a heavy-tailed variable to a bounded range caps its variance, at the price of some bias.
2 · Background and related work
2.1 Notation
We use uppercase letters to denote a probability measure (e.g., ) and corresponding lowercase letters to denote its density functions (e.g., ) unless specified otherwise.
Footnote 1: In the remainder of the paper, we slightly overload "density" for discrete random variables.
is the distribution itself, the object that assigns probabilities to events. is the function you can evaluate at a point. For discrete variables, "density" means the probability mass function . This is the "overloading" in the footnote.
We use to denote random variables with separable sample spaces denoted as and respectively, and (or ) to denote the set of all probability measures over the Borel σ-algebra on (or ).
- Sample space
- The set of possible values of , e.g. or the set of 28×28 images.
- Separable
- A technical condition (there is a countable dense subset) that rules out pathological spaces. Every space used in practice satisfies it, so you can ignore it.
- Borel σ-algebra
- The collection of "reasonable" events (sets) to which probabilities can be assigned: intervals, balls, and anything built from them by countable unions, intersections and complements.
- "All possible probability distributions on ".
Under , the p-norm of a function is defined as with . The set of locally p-integrable functions is defined as .
- : the average absolute value.
- : the root-mean-square.
- : essentially the largest value takes, ignoring sets of probability 0.
- : the bounded functions. The critics in the DV and NWJ bounds are taken from so that is guaranteed to have a finite average.
The space of probability measures wrt. is defined as ; we also call this the space of "valid density ratios" wrt. .
Any can be used to reweight into a new distribution with density . The new density is non-negative and integrates to . So is the density ratio between some legitimate distribution and . Conversely, every distribution that is absolutely continuous w.r.t. arises this way.
Example: let be uniform on and . Then and , so . It reweights the uniform distribution into the triangular density . By contrast, integrates to 1.5, so it is not valid.
We use to denote that is absolutely continuous with respect to . We use to denote an estimator for where we replace expectations with sample averages.
means "if says an event is impossible, then says so too". It guarantees that exists and has no division by zero where has mass. For MI, is the joint and is the product of marginals. Wherever the joint puts mass, both marginals are positive, so this condition is usually satisfied.
The hat convention: is a population quantity with true expectations. is the computable version with mini-batch averages. Keep the distinction in mind: Section 4 is entirely about the gap between the two.
2.2 Variational mutual information estimation
The mutual information between two random variables and is the KL divergence between the joint and the product of marginals:
which we wish to estimate using samples from ; in certain cases we may know the density of marginals (e.g. ). There are a wide range of variational approaches to variational MI estimation. Variational information maximization uses the following result (Barber & Agakov, 2003):
This is the setting: we get a dataset of pairs and nothing else, unless we happen to know exactly (for example, when is noise we generated ourselves).
Lemma 1 · Barber–Agakov (BA)
For two random variables and :
where is a valid conditional distribution over given and is the probability density function of the marginal distribution .
is a neural network that takes and outputs a distribution over . For instance, a decoder that outputs the mean and variance of a Gaussian. are its weights.
So the gap is exactly how badly the decoder approximates the true posterior. Training by maximum likelihood closes the gap.
You must be able to evaluate , the true marginal density. Equivalently , so you need to know the entropy . For real data such as images you don't.
Another family of approaches perform MI estimation through variational lower bounds to KL divergences. For example, the Mutual Information Neural Estimator (MINE, Belghazi et al. (2018)) applies the following lower bound to KL divergences (Donsker & Varadhan, 1975).
Lemma 2 · Donsker–Varadhan (DV), the basis of MINE
such that ,
is any bounded function, called the critic or "statistics network". To make the objective large, must be high on samples from (first term) while keeping small on average under (second term). In other words, learns to tell -samples from -samples.
Given any , define a new distribution by reweighting : . Then
Equality holds when , i.e. when for any constant . DV is shift-invariant: adding a constant to changes nothing, because the term subtracts it back out. Keep in mind: it is a normalized ratio and it reappears in Theorem 1.
One could set and , as a parametrized neural network (e.g. parametrized by ), and obtain the estimate by optimizing the above objective via stochastic gradient descent over mini-batches.
In practice, for a batch :
Here is a random permutation, or you can use all off-diagonal pairs , . Shuffling breaks the dependence between and and turns joint samples into product-of-marginal samples, exactly as in Widget 1.
However, the corresponding estimator (where we replace the expectations in Eq. (3) with sample averages) is biased, leading to biased gradient estimates; Belghazi et al. (2018) propose to reduce bias via estimating the partition function with exponential moving averages of mini-batches.
(a) Bias of the value. is an unbiased estimate of , but its log is not: by Jensen, . Since this term is subtracted, is biased upward relative to . That is unpleasant for something advertised as a lower bound.
(b) Bias of the gradient.
This is a ratio of expectations, and a ratio of two mini-batch averages is a biased estimate of the ratio of expectations. MINE's fix is to replace the denominator with an exponential moving average (EMA) over many past batches, which is effectively a larger batch.
The variational f-divergence estimation approach (Nguyen et al., 2010; Nowozin et al., 2016) considers lower bounds on f-divergences which can be specialize to KL divergence, and subsequently to mutual information estimation:
An f-divergence is for a convex with . KL is the case . Every f-divergence has a variational form , where is the convex conjugate (Fenchel dual). This is also the basis of f-GAN (Nowozin). For KL, , which gives:
Lemma 3 · Nguyen–Wainwright–Jordan (NWJ)
such that ,
and when .
For any , . The tangent line of at is , and is concave, so it lies below its tangent. Apply this with :
Check the optimum: with , we get and , so . ✓
Unlike DV, NWJ is not shift-invariant. The optimal is pinned to , with no free constant. That is the price for removing the log.
The supremum over is a invertible function of the density ratio , so one could use this approach to estimate density ratios by inverting the function (Nguyen et al., 2010; Nowozin et al., 2016; Grover & Ermon, 2017).
Since , once is trained you can read off a density-ratio estimate . This "estimator → density ratio" link is the seed of Section 3.
The corresponding mini-batch estimator (denoted as ) is unbiased, so unlike MINE, this approach does not require special care to reduce bias in gradients.
There is no log of an average, just two plain averages, and the average of i.i.d. terms is unbiased. Careful with the wording, though: "unbiased" means for the current . It does not mean the estimate equals the true MI. Section 4 shows that this unbiasedness comes with enormous variance.
Contrastive Predictive Coding (CPC / InfoNCE)
Contrastive Predictive Coding (CPC, van den Oord et al. (2018)) considers the following objective:
where is a neural network parametrized by and denotes the joint pdf for i.i.d. random variables sampled from .
Take a batch of real pairs. For each , line up all candidates . Exactly one of them, , is its real partner (the "positive"), and the other are "negatives". The fraction is a softmax probability of picking the right partner. So
Even a perfect classifier only gets the loss down to 0, so CPC can never report more than .
CPC generally has less variance but is more biased because its estimate does not exceed , where is the batch size (van den Oord et al., 2018; Poole et al., 2019). While one can further reduce the bias with larger , the number of evaluations needed for estimating each batch with is , which scales poorly. To address the high-bias issue of CPC, Poole et al. proposed an interpolation between and to obtain more fine-grained bias-variance trade-offs.
With the batch size used in the experiments, , CPC is capped at nats. In Figure 1 you will see CPC's curve flatten exactly at the dashed "log(bs)" line, while the true MI climbs to 10. To measure 10 nats you would need , which means critic evaluations per batch.
CPC has low variance because the softmax ratio lies in , so each term is bounded. It has high bias because of the cap. NWJ is the opposite. Poole et al.'s interpolation (often called α-InfoNCE) sits in between.
3 · Variational MI estimation as optimization over density ratios
In this section, we unify several existing methods for variational mutual information estimation. We first show that variational mutual information estimation can be formulated as a constrained optimization problem, where the feasible set is , i.e. the valid density ratios with respect to .
"Feasible set" is optimization language for "the set of candidates you are allowed to choose from". Here the candidates are the valid density ratios: non-negative functions with .
Theorem 1 · the unifying principle
such that we have
where the supremum is achived when .
"Among all ways of reweighting into a legitimate distribution, choose the one that makes the real samples from look most likely (on a log scale). The best you can do is exactly the KL divergence, and you get it by reweighting with the true ratio."
This is just maximum likelihood in disguise. Writing , the quantity is the log-likelihood of 's samples under , minus that under . The best model of is itself.
For , let be the distribution with . Then
with equality iff , i.e. . ∎ (If , then and the inequality is trivial.)
Drop the constraint and the supremum is : take , and regardless of and . An estimator is only as good as its enforcement of . Enforcing this needs , the partition function, which in practice comes from a mini-batch. Hold on to this: it is the root of the variance problem in Section 4.
We defer the proof in Appendix A. The above argument works for KL divergence between general distributions, but in this paper we focus on the special case of mutual information estimation. For the remainder of the paper, we use to represent the short-hand notation for the joint distribution and use to represent the short-hand notation for the product of marginals .
3.1 A summary of existing variational methods
From Theorem 1, we can describe a general approach to variational MI estimation:
1. Obtain a density ratio estimate – denote the solution as ;
2. Project to be close to – in practice we only have samples from , so we denote the solution as , where is the empirical distribution of i.i.d. samples from ;
3. Estimate mutual information with .
"Project" is used loosely: it means "fix so that it is (approximately) a valid density ratio", usually by dividing by an estimate of its mean under . The approximation comes from having only the samples instead of itself.
We illustrate two examples of variational mutual information estimation that can be summarized with this approach. In the case of Barber-Agakov, the proposed density ratio estimate is (assuming that is known), which is guaranteed to be in because
for all conditional distributions .
So BA's ratio is valid by construction and needs no normalization step: is the identity, there is no partition function, and there is no variance problem. That is the appeal of generative approaches. The price is that you must know .
In the case of MINE / Donsker-Varadahn, the logarithm of the density ratio is estimated with ; the corresponding density ratio might not be normalized, so one could apply the following normalization for samples:
where (the sample average) is an unbiased estimate of the partition function ; is only true when .
Step ① gives . Step ② divides by the batch average. Step ③:
The normalized ratio averages exactly 1 on the samples used to normalize it. Under the true it averages , which is a random number that is only for large . If the batch "missed" the rare points where is huge, the denominator is far too small, the ratio is over-inflated, and the estimate jumps. ( in the text is the same as ; the paper switches names.)
Similarly, we show is a lower bound to MI in Corollary 2, Appendix A, providing an alternative proof to the one in Poole et al. (2019).
CPC also fits the recipe. Its "ratio" is : it normalizes by an average over the batch of 's. The appendix shows this ratio is exactly valid (in a cleverly enlarged space), so Theorem 1 immediately makes CPC a lower bound. See Corollary 2 in the paper's Appendix A.
These examples demonstrate that different mutual information estimators can be obtained in a procedural manner by implementing the above steps, and one could involve different objectives at each step. For example, one could estimate density ratio via logistic regression (Hjelm et al., 2018; Poole et al., 2019; Mukherjee et al., 2019) while using or to estimate MI. While logistic regression does not optimize for a lower bound for KL divergence, it provides density ratio estimates between and which could be used for subsequent steps.
The recipe decouples training from evaluation. For example, train a binary classifier to tell real pairs (label 1) from shuffled pairs (label 0) with the ordinary cross-entropy loss. With balanced classes, the optimal classifier satisfies
So the classifier's logit is a log-density-ratio estimate (step ①). Plug it into the MINE or NWJ formula for steps ② and ③. SMILE does exactly this (Section 5). Logistic regression is numerically far more stable to train than DV or NWJ, because its loss is bounded and does not involve .
Table 1, rebuilt and annotated
Summarization of variational estimators of mutual information. The column denotes whether the estimator is a valid density ratio wrt. . (✓) means any parameterization is valid; () means any parameterization is valid as the batch size grows to infinity; () means only the optimal parametrization is valid (infinite training cost).
| Category | Estimator | Params | ? | Plain-English reading | |
|---|---|---|---|---|---|
| Generative | ✓ | Always valid (Eq. 7), but needs known | |||
| Generative | (Eq. 9) | Three separate density models. Valid only if all three are perfect. | |||
| Discriminative | Normalized by a noisy batch average | ||||
| Discriminative | ✓ | Valid in an enlarged space (Cor. 2), but capped at | |||
| Discriminative | (Eq. 17) | MINE with a clipped (tamed) normalizer |
3.2 Generative and discriminative approaches to MI estimation
The above discussed variational mutual information methods can be summarized into two broad categories based on how the density ratio is obtained.
• The discriminative approach estimates the density ratio directly; examples include the MINE, NWJ and CPC estimators.
• The generative approach estimates the densities of and separately; examples include the BA estimator where a conditional generative model is learned. In addition, we describe a generative approach that explicitly learns generative models (GM) for , and :
where are maximum likelihood estimates of , and respectively. We can learn the three distributions with generative models, such as VAE (Kingma & Welling, 2013) or Normalizing flows (Dinh et al., 2016), from samples.
The idea of Eq. (9): if you had all three densities, MI would just be . So fit three density models, one for the joint (trained on concatenated ), one for , one for , and plug them in. Normalizing flows give exact log-likelihoods. VAEs give only a lower bound on (the ELBO), so with VAEs is a difference of approximations and not a bound in either direction.
We summarize various generative and discriminative variational estimators in Table 1.
Differences between two approaches While both generative and discriminative approaches can be summarized with the procedure in Section 3.1, they imply different choices in modeling, estimation and optimization.
• On the modeling side, the generative approaches might require more stringent assumptions on the architectures (e.g. likelihood or evidence lower bound is tractable), whereas the discriminative approaches do not have such restrictions.
Flows must be invertible with a tractable Jacobian determinant, and VAEs need an encoder/decoder pair. A critic can be any network that outputs a number, so it is much more flexible.
• On the estimation side, generative approaches do not need to consider samples from the product of marginals (since it can model separately), yet the discriminative approaches require samples from ; if we consider a mini-batch of size , the number of evaluations for generative approaches is whereas that for discriminative approaches it could be .
means "at least on the order of". A generative model evaluates 3 log-densities per real pair, so about evaluations. A discriminative method that uses all pairs as negatives evaluates the critic times. (The separable critic from Appendix B reduces the network passes to plus one matrix product.)
• On the optimization side, discriminative approaches may need additional projection steps to be close to (such as ), while generative approaches might not need to perform this step (such as ).
This is the key asymmetry. Generative ratios are normalized by construction, because each model is a proper density. Discriminative ratios need step ②, and step ② is where exponential variance comes from (next section).
What comes next (not covered in this part)
This guide stops at the end of Section 3. Here is a preview of how the rest of the paper builds on what you have just read:
- §4.1 Theorem 2: with the optimal ratio , the variance of the batch normalizer satisfies . This is the "rare huge values" property from 0.4 made quantitative, and it means MINE/NWJ need batches of size about .
- §4.2 Self-consistency tests: independence (), data processing, and additivity ().
- §5 SMILE: , which tames step ② of the recipe.
- §6–7: experiments on Gaussians and images, and the conclusion that no estimator passes all the tests.