Jeffreys prior

\[ \pi(\theta) \propto \det(I(\theta))^{1 / 2}\,, \] for \[ I(\theta) = \textrm{E}_X \left( \nabla_\theta \ell(X \mid \theta)\nabla_\theta^\top \ell(X \mid \theta) \right) = - \textrm{E}_X\left(\nabla_\theta \nabla^\top \ell(X \mid \theta) \right) \,, \]

where \(\ell(x \mid \theta) = \log(f(x \mid \theta))\) is the model log-likelihood.

Recipe

1. Find model log-likelihood \(\ell(x \mid \theta)\)
2. Get Hessian (second derivative) of \(\ell(x \mid \theta)\)
3. Take expectation of the negative of 2. to get \(I(\theta)\)
4. Compute \(\det(I(\theta))^{1/2}\)

Question 1

A big magnetic roll tape needs tape. An experiment is being conducted in which each time \(1\) meter of the tape is examined randomly. The procedure is repeated \(5\) times and the number of defects is recorded to be \(2,2,6,0\) and \(3\) respectively. The researcher assumes a Poisson distribution with parameter \(\lambda\) for the number of defects. From previous experience, the beliefs of the researcher about \(\lambda\) can be expressed by a Gamma distribution with mean and variance equal to \(3\).

1. Derive the posterior distribution that will be obtained.

2. What would be the expected mean and variance of the number of the posterior distribution?

Question 1.a

1. \(x_1,\ldots,x_5\), where \(x_i\) is the number of defects on \(i\)th tested meter
2. \(x_i \sim \textrm{Pois}(\lambda)\)
3. \(\lambda \sim \textrm{Gamma}(a,b)\)

Question 1.a

Likelihood

\[ \begin{aligned} f(x_1, \ldots, x_n \mid \lambda) &= \prod_{i = 1}^n \frac{\lambda^{x_i} \exp\{- \lambda\}}{x_i !} \\ =& \frac{\exp\{- n\lambda\} \lambda^{ \sum_{i = 1}^n x_i} }{\prod_{i = 1}^n x_i !} \\ \end{aligned} \]

Question 1.a

Prior

Gamma distribution with parameters \(a,b\):

• Mean: \(a / b\)
• Variance: \(a / b^2\)

\(a = 3\), \(b = 1\).

\[ \pi(\lambda) = \frac{1}{\Gamma(3)} \lambda^2 \exp\{- \lambda\} \]

Question 1.a

Posterior

\[ \begin{aligned} \pi(\lambda \mid x_1,\ldots, x_n) &\propto f(x_1, \ldots, x_n \mid \lambda)\pi(\lambda) \\ &= \frac{\exp\{- n\lambda\} \lambda^{ \sum_{i = 1}^n x_i} }{\prod_{i = 1}^n x_i !} \frac{1}{\Gamma(3)} \lambda^2 \exp\{- \lambda\} \\ & \propto \exp\{- n\lambda\} \lambda^{ \sum_{i = 1}^n x_i} \lambda^2 \exp\{- \lambda\} \\ & = \exp\{- (n + 1)\lambda\} \lambda^{2 + \sum_{i = 1}^n x_i} \\ \end{aligned} \]

So we have a \(\textrm{Gamma}(3 + \sum_{i = 1}^n x_i, n + 1)\) posterior with \(\sum_{i = 1}^n x_i = 13\)

Question 1.b

Posterior

\(\textrm{Gamma}(3 + \sum_{i = 1}^n x_i, n + 1)\) posterior with \(\sum_{i = 1}^n x_i = 13\), \(n = 5\) so \(\textrm{Gamma}(16,6)\)

Posterior mean: \(16 / 6 = 8 / 3\)

Posterior variance: \(16 / 36 = 4 / 9\)

Question 2

Let \(x = (x_1, \ldots , x_n)\) be a random sample from a Negative Binomial(\(m\), \(\theta\)) distribution.

1. Set the prior for \(\theta\) to be a Beta(\(\alpha\), \(\beta\)) and derive its posterior distribution.

2. Then find the Jeffreys prior and derive the corresponding posterior.

Likelihood

\[ \begin{aligned} f(x \mid \theta) &= \prod_{i = 1}^n \binom{m + x_i - 1}{x_i} \theta^{x_i} (1 - \theta)^{m} \\ &= \left(\prod_{i = 1}^n \binom{m + x_i - 1}{x_i} \right) \theta^{\sum_{i = 1}^n x_i} (1 - \theta)^{nm} \\ & \propto \theta^{\sum_{i = 1}^n x_i} (1 - \theta)^{nm} \end{aligned} \]

Question 2.a

Prior

\[ \pi(\theta) \propto \theta^{\alpha - 1} (1 - \theta)^{\beta - 1} \,. \]

Posterior

\[ \pi(\theta \mid x) \propto \theta^{\sum_{i = 1}^n x_i} (1 - \theta)^{nm}\theta^{\alpha - 1} (1 - \theta)^{\beta - 1} = \theta^{\sum_{i = 1}^n x_i + \alpha - 1}(1 - \theta)^{nm + \beta - 1} \,, \]

which is \(\textrm{Beta}(\alpha + \sum_{i = 1}^n x_i, nm + \beta)\).

Question 2.b

log-likelihood

\[ \begin{aligned} \log (f(x \mid \theta)) &= \log\left( \prod_{i = 1}^n \binom{m + x_i - 1}{x_i} \right) + \left( \sum_{i = 1}^n x_i \right) \log(\theta) + nm \log(1 - \theta) \end{aligned} \]

\[ \frac{\partial}{\partial \theta}\log (f(x \mid \theta)) = \frac{\sum_{i = 1}^n x_i}{\theta} - \frac{nm}{1 - \theta} \]

\[ \frac{\partial^2}{\partial \theta^2}\log (f(x \mid \theta)) = -\frac{\sum_{i = 1}^n x_i}{\theta^2} - \frac{nm}{(1 - \theta)^2} \]

Question 2.b

Fisher information

\[ \begin{aligned} I(\theta) &= -\textrm{E}\left( \frac{\partial^2}{\partial \theta^2}\log (f(x \mid \theta)) \right) \\ &= \textrm{E}\left( \frac{\sum_{i = 1}^n x_i}{\theta^2} + \frac{nm}{(1 - \theta)^2} \right) \\ &= n \left( \frac{\textrm{E}(x_i)}{\theta^2} + \frac{m}{(1 - \theta)^2} \right) \\ &= n \left( \frac{\frac{m \theta}{1 - \theta}}{\theta^2} + \frac{m}{(1 - \theta)^2} \right) \\ &= \frac{nm}{1 - \theta} \left( \frac{1}{\theta} + \frac{1}{1 - \theta} \right) \\ &= \frac{nm}{\theta (1 - \theta)^2} \\ \end{aligned} \]

Question 2.b

Jeffreys prior

\[ \pi(\theta) \propto I(\theta)^{1 /2} \propto \theta^{-1/2}(1 - \theta)^{-1} \,, \]

which is proportional to a \(\textrm{Beta}(1/2, 0)\) distribution.

Hence, we get a \(\textrm{Beta}(1 / 2 + \sum_{i = 1}^n x_i, nm )\) posterior.

Question 3

Let \(x = (x_1, \ldots , x_n)\) be a random sample from a \(\mathrm{N}(\mu, \sigma^2)\) distribution with \(\mu\) known. Find the Jeffreys’ prior for \(\sigma^2\) and derive the corresponding posterior distribution.

log-likelihood

\[ \log(f(x \mid \mu, \sigma^2)) = - \frac{n}{2} \left( \log(2 \pi) + \log(\sigma^2) \right) -\frac{\sum_{i = 1}^n (x_i - \mu)^2}{2 \sigma^2} \]

\[ \frac{\partial}{\partial (\sigma^2) }\log (f(x \mid \theta)) = - \frac{n}{2 \sigma^2} + \frac{\sum_{i = 1}^n (x_i - \mu)^2}{2 (\sigma^2)^2} \]

\[ \frac{\partial^2}{\partial (\sigma^2)^2} \log (f(x \mid \theta)) = \frac{n}{2 (\sigma^2)^2} - \frac{\sum_{i = 1}^n (x_i - \mu)^2}{ (\sigma^2)^3} \]

Question 3

Fisher information

\[ \begin{aligned} I(\theta) &= -\textrm{E}\left( \frac{\partial^2}{\partial \theta^2}\log (f(x \mid \theta)) \right) \\ &= -\textrm{E}\left( \frac{n}{2 (\sigma^2)^2} - \frac{\sum_{i = 1}^n (x_i - \mu)^2}{ (\sigma^2)^3} \right) \\ &= -\frac{n}{2 (\sigma^2)^2} + \frac{n \textrm{E}[ (x_i - \mu)^2]}{ (\sigma^2)^3} \\ &= -\frac{n}{2 (\sigma^2)^2} + \frac{n \sigma^2} { (\sigma^2)^3} \\ &= \frac{n}{2 (\sigma^2)^2} \end{aligned} \]

Question 3

Jeffreys prior

\[ \pi(\sigma^2) \propto I(\sigma^2)^{1 /2} \propto (\sigma^2)^{-1} \,. \]

Posterior

\[ \begin{aligned} \pi(\sigma^2 \mid x ) \propto (\sigma^2)^{-n / 2 } \exp \left\{- \frac{\sum_{i = 1}^n (x_i - \mu)^2}{2 \sigma^2} \right\} (\sigma^2)^{-1} \\ \propto (\sigma^2)^{-(n / 2 + 1)} \exp \left\{- \frac{\sum_{i = 1}^n (x_i - \mu)^2}{2 \sigma^2} \right\} \end{aligned} \]

which is \(\textrm{IGamma}(n/2, \sum_{i = 1}^n (x_i - \mu)^2 / 2)\).

Question 4

Let \(x = (x_1, \ldots, x_n)\) be a random sample from a \(\mathrm{N}(\theta, \sigma^2)\). Assign the Jeffreys’ prior \(\pi(\theta, \sigma^2) \propto (\sigma^2)^{-1}\). Find the marginal posterior \(\pi(\theta \mid x)\).

Hint: Consider the joint posterior up to proportionality and integrate \(\sigma^2\) out, to get the marginal posterior up to proportionality. Then consider the centralised verion of \(\theta\), \(T = \frac{\theta - \bar{x}}{S / \sqrt{n}}\) and match its kernel to a \(t_{n−1}\) distribution.

Question 4

Likelihood

From Assignment 1,

\[ f(x \mid \theta, \sigma^2) \propto (\sigma^2)^{- n / 2 } \exp \left \{ - \frac{(n - 1)S^2 + n(\bar{x} - \theta)^2}{2 \sigma^2} \right\} \,. \]

Prior

\[ \pi(\theta, \sigma^2) \propto (\sigma^2)^{-1} \]

Question 4

Posterior

\[ \begin{aligned} \pi(\theta, \sigma^2 \mid x) &\propto (\sigma^2)^{- n / 2} \exp \left \{ - \frac{(n - 1)S^2 + n(\bar{x} - \theta)^2}{2 \sigma^2} \right\}(\sigma^2)^{-1} \\ &= (\sigma^2)^{- n / 2 - 1} \exp \left \{ - \frac{(n - 1) S^2 + n(\bar{x} - \theta)^2}{2 \sigma^2} \right\} \\ &= \tau^{- n / 2 - 1} \exp \left \{ - \frac{C(\theta)}{\tau} \right\} \end{aligned} \]

The last term is proportional to a \(\textrm{IGamma}(n / 2, C(\theta))\) distribution. Hence

\[ \int_0^\infty \tau^{- n / 2 - 1} \exp \left \{ - \frac{C(\theta)}{\tau} \right\} d \tau = \frac{\Gamma(n / 2)}{(C(\theta))^{n / 2}} \]

Question 4

Posterior

\[ \begin{aligned} \pi(\theta \mid x) &\propto \left((n - 1) S^2 + n(\theta - \bar{x})^2 \right)^{-n / 2} \\ &= [(n - 1)S^2]^{-n / 2} \left( 1 + \frac{1}{n - 1}\frac{(\theta - \bar{x})^2}{S^2 / n }\right)^{-n / 2} \\ & \propto \left( 1 + \frac{T^2}{n - 1} \right)^{-n / 2} \,, \end{aligned} \]

which is proportional to a \(T\)-distribution with \(n - 1\) degrees of freedom.