Normal-linear factor model

Data

\(x_1,x_2,\ldots,x_n \in \Re^p\)

Model

\[X_i = \mu + \Lambda Y_i + E_i\]

Normal-linear factor model

Log-likelihood

\(X_i \sim \mathrm{N}(\mu, \Sigma)\), for \(\Sigma = \Lambda \Lambda^\top + \Psi\) so that

\[ \ell(\Sigma; S) = - \frac{n}{2} \left \{ \mathrm{det}(\Sigma) + \mathrm{tr}\left(\Sigma^{-1}S \right) \right\} + C \,, \] where \(S = 1/n \sum_{i = 1}^n (x_i - \bar{x})(x_i - \bar{x})^\top\), \(\bar{x} = 1/n \sum_{i = 1}^n x_i\)

Maximum likelihood (ML) estimator

\(\hat{\Lambda}, \hat{\Psi} = \arg \max \,\ell(\Lambda \Lambda^\top + \Psi; S)\)

Job application data

Data from Hirose et al. (2011) which scores job applicants (n = 48) according to a range of criteria (p = 15)

Want to use exploratory factor analysis to find common factors underlying the data

Job application data

Fitting a normal-linear factor model with 4 factors in R using the psych package yields

fa_fit <- fa(job_data, nfactors = 4)

Estimated specific variances

fa_fit$uniquenesses

The ML estimate does not exist (Heywood (1931) case)

Maximum penalized likelihood

Add penalty function \(P(\Psi, \Lambda)\) that diverges to \(-\infty\) whenever \(\Psi \to 0\), or \(\Psi,\Lambda \to \infty\)

Penalized log-likelihood

\[\ell^*(\Lambda, \Psi; S) = \ell(\Lambda \Lambda^\top + \Psi; S) + P(\Lambda, \Psi)\]

Maximum penalized likelihood (MPL) estimator

\[\bar{\Lambda}, \bar{\Psi} = \arg \max \, \ell^*(\Lambda, \Psi; S) \tag{1}\]

Existence

If \(P(\cdot, \cdot)\) is continuous and bounded from above, then the MPL estimates of Equation 1 exist

Soft penalisation

Distort MPL estimates away from ML estimates through penalisation

Scale penalty by sequence \(c_n\) to make \(c_n P(\cdot, \cdot)\) asymtpotically negligible

Maximum softly-penalized likelihood (MSPL) estimator

\[\bar{\Lambda}, \bar{\Psi} = \arg \max \, \ell(\Lambda \Lambda^\top + \Psi; S) + c_n P(\Lambda, \Psi)\]

Soft penalisation

Choose \(c_n\) to preserve ML asymptotics whilst guaranteeing existence of estimates

Consistency

\(c_n = o_p(n)\)

\[(\bar{\Lambda}, \bar{\Psi}) \overset{p}{\to}( \Lambda, \Psi) \]

Asymptotic normality

\(c_n = o_p(n^{1/2})\)

\[\sqrt{n} \left( \begin{bmatrix} \bar{\Lambda} \\ \bar{\Psi} \end{bmatrix} - \begin{bmatrix} \hat{\Lambda} \\ \hat{\Psi} \end{bmatrix} \right) = o_p(1) \]

Candidate penalties

Huber loss

\[ P(\Psi) =- \sum_{j =1}^p g(\Psi_{ii}), \quad g(x) = \begin{cases} \frac{1}{2} \log(x)^2 & \textrm{if } |\log(x)| < 1, \\ |\log(x)| - \frac{1}{2} & \textrm{else} \end{cases} \]

Akaike (1987)

\[ P(\Lambda, \Psi) = -\frac{\rho n}{2} \mathrm{tr}\left(\Psi^{-1/2} \Lambda \Lambda^\top \Psi^{-1/2}\right) \]

Hirose et al. (2011)

\[ P(\Psi) = -\frac{\rho n}{2} \mathrm{tr}\left(\Psi^{-1/2} S \Psi^{-1/2}\right) \]

Simulation setup

Huber loss penalty with \(c_n = n^{-1/2}\) on \(\Psi\) (Huber[\(\Psi\)]) and \(\Psi,\Lambda \Lambda^\top\) (Huber[\(\Psi, \Lambda \Lambda'\)])

Akaike (1987), Hirose et al. (2011) penalties with \(\rho \in \{1, n^{-3/2} \}\)

% Used

Bias

Probabiliy of underestimation

Root mean-squared error

Model selection

Take away

Maximum softly-penalized likelihood framework is a robust alternative to ML estimation in factor analysis with desirable theoretical properties and strong finite-sample performance

• Penalisation guarantees existence of estimates

• Soft scaling preserves ML asymptotics

• Strong finite-sample & model selection performance

• Simple Huber loss penalty favourable to penalties of Hirose et al. (2011) and Akaike (1987)

Contact

Philipp Sterzinger

philipp.sterzinger@warwick.ac.uk
warwick.ac.uk/fac/sci/statistics/staff/research_students/sterzinger
psterzinger