% 431Assignment8.tex  Regression with measurement error, some identifiability
\documentclass[11pt]{article} 
%\usepackage{amsbsy} % for \boldsymbol and \pmb 
\usepackage{graphicx} % To include pdf files!
\usepackage{amsmath}
\usepackage{amsbsy}
\usepackage{amsfonts}
\usepackage[colorlinks=true, pdfstartview=FitV, linkcolor=blue, citecolor=blue, urlcolor=blue]{hyperref} % For links
\usepackage{fullpage}
%\pagestyle{empty} % No page numbers


\begin{document}

%\enlargethispage*{1000 pt} 


\begin{center}   
{\Large \textbf{STA 2101f19 Assignment Eight}}\footnote{This assignment was prepared by  \href{http://www.utstat.toronto.edu/~brunner}{Jerry Brunner},
Department of Statistical Sciences, University of Toronto. It is licensed under a 
\href{http://creativecommons.org/licenses/by-sa/3.0/deed.en_US}
     {Creative Commons Attribution - ShareAlike 3.0 Unported License}. Use any part of it as you like and share the result freely. The \LaTeX~source code is available from the course website:
\href{http://www.utstat.toronto.edu/~brunner/oldclass/2101f19} {\small\texttt{http://www.utstat.toronto.edu/$^\sim$brunner/oldclass/2101f19}}}
\vspace{1 mm}
\end{center}


\begin{enumerate} 

\item In the model pictured below, the explanatory variable $X$ is measured with error as well as being correlated with omitted variables. $Z$ is an instrumental variable.
\begin{center}
\includegraphics[width=2.2in]{2019A8path1}
\end{center}
        \begin{enumerate}
            \item Give the model equations without intercepts. Don't mention the expected values. 
            \item Guided by the symbols on the path diagram, provide notation for the variances and covariances of the error terms and exogenous variables. 
            \item Let $\boldsymbol{\theta}$ denote the vector of parameters you have witten down so far. These are the parameters that will appear in the covariance matrix of the observable data. What is $\boldsymbol{\theta}$?
            \item Does this model pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers. % 7 parameters, 6 equations. It would be just identifiable with kappa=0
(Notice that we are only trying to identify the parameters in $\boldsymbol{\theta}$, which is a function of the full parameter vector. The full parameter vector has intercepts and unknown probability distributions.)
            \item Calculate the covariance matrix $\boldsymbol{\Sigma}$ of $\mathbf{D_i}$, a single observable data vector. 
            \item Is the parameter $\beta$ identifiable provided $\phi_{12}\neq 0$? Answer Yes or No. If the answer is Yes, prove it. If the answer is No, give a simple numerical example of two parameter vectors with different $\beta$ values, yielding the same covariance matrix $\boldsymbol{\Sigma}$. 
            
            \item Why is is reasonable to assume $\phi_{12}\neq 0$?
            \item Now let's make the model more realistic and scary. The response variable is measured with error, so $V = Y + e_2$. Furthermore, because of omitted variables, all the error terms might be correlated with one another and with $X$. 
                \begin{enumerate}
                    \item Do your best to make a path diagram of the new model. You need not write symbols on the curved double-headed arrows you have added.
                    \item Show that $\beta$ is still identifiable.
                \end{enumerate} 
        \end{enumerate}
        

\pagebreak

\item Here is a model with two explanatory variables and two instrumental variables. The path diagram looks busy, but it has features that make sense once you think about them. The instrumental and explanatory variables have covariance matrix $\boldsymbol{\Phi} = [\phi_{ij}]$, so that for example $Var(X_1) = \phi_{33}$. No doubt there are omitted explanatory variables that are correlated with $X_1$ and $X_2$, and affect $Y$. That is the source of $c_1=Cov(X_1,\epsilon)$ and  $c_2=Cov(X_2,\epsilon)$. The variables in the latent regression model are all measured (once) with error. Because of omitted variables in the measurement process, the measurement errors are correlated, with $3 \times 3$ covariance matrix $\boldsymbol{\Omega} = [\omega_{ij}]$. 
\begin{center}
\includegraphics[width=4in]{2019A8path2} % Replace with DoubleInstrumental1 when this goes in the text.
\end{center}
        \begin{enumerate}
            \item Give the model equations without intercepts. Don't mention the expected values. 
            \item How many parameters appear in the covariance matrix of the observable data? Scanning from the bottom, I get 10+2+1+2+6=21.
            \item Does this model pass the test of the Parameter Count Rule? Answer Yes or No and give the numbers.
            \item The next step would be to calculate the covariance matrix $\boldsymbol{\Sigma}$ of the observable data vector, but that's a big job. To save work and also to reveal the essential features of the problem, please just calculate $cov\left( (Z_1,Z_2)^\top,(W_1,W_2,V)^\top \right)$.
            \item Are the parameters $\beta_1$ and $\beta_2$ identifiable? Answer Yes or No. If the answer is Yes, prove it. You don't have to finish solving for $\beta_1$ and $\beta_2$. You can stop once you have two linear equations in two unknowns, where the coefficients are $\sigma_{ij}$ quantities.  Presumably it's possible to solve two linear equations in two unknowns. To prove identifiability, you don't have to actually recover the parameters from the covariance matrix. All you have to do is show it can be done. In $\boldsymbol{\Sigma}$, please maintain the order  $Z_1,Z_2,W_1,W_2,V$ so we will have the same answer.
        \end{enumerate}

\item Here is a matrix version of instrumental variables. Independently for $i = 1, \ldots, n$, the  model equations are 
\begin{eqnarray*} 
    \mathbf{Y}_i & = &  \boldsymbol{\beta} \mathbf{X}_i + \boldsymbol{\epsilon}_i \\
    \mathbf{W}_i & = &  \mathbf{X}_i + \mathbf{e}_{i,1}              \\
    \mathbf{V}_i & = &  \mathbf{Y}_i + \mathbf{e}_{i,2}. 
\end{eqnarray*}
The random vectors $\mathbf{X}_i$ and $\mathbf{Y}_i$ are latent, while $\mathbf{W}_i$ and $\mathbf{V}_i$ are observable. In addition, there is a vector of observable instrumental variables $\mathbf{Z}_i$. The random vectors $\mathbf{X}_i$ and $\mathbf{Z}_i$ are $p \times 1$, while $\mathbf{Y}_i$ is $q \times 1$. This determines the sizes of all the matrices. The variances and covariances are as follows:
$cov(\mathbf{X}_i)=\boldsymbol{\Phi}_x$, $cov(\mathbf{Z}_i)=\boldsymbol{\Phi}_z$, $cov(\mathbf{Z}_i,\mathbf{X}_i)=\boldsymbol{\Phi}_{zx}$, 
$cov(\boldsymbol{\epsilon}_i)=\boldsymbol{\Psi}$, 
$cov(\mathbf{X}_i, \boldsymbol{\epsilon}_i)=\mathbf{C}$,
and $cov\left( \begin{array}{c} \mathbf{e}_{i,1} \\ \mathbf{e}_{i,2}  \end{array} \right) = \boldsymbol{\Omega}$. All variance-covariance matrices are positive definite (why not), and in addition, the $p \times p$ matrix of covariances $\boldsymbol{\Phi}_{zx}$ has an inverse. Covariances that are not specified are zero; in particular, the instrumental variables  have zero covariance with the error terms.

Collecting $\mathbf{Z}_i$, $\mathbf{W}_i$, $\mathbf{V}_i$ into a single long data vector $\mathbf{D}_i$, we write its variance-covariance matrix as a partitioned matrix:
\begin{displaymath} 
    \boldsymbol{\Sigma} = 
    \left( \begin{array}{c|c|c}
        \boldsymbol{\Sigma}_{11} & \boldsymbol{\Sigma}_{12} & 
        \boldsymbol{\Sigma}_{13}  \\ \hline
                                 & \boldsymbol{\Sigma}_{22} &     
        \boldsymbol{\Sigma}_{23} \\ \hline
             &  &  \boldsymbol{\Sigma}_{33} 
    \end{array} \right), 
\end{displaymath}   
where $cov(\mathbf{Z}_i, \mathbf{W}_i) = \boldsymbol{\Sigma}_{12}$, and so on.

    \begin{enumerate}
        \item Give the dimensions (number of rows and number of columns) of the following matrices: $\boldsymbol{\beta}$, $\boldsymbol{\Psi}$, $\boldsymbol{\Omega}$, $\boldsymbol{\Sigma}_{23}$.
        % qxp,              qxq,                 (p+q)x(p+q)              pxq
         
        \item This problem fails the test of the Parameter Count Rule, though you are not required to show it. Fortunately, all we care about is $\boldsymbol{\beta}$. Doing as little work as possible, prove that $\boldsymbol{\beta}$ is identifiable by showing how it can be recovered from the $\boldsymbol{\Sigma}_{ij}$ matrices.  
        \item Give the formula for an estimator of $\boldsymbol{\beta}$ and show that it is consistent.
    \end{enumerate}

% q*p + p*(p+1)/2 + p*(p+1)/2 + p^2 + q*(q+1)/2 +p*q + (p+q)*(p+q+1)/2 - (2+p+q)*(2*p+q+1)/2
% 3/2*p^2 + 3/2*p*q + 1/2*q^2 - p - 1/2*q - 1

 % Cutting out a nice section on extra response variables. It's in A8 of 431s17.

\newpage

\item For the General Structural Equation Model (see formula sheet), calculate 
    \begin{enumerate}
        \item $cov(\mathbf{Y}_i)$
        \item $cov(\mathbf{X}_i,\mathbf{Y}_i)$
    \end{enumerate}

\item In your calculation of $cov(\mathbf{Y}_i)$ and $cov(\mathbf{X}_i,\mathbf{Y}_i)$, you used the matrix $(\mathbf{I}-\boldsymbol{\beta})^{-1}$. As described in lecture, the existence of this matrix is implied by the model. Assume it does \emph{not} exist. Then the rows of $(\mathbf{I}-\boldsymbol{\beta})$ are linearly dependent, and there is a $q \times 1$ vector $\mathbf{v} \neq \mathbf{0}$ with $\mathbf{v}^\top (\mathbf{I}-\boldsymbol{\beta}) = \mathbf{0}$. Under this assumption, show 
        $\mathbf{v}^\top\boldsymbol{\Psi}\mathbf{v} = 0$, contradicting  $\boldsymbol{\Psi}$ positive definite.

    \item The following model has zero covariance between all pairs of exogenous variables, including error terms.  
\begin{eqnarray}
Y_1 &=& \gamma_1 X  + \epsilon_1 \nonumber \\
Y_2 &=& \beta Y_1 + \gamma_2 X  + \epsilon_2  \nonumber \\
W &=& X + e_1  \nonumber \\
V_1 &=& Y_1 + e_2  \nonumber \\
V_2 &=& Y_2 + e_3  \nonumber 
\end{eqnarray}

        \begin{enumerate}
            \item Draw the path diagram. Put a coefficient on each straight arrow that does not come from an error term, either the number one or a Greek letter. It is assumed that all straight arrows coming from error terms have a one.

            \item As the notation suggests, the observable variables are $W$, $V_1$ and $V_2$. Are the parameters of this model identifiable from the covariance matrix? Respond Yes or No and justify your answer.
        \end{enumerate}

% \pagebreak

    \item Consider the following model. \vspace{10mm} 

% Path diagram: Had to fiddle with this!
\begin{picture}(100,100)(0,0)           % Size of picture (does not matter), origin

    \put(116,46){$F$}  
    \put(120,50){\circle{20}}
    \put(127,59){\vector(1,1){40}} % Put at location, vector tow (1,1), length 40
    \put(135,75){$\lambda$}
    \put(129,44){\vector(1,-1){39}}      
    \put(147,25){$\lambda$}

    \put(30,46){\framebox{$D_1$}}  % Put at location, object framed by box
    \put(108,48){\vector(-1,0){52}}
    \put(35,100){$e_1$}
    \put(38,95){\vector(0,-1){35}}      

    \put(170,100){\framebox{$D_2$}}
    \put(245,100){$e_2$}
    \put(243,102){\vector(-1,0){52}}      


    \put(170,000){\framebox{$D_3$}}
    \put(245,000){$e_3$}
    \put(242,002){\vector(-1,0){50}}      

\end{picture}

        \begin{enumerate}
            \item Write the model equations without intercepts. Don't mention the expected values. Please start by writing ``Independently for $i=1, \ldots, n$, \ldots" and put a subscript $i$ on all the random variables. 
            \item Let $\boldsymbol{\theta}$ denote the vector of parameters that appear in the covariance matrix of the observable data. What is $\boldsymbol{\theta}$?
            \item Does this model pass the test of the parameter count rule? Answer Yes or No and give the numbers.
            \item  Are the elements of $\boldsymbol{\theta}$ identifiable from the covariance matrix? Answer Yes or No and prove it. If the answer is No, all you need is a simple numerical example of two distinct parameter vectors that yield the same covariance matrix of the observable data. 
            \item In a test of model fit, what would the degrees of freedom be? The answer is a single number. 
        \end{enumerate}

\item In  the following model, all random variables are normally distributed with expected value zero, and there are no intercepts. \vspace{10mm} % This is like a manipulation check problem.

% Path diagram: Had to fiddle with this!
\begin{picture}(100,100)(0,0)           % Size of picture (does not matter), origin


    \put(30,46){\framebox{$X$}}  % Put at location, object framed by box
    \put(50,50){\vector(1,0){59}}% Put at location, vector tow (1,0), length 59
    \put(75,55){$\gamma$}

    \put(112,100){\framebox{$V$}}
    \put(120,60){\vector(0,1){35}}
    \put(112,70){$1$}  
    
    \put(35,100){$e$}
    \put(45,102){\vector(1,0){65}}      

    \put(116,46){$Y_1$}  
    \put(120,50){\circle{20}}
    \put(127,59){\vector(1,1){40}} 

    %\put(75,000){$\epsilon_1$}
    %\put(82,007){\vector(1,1){32}} 

    \put(90,20){$\epsilon_1$}
    \put(97,27){\vector(1,1){15}} 


    \put(170,100){\framebox{$Y_2$}}
    \put(245,100){$\epsilon_2$}
    \put(243,102){\vector(-1,0){52}}      
    \put(135,80){$\beta_1$}

    \put(170,000){\framebox{$Y_3$}}
    \put(245,000){$\epsilon_3$}
    \put(242,002){\vector(-1,0){50}}
    \put(129,44){\vector(1,-1){39}}      
    \put(148,25){$\beta_2$}

    \put(270,52){\oval(20,100)[r]} % r for Right side of oval
    \put(270,102){\vector(-1,0){10}} % Top arrow on curve
    \put(270,002){\vector(-1,0){10}} % Bottom arrow on curve

\end{picture}

    \begin{enumerate}
        \item Write the model equations in scalar form. 
        \item What is the parameter vector $\boldsymbol{\theta}$ for this model? Use standard notation. Include unknown parameters in the covariance matrix only. 
        \item Does this model pass the test of the parameter count rule? Answer Yes or No and give both numbers.
        \item It's a bit time-consuming to write $\boldsymbol{\Sigma} = cov(X,V,Y_2,Y_3)^\top$, but it's worth it. Please do so.
        \item Verify that all the parameters are identifiable at points in the parameter space where $\gamma \neq 0$.
        \item Even where $\gamma=0$, you can tell whether $\beta_1$ and $\beta_2$ are zero, and if they are non-zero, you can identify the sign (a function of $\boldsymbol{\theta}$). Do you agree?
        \item Using the parameter count rule, there should be one model-induced equality constraint on the $\sigma_{i,j}$ quantities. Provided that $\gamma$, $\beta_1$ and $\beta_2$ are all non-zero, I can see what it is. What is the equality constraint?
    \end{enumerate}

\end{enumerate}





\end{document}




%%%%%%%%%%%%%%%%%%%%%%%%  %%%%%%%%%%%%%%%%%%%%%%%%