For \(A \in \mathbb{R}^{m \times n}\), \(b \in \mathbb{R}^m\) and \(x \in \mathbb{R}^n\), the normal equation is given by

\[ A^\top A x = A^\top b \]

Existence

There always exists a solution to the normal equation.

Proof

We want to prove that \(\text{Im}(A^\top A) = \text{Im}(A^\top)\).

First, it is clear that \(\text{Im}(A^\top A) \subset \text{Im}(A^\top)\). Then, we need to prove the equality of dimensions, which, with the property of finite-dimensional subspaces, allows us to conclude the equality of the two spaces.

By noting that \(\text{rank}(A) = \text{rank}(A^\top)\) (which can be justified by the \(J_r\) decomposition), it remains to show that \(\text{rank}(A) = \text{rank}(A^\top A)\).

First, we prove that \(\text{Ker}(A^\top A) = \text{Ker}(A)\).

It is clear that \(\text{Ker}(A) \subset \text{Ker}(A^\top A)\). Then, for \(x \in \text{Ker}(A^\top A)\), we have:

Thanks to the rank theorem, we have:

and

So, because \(\text{Ker}(A^\top A) = \text{Ker}(A)\), we also have the equality of dimensions and thus:

Finally, we have:

So, there exists \(x\) such that \(A^\top A x = A^\top b\).

Remarks

• \(x\) is a solution to the normal equation if and only if it minimizes the least square error \( \| Ax - b \|_2^2 \).
• For a full rank matrix \(A\), there exists a unique solution: \[ x = (A^\top A)^{-1} A^\top b \]
• \( x = (A^\top A)^+ A^\top b \) belongs to the set of solutions and minimizes the norm of \(x\).

Acknowledgment: Some insights and inspiration were taken from discussions with Maël Chaumette.

1. Install latexpand

# check whether it's already available
latexpand --version

# Ubuntu / Debian
sudo apt install texlive-extra-utils

# macOS (Homebrew)
brew install latexpand

2. Typical project layout

my_paper/
├── main.tex              ← main file
├── intro/
│   └── intro.tex
├── methods/
│   └── methods.tex
├── figures/
│   ├── fig1.pdf
│   └── fig2.pdf
├── icml.sty               ← conference style file
├── references.tex         ← or main.bbl if using BibTeX
└── bibliography.bib       ← local only, not for arXiv

3. Flatten the .tex files

From the project root:

cd my_paper/
latexpand main.tex > main_flat.tex

# keep comments
latexpand --keep-comments main.tex > main_flat.tex

# verbose, for debugging
latexpand --verbose main.tex > main_flat.tex

latexpand automatically resolves every \input{} and \include{}, including inlined bibliography files.

4. Check that it still compiles

pdflatex main_flat.tex
bibtex main_flat        # if using BibTeX
pdflatex main_flat.tex
pdflatex main_flat.tex

5. Handling citations without a .bib file

Case A — thebibliography directly in the .tex: nothing to do, latexpand already inlined it.

Case B — you used BibTeX (.bib): after compiling, a main_flat.bbl file is generated. That's what arXiv expects — not the .bib. You can also paste the .bbl content directly into main_flat.tex in place of \bibliography{refs}.

Case C — manual bibliography:

\begin{thebibliography}{99}
  \bibitem{ref1} Author A., \textit{Title}, Journal \textbf{12} (2023) 1--10.
  \bibitem{ref2} Author B., \textit{Book}, Publisher, 2021.
\end{thebibliography}

6. Clean up before submission

# preview what will be deleted
find . -type f \
  ! -name "*.tex" ! -name "*.pdf" ! -name "*.png" \
  ! -name "*.jpg" ! -name "*.eps" ! -name "*.bbl" ! -name "*.sty"

# delete — irreversible, check the preview first!
find . -type f \
  ! -name "*.tex" ! -name "*.pdf" ! -name "*.png" \
  ! -name "*.jpg" ! -name "*.eps" ! -name "*.bbl" ! -name "*.sty" \
  -delete

7. Build the arXiv archive

mkdir arxiv_submission
cp main_flat.tex  arxiv_submission/main.tex
cp main_flat.bbl  arxiv_submission/main.bbl   # if BibTeX
cp *.sty          arxiv_submission/
cp -r figures/    arxiv_submission/figures/

tar -czf submission.tar.gz arxiv_submission/

8. What arXiv expects

9. Common pitfalls

One-liner

latexpand main.tex > main_flat.tex && pdflatex main_flat.tex

Notes on running MLflow locally to track training runs for the path-conditioning experiments, using the local file store (simpler and more stable than a remote tracking server).

Recovering from a stuck server

# check what's holding the port
lsof -i :5000

# kill any lingering mlflow processes
pkill -f mlflow

# clear the tracking URI env var (avoids the UI hanging at 0%)
unset MLFLOW_TRACKING_URI

# restart cleanly from the project root
mlflow ui --host 127.0.0.1 --port 8008

Python setup

At the top of the training script, to keep runs consistent:

import mlflow

# use the local 'mlruns' folder
mlflow.set_experiment("PathCond_Stability_Analysis")

# log GPU / RAM usage alongside metrics
mlflow.config.enable_system_metrics_logging()

Grouping runs by method and architecture

To compare methods, group by architecture and average over seeds:

# df is the DataFrame returned by mlflow.search_runs()
df_stats = df.groupby(['params.method', 'num_params']).agg(
    mean_epochs=('epochs_to_target', 'mean'),
    std_epochs=('epochs_to_target', 'std')
).reset_index()

Viewing the UI from a remote server

When experiments run on a compute server but the UI should be viewed locally:

# on the server, inside tmux
mlflow ui --port 8008

# on the local machine
ssh -L 8008:localhost:8008 user@server_ip

# then open http://localhost:8008 in a browser

Long-running jobs with tmux

tmux new -s training
python train.py
# detach with Ctrl-b d

tmux new -s mlflow-server
mlflow ui --port 8008

tmux kill-server

Checklist

• No trailing slash in tracking URIs.
• mlruns/ is in .gitignore — never push raw run data to git.
• Use mlflow.pytorch.load_model(model_uri) to reload the exact model of a given run instead of recreating it by hand.