tags/toolsunfolding disastershttp://blog.tremily.us//tags/tools/unfolding disastersikiwiki2013-09-20T04:04:58ZNodehttp://blog.tremily.us//posts/Node/2013-09-20T04:04:58Z2013-09-20T04:04:58Z
<p><a href="http://nodejs.org/">Node</a> is a server-side <a href="http://en.wikipedia.org/wiki/JavaScript">JavaScript</a> engine (i.e. it executes
JavaScript without using a browser). This means that JavaScript
developers can now develop tools in their native language, so it's not
a surprise that the <a href="http://getbootstrap.com/">Bootstrap</a> folks use <a href="http://gruntjs.com/">Grunt</a> for their build
system. I'm new to the whole Node ecosystem, so here are my notes on
how it works.</p>
<p>Start off by installing <a href="https://npmjs.org/">npm</a>, the Node package manager. On Gentoo,
that's:</p>
<pre><code># USE=npm emerge -av net-libs/nodejs
</code></pre>
<p>[Configure npm][npm-config] to make <a href="https://npmjs.org/doc/files/npm-folders.html#prefix-Configuration">"global"
installs</a> in your personal space:</p>
<pre><code># npm config set prefix ~/.local/
</code></pre>
<p>Install the <a href="http://gruntjs.com/">Grunt</a> command line interface for building
<a href="http://getbootstrap.com/">Bootstrap</a>:</p>
<pre><code>$ npm install -g grunt-cli
</code></pre>
<p>That installs the libraries under <code>~/.local/lib/node_modules</code> and
drops symlinks to binaries in <code>~/.local/bin</code> (which is <a href="http://git.tremily.us/?p=dotfiles-public.git;a=blob;f=src/.bash_profile;hb=HEAD">already in my
<code>PATH</code></a> thanks to my <a href="http://blog.tremily.us//tags/tools/../../posts/dotfiles/">dotfiles</a>).</p>
<p>Clone Boostrap and <a href="https://github.com/twbs/bootstrap/blob/master/README.md#install-grunt">install it's
dependencies</a>:</p>
<pre><code>$ git clone git://github.com/twbs/bootstrap.git
$ cd bootstrap
$ npm install
</code></pre>
<p>This looks in the local [package.json][] to extract a list of
dependencies, and <a href="https://npmjs.org/doc/files/npm-folders.html">installs each of them</a> under
<a href="http://nodejs.org/api/modules.html#modules_loading_from_node_modules_folders">node_modules</a>. Node likes to isolate its packages, so every
dependency for a given package is installed underneath that package.
This leads to some crazy nesting:</p>
<pre><code>$ find node_modules/ -name graceful-fs
node_modules/grunt/node_modules/glob/node_modules/graceful-fs
node_modules/grunt/node_modules/rimraf/node_modules/graceful-fs
node_modules/grunt-contrib-clean/node_modules/rimraf/node_modules/graceful-fs
node_modules/grunt-contrib-qunit/node_modules/grunt-lib-phantomjs/node_modules/phantomjs/node_modules/rimraf/node_modules/graceful-fs
node_modules/grunt-contrib-watch/node_modules/gaze/node_modules/globule/node_modules/glob/node_modules/graceful-fs
</code></pre>
<p>Sometimes the redundancy is due to different version requirements, but
sometimes the redundancy is just redundant :p. Let's look with <a href="https://npmjs.org/doc/cli/npm-ls.html">npm
ls</a>.</p>
<pre><code>$ npm ls graceful-fs
bootstrap@3.0.0 /home/wking/src/bootstrap
├─┬ grunt@0.4.1
│ ├─┬ glob@3.1.21
│ │ └── graceful-fs@1.2.3
│ └─┬ rimraf@2.0.3
│ └── graceful-fs@1.1.14
├─┬ grunt-contrib-clean@0.5.0
│ └─┬ rimraf@2.2.2
│ └── graceful-fs@2.0.1
├─┬ grunt-contrib-qunit@0.2.2
│ └─┬ grunt-lib-phantomjs@0.3.1
│ └─┬ phantomjs@1.9.2-1
│ └─┬ rimraf@2.0.3
│ └── graceful-fs@1.1.14
└─┬ grunt-contrib-watch@0.5.3
└─┬ gaze@0.4.1
└─┬ globule@0.1.0
└─┬ glob@3.1.21
└── graceful-fs@1.2.3
</code></pre>
<p>Regardless of on-disk duplication, Node <a href="http://nodejs.org/api/modules.html#modules_caching">caches modules</a> so a
given module only loads once. If it really bothers you, you can
<a href="https://npmjs.org/doc/files/npm-folders.html#Cycles-Conflicts-and-Folder-Parsimony">avoid some duplicates by installing duplicated packages higher up in
the local tree</a>:</p>
<pre><code>$ rm -rf node_modules
$ npm install graceful-fs@1.1.14
$ npm install
$ npm ls graceful-fs
bootstrap@3.0.0 /home/wking/src/bootstrap
├── graceful-fs@1.1.14 extraneous
├─┬ grunt@0.4.1
│ └─┬ glob@3.1.21
│ └── graceful-fs@1.2.3
├─┬ grunt-contrib-clean@0.5.0
│ └─┬ rimraf@2.2.2
│ └── graceful-fs@2.0.1
└─┬ grunt-contrib-watch@0.5.3
└─┬ gaze@0.4.1
└─┬ globule@0.1.0
└─┬ glob@3.1.21
└── graceful-fs@1.2.3
</code></pre>
<p>This is probably not worth the trouble.</p>
<p>Now that we have Grunt and the Bootstrap dependencies, we can build
the distributed libraries:</p>
<pre><code>$ ~/src/node_modules/.bin/grunt dist
Running "clean:dist" (clean) task
Cleaning dist...OK
Running "recess:bootstrap" (recess) task
File "dist/css/bootstrap.css" created.
Running "recess:min" (recess) task
File "dist/css/bootstrap.min.css" created.
Original: 121876 bytes.
Minified: 99741 bytes.
Running "recess:theme" (recess) task
File "dist/css/bootstrap-theme.css" created.
Running "recess:theme_min" (recess) task
File "dist/css/bootstrap-theme.min.css" created.
Original: 18956 bytes.
Minified: 17003 bytes.
Running "copy:fonts" (copy) task
Copied 4 files
Running "concat:bootstrap" (concat) task
File "dist/js/bootstrap.js" created.
Running "uglify:bootstrap" (uglify) task
File "dist/js/bootstrap.min.js" created.
Original: 58543 bytes.
Minified: 27811 bytes.
Done, without errors.
</code></pre>
<p>Wohoo!</p>
<p>Unfortunately, like all <a href="https://pypi.python.org/pypi">language-specific</a> <a href="http://rubygems.org/">packing</a>
<a href="http://www.cpan.org/">systems</a>, npm has trouble installing packages that aren't
written in its native language. This means you get things like:</p>
<pre><code>$ ~/src/node_modules/.bin/grunt
…
Running "jekyll:docs" (jekyll) task
`jekyll build` was initiated.
Jekyll output:
Warning: Command failed: /bin/sh: jekyll: command not found
Use --force to continue.
Aborted due to warnings.
</code></pre>
<p>Once everybody wises up and starts writing packages for <a href="http://blog.tremily.us//tags/tools/../../posts/Gentoo_Prefix_overlay/">Gentoo
Prefix</a>, we can stop worrying about
installation and get back to work developing :p.</p>
Package managementhttp://blog.tremily.us//posts/Package_management/2013-05-31T19:13:43Z2013-05-31T19:02:36Z
<p>Lex Nederbragt posted <a href="http://lists.software-carpentry.org/pipermail/discuss-software-carpentry.org/2013-May/000529.html">a question about version control and
provenance</a> on the <a href="http://software-carpentry.org/">Software Carpentry</a> <a href="http://lists.software-carpentry.org/listinfo.cgi/discuss-software-carpentry.org">discussion
list</a>. I responded with my <a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1">Portage</a>-based
<a href="http://lists.software-carpentry.org/pipermail/discuss-software-carpentry.org/2013-May/000533.html">workflow</a>, but C. Titus Brown <a href="http://lists.software-carpentry.org/pipermail/discuss-software-carpentry.org/2013-May/000534.html">pointed out a number of reasons
why this approach isn't more widely used</a>, which seem to boil
down to “that sounds like more trouble than it's worth”. Because
recording the state of a system is important for <a href="http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=4720211">reproducible
research</a>, it is worth doing <em>something</em> to clean up the current
seat-of-the-pants approach.</p>
<p>Figuring out what software you have intalled on your system is
actually a (mostly) solved problem. There is a long history in the
Linux ecosystem for <a href="http://en.wikipedia.org/wiki/Package_management_system">package management systems</a> that track
installed packages and install new software (and any dependencies)
automatically. Unfortunately, there is not a consensus package
manager across distributions, with <a href="http://www.debian.org/">Debian</a>-based distributions
using <a href="http://wiki.debian.org/Apt">apt</a>, <a href="http://fedoraproject.org/">Fedora</a>-based distributions using <a href="http://yum.baseurl.org/">yum</a>, …. If
you are not the system administrator for your computer, you can either
talk your sysadmin into installing the packages you need, or use one
of a number of guest package managers (<a href="http://www.gentoo.org/proj/en/gentoo-alt/prefix/">Gentoo Prefix</a>,
<a href="http://mxcl.github.io/homebrew/">homebrew</a>, …). The guest package managers also work if you're
committed to an OS that doesn't have an existing native package
manager.</p>
<p>Despite the existence of many high quality package managers, I know
many people who continue to install significant amounts of software by
hand. While this is sustainable for a handful of packages, I see no
reason to struggle through manual installations (subsequent upgrades,
dependencies, …) when existing tools can automate the procedure. A
stopgap solution is to use language specific package managers (<a href="http://www.pip-installer.org/en/latest/">pip</a>
for <a href="http://blog.tremily.us//tags/tools/../../posts/Python/">Python</a>, <a href="http://rubygems.org/">gem</a> for <a href="http://www.ruby-lang.org/">Ruby</a>, …). This works fairly well, but
once you reach a certain level of complexity (e.g. integrating
<a href="http://en.wikipedia.org/wiki/Fortran">Fortran</a> and <a href="http://blog.tremily.us//tags/tools/../../posts/C/">C</a> extensions with <a href="http://blog.tremily.us//tags/tools/../../posts/Python/">Python</a> in <a href="http://www.scipy.org/">SciPy</a>), things
<a href="http://www.scipy.org/Installing_SciPy/BuildingGeneral#head-fe5a03a1d9ad7414becca62672c1316dc91b5f88">get difficult</a>. While language-specific packaging
standards ease automation, they are not a substitute for a
language-agnostic package manager.</p>
<p>Many distributions distribute pre-compiled, binary packages, which
give fast, stable installs without the need to have a full build
system on your local machine. When the package you need is in the
official repository (or a third-party repository), this approach works
quite well. There's no need to go through the time or effort of
compiling <a href="http://www.mozilla.org/en-US/firefox/new/">Firefox</a>, <a href="http://blog.tremily.us//tags/tools/../../posts/LaTeX/">LaTeX</a>, <a href="http://www.libreoffice.org/">LibreOffice</a>, or other software
that I interact with as a general a user. However, my own packages
(or actively developed libraries that use from my own software) are
rarely available as pre-compiled binaries. If you find yourself in
this situation, it is useful to use a package manager that makes it
easy to write source-based packages (<a href="http://www.gentoo.org/">Gentoo</a>'s <a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1">Portage</a>,
<a href="http://www.exherbo.org/">Exherbo</a>'s <a href="http://paludis.exherbo.org/">Paludis</a>, <a href="http://www.archlinux.org/">Arch</a>'s <a href="https://www.archlinux.org/pacman/">packman</a>, …).</p>
<p>With source-based packaging systems, packaging an existing Python
package is usually a matter of <a href="http://git.tremily.us/?p=wtk-overlay.git;a=blob;f=dev-python/ipythonblocks/ipythonblocks-9999.ebuild;hb=HEAD">listing a bit of
metadata</a>. With <a href="http://layman.sourceforge.net/">layman</a>, integrating your <a href="http://blog.tremily.us//tags/tools/../../posts/Gentoo_overlay/">local
packages</a> into your <a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1">Portage</a> tree is extremely
simple. Does your package depend on some other package in another
oddball language? Some wonky build tool? No problem! Just list the
new dependency in your <a href="http://www.gentoo.org/proj/en/qa/pms.xml">ebuild</a> (it probably <a href="http://gpo.zugaina.org/">already
exists</a>). Source-based package managers also make it easy
to stay up to date with ongoing development. <a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1">Portage</a> supports
live ebuilds that build fresh checkouts from a project's version
control repository (use <a href="http://blog.tremily.us//tags/tools/../../posts/Git/">Git</a>!). There is no need to dig out your
old installation notes or reread the projects installation
instructions.</p>
<p>Getting back to the goals of reproducible research, I think that
existing package managers are an excellent solution for tracking the
software used to perform experiments or run simulations and analysis.
The main stumbling block is the lack of market penetration ;).
Building a lightweight package manager that can work easily at both
the system-wide and per-user levels across a range of host OSes is
hard work. With the current fractured packaging ecosystem, I doubt
that rolling a new package manager from scratch would be an effective
approach. Existing package managers have mostly satisfied their
users, and the fundamental properties haven't changed much in over a
decade. Writing a system appealing enough to drag these satisfied
users over to your new system is probably not going to happen.</p>
<p><a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1">Portage</a> (and <a href="http://www.gentoo.org/proj/en/gentoo-alt/prefix/">Gentoo Prefix</a>) get you most of the way
there, with the help of well written <a href="http://www.gentoo.org/proj/en/qa/pms.xml">specifications</a> and
<a href="http://devmanual.gentoo.org/quickstart/">documentation</a>. However, compatibility and testing in the
prefix configuration still need some polishing, as does <a href="http://article.gmane.org/gmane.linux.gentoo.devel/84964/">robust binary
packaging support</a>. These issues are less interesting to most
<a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1">Portage</a> developers, as they usually run Portage as the native
package manager and avoid binary packages. If the broader scientific
community is interested in sustainable software, I think effort
channeled into polishing these use-cases would be time well spent.</p>
<p>For those less interested in adopting a full-fledged package manager,
you should at least make <em>some</em> effort to package your software. I
have used software that didn't even have a <code>README</code> with build
instructions, but compiling it was awful. If you're publishing your
software in the hopes that others will find it, use it, and cite you
in their subsequent paper, it behooves you to make the installation as
easy as possible. Until your community coalesces around a single
package management framework, picking a standard build system
(<a href="http://www.gnu.org/savannah-checkouts/gnu/automake/manual/html_node/Autotools-Introduction.html">Autotools</a>, <a href="http://docs.python.org/3/distutils/">Distutils</a>, …) will at least make it easier for
folks to install your software by hand.</p>
Factor analysishttp://blog.tremily.us//posts/Factor_analysis/2013-09-20T04:04:58Z2013-03-16T17:04:21Z
<p>I've been trying to wrap my head around <a href="http://en.wikipedia.org/wiki/Factor_analysis">factor analysis</a> as a
theory for designing and understanding test and survey results. This
has turned out to be <a href="http://blog.tremily.us//tags/tools/../../posts/Gumbel-Fisher-Tippett_distributions/">another</a>
one of those fields where the going has been a bit rough. I think the
key factors in making these older topics difficult are:</p>
<ul>
<li>“Everybody knows this, so we don't need to write up the details.”</li>
<li>“Hey, I can do better than Bob if I just tweak this knob…”</li>
<li>“I'll just publish this seminal paper behind a paywall…”</li>
</ul>
<p>The resulting discussion ends up being overly complicated, and it's
hard for newcomers to decide if people using similar terminology are
in fact talking about the same thing.</p>
<p>Some of the better open sources for background has been <a href="http://www.unc.edu/~rcm/book/factornew.htm">Tucker and
MacCallum's “Exploratory Factor Analysis” manuscript</a> and <a href="http://www.ics.uci.edu/~welling/classnotes/classnotes.html">Max
Welling's notes</a>. I'll use Welling's terminology for this
discussion.</p>
<p>The basic idea of factor analsys is to model <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>d</mi></math> measurable attributes
as generated by <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>k</mi><mi><</mi><mi>d</mi></math> common factors and <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>d</mi></math> unique factors.
With <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>n</mi><mo>=</mo><mn>4</mn></math> and <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>k</mi><mo>=</mo><mn>2</mn></math>, you get something like:</p>
<table class="img"><caption>Relationships between factors and measured attributes
(adapted from Tucker and MacCallum's Figure 1.2)</caption><tr><td><a href="http://blog.tremily.us//tags/tools/../../posts/Factor_analysis/factors.png"><img src="http://blog.tremily.us//tags/tools/../../posts/Factor_analysis/factors.png" width="520" height="472" alt="Relationships between factors and measured attributes" class="img" /></a></td></tr></table>
<p>Corresponding to the equation (<a href="http://www.ics.uci.edu/~welling/classnotes/papers_class/LinMod.ps.gz">Welling's eq. 1</a>):</p>
<div class="numberedEq"><span>(1)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>=</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo>+</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo>+</mo><mstyle mathvariant="bold"><mi>ν</mi></mstyle></math></div>
<p>The independent random variables <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>y</mi></mstyle></math> are distributed
according to a Gaussian with zero mean and unit
variance <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msub><mi>𝒢</mi> <mstyle mathvariant="bold"><mi>y</mi></mstyle></msub><mo stretchy="false">[</mo><mn>0</mn><mo>,</mo><mstyle mathvariant="bold"><mi>I</mi></mstyle><mo stretchy="false">]</mo></math> (zero mean because
constant offsets are handled by <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>μ</mi></mstyle></math>; unit variance because
scaling is handled by <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math>). The independent random
variables <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>ν</mi></mstyle></math> are distributed according
to <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msub><mi>𝒢</mi> <mstyle mathvariant="bold"><mi>ν</mi></mstyle></msub><mo stretchy="false">[</mo><mn>0</mn><mo>,</mo><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><mo stretchy="false">]</mo></math>, with (Welling's
eq. 2):</p>
<div class="numberedEq"><span>(2)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><mo>≡</mo><mtext>diag</mtext><mo stretchy="false">[</mo><msubsup><mi>σ</mi> <mn>1</mn> <mn>2</mn></msubsup><mo>,</mo><mi>…</mi><mo>,</mo><msubsup><mi>σ</mi> <mi>d</mi> <mn>2</mn></msubsup><mo stretchy="false">]</mo></math></div>
<p>The matrix <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math> (linking common factors with measured
attributes <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo><mi>is</mi><mi>refered</mi><mi>to</mi><mi>as</mi><mi>the</mi><mo><em></mo><mi>factor</mi><mi>weights</mi><mo></em></mo><mi>or</mi><mo><em></mo><mi>factor</mi><mi>loadings</mi><mo></em></mo><mo>.</mo><mi>Because</mi><mi>the</mi><mi>only</mi><mi>source</mi><mi>of</mi><mi>constant</mi><mi>offset</mi><mi>is</mi></math>\mathbf{\mu}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>,</mo><mi>we</mi><mi>can</mi><mi>calculate</mi><mi>it</mi><mi>by</mi><mi>averaging</mi><mi>out</mi><mi>the</mi><mi>random</mi><mi>noise</mi><mo stretchy="false">(</mo><mi>Welling</mi><mo>′</mo><mi>s</mi><mi>eq</mi><mo>.</mo><mn>6</mn><mo stretchy="false">)</mo><mo>:</mo><merror><mtext>Unknown character</mtext></merror><mi>div</mi><mi>class</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mi>numberedEq</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>span</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">(</mo><mn>3</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>span</mi><merror><mtext>Unknown character</mtext></merror></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><msubsup><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow> <mi>N</mi></msubsup><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>div</mi><merror><mtext>Unknown character</mtext></merror><mi>where</mi></math>N<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>is</mi><mi>the</mi><mi>number</mi><mi>of</mi><mi>measurements</mi><mo stretchy="false">(</mo><mi>survey</mi><mi>responders</mi><mo stretchy="false">)</mo><mi>and</mi></math>\mathbf{x}<em>n<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>is</mi><mi>the</mi><mi>response</mi><mi>vector</mi><mi>for</mi><mi>the</mi></math>n^\text{th}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>responder</mi><mo>.</mo><mi>How</mi><mi>do</mi><mi>we</mi><mi>find</mi></math>\mathbf{A}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>and</mi></math>\mathbf{\Sigma}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>?</mo><mi>This</mi><mi>is</mi><mi>the</mi><mi>tricky</mi><mi>bit</mi><mo>,</mo><mi>and</mi><mi>there</mi><mi>are</mi><mi>a</mi><mi>number</mi><mi>of</mi><mi>possible</mi><mi>approaches</mi><mo>.</mo><mi>Welling</mi><mi>suggests</mi><mi>using</mi><mi>expectation</mi><mi>maximization</mi><mo stretchy="false">(</mo><mi>EM</mi><mo stretchy="false">)</mo><mo>,</mo><mi>and</mi><mi>there</mi><mo>′</mo><mi>s</mi><mi>an</mi><mi>excellent</mi><mi>example</mi><mi>of</mi><mi>the</mi><mi>procedure</mi><mi>with</mi><mi>a</mi><mi>colorblind</mi><mi>experimenter</mi><mi>drawing</mi><mi>colored</mi><mi>balls</mi><mi>in</mi><mi>his</mi><mo stretchy="false">[</mo><mi>EM</mi><mi>notes</mi><mo stretchy="false">]</mo><mo stretchy="false">[</mo><mi>EM</mi><mo stretchy="false">]</mo><mo stretchy="false">(</mo><mi>to</mi><mi>test</mi><mi>my</mi><mi>understanding</mi><mo>,</mo><mi>I</mi><mi>wrote</mi><merror><mtext>Unknown character</mtext></merror><mi>a</mi><mi>href</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mo>.</mo><mo>.</mo><mo stretchy="false">/</mo><mo>.</mo><mo>.</mo><mo stretchy="false">/</mo><mi>posts</mi><mo stretchy="false">/</mo><msub><mi>Factor</mi> <mi>analysis</mi></msub><mo stretchy="false">/</mo><mi>color</mi><mo>−</mo><mi>ball</mi><mo>.</mo><mi>py</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>color</mi><mo>−</mo><mi>ball</mi><mo>.</mo><mi>py</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>a</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">)</mo><mo>.</mo><mi>To</mi><mi>simplify</mi><mi>calculations</mi><mo>,</mo><mi>Welling</mi><mi>defines</mi><mo stretchy="false">(</mo><mi>before</mi><mi>eq</mi><mo>.</mo><mn>15</mn><mo stretchy="false">)</mo><mo>:</mo><merror><mtext>Unknown character</mtext></merror><mi>div</mi><mi>class</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mi>numberedEq</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>span</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">(</mo><mn>4</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>span</mi><merror><mtext>Unknown character</mtext></merror></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mrow><mtable columnalign="right left right left right left right left right left" columnspacing="0em"><mtr><mtd><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo>′</mo></mtd> <mtd><mo>≡</mo><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo>,</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo stretchy="false">]</mo></mtd></mtr> <mtr><mtd><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo>′</mo></mtd> <mtd><mo>≡</mo><mo stretchy="false">[</mo><msup><mstyle mathvariant="bold"><mi>y</mi></mstyle> <mi>T</mi></msup><mo>,</mo><mn>1</mn><msup><mo stretchy="false">]</mo> <mi>T</mi></msup></mtd></mtr></mtable></mrow></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>div</mi><merror><mtext>Unknown character</mtext></merror><mi>which</mi><mi>reduce</mi><mi>the</mi><mi>model</mi><mi>to</mi><merror><mtext>Unknown character</mtext></merror><mi>div</mi><mi>class</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mi>numberedEq</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>span</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">(</mo><mn>5</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>span</mi><merror><mtext>Unknown character</mtext></merror></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>=</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo>′</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo>′</mo><mo lspace="verythinmathspace" rspace="0em">+</mo><mstyle mathvariant="bold"><mi>ν</mi></mstyle></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>div</mi><merror><mtext>Unknown character</mtext></merror><mi>After</mi><mi>some</mi><mi>manipulation</mi><mi>Welling</mi><mi>works</mi><mi>out</mi><mi>the</mi><mi>maximizing</mi><mi>updates</mi><mo stretchy="false">(</mo><mi>eq</mi><mo>′</mo><mi>ns</mi><mn>16</mn><mi>and</mi><mn>17</mn><mo stretchy="false">)</mo><mo>:</mo><merror><mtext>Unknown character</mtext></merror><mi>div</mi><mi>class</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mi>numberedEq</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>span</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">(</mo><mn>6</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>span</mi><merror><mtext>Unknown character</mtext></merror></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mrow><mtable columnalign="right left right left right left right left right left" columnspacing="0em"><mtr><mtd><mstyle mathvariant="bold"><mi>A</mi></mstyle><msup><mo>′</mo> <mtext>new</mtext></msup></mtd> <mtd><mo>=</mo><mrow><mo>(</mo><msubsup><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow> <mi>N</mi></msubsup><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo>′</mo><mo stretchy="false">∣</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><msup><mo stretchy="false">]</mo> <mi>T</mi></msup><mo>)</mo></mrow><msup><mrow><mo>(</mo><msubsup><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow> <mi>N</mi></msubsup><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo>′</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><msup><mo>′</mo> <mi>T</mi></msup><mo stretchy="false">∣</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mo stretchy="false">]</mo><mo>)</mo></mrow> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup></mtd></mtr> <mtr><mtd><msup><mstyle mathvariant="bold"><mi>Σ</mi></mstyle> <mtext>new</mtext></msup></mtd> <mtd><mo>=</mo><mfrac><mn>1</mn><mi>N</mi></mfrac><msubsup><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>n</mi><mo>=</mo><mn>1</mn></mrow> <mi>N</mi></msubsup><mtext>diag</mtext><mo stretchy="false">[</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><msubsup><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi> <mi>T</mi></msubsup><mo>−</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><msup><mo>′</mo> <mtext>new</mtext></msup><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo>′</mo><mo stretchy="false">∣</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mo stretchy="false">]</mo><msubsup><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi> <mi>T</mi></msubsup><mo stretchy="false">]</mo></mtd></mtr></mtable></mrow></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>div</mi><merror><mtext>Unknown character</mtext></merror><mi>The</mi><mi>expectation</mi><mi>values</mi><mi>used</mi><mi>in</mi><mi>these</mi><mi>updates</mi><mi>are</mi><mi>given</mi><mi>by</mi><mo stretchy="false">(</mo><mi>Welling</mi><mo>′</mo><mi>s</mi><mi>eq</mi><mo>′</mo><mi>ns</mi><mn>12</mn><mi>and</mi><mn>13</mn><mo stretchy="false">)</mo><mo>:</mo><merror><mtext>Unknown character</mtext></merror><mi>div</mi><mi>class</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mi>numberedEq</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>span</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">(</mo><mn>7</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>span</mi><merror><mtext>Unknown character</mtext></merror></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mrow><mtable columnalign="right left right left right left right left right left" columnspacing="0em"><mtr><mtd><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">∣</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mo stretchy="false">]</mo></mtd> <mtd><mo>=</mo><msup><mstyle mathvariant="bold"><mi>A</mi></mstyle> <mi>T</mi></msup><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><msup><mstyle mathvariant="bold"><mi>A</mi></mstyle> <mi>T</mi></msup><mo>+</mo><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><msup><mo stretchy="false">)</mo> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup><mo stretchy="false">(</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mo>−</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo stretchy="false">)</mo></mtd></mtr> <mtr><mtd><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><msup><mstyle mathvariant="bold"><mi>y</mi></mstyle> <mi>T</mi></msup><mo stretchy="false">∣</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mo stretchy="false">]</mo></mtd> <mtd><mo>=</mo><mstyle mathvariant="bold"><mi>I</mi></mstyle><mo>−</mo><msup><mstyle mathvariant="bold"><mi>A</mi></mstyle> <mi>T</mi></msup><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><msup><mstyle mathvariant="bold"><mi>A</mi></mstyle> <mi>T</mi></msup><mo>+</mo><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><msup><mo stretchy="false">)</mo> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo>+</mo><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">∣</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><mo stretchy="false">]</mo><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">∣</mo><msub><mstyle mathvariant="bold"><mi>x</mi></mstyle> <mi>n</mi></msub><msup><mo stretchy="false">]</mo> <mi>T</mi></msup></mtd></mtr></mtable></mrow></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>div</mi><merror><mtext>Unknown character</mtext></merror><mi>Survey</mi><mi>analysis</mi><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mi>Enough</mi><mi>abstraction</mi><mo>!</mo><mi>Let</mi><mo>′</mo><mi>s</mi><mi>look</mi><mi>at</mi><mi>an</mi><mi>example</mi><mo>:</mo><mo stretchy="false">[</mo><mi>survey</mi><mi>results</mi><mo stretchy="false">]</mo><mo stretchy="false">[</mo><mi>survey</mi><mo stretchy="false">]</mo><mo>:</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>import</mi><mi>numpy</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>scores</mi><mo>=</mo><mi>numpy</mi><mo>.</mo><mi>genfromtxt</mi><mo stretchy="false">(</mo><mo>′</mo><msub><mi>Factor</mi> <mi>analysis</mi></msub><mo stretchy="false">/</mo><mi>survey</mi><mo>.</mo><mi>data</mi><mo>′</mo><mo>,</mo><mi>delimiter</mi><mo>=</mo><mo>′</mo><mo lspace="0em" rspace="thinmathspace">t</mo><mo>′</mo><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>scores</mi><mi>array</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mo stretchy="false">[</mo><mn>1</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>1</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo>,</mo><mn>4</mn><mo>.</mo><mo stretchy="false">]</mo><mo>,</mo><mo stretchy="false">[</mo><mn>2</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>5</mn><mo>.</mo><mo>,</mo><mn>7</mn><mo>.</mo><mo>,</mo><mn>6</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo>,</mo><mn>3</mn><mo>.</mo><mo stretchy="false">]</mo><mo stretchy="false">]</mo><mo stretchy="false">)</mo><mo><code></mo><mi>scores</mi><mo stretchy="false">[</mo><mi>i</mi><mo>,</mo><mi>j</mi><mo stretchy="false">]</mo><mo></code></mo><mi>is</mi><mi>the</mi><mi>answer</mi><mi>the</mi><mo><code></mo><mi>i</mi><mo></code></mo><mi>th</mi><mi>respondent</mi><mi>gave</mi><mi>for</mi><mi>the</mi><mo><code></mo><mi>j</mi><mo></code></mo><mi>th</mi><mi>question</mi><mo>.</mo><mi>We</mi><mo>′</mo><mi>re</mi><mi>looking</mi><mi>for</mi><mi>underlying</mi><mi>factors</mi><mi>that</mi><mi>can</mi><mi>explain</mi><mi>covariance</mi><mi>between</mi><mi>the</mi><mi>different</mi><mi>questions</mi><mo>.</mo><mi>Do</mi><mi>the</mi><mi>question</mi><mi>answers</mi><mo stretchy="false">(</mo></math>\mathbf{x}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo stretchy="false">)</mo><mi>represent</mi><mi>some</mi><mi>underlying</mi><mi>factors</mi><mo stretchy="false">(</mo></math>\mathbf{y}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo stretchy="false">)</mo><mo>?</mo><mi>Let</mi><mo>′</mo><mi>s</mi><mi>start</mi><mi>off</mi><mi>by</mi><mi>calculating</mi></math>\mathbf{\mu}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>:</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>def</mi><msub><mi>print</mi> <mi>row</mi></msub><mo stretchy="false">(</mo><mi>row</mi><mo stretchy="false">)</mo><mo>:</mo><mo>.</mo><mo>.</mo><mo>.</mo><mi>print</mi><mo stretchy="false">(</mo><mo>′</mo><mo>′</mo><mo>.</mo><mi>join</mi><mo stretchy="false">(</mo><mo>′</mo><mrow><mo>:</mo><mn>0.2</mn><mi>f</mi></mrow><mo>′</mo><mo>.</mo><mi>format</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi>for</mi><mi>x</mi><mi>in</mi><mi>row</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>mu</mi><mo>=</mo><mi>scores</mi><mo>.</mo><mi>mean</mi><mo stretchy="false">(</mo><mi>axis</mi><mo>=</mo><mn>0</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>print</mi> <mi>row</mi></msub><mo stretchy="false">(</mo><mi>mu</mi><mo stretchy="false">)</mo><mn>2.42</mn><mn>3.58</mn><mn>5.08</mn><mn>5.75</mn><mn>6.08</mn><mn>3.42</mn><mn>3.92</mn><mn>4.25</mn><mi>Next</mi><mi>we</mi><mi>need</mi><mi>priors</mi><mi>for</mi></math>\mathbf{A}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>and</mi></math>\mathbf{\Sigma}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>.</mo><merror><mtext>Unknown character</mtext></merror><mi>span</mi><mi>class</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mi>createlink</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>MDP</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>span</mi><merror><mtext>Unknown character</mtext></merror><mi>has</mi><mi>an</mi><mi>implementation</mi><mi>for</mi><merror><mtext>Unknown character</mtext></merror><mi>a</mi><mi>href</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mo>.</mo><mo>.</mo><mo stretchy="false">/</mo><mo>.</mo><mo>.</mo><mo stretchy="false">/</mo><mi>posts</mi><mo stretchy="false">/</mo><mi>Python</mi><mo stretchy="false">/</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>Python</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>a</mi><merror><mtext>Unknown character</mtext></merror><mo>,</mo><mi>and</mi><mi>their</mi><mo stretchy="false">[</mo><mi>FANode</mi><mo stretchy="false">]</mo><mo stretchy="false">[</mo><mo stretchy="false">]</mo><mi>uses</mi><mi>a</mi><mi>Gaussian</mi><mi>random</mi><mi>matrix</mi><mi>for</mi></math>\mathbf{A}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>and</mi><mi>the</mi><mi>diagonal</mi><mi>of</mi><mi>the</mi><mi>score</mi><mi>covariance</mi><mi>for</mi></math>\mathbf{\Sigma}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>.</mo><mi>They</mi><mi>also</mi><mi>use</mi><mi>the</mi><mi>score</mi><mi>covariance</mi><mi>to</mi><mi>avoid</mi><mi>repeated</mi><mi>summations</mi><mi>over</mi></math>n<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>.</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>import</mi><mi>mdp</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>def</mi><msub><mi>print</mi> <mi>matrix</mi></msub><mo stretchy="false">(</mo><mi>matrix</mi><mo stretchy="false">)</mo><mo>:</mo><mo>.</mo><mo>.</mo><mo>.</mo><mi>for</mi><mi>row</mi><mi>in</mi><mi>matrix</mi><mo>:</mo><mo>.</mo><mo>.</mo><mo>.</mo><msub><mi>print</mi> <mi>row</mi></msub><mo stretchy="false">(</mo><mi>row</mi><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>fa</mi><mo>=</mo><mi>mdp</mi><mo>.</mo><mi>nodes</mi><mo>.</mo><mi>FANode</mi><mo stretchy="false">(</mo><msub><mi>output</mi> <mi>dim</mi></msub><mo>=</mo><mn>3</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>numpy</mi><mo>.</mo><mi>random</mi><mo>.</mo><mi>seed</mi><mo stretchy="false">(</mo><mn>1</mn><mo stretchy="false">)</mo><mo>#</mo><mi>for</mi><mi>consistend</mi><mi>doctest</mi><mi>results</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>responder</mi> <mi>scores</mi></msub><mo>=</mo><mi>fa</mi><mo stretchy="false">(</mo><mi>scores</mi><mo stretchy="false">)</mo><mo>#</mo><mi>common</mi><mi>factors</mi><mi>for</mi><mi>each</mi><mi>responder</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>print</mi> <mi>matrix</mi></msub><mo stretchy="false">(</mo><msub><mi>responder</mi> <mi>scores</mi></msub><mo stretchy="false">)</mo><mo>−</mo><mn>1.92</mn><mo>−</mo><mn>0.45</mn><mn>0.00</mn><mn>0.67</mn><mn>1.97</mn><mn>1.96</mn><mn>0.70</mn><mn>0.03</mn><mo>−</mo><mn>2.00</mn><mn>0.29</mn><mn>0.03</mn><mo>−</mo><mn>0.60</mn><mo>−</mo><mn>1.02</mn><mn>1.79</mn><mo>−</mo><mn>1.43</mn><mn>0.82</mn><mn>0.27</mn><mo>−</mo><mn>0.23</mn><mo>−</mo><mn>0.07</mn><mo>−</mo><mn>0.08</mn><mn>0.82</mn><mo>−</mo><mn>1.38</mn><mo>−</mo><mn>0.27</mn><mn>0.48</mn><mn>0.79</mn><mo>−</mo><mn>1.17</mn><mn>0.50</mn><mn>1.59</mn><mo>−</mo><mn>0.30</mn><mo>−</mo><mn>0.41</mn><mn>0.01</mn><mo>−</mo><mn>0.48</mn><mn>0.73</mn><mo>−</mo><mn>0.46</mn><mo>−</mo><mn>1.34</mn><mn>0.18</mn><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>print</mi> <mi>row</mi></msub><mo stretchy="false">(</mo><mi>fa</mi><mo>.</mo><mi>mu</mi><mo>.</mo><mi>flat</mi><mo stretchy="false">)</mo><mn>2.42</mn><mn>3.58</mn><mn>5.08</mn><mn>5.75</mn><mn>6.08</mn><mn>3.42</mn><mn>3.92</mn><mn>4.25</mn><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>fa</mi><mo>.</mo><mi>mu</mi><mo>.</mo><mi>flat</mi><mo>=</mo><mo>=</mo><mi>mu</mi><mo>#</mo><mi>MDP</mi><mi>agrees</mi><mi>with</mi><mi>our</mi><mi>earlier</mi><mi>calculation</mi><mi>array</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>True</mi><mo>,</mo><mi>True</mi><mo>,</mo><mi>True</mi><mo>,</mo><mi>True</mi><mo>,</mo><mi>True</mi><mo>,</mo><mi>True</mi><mo>,</mo><mi>True</mi><mo>,</mo><mi>True</mi><mo stretchy="false">]</mo><mo>,</mo><mi>dtype</mi><mo>=</mo><mi>bool</mi><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>print</mi> <mi>matrix</mi></msub><mo stretchy="false">(</mo><mi>fa</mi><mo>.</mo><mi>A</mi><mo stretchy="false">)</mo><mo>#</mo><mi>factor</mi><mi>weights</mi><mi>for</mi><mi>each</mi><mi>question</mi><mn>0.80</mn><mo>−</mo><mn>0.06</mn><mo>−</mo><mn>0.45</mn><mn>0.17</mn><mn>0.30</mn><mo>−</mo><mn>0.65</mn><mn>0.34</mn><mo>−</mo><mn>0.13</mn><mo>−</mo><mn>0.25</mn><mn>0.13</mn><mo>−</mo><mn>0.73</mn><mo>−</mo><mn>0.64</mn><mn>0.02</mn><mo>−</mo><mn>0.32</mn><mo>−</mo><mn>0.70</mn><mn>0.61</mn><mn>0.23</mn><mn>0.86</mn><mn>0.08</mn><mn>0.63</mn><mn>0.59</mn><mo>−</mo><mn>0.09</mn><mn>0.67</mn><mn>0.13</mn><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>print</mi> <mi>row</mi></msub><mo stretchy="false">(</mo><mi>fa</mi><mo>.</mo><mi>sigma</mi><mo stretchy="false">)</mo><mo>#</mo><mi>unique</mi><mi>noise</mi><mi>for</mi><mi>each</mi><mi>question</mi><mn>0.04</mn><mn>0.02</mn><mn>0.38</mn><mn>0.55</mn><mn>0.30</mn><mn>0.05</mn><mn>0.48</mn><mn>0.21</mn><mi>Because</mi><mi>the</mi><mi>covariance</mi><mi>is</mi><mi>unaffected</mi><mi>by</mi><mi>the</mi><mi>rotation</mi></math>\mathbf{A}\rightarrow\mathbf{A}\mathbf{R}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>,</mo><mi>the</mi><mi>estimated</mi><mi>weights</mi></math>\mathbf{A}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>and</mi><mi>responder</mi><mi>scores</mi></math>\mathbf{y}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>can</mi><mi>be</mi><mi>quite</mi><mi>sensitive</mi><mi>to</mi><mi>the</mi><mi>seed</mi><mi>priors</mi><mo>.</mo><mi>The</mi><mi>width</mi></math>\mathbf{\Sigma}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>of</mi><mi>the</mi><mi>unique</mi><mi>noise</mi></math>\mathbf{\nu}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>is</mi><mi>more</mi><mi>robust</mi><mo>,</mo><mi>because</mi></math>\mathbf{\Sigma}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>is</mi><mi>unaffected</mi><mi>by</mi><mi>rotations</mi><mi>on</mi></math>\mathbf{A}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>.</mo><mi>Related</mi><mi>tidbits</mi><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mo>=</mo><mi>Communality</mi><mo>−</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mo>−</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mo>−</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mo>−</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mo>−</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mo>−</mo><mi>The</mi><mo stretchy="false">[</mo><mi>communality</mi><mo stretchy="false">]</mo><mo stretchy="false">[</mo><mo stretchy="false">]</mo></math>h</em>i^2<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>of</mi><mi>the</mi></math>i^\text{th}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>measured</mi><mi>attribute</mi></math>x_i<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>is</mi><mi>the</mi><mi>fraction</mi><mi>of</mi><mi>variance</mi><mi>in</mi><mi>the</mi><mi>measured</mi><mi>attribute</mi><mi>which</mi><mi>is</mi><mi>explained</mi><mi>by</mi><mi>the</mi><mi>set</mi><mi>of</mi><mi>common</mi><mi>factors</mi><mo>.</mo><mi>Because</mi><mi>the</mi><mi>common</mi><mi>factors</mi></math>\mathbf{y}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>have</mi><mi>unit</mi><mi>variance</mi><mo>,</mo><mi>the</mi><mi>communality</mi><mi>is</mi><mi>given</mi><mi>by</mi><mo>:</mo><merror><mtext>Unknown character</mtext></merror><mi>div</mi><mi>class</mi><mo>=</mo><merror><mtext>Unknown character</mtext></merror><mi>numberedEq</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>span</mi><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">(</mo><mn>8</mn><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>span</mi><merror><mtext>Unknown character</mtext></merror></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msub><mi>h</mi> <mi>i</mi></msub><mo>=</mo><mfrac><mrow><msubsup><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow> <mi>k</mi></msubsup><msubsup><mi>A</mi> <mi>ij</mi> <mn>2</mn></msubsup></mrow><mrow><msubsup><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow> <mi>k</mi></msubsup><msubsup><mi>A</mi> <mi>ij</mi> <mn>2</mn></msubsup><mo>+</mo><msubsup><mi>σ</mi> <mn>1</mn> <mn>2</mn></msubsup></mrow></mfrac></math><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><merror><mtext>Unknown character</mtext></merror><mo stretchy="false">/</mo><mi>div</mi><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>factor</mi> <mi>variance</mi></msub><mo>=</mo><mi>numpy</mi><mo>.</mo><mi>array</mi><mo stretchy="false">(</mo><mo stretchy="false">[</mo><mi>sum</mi><mo stretchy="false">(</mo><mi>row</mi><mo><em></mo><mo></em></mo><mn>2</mn><mo stretchy="false">)</mo><mi>for</mi><mi>row</mi><mi>in</mi><mi>fa</mi><mo>.</mo><mi>A</mi><mo stretchy="false">]</mo><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><mi>h</mi><mo>=</mo><mi>numpy</mi><mo>.</mo><mi>array</mi><mo stretchy="false">(</mo><mo>.</mo><mo>.</mo><mo>.</mo><mo stretchy="false">[</mo><mi>var</mi><mo stretchy="false">/</mo><mo stretchy="false">(</mo><mi>var</mi><mo>+</mo><mi>sig</mi><mo stretchy="false">)</mo><mi>for</mi><mi>var</mi><mo>,</mo><mi>sig</mi><mi>in</mi><mi>zip</mi><mo stretchy="false">(</mo><msub><mi>factor</mi> <mi>variance</mi></msub><mo>,</mo><mi>fa</mi><mo>.</mo><mi>sigma</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo stretchy="false">)</mo><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><merror><mtext>Unknown character</mtext></merror><msub><mi>print</mi> <mi>row</mi></msub><mo stretchy="false">(</mo><mi>h</mi><mo stretchy="false">)</mo><mn>0.95</mn><mn>0.97</mn><mn>0.34</mn><mn>0.64</mn><mn>0.66</mn><mn>0.96</mn><mn>0.61</mn><mn>0.69</mn><mi>There</mi><mi>may</mi><mi>be</mi><mi>some</mi><mi>scaling</mi><mi>issues</mi><mi>in</mi><mi>the</mi><mi>communality</mi><mi>due</mi><mi>to</mi><mi>deviations</mi><mi>between</mi><mi>the</mi><mi>estimated</mi></math>\mathbf{A}<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>and</mi><mstyle mathvariant="bold"><mi>Σ</mi></mstyle></math> and the
variations contained in the measured scores (why?):</p>
<pre><code>>>> print_row(factor_variance + fa.sigma)
0.89 0.56 0.57 1.51 0.89 1.21 1.23 0.69
>>> print_row(scores.var(axis=0, ddof=1)) # total variance for each question
0.99 0.63 0.63 1.66 0.99 1.36 1.36 0.75
</code></pre>
<p>The <a href="http://sites.stat.psu.edu/~ajw13/stat505/fa06/17_factor/07_factor_commun.html">proportion of total variation</a> explained by the
common factors is given by:</p>
<div class="numberedEq"><span>(9)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mfrac><mrow><munderover><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow> <mi>k</mi></munderover><msub><mi>h</mi> <mi>i</mi></msub></mrow><mrow></mrow></mfrac></math></div>
<h2>Varimax rotation</h2>
<p>As mentioned earlier, factor analysis generated loadings <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math>
that are unique up to an arbitrary rotation <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>R</mi></mstyle></math> (as you'd
expect for a <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>k</mi></math>-dimensional Gaussian ball of factors <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>y</mi></mstyle></math>).
A number of of schemes have been proposed to simplify the initial
loadings by rotating <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math> to reduce off-diagonal terms. One
of the more popular approaches is Henry Kaiser's <em>varimax rotation</em>
(unfortunately, I don't have access to either his <a href="http://oskicat.berkeley.edu/record=b12138762~S1">thesis</a> or
the <a href="http://dx.doi.org/10.1007%2FBF02289233">subsequent paper</a>). I did find (via
<a href="http://en.wikipedia.org/wiki/Varimax_rotation">Wikipedia</a>) <a href="http://www.stat.ufl.edu/~tpark/Research/about_varimax.html">Trevor Park's notes</a> which have been
very useful.</p>
<p>The idea is to iterate rotations to maximize the raw varimax criterion
(Park's eq. 1):</p>
<div class="numberedEq"><span>(10)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mi>V</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo stretchy="false">)</mo><mo>=</mo><munderover><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow> <mi>k</mi></munderover><mrow><mo>(</mo><mfrac><mn>1</mn><mi>d</mi></mfrac><munderover><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow> <mi>d</mi></munderover><msubsup><mi>A</mi> <mi>ij</mi> <mn>4</mn></msubsup><mo>−</mo><msup><mrow><mo>(</mo><mfrac><mn>1</mn><mi>d</mi></mfrac><munderover><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow> <mi>d</mi></munderover><msubsup><mi>A</mi> <mi>ij</mi> <mn>4</mn></msubsup><mo>)</mo></mrow> <mn>2</mn></msup><mo>)</mo></mrow></math></div>
<p>Rather than computing a <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>k</mi></math>-dimensional rotation in one sweep, we'll
iterate through 2-dimensional rotations (on successive column pairs)
until convergence. For a particular column pair <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo stretchy="false">(</mo><mi>p</mi><mo>,</mo><mi>q</mi><mo stretchy="false">)</mo></math>, the
rotation matrix <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msup><mi>R</mi> <mo>*</mo></msup></math> is the usual rotation matrix:</p>
<div class="numberedEq"><span>(11)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><msup><mi>R</mi> <mo>*</mo></msup><mo>=</mo><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><mi>cos</mi><mo stretchy="false">(</mo><msup><mi>ϕ</mi> <mo>*</mo></msup><mo stretchy="false">)</mo></mtd> <mtd><mo lspace="verythinmathspace" rspace="0em">−</mo><mi>sin</mi><mo stretchy="false">(</mo><msup><mi>ϕ</mi> <mo>*</mo></msup><mo stretchy="false">)</mo></mtd></mtr> <mtr><mtd><mi>sin</mi><mo stretchy="false">(</mo><msup><mi>ϕ</mi> <mo>*</mo></msup><mo stretchy="false">)</mo></mtd> <mtd><mi>cos</mi><mo stretchy="false">(</mo><msup><mi>ϕ</mi> <mo>*</mo></msup><mo stretchy="false">)</mo></mtd></mtr></mtable></mrow><mo>)</mo></mrow></math></div>
<p>where the optimum rotation angle <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msup><mi>ϕ</mi> <mo>*</mo></msup></math> is (Park's eq. 3):</p>
<div class="numberedEq"><span>(12)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><msup><mi>ϕ</mi> <mo>*</mo></msup><mo>=</mo><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>∠</mo><mrow><mo>(</mo><mfrac><mn>1</mn><mi>d</mi></mfrac><munderover><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow> <mi>d</mi></munderover><msup><mrow><mo>(</mo><msub><mi>A</mi> <mi>jp</mi></msub><mo>+</mo><msub><mi>iA</mi> <mi>jq</mi></msub><mo>)</mo></mrow> <mn>4</mn></msup><mo>−</mo><msup><mrow><mo>(</mo><mfrac><mn>1</mn><mi>d</mi></mfrac><munderover><mo lspace="thinmathspace" rspace="thinmathspace">∑</mo> <mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow> <mi>d</mi></munderover><msup><mrow><mo>(</mo><msub><mi>A</mi> <mi>jp</mi></msub><mo>+</mo><msub><mi>iA</mi> <mi>jq</mi></msub><mo>)</mo></mrow> <mn>2</mn></msup><mo>)</mo></mrow> <mn>2</mn></msup><mo>)</mo></mrow></math></div>
<p>where <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>i</mi><mo>≡</mo><msqrt><mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msqrt></math>.</p>
<h1>Nomenclature</h1>
<dl>
<dt id="subsripts"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msub><mi>A</mi> <mi>ij</mi></msub></math></dt>
<dd>The element from the <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msup><mi>i</mi> <mtext>th</mtext></msup></math> row and <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msup><mi>j</mi> <mtext>th</mtext></msup></math>
column of a matrix <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math>. For example here is a 2-by-3
matrix terms of components:
<div class="numberedEq"><span>(13)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo>=</mo><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><msub><mi>A</mi> <mn>11</mn></msub></mtd> <mtd><msub><mi>A</mi> <mn>12</mn></msub></mtd> <mtd><msub><mi>A</mi> <mn>13</mn></msub></mtd></mtr> <mtr><mtd><msub><mi>A</mi> <mn>21</mn></msub></mtd> <mtd><msub><mi>A</mi> <mn>22</mn></msub></mtd> <mtd><msub><mi>A</mi> <mn>23</mn></msub></mtd></mtr></mtable></mrow><mo>)</mo></mrow></math></div>
</dd>
<dt id="transpose"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msup><mstyle mathvariant="bold"><mi>A</mi></mstyle> <mi>T</mi></msup></math></dt>
<dd>The transpose of a matrix (or vector) <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math>.
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msubsup><mi>A</mi> <mi>ij</mi> <mi>T</mi></msubsup><mo>=</mo><msub><mi>A</mi> <mi>ji</mi></msub></math></dd>
<dt id="inverse"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msup><mstyle mathvariant="bold"><mi>A</mi></mstyle> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup></math></dt>
<dd>The inverse of a matrix <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math>.
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msup><mstyle mathvariant="bold"><mi>A</mi></mstyle> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup><mover><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo>˙</mo></mover><mo>=</mo><mn>1</mn></math></dd>
<dt id="diag"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mtext>diag</mtext><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>A</mi></mstyle><mo stretchy="false">]</mo></math></dt>
<dd>A matrix containing only the diagonal elements of
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>A</mi></mstyle></math>, with the off-diagonal values set to zero.</dd>
<dt id="expect"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo><mo stretchy="false">]</mo></math></dt>
<dd>Expectation value for a function <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>f</mi></math> of a random variable
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle></math>. If the probability density of <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle></math> is
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo></math>, then <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mo>∫</mo><mi>d</mi><mstyle mathvariant="bold"><mi>x</mi></mstyle><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo><mi>f</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo></math>. For example,
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mn>1</mn></math>.</dd>
<dt id="mean"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>μ</mi></mstyle></math></dt>
<dd>The mean of a random variable <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle></math> is given by
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo>=</mo><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">]</mo></math>.</dd>
<dt id="variance"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>Σ</mi></mstyle></math></dt>
<dd>The covariance of a random variable <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle></math> is given by
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><mo>=</mo><mstyle mathvariant="bold"><mi>E</mi></mstyle><mo stretchy="false">[</mo><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>−</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo stretchy="false">)</mo><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>−</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><msup><mo stretchy="false">)</mo> <mstyle mathvariant="bold"><mi>T</mi></mstyle></msup><mo stretchy="false">]</mo></math>. In the factor analysis
model discussed above, <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>Σ</mi></mstyle></math> is restricted to a
diagonal matrix.</dd>
<dt id="gaussian"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><msub><mi>𝒢</mi> <mstyle mathvariant="bold"><mi>x</mi></mstyle></msub><mo stretchy="false">[</mo><mi>μ</mi><mo>,</mo><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><mo stretchy="false">]</mo></math></dd>
<dd>A Gaussian probability density for the random variables
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle></math> with a mean <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>μ</mi></mstyle></math> and a covariance
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>Σ</mi></mstyle></math>.
<div class="numberedEq"><span>(14)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><msub><mi>𝒢</mi> <mstyle mathvariant="bold"><mi>x</mi></mstyle></msub><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo>,</mo><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><mo stretchy="false">]</mo><mo>=</mo><mfrac><mn>1</mn><mrow><mo stretchy="false">(</mo><mn>2</mn><mi>π</mi><msup><mo stretchy="false">)</mo> <mfrac><mi>D</mi><mn>2</mn></mfrac></msup><msqrt><mrow><mi>det</mi><mo stretchy="false">[</mo><mstyle mathvariant="bold"><mi>Σ</mi></mstyle><mo stretchy="false">]</mo></mrow></msqrt></mrow></mfrac><msup><mi>e</mi> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mfrac><mn>1</mn><mn>2</mn></mfrac><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>−</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><msup><mo stretchy="false">)</mo> <mi>T</mi></msup><msup><mstyle mathvariant="bold"><mi>Σ</mi></mstyle> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>−</mo><mstyle mathvariant="bold"><mi>μ</mi></mstyle><mo stretchy="false">)</mo></mrow></msup></math></div>
</dd>
<dt id="probability-of-y-given-x"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">∣</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">)</mo></math></dt>
<dd>Probability of <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>y</mi></mstyle></math> occurring given that <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle></math>
occured. This is commonly used in <a
href="http://en.wikipedia.org/wiki/Bayes%27_theorem">Bayesian
statistics</a>.</dd>
<dt id="probability-of-x-and-y"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>,</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">)</mo></math></dt>
<dd>Probability of <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>y</mi></mstyle></math> and <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mstyle mathvariant="bold"><mi>x</mi></mstyle></math> occuring
simultaneously (the joint density).
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo>,</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">)</mo><mo>=</mo><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>x</mi></mstyle><mo stretchy="false">∣</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">)</mo><mi>p</mi><mo stretchy="false">(</mo><mstyle mathvariant="bold"><mi>y</mi></mstyle><mo stretchy="false">)</mo></math></dd>
<dt id="angle"><math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>∠</mo><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></math></dt>
<dd>The angle of <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>z</mi></math> in the complex plane.
<math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mo>∠</mo><mo stretchy="false">(</mo><msup><mi>re</mi> <mrow><mi>i</mi><mi>θ</mi></mrow></msup><mo stretchy="false">)</mo><mo>=</mo><mi>θ</mi></math>.</dd>
</dl>
<p>Note: if you have trouble viewing some of the more obscure <a href="http://en.wikipedia.org/wiki/Unicode">Unicode</a>
used in this post, you might want to install the <a href="http://www.stixfonts.org/">STIX fonts</a>.</p>
catalysthttp://blog.tremily.us//posts/catalyst/2013-03-09T17:54:16Z2013-03-06T17:50:45Z
<p><span class="infobox">
Available in a <a href="http://blog.tremily.us//tags/tools/../git/">git</a> repository.<br />
Repository: <a href="git://tremily.us/catalyst-swc.git" rel="vcs-git" title="catalyst-swc repository">catalyst-swc</a><br />
Browsable repository: <a href="http://git.tremily.us/?p=catalyst-swc.git" rel="vcs-git" title="catalyst-swc repository">catalyst-swc</a><br />
Author: W. Trevor King<br />
</span></p>
<p><a href="http://www.gentoo.org/proj/en/releng/catalyst/">Catalyst</a> is a release-building tool for Gentoo. If you use Gentoo
and want to roll your own live CD or bootable USB drive, this is the
way to go. As I've been wrapping my head around catalyst, I've been
<a href="http://thread.gmane.org/gmane.linux.gentoo.catalyst/2006">pushing</a> <a href="http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=commit;h=0a99c494d4602ebb92f6d352f86ec519903fdc88">my</a> <a href="http://thread.gmane.org/gmane.linux.gentoo.catalyst/2107">notes</a>
<a href="http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=commit;h=b7f2cf0f4c34af8d2dd9dcfa186930c17323e298">upstream</a>. This post builds on those notes to discuss the
construction of <a href="http://lists.software-carpentry.org/pipermail/discuss-software-carpentry.org/2013-March/000210.html">a bootable ISO</a> for <a href="http://software-carpentry.org/">Software Carpentry</a>
boot camps.</p>
<h1>Getting a patched up catalyst</h1>
<p>Catalyst has been around for a while, but the user base has been
fairly small. If you try to do something that <a href="http://www.gentoo.org/proj/en/releng/">Gentoo's Release
Engineering team</a> doesn't do on a regular basis, built in
catalyst support can be spotty. There's been a fair amount of patch
submissions an <a href="http://www.gentoo.org/main/en/lists.xml#doc_chap1_sect2">gentoo-catalyst@</a> recently, but patch acceptance can
be slow. For the SWC ISO, I applied versions of the following patches
(or patch series) to <a href="http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=commit;h=37540ff5e9db28e65a4f55b71c16cec1e7d294ec">37540ff</a>:</p>
<ul>
<li><a href="http://thread.gmane.org/gmane.linux.gentoo.catalyst/2085/focus=2093">chmod +x all sh scripts so they can run from the git checkout</a></li>
<li><a href="http://thread.gmane.org/gmane.linux.gentoo.catalyst/2182">livecdfs-update.sh: Set XSESSION in /etc/env.d/90xsession</a></li>
<li><a href="http://thread.gmane.org/gmane.linux.gentoo.catalyst/2187/focus=2191">Fix livecdfs-update.sh startx handling</a></li>
</ul>
<h1>Configuring catalyst</h1>
<p>The easiest way to run catalyst from a Git checkout is to setup a
<a href="http://article.gmane.org/gmane.linux.gentoo.catalyst/2111">local config file</a>. I didn't have enough hard drive space on
my local system (~16 GB) for this build, so I set things up in a
temporary directory on an external hard drive:</p>
<pre><code>$ cat catalyst.conf | grep -v '^#\|^$'
digests="md5 sha1 sha512 whirlpool"
contents="auto"
distdir="/usr/portage/distfiles"
envscript="/etc/catalyst/catalystrc"
hash_function="crc32"
options="autoresume kerncache pkgcache seedcache snapcache"
portdir="/usr/portage"
sharedir="/home/wking/src/catalyst"
snapshot_cache="/mnt/d/tmp/catalyst/snapshot_cache"
storedir="/mnt/d/tmp/catalyst"
</code></pre>
<p>I used the default values for everything except <code>sharedir</code>,
<code>snapshot_cache</code>, and <code>storedir</code>. Then I cloned the <code>catalyst-swc</code>
repository into <code>/mnt/d/tmp/catalyst</code>.</p>
<h1>Portage snapshot and a seed stage</h1>
<p>Take a snapshot of the current Portage tree:</p>
<pre><code># catalyst -c catalyst.conf --snapshot 20130208
</code></pre>
<p>Download a seed stage3 from a <a href="http://www.gentoo.org/main/en/mirrors.xml">Gentoo mirror</a>:</p>
<pre><code># wget -O /mnt/d/tmp/catalyst/builds/default/stage3-i686-20121213.tar.bz2 \
> http://distfiles.gentoo.org/releases/x86/current-stage3/stage3-i686-20121213.tar.bz2
</code></pre>
<h1>Building the live CD</h1>
<pre><code># catalyst -c catalyst.conf -f /mnt/d/tmp/catalyst/spec/default-stage1-i686-2013.1.spec
# catalyst -c catalyst.conf -f /mnt/d/tmp/catalyst/spec/default-stage2-i686-2013.1.spec
# catalyst -c catalyst.conf -f /mnt/d/tmp/catalyst/spec/default-stage3-i686-2013.1.spec
# catalyst -c catalyst.conf -f /mnt/d/tmp/catalyst/spec/default-livecd-stage1-i686-2013.1.spec
# catalyst -c catalyst.conf -f /mnt/d/tmp/catalyst/spec/default-livecd-stage2-i686-2013.1.spec
</code></pre>
<h1>isohybrid</h1>
<p>To make the ISO bootable from a USB drive, I used <a href="http://www.syslinux.org/wiki/index.php/Doc/isolinux#HYBRID_CD-ROM.2FHARD_DISK_MODE">isohybrid</a>:</p>
<pre><code># cp swc-x86.iso swc-x86-isohybrid.iso
# isohybrid iso-x86-isohybrid.iso
</code></pre>
<p>You can install the resulting ISO on a USB drive with:</p>
<pre><code># dd if=iso-x86-isohybrid.iso of=/dev/sdX
</code></pre>
<p>replacing replacing <code>X</code> with the appropriate drive letter for your USB
drive.</p>
<p>With versions of catalyst after <a href="http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=commit;h=d1c2ba9bdaf1a69078fcbcd6ec4210e829f99f0a">d1c2ba9</a>, the <code>isohybrid</code> call is
built into catalysts ISO construction.</p>
SymPyhttp://blog.tremily.us//posts/SymPy/2013-02-27T13:47:15Z2013-02-27T12:52:03Z
<p><a href="http://sympy.org/">SymPy</a> is a <a href="http://blog.tremily.us//tags/tools/../../posts/Python/">Python</a> library for symbolic mathematics. To give
you a feel for how it works, lets extrapolate the extremum location
for <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></math> given a quadratic model:</p>
<div class="numberedEq"><span>(1)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mi>f</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mi>A</mi><msup><mi>x</mi> <mn>2</mn></msup><mo>+</mo><mi>B</mi><mi>x</mi><mo>+</mo><mi>C</mi></math></div>
<p>and three known values:</p>
<div class="numberedEq"><span>(2)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mrow><mtable columnalign="right left right left right left right left right left" columnspacing="0em"><mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo></mtd> <mtd><mo>=</mo><mi>A</mi><msup><mi>a</mi> <mn>2</mn></msup><mo>+</mo><mi>B</mi><mi>a</mi><mo>+</mo><mi>C</mi></mtd></mtr> <mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>b</mi><mo stretchy="false">)</mo></mtd> <mtd><mo>=</mo><mi>A</mi><msup><mi>b</mi> <mn>2</mn></msup><mo>+</mo><mi>B</mi><mi>b</mi><mo>+</mo><mi>C</mi></mtd></mtr> <mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo></mtd> <mtd><mo>=</mo><mi>A</mi><msup><mi>c</mi> <mn>2</mn></msup><mo>+</mo><mi>B</mi><mi>c</mi><mo>+</mo><mi>C</mi></mtd></mtr></mtable></mrow></math></div>
<p>Rephrase as a matrix equation:</p>
<div class="numberedEq"><span>(3)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo></mtd></mtr> <mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>b</mi><mo stretchy="false">)</mo></mtd></mtr> <mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo></mtd></mtr></mtable></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><msup><mi>a</mi> <mn>2</mn></msup></mtd> <mtd><mi>a</mi></mtd> <mtd><mn>1</mn></mtd></mtr> <mtr><mtd><msup><mi>b</mi> <mn>2</mn></msup></mtd> <mtd><mi>b</mi></mtd> <mtd><mn>1</mn></mtd></mtr> <mtr><mtd><msup><mi>c</mi> <mn>2</mn></msup></mtd> <mtd><mi>c</mi></mtd> <mtd><mn>1</mn></mtd></mtr></mtable></mrow><mo>)</mo></mrow><mo>⋅</mo><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><mi>A</mi></mtd></mtr> <mtr><mtd><mi>B</mi></mtd></mtr> <mtr><mtd><mi>C</mi></mtd></mtr></mtable></mrow><mo>)</mo></mrow></math></div>
<p>So the solutions for <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>A</mi></math>, <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>B</mi></math>, and <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>C</mi></math> are:</p>
<div class="numberedEq"><span>(4)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><mi>A</mi></mtd></mtr> <mtr><mtd><mi>B</mi></mtd></mtr> <mtr><mtd><mi>C</mi></mtd></mtr></mtable></mrow><mo>)</mo></mrow><mo>=</mo><msup><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><msup><mi>a</mi> <mn>2</mn></msup></mtd> <mtd><mi>a</mi></mtd> <mtd><mn>1</mn></mtd></mtr> <mtr><mtd><msup><mi>b</mi> <mn>2</mn></msup></mtd> <mtd><mi>b</mi></mtd> <mtd><mn>1</mn></mtd></mtr> <mtr><mtd><msup><mi>c</mi> <mn>2</mn></msup></mtd> <mtd><mi>c</mi></mtd> <mtd><mn>1</mn></mtd></mtr></mtable></mrow><mo>)</mo></mrow> <mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mn>1</mn></mrow></msup><mo>⋅</mo><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo></mtd></mtr> <mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>b</mi><mo stretchy="false">)</mo></mtd></mtr> <mtr><mtd><mi>f</mi><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo></mtd></mtr></mtable></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>(</mo><mrow><mtable rowspacing="0.5ex"><mtr><mtd><mtext>long</mtext></mtd></mtr> <mtr><mtd><mtext>complicated</mtext></mtd></mtr> <mtr><mtd><mtext>stuff</mtext></mtd></mtr></mtable></mrow><mo>)</mo></mrow></math></div>
<p>Now that we've found the model parameters, we need to find the <math xmlns='http://www.w3.org/1998/Math/MathML' display='inline'><mi>x</mi></math>
coordinate of the extremum.</p>
<div class="numberedEq"><span>(5)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mfrac><mrow><mi mathvariant="normal">d</mi><mi>f</mi></mrow><mrow><mi mathvariant="normal">d</mi><mi>x</mi></mrow></mfrac><mo>=</mo><mn>2</mn><mi>A</mi><mi>x</mi><mo>+</mo><mi>B</mi><mspace width="thickmathspace"></mspace><mo>,</mo></math></div>
<p>which is zero when</p>
<div class="numberedEq"><span>(6)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mrow><mtable columnalign="right left right left right left right left right left" columnspacing="0em"><mtr><mtd><mn>2</mn><mi>A</mi><mi>x</mi></mtd> <mtd><mo>=</mo><mo lspace="verythinmathspace" rspace="0em">−</mo><mi>B</mi></mtd></mtr> <mtr><mtd><mi>x</mi></mtd> <mtd><mo>=</mo><mfrac><mrow><mo lspace="verythinmathspace" rspace="0em">−</mo><mi>B</mi></mrow><mrow><mn>2</mn><mi>A</mi></mrow></mfrac></mtd></mtr></mtable></mrow></math></div>
<p>Here's the solution in SymPy:</p>
<pre><code>>>> from sympy import Symbol, Matrix, factor, expand, pprint, preview
>>> a = Symbol('a')
>>> b = Symbol('b')
>>> c = Symbol('c')
>>> fa = Symbol('fa')
>>> fb = Symbol('fb')
>>> fc = Symbol('fc')
>>> M = Matrix([[a**2, a, 1], [b**2, b, 1], [c**2, c, 1]])
>>> F = Matrix([[fa],[fb],[fc]])
>>> ABC = M.inv() * F
>>> A = ABC[0,0]
>>> B = ABC[1,0]
>>> x = -B/(2*A)
>>> x = factor(expand(x))
>>> pprint(x)
2 2 2 2 2 2
a *fb - a *fc - b *fa + b *fc + c *fa - c *fb
---------------------------------------------
2*(a*fb - a*fc - b*fa + b*fc + c*fa - c*fb)
>>> preview(x, viewer='pqiv')
</code></pre>
<p>Where <code>pqiv</code> is the executable for <a href="http://www.pberndt.com/Programme/Linux/pqiv/">pqiv</a>, my preferred image
viewer. With a bit of additional factoring, that is:</p>
<div class="numberedEq"><span>(7)</span><math xmlns='http://www.w3.org/1998/Math/MathML' display='block'><mi>x</mi><mo>=</mo><mfrac><mrow><msup><mi>a</mi> <mn>2</mn></msup><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>b</mi><mo stretchy="false">)</mo><mo>−</mo><mi>f</mi><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>+</mo><msup><mi>b</mi> <mn>2</mn></msup><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo><mo>−</mo><mi>f</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>+</mo><msup><mi>c</mi> <mn>2</mn></msup><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo><mo>−</mo><mi>f</mi><mo stretchy="false">(</mo><mi>b</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow><mrow><mn>2</mn><mo>⋅</mo><mo stretchy="false">{</mo><mi>a</mi><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>b</mi><mo stretchy="false">)</mo><mo>−</mo><mi>f</mi><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>+</mo><mi>b</mi><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>c</mi><mo stretchy="false">)</mo><mo>−</mo><mi>f</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>+</mo><mi>c</mi><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>a</mi><mo stretchy="false">)</mo><mo>−</mo><mi>f</mi><mo stretchy="false">(</mo><mi>b</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo stretchy="false">}</mo></mrow></mfrac></math></div>
One-off Git daemonhttp://blog.tremily.us//posts/One-off_Git_daemon/2013-04-14T00:44:55Z2013-02-19T00:00:45Z
<p>In my <a href="http://blog.tremily.us//tags/tools/../../posts/gitweb/">gitweb</a> post, I explain how to setup <code>git daemon</code> to serve
<code>git://</code> requests under <a href="http://blog.tremily.us//tags/tools/../../posts/Nginx/">Nginx</a> on <span class="createlink">Gentoo</span>. This post talks
about a different situation, where you want to toss up a Git daemon
for collaboration on your LAN. This is useful when you're teaching
Git to a room full of LAN-sharing students, and you don't want to
bother setting up public repositories more permanently.</p>
<h1>Serving a few repositories</h1>
<p>Say you have a repository that you want to serve:</p>
<pre><code>$ mkdir -p ~/src/my-project
$ cd ~/src/my-project
$ git init
$ …hack hack hack…
</code></pre>
<p>Fire up the daemon (probably in another terminal so you can keep
hacking in your original terminal) with:</p>
<pre><code>$ cd ~/src
$ git daemon --export-all --base-path=. --verbose ./my-project
</code></pre>
<p>Then you can clone with:</p>
<pre><code>$ git clone git://192.168.1.2/my-project
</code></pre>
<p>replacing <code>192.168.1.2</code> with your public IP address (e.g. from <code>ip
addr show scope global</code>). Add additional repository paths to the <code>git
daemon</code> call to serve additional repositories.</p>
<h1>Serving a single repository</h1>
<p>If you don't want to bother listing <code>my-project</code> in your URLs, you can
base the daemon in the project itself (instead of in the parent
directory):</p>
<pre><code>$ cd
$ git daemon --export-all --base-path=src/my-project --verbose
</code></pre>
<p>Then you can clone with:</p>
<pre><code>$ git clone git://192.168.1.2/
</code></pre>
<p>This may be more convenient if you're only sharing a single
repository.</p>
<h1>Enabling pushes</h1>
<p>If you want your students to be able to push to your repository during
class, you can run:</p>
<pre><code>$ git daemon --enable=receive-pack …
</code></pre>
<p>Only do this on a trusted LAN with a junk test repository, because it
will allow <em>anybody</em> to push <em>anything</em> or remove references.</p>
PDF formshttp://blog.tremily.us//posts/PDF_forms/2012-09-28T02:20:53Z2012-09-20T20:42:02Z
<p>You can use <a href="http://blog.tremily.us//tags/tools/../../posts/pdftk/">pdftk</a> to fill out <a href="http://blog.tremily.us//tags/tools/../../posts/PDF/">PDF</a> forms (thanks for the
inspiration, <a href="http://www.myown1.com/linux/pdf_formfill.shtml">Joe Rothweiler</a>). The syntax is simple:</p>
<pre><code>$ pdftk input.pdf fill_form data.fdf output output.pdf
</code></pre>
<p>where <code>input.pdf</code> is the input PDF containing the form, <code>data.fdf</code> is
an <a href="http://en.wikipedia.org/wiki/Forms_Data_Format#Forms_Data_Format_.28FDF.29">FDF</a> or <a href="http://en.wikipedia.org/wiki/Forms_Data_Format#XML_Forms_Data_Format_.28XFDF.29">XFDF</a> file containing your data, and <code>output.pdf</code> is
the name of the PDF you're creating. The tricky part is figuring out
what to put in <code>data.fdf</code>. There's a useful comparison of the Forms
Data Format (FDF) and it's XML version (XFDF) in the <a href="http://partners.adobe.com/public/developer/en/xml/xfdf_2.0.pdf">XFDF
specification</a>. XFDF only covers a subset of FDF, so I
won't worry about it here. FDF is defined in section 12.7.7 of <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">ISO
32000-1:2008</a>, the PDF 1.7 specification, and it has been in
PDF specifications since version 1.2.</p>
<h1>Forms Data Format (FDF)</h1>
<p>FDF files are basically stripped down PDFs (§12.7.7.1). A simple FDF
file will look something like:</p>
<pre><code>%FDF-1.2
1 0 obj<</FDF<</Fields[
<</T(FIELD1_NAME)/V(FIELD1_VALUE)>>
<</T(FIELD2_NAME)/V(FIELD2_VALUE)>>
…
] >> >>
endobj
trailer
<</Root 1 0 R>>
%%EOF
</code></pre>
<p>Broken down into the lingo of ISO 32000, we have a header
(§12.7.7.2.2):</p>
<pre><code>%FDF-1.2
</code></pre>
<p>followed by a body with a single object (§12.7.7.2.3):</p>
<pre><code>1 0 obj<</FDF<</Fields[
<</T(FIELD1_NAME)/V(FIELD1_VALUE)>>
<</T(FIELD2_NAME)/V(FIELD2_VALUE)>>
…
] >> >>
endobj
</code></pre>
<p>followed by a trailer (§12.7.7.2.4):</p>
<pre><code>trailer
<</Root 1 0 R>>
%%EOF
</code></pre>
<p>Despite the claims in §12.7.7.2.1 that the trailer is optional, pdftk
choked on files without it:</p>
<pre><code>$ cat no-trailer.fdf
%FDF-1.2
1 0 obj<</FDF<</Fields[
<</T(Name)/V(Trevor)>>
<</T(Date)/V(2012-09-20)>>
] >> >>
endobj
$ pdftk input.pdf fill_form no-trailer.fdf output output.pdf
Error: Failed to open form data file:
data.fdf
No output created.
</code></pre>
<p>Trailers are easy to add, since all they reqire is a reference to the
root of the FDF catalog dictionary. If you only have one dictionary,
you can always use the simple trailer I gave above.</p>
<h2>FDF Catalog</h2>
<p>The meat of the FDF file is the catalog (§12.7.7.3). Lets take a
closer look at the catalog structure:</p>
<pre><code>1 0 obj<</FDF<</Fields[
…
] >> >>
</code></pre>
<p>This defines a new object (the FDF catalog) which contains one key
(the <code>/FDF</code> dictionary). The FDF dictionary contains one key
(<code>/Fields</code>) and its associated array of fields. Then we close the
<code>/Fields</code> array (<code>]</code>), close the FDF dictionary (<code>>></code>) and close the
FDF catalog (<code>>></code>).</p>
<p>There are a number of interesting entries that you can add to the FDF
dictionary (§12.7.7.3.1, table 243), some of which require a more
advanced FDF version. You can use the <code>/Version</code> key to the FDF
catalog (§12.7.7.3.1, table 242) to specify the of data in the
dictionary:</p>
<pre><code>1 0 obj<</Version/1.3/FDF<</Fields[…
</code></pre>
<p>Now you can extend the dictionary using table 244. Lets set things up
to use <a href="http://en.wikipedia.org/wiki/UTF-8">UTF-8</a> for the field values (<code>/V</code>) or options (<code>/Opt</code>):</p>
<pre><code>1 0 obj<</Version/1.3/FDF<</Encoding/utf_8/Fields[
<</T(FIELD1_NAME)/V(FIELD1_VALUE)>>
<</T(FIELD2_NAME)/V(FIELD2_VALUE)>>
…
] >> >>
endobj
</code></pre>
<p>pdftk understands raw text in the specified encoding (<code>(…)</code>), raw
UTF-16 strings starting with a <a href="http://en.wikipedia.org/wiki/Byte_order_mark">BOM</a> (<code>(\xFE\xFF…)</code>), or UTF-16BE
strings encoded as ASCII hex (<code><FEFF…></code>). You can use
<a href="http://blog.tremily.us//tags/tools/../../posts/PDF_bookmarks_with_Ghostscript/pdf-merge.py">pdf-merge.py</a> and its
<code>--unicode</code> option to find the latter. Support for the <code>/utf_8</code>
encoding in pdftk is new. I mailed a
<a href="http://blog.tremily.us//tags/tools/../../posts/PDF_forms/0001-Add-support-for-Encoding-utf_8-to-the-FDF-reader.patch">patch</a>
to pdftk's Sid Steward and posted a <a href="https://sourceforge.net/p/itext/patches/101/">patch request</a> to
the underlying iText library. Until those get accepted, you're stuck
with the less convenient encodings.</p>
<h2>Fonts</h2>
<p>Say you fill in some Unicode values, but your PDF reader is having
trouble rendering some funky glyphs. Maybe it doesn't have access to
the right font? You can see which fonts are embedded in a given PDF
using <a href="http://poppler.freedesktop.org/">pdffonts</a>.</p>
<pre><code>$ pdffonts input.pdf
name type emb sub uni object ID
------------------------------------ ----------------- --- --- --- ---------
MMXQDQ+UniversalStd-NewswithCommPi CID Type 0C yes yes yes 1738 0
MMXQDQ+ZapfDingbatsStd CID Type 0C yes yes yes 1749 0
MMXQDQ+HelveticaNeueLTStd-Roman Type 1C yes yes no 1737 0
CPZITK+HelveticaNeueLTStd-BlkCn Type 1C yes yes no 1739 0
…
</code></pre>
<p>If you don't have the right font for your new data, you can add it
<a href="http://itextpdf.com/examples/iia.php?id=158">using current versions of iText</a>. However,
pdftk uses an older version, so I'm not sure how to translate this
idea for pdftk.</p>
<h2>FDF templates and field names</h2>
<p>You can use pdftk itself to create an FDF template, which it does with
embedded UTF-16BE (you can see the FE FF BOMS at the start of each
string value).</p>
<pre><code>$ pdftk input.pdf generate_fdf output template.fdf
$ hexdump -C template.fdf | head
00000000 25 46 44 46 2d 31 2e 32 0a 25 e2 e3 cf d3 0a 31 |%FDF-1.2.%.....1|
00000010 20 30 20 6f 62 6a 20 0a 3c 3c 0a 2f 46 44 46 20 | 0 obj .<<./FDF |
00000020 0a 3c 3c 0a 2f 46 69 65 6c 64 73 20 5b 0a 3c 3c |.<<./Fields [.<<|
00000030 0a 2f 56 20 28 fe ff 29 0a 2f 54 20 28 fe ff 00 |./V (..)./T (...|
00000040 50 00 6f 00 73 00 74 00 65 00 72 00 4f 00 72 00 |P.o.s.t.e.r.O.r.|
…
</code></pre>
<p>You can also dump a more human friendly version of the PDF's fields
(without any default data):</p>
<pre><code>$ pdftk input.pdf dump_data_fields_utf8 output data.txt
$ cat data.txt
---
FieldType: Text
FieldName: Name
FieldNameAlt: Name:
FieldFlags: 0
FieldJustification: Left
---
FieldType: Text
FieldName: Date
FieldNameAlt: Date:
FieldFlags: 0
FieldJustification: Left
---
FieldType: Text
FieldName: Advisor
FieldNameAlt: Advisor:
FieldFlags: 0
FieldJustification: Left
---
…
</code></pre>
<p>If the fields are poorly named, you may have to fill the entire form
with unique values and then see which values appeared where in the
output PDF (for and example, see codehero's
<a href="https://github.com/codehero/OpenTaxFormFiller/blob/master/script/identify_pdf_fields.js">identify_pdf_fields.js</a>).</p>
<h1>Conclusions</h1>
<p>This would be so much easier if people just used <a href="http://www.yaml.org/">YAML</a> or <a href="http://www.json.org/">JSON</a>
instead of bothering with PDFs ;).</p>
Portagehttp://blog.tremily.us//posts/Portage/2012-06-23T18:50:41Z2012-06-23T18:50:41Z
<p><a href="http://www.gentoo.org/proj/en/portage/index.xml">Portage</a> is <span class="createlink">Gentoo</span>'s default package manager. This post isn't
supposed to be a tutorial, the <a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=1">handbook</a> does a pretty good job of
that already. I'm just recording a few tricks so I don't forget them.</p>
<h1>User patches</h1>
<p>While playing around with <a href="http://blog.tremily.us//tags/tools/../../posts/LDAP/">LDAP</a>, I was trying to troubleshoot the
<code>SASL_NOCANON</code> handling. “Gee,” I thought, “wouldn't it be nice to be
able to add debugging printfs to figure out what was happening?”
Unfortunately, I had trouble getting <code>ldapwhoami</code> working when I
compiled it by hand. “Grrr,” I though, “I just want to add a simple
patch and do whatever the ebuild already does.” This is actually
pretty easy to do, once you're looking in the right places.</p>
<h2>Write your patch</h2>
<p>I'm not going to cover that here.</p>
<h2>Place your patch where <code>epatch_user</code> will find it</h2>
<p>This would be under</p>
<pre><code>/etc/portage/patches/<CATEGORY>/<PF|P|PN>/
</code></pre>
<p>If your ebuild already calls <code>epatch_user</code>, or it uses an eclass like
<code>base</code> that calls <code>epatch_user</code> internally, you're done. If not, read
on…</p>
<h2>Forcing <code>epatch_user</code></h2>
<p>While you could always write an <a href="http://blog.tremily.us//tags/tools/../../posts/Gentoo_overlay/">overlay</a> with an
improved ebuild, a quicker fix for this kind of hack is
<a href="http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=3&chap=6#doc_chap3">/etc/portage/bashrc</a>. I used:</p>
<pre><code>if [ "${EBUILD_PHASE}" == "prepare" ]; then
echo ":: Calling epatch_user";
pushd "${S}"
epatch_user
popd
fi
</code></pre>
<p>to insert my patches at the beginning of the <code>prepare</code> phase.</p>
<h2>Cleaning up</h2>
<p>It's safe to call <code>epatch_user</code> multiple times, so you can leave this
setup in place if you like. However, you might run into problems <a href="http://comments.gmane.org/gmane.linux.gentoo.devel/76466">if
you touch autoconf files</a>, so you may want to move your
<code>bashrc</code> somewhere else until you need it again!</p>
DVD Backuphttp://blog.tremily.us//posts/DVD_Backup/2013-09-20T04:04:58Z2012-04-27T14:08:45Z
<p>I've been using <a href="http://code.google.com/p/abcde/">abcde</a> to rip our audio CD collection onto our
fileserver for a few years now. Then I can play songs from across the
collection using <a href="http://blog.tremily.us//tags/tools/../../posts/MPD/">MPD</a> without having to dig the original CDs out of
the closet. I just picked up a large external hard drive and thought
it might be time to take a look at ripping our DVD collection as well.</p>
<p>There is an excellent <a href="http://www.scottro.net/qnd/qnd-dvd-backup.html">Quick-n-Dirty Guide</a> that goes into more
detail on all of this, but here's an executive summary.</p>
<p>Make sure your kernel understands the <a href="http://en.wikipedia.org/wiki/Universal_Disk_Format">UDF file system</a>:</p>
<pre><code>$ grep CONFIG_UDF_FS /usr/src/linux/.config
</code></pre>
<p>If your kernel was compiled with <code>CONFIG_IKCONFIG_PROC</code> enabled, you
could use</p>
<pre><code>$ zcat /proc/config.gz | grep CONFIG_UDF_FS
</code></pre>
<p>instead, to make sure you're checking the configuration of the
currently running kernel. If the <code>udf</code> driver was compiled as a
module, make sure it's loaded.</p>
<pre><code>$ sudo modprobe udf
</code></pre>
<p>Mount your DVD somewhere:</p>
<pre><code>$ sudo mount /dev/dvd /mnt/dvd
</code></pre>
<p>Now you're ready to rip. You've got two options: you can copy the
VOBs over directly, or rip the DVD into an alternative container
format such as <a href="http://www.matroska.org/">Matroska</a>.</p>
<h2>Vobcopy</h2>
<p>Mirror the disc with <a href="http://lpn.rnbhq.org/">vobcopy</a> (<code>media-video/vobcopy</code> on <span class="createlink">Gentoo</span>):</p>
<pre><code>$ vobcopy -m -t "Awesome_Movie" -v -i /mnt/dvd -o ~/movies/
</code></pre>
<p>Play with <a href="http://www.mplayerhq.hu/">Mplayer</a> (<code>media-video/mplayer</code> on <span class="createlink">Gentoo</span>):</p>
<pre><code>$ mplayer -nosub -fs -dvd-device ~/movies/Awesome_Movie dvd://1
</code></pre>
<p>where <code>-nosub</code> and <code>-fs</code> are optional.</p>
<h2>Matroska</h2>
<p>Remux the disc (without reencoding) with <code>mkvmerge</code> (from
<a href="http://www.bunkus.org/videotools/mkvtoolnix">MKVToolNix</a>, <code>media-video/mkvtoolnix</code> on <span class="createlink">Gentoo</span>):</p>
<pre><code>$ mkvmerge -o ~/movies/Awesome_Movie.mkv /mnt/dvd/VIDEO_TS/VTS_01_1.VOB
(Processing the following files as well: "VTS_01_2.VOB", "VTS_01_3.VOB", "VTS_01_4.VOB", "VTS_01_5.VOB")
</code></pre>
<p>Then you can do all the usual tricks. Here's an example of extracting
a slice of the Matroska file as silent video in an AVI container with
<code>mencoder</code> (from <a href="http://www.mplayerhq.hu/">Mplayer</a>, <code>media-video/mplayer</code> on <span class="createlink">Gentoo</span>):</p>
<pre><code>$ mencoder -ss 00:29:20.3 -endpos 00:00:21.6 Awesome_Movie.mkv -nosound -of avi -ovc copy -o silent-clip.avi
</code></pre>
<p>Here's an example of extracting a slice of the Matroska file as audio
in an AC3 container:</p>
<pre><code>$ mencoder -ss 51.1 -endpos 160.9 Awesome_Movie.mkv -of rawaudio -ovc copy -oac copy -o audio-clip.ac3
</code></pre>
<p>You can also take a look through the <a href="http://en.gentoo-wiki.com/wiki/Ripping_DVD_to_Matroska_and_H.264">Gentoo wiki</a> and <a href="http://ubuntuforums.org/showthread.php?s=e45e01b671c1dd08351876fda432f04a&t=1400598&page=2">this
Ubuntu thread</a> for more ideas.</p>
Screenhttp://blog.tremily.us//posts/Screen/2012-07-08T17:03:30Z2012-03-10T14:04:04Z
<p><a href="http://www.gnu.org/software/screen/">Screen</a> is a <a href="http://www.gnu.org/software/ncurses/">ncurses</a>-based terminal multiplexer. There are
tons of useful things you can do with it, and innumerable blog posts
describing them. I have two common use cases:</p>
<ul>
<li>On my local host when I don't start X Windows, I login to a
virtual terminal and run <code>screen</code>. Then I can easily open several
windows (e.g. for <a href="http://blog.tremily.us//tags/tools/../../posts/Emacs/">Emacs</a>, <a href="http://blog.tremily.us//tags/tools/../../posts/Mutt/">Mutt</a>, <a href="http://blog.tremily.us//tags/tools/../../posts/irssi/">irssi</a>, …) without having
to log in on another virtual terminal.</li>
<li>On remote hosts when I'm doing anything serious, I start <code>screen</code>
immediately aftering <a href="http://blog.tremily.us//tags/tools/../../posts/SSH/">SSH</a>-ing into the remote host. Then if my
connection is dropped (or I need to disconnect while I take the
train in to work), my remote work is waiting for me to pick up where
I left off.</li>
</ul>
<h1>Treehouse X</h1>
<p>Those are useful things, but they are well covered by others. A few
days ago I though of a cute trick, for increasing security on my local
host, which lead me to finally write up a <code>screen</code> post. I call it
“treehouse X”. Here's the problem:</p>
<p>You don't like waiting for X to start up when a virtual terminal is
sufficient for your task at hand, so you've set your box up without a
graphical login manager. However, sometimes you <em>do</em> need a graphical
interface (e.g. to use fancy characters via <a href="http://blog.tremily.us//tags/tools/../../posts/Xmodmap/">Xmodmap</a> or the
<a href="http://blog.tremily.us//tags/tools/../../posts/Compose_key/">Compose key</a>), so you fire up X with <code>startx</code>, and get on with your
life. But wait! You have to leave the terminal to do something else
(e.g. teach a class, eat dinner, sleep?). Being a security-concious
bloke, you lock your screen with <a href="http://www.tux.org/~bagleyd/xlockmore.html">xlockmore</a> (using your <a href="http://blog.tremily.us//tags/tools/../../posts/Fluxbox/">Fluxbox</a>
<a href="http://git.tremily.us/?p=dotfiles-public.git;a=blob;f=src/.fluxbox/keys;hb=HEAD">hotkeys</a>). You leave to complete your task. While you're gone
Mallory sneaks into your lab. You've locked your X server, so you
think you're safe, but Mallory jumps to the virtual terminal from
which you started X (using <code>Ctrl-Alt-F1</code>, or similar), and kills your
<code>startx</code> process with <code>Ctrl-c</code>. Now Mallory can do evil things in
your name, like adding <code>export EDITOR=vim</code> to your <code>.bashrc</code>.</p>
<p>So how do you protect yourself against this attack? Enter <code>screen</code>
and treehouse X. If you run <code>startx</code> from within a <code>screen</code> session,
you can jump back to the virtual terminal yourself, detach from the
sesion, and log out of the virtual terminal. This is equivalent to
climing into your treehouse (X) and pulling up your rope ladder
(<code>startx</code>) behind you, so that you are no longer vulnerable from the
ground (the virtual terminal). For kicks, you can reattach to the
screen session from an <code>xterm</code>, which leads to a fun chicken-and-egg
picture:</p>
<table class="img"><caption>startx → X → Xterm → Screen → startx cycle</caption><tr><td><a href="http://blog.tremily.us//tags/tools/../../posts/Screen/treehouse-X.png"><img src="http://blog.tremily.us//tags/tools/../../posts/Screen/treehouse-X.png" width="884" height="645" alt="startx → X → Xterm → Screen → startx cycle" class="img" /></a></td></tr></table>
<p>Of course the whole situation makes sense when you realize that it's
really:</p>
<pre><code>$ pstree 14542
screen───bash───startx───xinit─┬─X
└─fluxbox───xterm───bash───screen
</code></pre>
<p>where the first <code>screen</code> is the server and the second <code>screen</code> is the
client.</p>