A First Course in Linear Algebra

http://linear.ups.edu/

by ColinWright • 14 years ago

114 37 14 years ago

linear.ups.edu

Linear Algebra is a basis for so much in Machine Learning, and this is a free, open textbook that takes the beginner through the basics. I've not read it myself - I'm not the intended audience - but I've seen several reviews from people whose opinions I trust, and it seems to be well respected.

dbbolton• 14 years ago

Honestly, I felt like linear algebra was the low point in my math career-- even behind axiomatic set theory. I still have nightmares about determining whether a set of vectors is linearly independent.

shurane• 14 years ago

Maybe you just had a bad approach? Linear algebra is loads of fun, methinks. And you need a good teacher or appreciation of the material for that.

And it's something you slowly forget if you don't put it to use.

impendia5• 14 years ago

Linear algebra is frequently taught very poorly. I think most university courses in the subject are kind of watered down and reduced to algorithms you can memorize -- which, ironically, makes them tough going for the strong students.

wickedchicken2• 14 years ago

> I felt like linear algebra was the low point in my math career

What do you consider to be the high point?

paperwork• 14 years ago

Are there any linear algebra books which use machine learning as the motivating example? Or a book which teaches linear algebra and machine learning together?

j2kun2• 14 years ago

Sadly you need a hefty amount of it to get to machine learning that uses linear algebra in a nontrivial way.

kylebrown2• 14 years ago

Funny, I just spent last night implementing a "principal compnents analysis" using numeric.js, for a character recognition project. PCA / SVD seems like a great motivating topic.

btw, I couldn't use Singular Value Decomposition in numeric.js for PCA because the method, numeric.svd, uses the "thin" algorithm, and throws an error if there are more columns than rows. I calculate way more features (50-200+ columns) than I have training samples (rows, 10-30 written manually). without svd I had to use the "covariance method", which I guess can sometimes present approximation issues, but seems to be working well for me.

The purpose of the PCA is dimensionality reduction (google "curse of dimensionality"). I had used Mahalanobis Distance as a p-value score to detect outliers (p-val < 0.05), and it worked well when there were only 6 features. Curse of dimensionality makes MD useless when there are 50 or 100 features, and PCA reduces them to 3-10 features which carry the most information with the rest approaching zero. So I do MD on the projected (reduced) features, and its working great.

If I do it all over again I might try a "one-class SVM" (which, sadly, I had not heard of until only recently and very late in the project). SVM's are non-linear like most machine learning algos, but the linear PCA can still be used to complement other methods, eg to do pre-processing before feeding to a neural network.[1]

For deeper linear methods, check out the PCA-related extensions like Fisher Linear Discriminants or Projection Pursuit.

1. "Many neural networks are susceptible to the Curse of Dimensionality though less so than the statistical techniques. The neural networks attempt to fit a surface over the data and there must be sufficient data density to discern the surface. Most neural networks automatically reduce the input features to focus on the key attributes. But nevertheless, they still benefit from feature selection or lower dimensionality data projections."

kylebrown2• 14 years ago

Just wanted to share something I ran into the other day in the course of research on PCA, a very cool motivating example, sports-related.[1] On page 13, they take some NBA player stats and graph the projection of the third PC (y-axis) against the second PC (x-axis). The third PC is a negative correlation of rebounds with assists and steals. Turns out to sort players roughly by height, placing Karl Malone (6'8") and Mugsy Bogues (5'3") at opposite extremes.

1. Faloutsos et al, Quantifiable Data Mining Using Principal Component Analysis, 1997

j2kun2• 14 years ago

This book spends three chapters on the mechanics of matrix operations before defining a vector space, and then it jumps into determinants.

IMO this is the wrong way to understand linear algebra, and the typical path of a lot of introductory text books. A better text: Axler's "Linear Algebra Done Right"

pvarangot• 14 years ago

A million times this. Its not trivial to geometrically understand linear algebra without getting into topology if you go "determinants first". An algebraic understanding of linear algebra is not the most adequate if you are learning it to further understand topics such as machine learning or signal processing.

PD: If you really do like or need the algebraic approach, Shilov's book is the way to go after this course... I think the first chapter is about determinants and he builds the rest of the book from there. It also includes a chapter on tensor algebra.

impendia5• 14 years ago

Definitely agreed.

As a math professor, I think you can get by without defining vector spaces -- I think it is a little bit difficult to come up with examples, other than R^n or C^n, which seem motivated and interesting to the beginner. The truly important example (IMHO) is R^n without a choice of basis -- but I think this can only be well motivated after you've seen a lot of linear algebra, not before.

Mechanics of matrix operations are not pleasant to teach, they make the subject seem like a bunch of contrived and confusing examples. Same for the "row echelon" stuff. Alright, you now have an algorithm for solving systems of equations... but presenting linear algebra as an algorithm sells it short.

What is really important in my view is the geometry of the subject, which is already very interesting in two dimensions. Problem: Here is a linear transformation, given as a 2x2 matrix. Draw a picture which illustrates what this does to the plane. If you ask me, this is much more important than most of the crap that gets taught and tested in most courses in linear algebra.

alecdibble• 14 years ago

I completely agree. In college, my first set of Linear Algebra courses were taught by the Math department and were completely algebra based. Later on, taking the engineering LA courses where they stressed the geometry of all the operations really cemented the concepts and simplicity of LA into my mind.

kd0amg• 14 years ago

I think it is a little bit difficult to come up with examples, other than R^n or C^n, which seem motivated and interesting to the beginner

My introductory linear algebra class also used polynomials over R, which led to examples like using projection to construct a polynomial approximation of a non-polynomial function.

AUmrysh• 14 years ago

This is how I learned linear algebra, as a series of algorithmic steps to perform to reach the solution. Only a handful of homework examples even hinted that matrices were powerful and useful. I learned the semester before that class how matrices can be used to represent physical locations and how multiplying these matrices can move you along the coordinate frames for the joints in a robot arm all the way to the end effector. In fact, you can take your final transformation matrix after you multiply each joint together and use that to translate your points in your native/world coordinate system into coordinates in your tool coordinate system. I later learned that matrices can be used for quantum computation/pyhsics and all sorts of other things like weather prediction, physics, graph theory, Bayesian networks, and so on.

Now that I'm out of school, I really want to learn how to properly use matrix algebra. I don't care about HOW to find a determinant or HOW LU decomposition works as I've already done those, but rather why I would want to perform that operation on the matrix in the first place.

The ability for matrices to model real processes is something that really fascinates me.

Do you have any other good reading material for someone like myself?

purephase• 14 years ago

Anyone have any recommendations for linear algebra courses through Coursera or similar sites? It is a subject I've always been fascinated by but my self-directed learning endeavours have not yielded much usable knowledge.

cottonseed• 14 years ago

MIT OCW has 18.06 Linear Algebra including lectures by Strang.

rrmm• 14 years ago

http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-...

I agree, it's very useful. He gives nice explanations of some of the decompositions and why they're useful.

shriphani• 14 years ago

Ruslan Sharipov wrote a text on the subject : http://arxiv.org/pdf/math/0405323v1.pdf

It is very good and explains all the proofs and definitions very well.

mindcrime2• 14 years ago

On a slightly related note, if anyone is looking for more free resources for learning maths of various sorts, check out:

http://mathbooks.reddit.com

and/or

http://people.math.gatech.edu/~cain/textbooks/onlinebooks.ht...

iamalexalright• 14 years ago

I'll post my favourite (sorted by topic - many of them free)

http://hbpms.blogspot.com/

mindcrime2• 14 years ago

Oooh, very cool resource. I'd been looking for something like that for a while actually. Something that gives a little bit of structure in terms of what sequence to do things in... that's cool.

Miserlou57• 14 years ago

See how everyone has their own favorite teacher / explanation? Let's aggregate them all.

http://www.reddit.com/r/newschool/comments/wmu5q/read_this_h...

hobbyist• 14 years ago

Khan Academy's lectures on linear algebra is more than enough for a hacker to get the gory math details and then jump into machine learning.

santigepigon• 14 years ago

What impresses me is the accessibility of this textbook. Not only is it available online (via XML and MathML) but you can download the TeX source and multiple PDFs optimized for printing or on-screen viewing [1].

Bravo.

[1] http://linear.ups.edu/download.html

tmh88j• 14 years ago

This is the best resource I've used for higher level math. I'm not sure why more people don't know about it because everything is VERY in depth with great examples and explanations.

http://tutorial.math.lamar.edu/Classes/LinAlg/LinAlg.aspx

Watabou• 14 years ago

I've used Paul's Online Notes for all my higher level math classes starting from Calculus 2. It has been more useful to me, even more than Khan's Academy. I like the little tricks and mnemonics that he teaches, especially in Linear Algebra. Example here: http://tutorial.math.lamar.edu/Classes/LinAlg/DeterminantRev.... Much easier way of calculating determinants.

Another resource I've used is Patrick JMT's excellent videos found at: http://patrickjmt.com. He really goes over problems slowly. Best math teacher I've ever had.

wickedchicken2• 14 years ago

Thank you for this. I had wanted to brush up my knowledge on vector spaces for a while and this looks like the kick I needed.

A First Course in Linear Algebra

Related Stories