From Scratch on OranLooney.com
http://www.oranlooney.com/tags/fromscratch/
Recent content in From Scratch on OranLooney.com
Hugo  gohugo.io
en
© Copyright {year} Oran Looney
Mon, 16 Sep 2019 00:00:00 +0000

ML From Scratch, Part 6: Principal Component Analysis
http://www.oranlooney.com/post/mlfromscratchpart6pca/
Mon, 16 Sep 2019 00:00:00 +0000
http://www.oranlooney.com/post/mlfromscratchpart6pca/
In the previous article in this series we distinguished between two kinds of unsupervised learning (cluster analysis and dimensionality reduction) and discussed the former in some detail. In this installment we turn our attention to the later.
In dimensionality reduction we seek a function \(f : \mathbb{R}^n \mapsto \mathbb{R}^m\) where \(n\) is the dimension of the original data \(\mathbf{X}\) and \(m\) is less than or equal to \(n\). That is, we want to map some high dimensional space into some lower dimensional space.

ML From Scratch, Part 5: Gaussian Mixture Models
http://www.oranlooney.com/post/mlfromscratchpart5gmm/
Wed, 05 Jun 2019 00:00:00 +0000
http://www.oranlooney.com/post/mlfromscratchpart5gmm/
Consider the following motivating dataset:
Unlabled Data
It is apparent that these data have some kind of structure; which is to say, they certainly are not drawn from a uniform or other simple distribution. In particular, there is at least one cluster of data in the lower right which is clearly separate from the rest. The question is: is it possible for a machine learning algorithm to automatically discover and model these kinds of structures without human assistance?

ML From Scratch, Part 4: Decision Trees
http://www.oranlooney.com/post/mlfromscratchpart4decisiontree/
Fri, 01 Mar 2019 00:00:00 +0000
http://www.oranlooney.com/post/mlfromscratchpart4decisiontree/
So far in this series we’ve followed one particular thread: linear regression > logistic regression > neural network. This is a very natural progression of ideas, but it really represents only one possible approach. Today we’ll switch gears and look at a model with completely different pedigree: the decision tree, sometimes also referred to as Classification and Regression Trees, or simply CART models. In contrast to the earlier progression, decision trees are designed from the start to represent nonlinear features and interactions.

ML From Scratch, Part 3: Backpropagation
http://www.oranlooney.com/post/mlfromscratchpart3backpropagation/
Sun, 03 Feb 2019 00:00:00 +0000
http://www.oranlooney.com/post/mlfromscratchpart3backpropagation/
In today’s installment of Machine Learning From Scratch we’ll build on the logistic regression from last time to create a classifier which is able to automatically represent nonlinear relationships and interactions between features: the neural network. In particular I want to focus on one central algorithm which allows us to apply gradient descent to deep neural networks: the backpropagation algorithm. The history of this algorithm appears to be somewhat complex (as you can hear from Yann LeCun himself in this 2018 interview) but luckily for us the algorithm in its modern form is not difficult  although it does require a solid handle on linear algebra and calculus.

ML From Scratch, Part 2: Logistic Regression
http://www.oranlooney.com/post/mlfromscratchpart2logisticregression/
Thu, 27 Dec 2018 00:00:00 +0000
http://www.oranlooney.com/post/mlfromscratchpart2logisticregression/
In this second installment of the machine learning from scratch we switch the point of view from regression to classification: instead of estimating a number, we will be trying to guess which of 2 possible classes a given input belongs to. A modern example is looking at a photo and deciding if its a cat or a dog.
In practice, its extremely common to need to decide between \(k\) classes where \(k > 2\) but in this article we’ll limit ourselves to just two classes  the socalled binary classification problem  because generalizations to many classes are usually both tedious and straightforward.

ML From Scratch, Part 1: Linear Regression
http://www.oranlooney.com/post/mlfromscratchpart1linearregression/
Thu, 29 Nov 2018 00:00:00 +0000
http://www.oranlooney.com/post/mlfromscratchpart1linearregression/
To kick off this series, will start with something simple yet foundational: linear regression via ordinary least squares.
While not exciting, linear regression finds widespread use both as a standalone learning algorithm and as a building block in more advanced learning algorithms. The output layer of a deep neural network trained for regression with MSE loss, simple AR time series models, and the “local regression” part of LOWESS smoothing are all examples of linear regression being used as an ingredient in a more sophisticated model.

ML From Scratch, Part 0: Introduction
http://www.oranlooney.com/post/mlfromscratchpart0introduction/
Sun, 11 Nov 2018 00:00:00 +0000
http://www.oranlooney.com/post/mlfromscratchpart0introduction/
Motivation
“As an apprentice, every new magician must prove to his own satisfaction, at least once, that there is truly great power in magic.”  The Flying Sorcerers, by David Gerrold and Larry Niven
How do you know if you really understand something? You could just rely on the subjective experience of feeling like you understand. This sounds plausible  surely you of all people should know, right? But this runs headfirst into in the DunningKruger effect.