1.1 Imports

import matplotlib.pyplot as pyplot
import pandas
import seaborn
from sklearn.linear_model import (
    Lasso,
    LinearRegression,
    Ridge,
    )
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split

%matplotlib inline
seaborn.set_style("whitegrid")

1.2 The Data

This is the same data I used for k-nearest neighbors regression.

boston = load_boston()
print("Boston data-shape: {0}".format(boston.data.shape))

Boston data-shape: (506, 13)

X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target)

1.3 The Model

model = LinearRegression()
model.fit(X_train, y_train)

print("coefficients: {0}".format(model.coef_))
print("intercept: {0}".format(model.intercept_))

coefficients: [ -5.29188465e-02   3.27516047e-02   5.15495287e-02   1.96191849e+00
  -1.70355026e+01   4.26984342e+00  -4.66261395e-03  -1.24731581e+00
   2.40316945e-01  -1.12757320e-02  -9.67653044e-01   1.07129222e-02
  -4.58665079e-01]
intercept: 31.315219281412134

pollution_index = 4
names = ["Crime", "Large Lots", "Non-Retail Businesses",
         "Charles River adjacent", "Nitric Oxide", "Rooms", "Old Homes",
         "Distance to Employment", "Access to Highways", "Tax Rate",
         "Pupil-Teacher Ratio", "Blacks", "Lower Status"]
pandas.Series(model.coef_, index=names)

Crime                     -0.052919
Large Lots                 0.032752
Non-Retail Businesses      0.051550
Charles River adjacent     1.961918
Nitric Oxide             -17.035503
Rooms                      4.269843
Old Homes                 -0.004663
Distance to Employment    -1.247316
Access to Highways         0.240317
Tax Rate                  -0.011276
Pupil-Teacher Ratio       -0.967653
Blacks                     0.010713
Lower Status              -0.458665
dtype: float64

The price of homes in Boston is negatively correlated with Crime, Nitric Oxide (pollution), Distance to employment centes, Tax Rate, Pupil:Teacher ratio and the Lower status of the residents, with pollution being the overall largest factor (positive or negative). The most positive factors were the number of rooms the house had and whether the house was adjacent to the Charles River.

print("Training r2: {:.2f}".format(model.score(X_train, y_train)))
print("Testing r2: {0:.2f}".format(model.score(X_test, y_test)))

Training r2: 0.74
Testing r2: 0.73

The training and testing scores were oddly close, suggesting that this model generalizes well.

training = pandas.DataFrame(X_train, columns=names)

seaborn.pairplot(training)

Imports

from numba import jit
import numpy
import matplotlib.pyplot as pyplot
import seaborn
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor

%matplotlib inline
seaborn.set_style("whitegrid")

The Model

def get_r_squared(max_neighbors=10, samples=100):
    train_score = []
    test_score = []
    models = []
    inputs, values = make_regression(n_samples=samples)
    X_train, X_test, y_train, y_test = train_test_split(inputs, values)

    for neighbors in range(1, max_neighbors+1):
        model = KNeighborsRegressor(n_neighbors=neighbors, n_jobs=4)
        model.fit(X_train, y_train)
        train_score.append(model.score(X_train, y_train))
        test_score.append(model.score(X_test, y_test))
        models.append(model)
    return train_score, test_score, models

def plot_r_squared(neighbors=20, samples=100):
    train_score, test_score, models = get_r_squared(neighbors, samples)
    neighbors = range(1, neighbors+1)
    pyplot.plot(neighbors, train_score, label="Training $r^2$")
    pyplot.plot(neighbors, test_score, label="Testing $r^2$")
    pyplot.xlabel("Neighbors")
    pyplot.ylabel("$r^2$")
    pyplot.title("KNN Synthetic Data")
    pyplot.legend()
    return train_score, test_score, models
plot_r_squared()

I originally had it set to a maximum of 10 neighbors, which made it appear that 9 was the peak, but expanding it shows that it was 15. It had a fairly low \(r^2\) score, even at its best. There appears to be more variance in the make_regression function than I had thought. When I ran it earlier the testing score never exceeded the training score and the best k was 12. The actual best score was the same, though.

print("Max r2: {:.2f}".format(max(test_score)))

Max r2: 0.47

The default for the make_regression function is to create 100 samples (which I mimicked by passing in 100 explicitly). By statistics standards this is a reasonable dataset (I believe 20 samples was the minimum for a long time) but it is very small by machine learning samples. Will it do better if it has a larger sample size?

plot_r_squared(samples=1000)

It didn't, but maybe because I didn't increase the number of neighbors.

plot_r_squared(neighbors=100, samples=1000)

No, that didn't help, and after re-looking at the plot above I realized that it was getting worse at the end, so I shouldn't have expected that to help. So why does it do worse with more data?

train, test, models = plot_r_squared(samples=10000, neighbors=100)

Having even more data seems to have improved the amount the testing score goes down with the number of neighbors. Maybe there's an ideal neighbors to data points ratio that I'm missing, and too many neighbors means you need more data.

@jit
def find_first(array, match):
    """find the index of the first match

    Expects a 1-dimensional array or list

    Args:
     array (numpy.array): thing to search
     match: thing to match

    Returns:
     int: index of the first match found (or None)
    """
    for index in range(len(array)):
        if array[index] == match:
            return index
    return

best = max(test)
print("Best Test r2: {:.2f}".format(best))
test = numpy.array(test)
index = find_first(test, best)
print("Best Neighbors: {0}".format(index + 1))

Best Test r2: 0.39
Best Neighbors: 18

Imports

import matplotlib.pyplot as pyplot
import seaborn
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor

%matplotlib inline
seaborn.set_style("whitegrid")

The Data

boston = load_boston()
print("Boston data-shape: {0}".format(boston.data.shape))

Boston data-shape: (506, 13)

print(boston.keys())

dict_keys(['target', 'feature_names', 'data', 'DESCR'])

X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target)

Model Performance

def get_r_squared(max_neighbors=10):
    train_score = []
    test_score = []
    models = []
    for neighbors in range(1, max_neighbors+1):
        model = KNeighborsRegressor(n_neighbors=neighbors)
        model.fit(X_train, y_train)
        train_score.append(model.score(X_train, y_train))
        test_score.append(model.score(X_test, y_test))
        models.append(model)
    return train_score, test_score, models

train_score, test_score, models = get_r_squared()
neighbors = range(1, 11)
pyplot.plot(neighbors, train_score, label="Training $r^2$")
pyplot.plot(neighbors, test_score, label="Testing $r^2$")
pyplot.xlabel("Neighbors")
pyplot.ylabel("$r^2$")
pyplot.title("KNN Boston Housing Prices")
pyplot.legend()

The testing score seems to peak at 2 neighbors and then go down from there.

print("Training r2 for 2 neigbors: {:.2f}".format(train_score[1]))
print("Testing r2 for 2 neighbors: {:.2f}".format(test_score[1]))
assert max(test_score) == test_score[1]

Training r2 for 2 neigbors: 0.84
Testing r2 for 2 neighbors: 0.63

In this case the K-Nearest Neighbors didn't seem to do as well with regression as it did with classification.

2.1 Imports

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
import matplotlib.pyplot as pyplot
import seaborn

%matplotlib inline
seaborn.set_style("whitegrid")

2.2 The Data

total = 569
positive_fraction = 212/total
negative_fraction = 1 - positive_fraction
inputs, classifications = make_classification(n_samples=total, n_features=30,
                                              weights=[positive_fraction,
                                                       negative_fraction])
print(inputs.shape)
print(classifications.shape)

(569, 30)
(569,)

positive = classifications.sum()
print("Positives: {}".format(positive))
print("Negatives: {}".format(classifications.size - positive))

Positives: 355
Negatives: 214

X_train, X_test, y_train, y_test = train_test_split(inputs, classifications)

2.3 The model

model = KNeighborsClassifier()

def get_accuracies(max_neighbors=10):
    train_accuracies = []
    test_accuracies = []
    for neighbors in range(1,  max_neighbors+1):
        classifier = KNeighborsClassifier(n_neighbors=neighbors)
        classifier.fit(X_train, y_train)
        train_accuracies.append(classifier.score(X_train, y_train))
        test_accuracies.append(classifier.score(X_test, y_test))
    return train_accuracies, test_accuracies

training_accuracies, testing_accuracies = get_accuracies()

neighbors = range(1, 11)
pyplot.plot(neighbors, training_accuracies, label="Training Accuracy")
pyplot.plot(neighbors, testing_accuracies, label="Testing Accuracy")
pyplot.ylabel("Accuracy")
pyplot.xlabel("Neighbors")
pyplot.title("KNN Cancer Accuracy")
pyplot.legend()

At k=1, the training set does perfectly while the test set does okay, but not as well as it does at k=9, what appears to be the best value.

3.1 Imports

import matplotlib.pyplot as pyplot
import seaborn
import pandas
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
from sklearn.neighbors import KNeighborsClassifier

%matplotlib inline
seaborn.set_style("whitegrid")

3.2 The Dataset

cancer = load_breast_cancer()
print("Keys in the cancer bunch: {}".format(",".join(cancer.keys())))
print("Training Data Shape: {}".format(cancer.data.shape))
print("Target Names: {}".format(','.join(cancer.target_names)))

Keys in the cancer bunch: feature_names,target_names,target,data,DESCR
Training Data Shape: (569, 30)
Target Names: malignant,benign

This is from the description.

Data Set Characteristics:

Number of Instances:
	569

• radius (mean of distances from center to points on the perimeter)
• texture (standard deviation of gray-scale values)
• perimeter
• area
• smoothness (local variation in radius lengths)
• compactness (perimeter^2 / area - 1.0)
• concavity (severity of concave portions of the contour)
• concave points (number of concave portions of the contour)
• symmetry
• fractal dimension ("coastline approximation" - 1)

The mean, standard error, and "worst" or largest (mean of the three largest values) of these features were computed for each image, resulting in 30 features. For instance, field 3 is Mean Radius, field 13 is Radius SE, field 23 is Worst Radius.

• class:
  • WDBC-Malignant
  • WDBC-Benign

This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets. https://goo.gl/U2Uwz2

Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.

Separating plane described above was obtained using Multisurface Method-Tree (MSM-T) [K. P. Bennett, "Decision Tree Construction Via Linear Programming." Proceedings of the 4th Midwest Artificial Intelligence and Cognitive Science Society, pp. 97-101, 1992], a classification method which uses linear programming to construct a decision tree. Relevant features were selected using an exhaustive search in the space of 1-4 features and 1-3 separating planes.

The actual linear program used to obtain the separating plane in the 3-dimensional space is that described in: [K. P. Bennett and O. L. Mangasarian: "Robust Linear Programming Discrimination of Two Linearly Inseparable Sets", Optimization Methods and Software 1, 1992, 23-34].

This database is also available through the UW CS ftp server:

ftp ftp.cs.wisc.edu cd math-prog/cpo-dataset/machine-learn/WDBC/

References

• W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on Electronic Imaging: Science and Technology, volume 1905, pages 861-870, San Jose, CA, 1993.
• O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and prognosis via linear programming. Operations Research, 43(4), pages 570-577, July-August 1995.
• W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994) 163-171.

Loading the Data

target = pandas.DataFrame(dict(target=cancer.target))
target_map = dict(zip(range(len(cancer.target_names)), cancer.target_names))
target['name'] = target.target.apply(lambda entry: target_map[entry])
print(target.name.value_counts())

benign       357
malignant    212
Name: name, dtype: int64

3.3 Splitting the Data

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target, stratify=cancer.target)
print("Trainining percent: {0:.2f} %".format(100 * len(y_train)/len(cancer.target)))
print("Testing percent: {0:.2f}".format(100 * len(y_test)/len(cancer.target)))

Trainining percent: 74.87 %
Testing percent: 25.13

3.4 Model Performance

def get_accuracies(max_neighbors=10):
    train_accuracies = []
    test_accuracies = []
    for neighbors in range(1,  max_neighbors+1):
        classifier = KNeighborsClassifier(n_neighbors=neighbors)
        classifier.fit(X_train, y_train)
        train_accuracies.append(classifier.score(X_train, y_train))
        test_accuracies.append(classifier.score(X_test, y_test))
    return train_accuracies, test_accuracies

training_accuracies, testing_accuracies = get_accuracies()

neighbors = range(1, 11)
pyplot.plot(neighbors, training_accuracies, label="Training Accuracy")
pyplot.plot(neighbors, testing_accuracies, label="Testing Accuracy")
pyplot.ylabel("Accuracy")
pyplot.xlabel("Neighbors")
pyplot.title("KNN Cancer Accuracy")
pyplot.legend()

It looks like five neighbors would be what you'd want.

print("Minimum test accuracy (n=1): {:.2f}".format(min(testing_accuracies)))
print("Maximum test accuracy (n=5): {:.2f}".format(max(testing_accuracies)))
assert max(testing_accuracies == testing_accuracies[4])

Minimum test accuracy (n=1): 0.90
Maximum test accuracy (n=5): 0.92

The original paper that used this data-set got a cross-validation error-rate of 3%, but it sounds like they didn't split the data into training and testing sets (I'll have to re-read the paper to be sure).