The Bike Sharing Project

This project builds a neural network and uses it to predict daily bike rental ridership.

This sets some "magic" jupyter values.

Display Matplotlib plots.

get_ipython().run_line_magic('matplotlib', 'inline')

Reload code from other modules that has changed (otherwise even if you re-run the import the changes won't get picked up).

get_ipython().run_line_magic('load_ext', 'autoreload')
get_ipython().run_line_magic('autoreload', '2')

I couldn't find any documentation on this other than people asking how to get it to work. I think it means to use a higher resolution if your display supports it.

# get_ipython().run_line_magic('config', "InlineBackend.figure_format = 'retina'")

The data comes from the UCI Machine Learning Repository (I think). It combines Capital Bikeshare data, Weather Data from i-Weather, and Washington D.C. holiday information. The authors note that becaus the bikes are tracked when they are checked out and when they arrive they have become a "virtual sensor network" that tracks how people move through the city (by shared bicycle, at least).

This first bit is the original path that you need for a submission, which is different from where I'm keeping it while working on this. I'm adding an EXPECTED_DATA_PATH variable because that's being checked in the unit-test for some reason, and I'll need to change it for the submission.

data_path = 'Bike-Sharing-Dataset/hour.csv'
EXPECTED_DATA_PATH = data_path.lower()

This is where it's kept for this post.

path = DataPath("hour.csv")
data_path = str(path.from_folder)
EXPECTED_DATA_PATH = data_path.lower()
print(path.from_folder)

../../../data/bike-sharing/hour.csv

rides = pandas.read_csv(data_path)

print(table(rides.head()))

print(len(rides.dteday.unique()))

731

print(table(rides.describe(), showindex=True))

print(len(rides.dteday.unique()) * 24)

17544

So there appear to be some hours missing, since there aren't enough rows in the data set.

print("First Hour: {} {}".format(
    rides.dteday.min(),
    rides[rides.dteday == rides.dteday.min()].hr.min()))
print("Last Hour: {} {}".format(
    rides.dteday.max(),
    rides[rides.dteday == rides.dteday.max()].hr.max()))

First Hour: 2011-01-01 0
Last Hour: 2012-12-31 23

Well, that's odd. It looks like the span is complete, why are there missing hours?

figure, axe = pyplot.subplots(figsize=FIGURE_SIZE)
counts = rides.groupby(["dteday"]).hr.count()
axe.set_title("Hours Recorded Per Day")
axe.set_xlabel("Day")
axe.set_ylabel("Count")
ax = axe.plot(range(len(counts.index)), counts.values, "o", markerfacecolor='None')

So it looks like some days they didn't manage to record all the hours.

Assuming this is the UC Irvine dataset, this is the description of the variables.

weathersit

This dataset has the number of riders for each hour of each day from January 1, 2011 to December 31, 2012. The number of riders is split between casual and registered and summed up in the cnt column. You can see the first few rows of the data above.

Below is a plot showing the number of bike riders over the first 10 days or so in the data set (some days don't have exactly 24 entries in the data set, so it's not exactly 10 days). You can see the hourly rentals here. This data is pretty complicated! The weekends have lower over all ridership and there are spikes when people are biking to and from work during the week. Looking at the data above, we also have information about temperature, humidity, and windspeed, all of these likely affecting the number of riders. You'll be trying to capture all this with your model.

figure, axe = pyplot.subplots(figsize=FIGURE_SIZE)
axe.set_title("Rides For the First Ten Days")
first_ten = rides[:24*10]
plot_lines = axe.plot(range(len(first_ten)), first_ten.cnt, label="Count")
lines = axe.plot(range(len(first_ten)), first_ten.cnt,
                 '.',
                 markeredgecolor="r")
axe.set_xlabel("Day")
legend = axe.legend(plot_lines, ["Count"], loc="upper left")

Here we have some categorical variables like season, weather, month. To include these in our model, we'll need to make binary dummy variables. This is simple to do with Pandas thanks to get_dummies.

dummy_fields = ['season', 'weathersit', 'mnth', 'hr', 'weekday']
for each in dummy_fields:
    dummies = pandas.get_dummies(rides[each], prefix=each, drop_first=False)
    rides = pandas.concat([rides, dummies], axis=1)

fields_to_drop = ['instant', 'dteday', 'season', 'weathersit', 
                  'weekday', 'atemp', 'mnth', 'workingday', 'hr']
data = rides.drop(fields_to_drop, axis=1)

print(data.head())

   yr  holiday  temp   hum  windspeed  casual  registered  cnt  season_1  \
0   0        0  0.24  0.81        0.0       3          13   16         1   
1   0        0  0.22  0.80        0.0       8          32   40         1   
2   0        0  0.22  0.80        0.0       5          27   32         1   
3   0        0  0.24  0.75        0.0       3          10   13         1   
4   0        0  0.24  0.75        0.0       0           1    1         1   

   season_2    ...      hr_21  hr_22  hr_23  weekday_0  weekday_1  weekday_2  \
0         0    ...          0      0      0          0          0          0   
1         0    ...          0      0      0          0          0          0   
2         0    ...          0      0      0          0          0          0   
3         0    ...          0      0      0          0          0          0   
4         0    ...          0      0      0          0          0          0   

   weekday_3  weekday_4  weekday_5  weekday_6  
0          0          0          0          1  
1          0          0          0          1  
2          0          0          0          1  
3          0          0          0          1  
4          0          0          0          1  

[5 rows x 59 columns]

To make training the network easier, we'll standardize each of the continuous variables. That is, we'll shift and scale the variables such that they have zero mean and a standard deviation of 1.

The scaling factors are saved so we can go backwards when we use the network for predictions.

quant_features = ['casual', 'registered', 'cnt', 'temp', 'hum', 'windspeed']
# Store scalings in a dictionary so we can convert back later
scaled_features = {}
for each in quant_features:
    mean, std = data[each].mean(), data[each].std()
    scaled_features[each] = [mean, std]
    data.loc[:, each] = (data[each] - mean)/std

We'll save the data for the last approximately 21 days to use as a test set after we've trained the network. We'll use this set to make predictions and compare them with the actual number of riders.

Save data for approximately the last 21 days.

LAST_TWENTY_ONE = -21 * 24 
test_data = data[LAST_TWENTY_ONE:]

Now remove the test data from the data set .

data = data[:LAST_TWENTY_ONE]

Separate the data into features and targets.

target_fields = ['cnt', 'casual', 'registered']
features, targets = data.drop(target_fields, axis=1), data[target_fields]
test_features, test_targets = (test_data.drop(target_fields, axis=1),
                               test_data[target_fields])

We'll split the data into two sets, one for training and one for validating as the network is being trained. Since this is time series data, we'll train on historical data, then try to predict on future data (the validation set).

Hold out the last 60 days or so of the remaining data as a validation set

LAST_SIXTY = -60 * 24
train_features, train_targets = features[:LAST_SIXTY], targets[:LAST_SIXTY]
val_features, val_targets = features[LAST_SIXTY:], targets[LAST_SIXTY:]

Below you'll build your network. We've built out the structure. You'll implement both the forward pass and backwards pass through the network. You'll also set the hyperparameters: the learning rate, the number of hidden units, and the number of training passes.

graph = Digraph(comment="Neural Network", format="png")
graph.attr(rankdir="LR")

with graph.subgraph(name="cluster_input") as cluster:
    cluster.attr(label="Input")
    cluster.node("a", "")
    cluster.node("b", "")
    cluster.node("c", "")

with graph.subgraph(name="cluster_hidden") as cluster:
    cluster.attr(label="Hidden")
    cluster.node("d", "")
    cluster.node("e", "")
    cluster.node("f", "")
    cluster.node("g", "")

with graph.subgraph(name="cluster_output") as cluster:
    cluster.attr(label="Output")
    cluster.node("h", "")


graph.edges(["ad", "ae", "af", "ag",
             "bd", "be", "bf", "bg",
             "cd", "ce", "cf", "cg"])

graph.edges(["dh", 'eh', "fh", "gh"])

graph.render("graphs/network.dot")
graph

The network has two layers, a hidden layer and an output layer. The hidden layer will use the sigmoid function for activations. The output layer has only one node and is used for the regression, the output of the node is the same as the input of the node. That is, the activation function is \(f(x)=x\). A function that takes the input signal and generates an output signal, but takes into account the threshold, is called an activation function. We work through each layer of our network calculating the outputs for each neuron. All of the outputs from one layer become inputs to the neurons on the next layer. This process is called forward propagation.

We use the weights to propagate signals forward from the input to the output layers in a neural network. We use the weights to also propagate error backwards from the output back into the network to update our weights. This is called backpropagation.

Hint: You'll need the derivative of the output activation function (\(f(x) = x\)) for the backpropagation implementation. If you aren't familiar with calculus, this function is equivalent to the equation \(y = x\). What is the slope of that equation? That is the derivative of \(f(x)\).

Below, you have these tasks:

1. Implement the sigmoid function to use as the activation function. Set `self.activation_function` in `__init__` to your sigmoid function.
2. Implement the forward pass in the `train` method.
3. Implement the backpropagation algorithm in the `train` method, including calculating the output error.
4. Implement the forward pass in the `run` method.

In the my_answers.py file, fill out the TODO sections as specified

from my_answers import NeuralNetwork

Run these unit tests to check the correctness of your network implementation. This will help you be sure your network was implemented correctly befor you starting trying to train it. These tests must all be successful to pass the project.

inputs = numpy.array([[0.5, -0.2, 0.1]])
targets = numpy.array([[0.4]])

test_w_i_h = numpy.array([[0.1, -0.2],
                          [0.4, 0.5],
                          [-0.3, 0.2]])
test_w_h_o = numpy.array([[0.3],
                          [-0.1]])

Here you'll set the hyperparameters for the network. The strategy here is to find hyperparameters such that the error on the training set is low, but you're not overfitting to the data. If you train the network too long or have too many hidden nodes, it can become overly specific to the training set and will fail to generalize to the validation set. That is, the loss on the validation set will start increasing as the training set loss drops.

You'll also be using a method know as Stochastic Gradient Descent (SGD) to train the network. The idea is that for each training pass, you grab a random sample of the data instead of using the whole data set. You use many more training passes than with normal gradient descent, but each pass is much faster. This ends up training the network more efficiently. You'll learn more about SGD later.

Here, use the test data to view how well your network is modeling the data. If something is completely wrong here, make sure each step in your network is implemented correctly.

fig, axe = pyplot.subplots(figsize=FIGURE_SIZE)

mean, std = scaled_features['cnt']
predictions = network.run(test_features) * std + mean
expected = (test_targets['cnt'] * std + mean).values
axe.plot(expected, '.', label='Data')
axe.plot(expected, linestyle="--", color="tab:blue", label=None)
axe.plot(predictions,linestyle="--", color="tab:orange", label=None)
axe.plot(predictions, ".", label='Prediction')
axe.set_xlim(right=len(predictions))
legend = axe.legend()

dates = pandas.to_datetime(rides.loc[test_data.index]['dteday'])
dates = dates.apply(lambda d: d.strftime('%b %d'))
axe.set_xticks(numpy.arange(len(dates))[12::24])
_ = axe.set_xticklabels(dates[12::24], rotation=45)

def train_this(hidden_nodes:int, learning_rate:float,
               output_nodes:int=1, 
               input_nodes: int=N_i,
               repetitions: int=100,
               emit: bool=True):
    """Trains the network using the given values

    Args:
     hidden_nodes: number of nodes in the hidden layer
     learning_rate: amount to change the weights during backpropagation
     output_nodes: number of nodes in the output layer
     input_nodes: number of nodes in the input layer
     repetitions: number of times to train the model
     emit: print information

    Returns:
     test error, losses: MSE against test, dict of losses
    """
    network = NeuralNetwork(input_nodes, hidden_nodes, output_nodes,
                            learning_rate)
    losses = {'train':[], 'validation':[]}
    last_validation_loss = -1
    if emit:        
        print(
            ("Inputs: {}, Hidden: {}, Output: {}, Learning Rate: {}, "
             "Repetitions: {}").format(
                 input_nodes,
                 hidden_nodes,
                 output_nodes,
                 learning_rate, 
                 repetitions))
    reported = False
    for iteration in range(repetitions):
        # Go through a random batch of 128 records from the training data set
        batch = numpy.random.choice(train_features.index, size=128)
        X, y = (train_features.iloc[batch].values,
                train_targets.iloc[batch]['cnt'])
        network.train(X, y)

        train_loss = MSE(network.run(train_features).T, train_targets['cnt'].values)
        val_loss = MSE(network.run(val_features).T, val_targets['cnt'].values)
        losses['train'].append(train_loss)
        losses['validation'].append(val_loss)
        if last_validation_loss == -1:
            last_validation_loss = val_loss[0] 
        if val_loss[0] > last_validation_loss and not reported:
            reported = True
            if emit:
                print("Repetition {} Validation Loss went up by {}".format(
                iteration + 1,
                val_loss[0] - last_validation_loss))
        last_validation_loss = val_loss[0]

    predictions = network.run(test_features)
    expected = (test_targets['cnt']).values
    test_error = MSE(predictions.T, expected)[0]
    if emit:
        print(("Training Error: {:.5f}, "
               "Validation Error: {:.5f}, "
               "Test Error: {:.2f}").format(
                   losses["train"][-1][0],
                   losses["validation"][-1][0],
                   test_error))
    return test_error, losses, network

Parameters = namedtuple(
    "Parameters",
    "hidden_nodes learning_rate trials losses test_error network".split())

def grid_search(hidden_nodes: list, learning_rates: list, trials: list,
                max_train_error = 0.09, max_validation_error=0.18,
                emit_training:bool=False):
    """does a search for the best parameters

    Args:
     hidden_nodes: list of number of hidden nodes
     learning rates: list of how much to update the weights
     trials: list of number of times to train
     max_train_error: upper ceiling for training error
     max_validation_error: upper ceilining for acceptable validation error
     emit_training: print the statements during training
    """
    best = 1000
    if not type(trials) is list:
        trials = [trials]
    for node_count in hidden_nodes:
        for rate in learning_rates:
            for trial in trials:
                test_error, losses, network = train_this(node_count, rate,
                                                         repetitions=trial,
                                                         emit=emit_training)
                if test_error < best:
                    print("New Best: {:.2f} (Hidden: {}, Learning Rate: {:.2f})".format(
                        test_error,
                        node_count,
                        rate))
                    best = test_error
                    best_parameters = Parameters(hidden_nodes=node_count,
                                                 learning_rate=rate,
                                                 trials=trials,
                                                 losses=losses,
                                                 test_error=test_error,
                                                 network=network)
                print()
    return best_parameters

parameters = grid_search(
    hidden_nodes=[14, 28, 42, 56],
    learning_rates=[0.1, 0.01, 0.001],
    trials=200)

New Best: 0.53 (Hidden: 14, Learning Rate: 0.10)



New Best: 0.47 (Hidden: 28, Learning Rate: 0.10)



New Best: 0.44 (Hidden: 42, Learning Rate: 0.10)



New Best: 0.42 (Hidden: 56, Learning Rate: 0.10)

parameters = grid_search([42, 56], [0.1, 0.01, 0.001], trials=300)

New Best: 0.45 (Hidden: 42, Learning Rate: 0.10)



New Best: 0.42 (Hidden: 56, Learning Rate: 0.10)

So, I wouldn't have guessed it, but the 56 node model does best with a reasonably large learning rate.

parameters = grid_search([56, 112], [0.1, 0.2], trials=200)

New Best: 0.44 (Hidden: 56, Learning Rate: 0.10)

New Best: 0.41 (Hidden: 56, Learning Rate: 0.20)

Weird.

parameters = grid_search([56], [0.2, 0.3], trials=300, emit_training=True)

Inputs: 56, Hidden: 56, Output: 1, Learning Rate: 0.2, Repetitions: 300
Repetition 2 Validation Loss went up by 1.2119413344400733
Training Error: 0.42973, Validation Error: 0.71298, Test Error: 0.36
New Best: 0.36 (Hidden: 56, Learning Rate: 0.20)

Inputs: 56, Hidden: 56, Output: 1, Learning Rate: 0.3, Repetitions: 300
Repetition 2 Validation Loss went up by 47.942730166612094
Training Error: 0.69836, Validation Error: 1.20312, Test Error: 0.63

figure, axe = pyplot.subplots(figsize=FIGURE_SIZE)
axe.set_title("Error Over Time")
axe.set_ylabel("MSE")
axe.set_xlabel("Repetition")
losses = parameters.losses
axe.plot(range(len(losses["train"])), losses['train'], label='Training loss')
lines = axe.plot(range(len(losses["validation"])), losses['validation'], label='Validation loss')
legend = axe.legend()

It looks like going over 100 doesn't really help the model a lot, or at all, really.

parameters = grid_search([56], [0.2, 0.3], trials=300, emit_training=True)

Inputs: 56, Hidden: 56, Output: 1, Learning Rate: 0.2, Repetitions: 300
Repetition 2 Validation Loss went up by 0.29155647125413564
Training Error: 0.46915, Validation Error: 0.77756, Test Error: 0.36
New Best: 0.36 (Hidden: 56, Learning Rate: 0.20)

Inputs: 56, Hidden: 56, Output: 1, Learning Rate: 0.3, Repetitions: 300
Repetition 2 Validation Loss went up by 68.10510030432465
Training Error: 0.63604, Validation Error: 1.07500, Test Error: 0.58

start = datetime.now()
parameters = grid_search([56], [0.2], 2000)
print("Elapsed: {}".format(datetime.now() - start))

New Best: 0.27 (Hidden: 56, Learning Rate: 0.20)

Elapsed: 0:02:20.654989

start = datetime.now()
parameters = grid_search([56], [0.2], 1000)
print("Elapsed: {}".format(datetime.now() - start))

New Best: 0.29 (Hidden: 56, Learning Rate: 0.20)

Elapsed: 0:01:51.175404

I just checked the rubric and you need a training loss below 0.09 and a validation loss below 0.18, regardless of the test loss.

start = datetime.now()
parameters = grid_search([28, 42, 56], [0.01, 0.1, 0.2], 100, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.01, Repetitions: 100
Repetition 10 Validation Loss went up by 0.0005887216742490597
Training Error: 0.92157, Validation Error: 1.36966, Test Error: 0.69
New Best: 0.69 (Hidden: 28, Learning Rate: 0.01)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.1, Repetitions: 100
Repetition 7 Validation Loss went up by 0.06008857123483735
Training Error: 0.65584, Validation Error: 1.09581, Test Error: 0.52
New Best: 0.52 (Hidden: 28, Learning Rate: 0.10)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.2, Repetitions: 100
Repetition 4 Validation Loss went up by 0.010135891697978794
Training Error: 0.61391, Validation Error: 0.99344, Test Error: 0.47
New Best: 0.47 (Hidden: 28, Learning Rate: 0.20)

Inputs: 56, Hidden: 42, Output: 1, Learning Rate: 0.01, Repetitions: 100
Repetition 2 Validation Loss went up by 0.0016900520112179684
Training Error: 0.87851, Validation Error: 1.31872, Test Error: 0.66

Inputs: 56, Hidden: 42, Output: 1, Learning Rate: 0.1, Repetitions: 100
Repetition 3 Validation Loss went up by 0.08058311852052547
Training Error: 0.66637, Validation Error: 1.09371, Test Error: 0.53

Inputs: 56, Hidden: 42, Output: 1, Learning Rate: 0.2, Repetitions: 100
Repetition 3 Validation Loss went up by 0.08181869722819135
Training Error: 0.60174, Validation Error: 0.99664, Test Error: 0.47

Inputs: 56, Hidden: 56, Output: 1, Learning Rate: 0.01, Repetitions: 100
Repetition 4 Validation Loss went up by 0.0015646480595301604
Training Error: 0.93732, Validation Error: 1.36061, Test Error: 0.72

Inputs: 56, Hidden: 56, Output: 1, Learning Rate: 0.1, Repetitions: 100
Repetition 2 Validation Loss went up by 0.03097966349747283
Training Error: 0.67087, Validation Error: 1.07306, Test Error: 0.51

Inputs: 56, Hidden: 56, Output: 1, Learning Rate: 0.2, Repetitions: 100
Repetition 2 Validation Loss went up by 9.947099886932289
Training Error: 0.65815, Validation Error: 1.17712, Test Error: 0.52

Elapsed: 0:00:47.842652

start = datetime.now()
parameters = grid_search([28], [0.01, 0.1, 0.2], 1000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.01, Repetitions: 1000
Repetition 2 Validation Loss went up by 0.002750823237845479
Training Error: 0.71460, Validation Error: 1.28385, Test Error: 0.59
New Best: 0.59 (Hidden: 28, Learning Rate: 0.01)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.1, Repetitions: 1000
Repetition 5 Validation Loss went up by 0.09885352252565549
Training Error: 0.31086, Validation Error: 0.48591, Test Error: 0.33
New Best: 0.33 (Hidden: 28, Learning Rate: 0.10)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.2, Repetitions: 1000
Repetition 2 Validation Loss went up by 0.09560269140958688
Training Error: 0.28902, Validation Error: 0.44905, Test Error: 0.33

Elapsed: 0:01:59.136160

start = datetime.now()
parameters = grid_search([28], [0.1, 0.2], 2000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.1, Repetitions: 2000
Repetition 2 Validation Loss went up by 0.06973195973646384
Training Error: 0.28625, Validation Error: 0.45083, Test Error: 0.29
New Best: 0.29 (Hidden: 28, Learning Rate: 0.10)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.2, Repetitions: 2000
Repetition 3 Validation Loss went up by 0.037295970545350166
Training Error: 0.26864, Validation Error: 0.43831, Test Error: 0.32

Elapsed: 0:02:35.122622

start = datetime.now()
parameters = grid_search([28], [0.05], 4000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.1, Repetitions: 4000
Repetition 4 Validation Loss went up by 0.039021584745929205
Training Error: 0.27045, Validation Error: 0.44738, Test Error: 0.29
New Best: 0.29 (Hidden: 28, Learning Rate: 0.10)

Elapsed: 0:02:36.990482

start = datetime.now()
parameters = grid_search([28], [0.2], 5000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.2, Repetitions: 5000
Repetition 4 Validation Loss went up by 0.32617394730848104
Training Error: 0.18017, Validation Error: 0.32432, Test Error: 0.24
New Best: 0.24 (Hidden: 28, Learning Rate: 0.20)

Elapsed: 0:03:05.664176

start = datetime.now()
parameters = grid_search([28], [0.2, 0.3], 6000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.2, Repetitions: 6000
Repetition 3 Validation Loss went up by 0.18572519005609722
Training Error: 0.22969, Validation Error: 0.38789, Test Error: 0.35
New Best: 0.35 (Hidden: 28, Learning Rate: 0.20)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.3, Repetitions: 6000
Repetition 3 Validation Loss went up by 1.6850265407570482
Training Error: 0.08168, Validation Error: 0.20003, Test Error: 0.24
New Best: 0.24 (Hidden: 28, Learning Rate: 0.30)

Elapsed: 0:07:30.082137

start = datetime.now()
parameters = grid_search([28], [0.3, 0.4], 7000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.3, Repetitions: 7000
Repetition 3 Validation Loss went up by 0.4652683795507646
Training Error: 0.07100, Validation Error: 0.19299, Test Error: 0.29
New Best: 0.29 (Hidden: 28, Learning Rate: 0.30)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.4, Repetitions: 7000
Repetition 2 Validation Loss went up by 8.612689644792866
Training Error: 0.05771, Validation Error: 0.19188, Test Error: 0.22
New Best: 0.22 (Hidden: 28, Learning Rate: 0.40)

Elapsed: 0:09:06.729922

start = datetime.now()
parameters = grid_search([28], [0.4, 0.5], 7500, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.4, Repetitions: 7500
Repetition 3 Validation Loss went up by 3.6207932112624386
Training Error: 0.05942, Validation Error: 0.13644, Test Error: 0.16
New Best: 0.16 (Hidden: 28, Learning Rate: 0.40)

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.5, Repetitions: 7500
Repetition 2 Validation Loss went up by 4.101532160572686
Training Error: 0.05710, Validation Error: 0.14214, Test Error: 0.22

Elapsed: 0:09:56.116403

start = datetime.now()
parameters = grid_search([28], [0.4], 8000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.4, Repetitions: 8000
Repetition 2 Validation Loss went up by 0.6450181997021212
Training Error: 0.05479, Validation Error: 0.14289, Test Error: 0.24
New Best: 0.24 (Hidden: 28, Learning Rate: 0.40)

Elapsed: 0:05:18.546979

That did worse so it probably overtrains at the 0.4 learning rate. What about 0.3?

start = datetime.now()
parameters = grid_search([28], [0.3], 8000, emit_training=True)
print("Elapsed: {}".format(datetime.now() - start))

Inputs: 56, Hidden: 28, Output: 1, Learning Rate: 0.3, Repetitions: 8000
Repetition 2 Validation Loss went up by 0.3336478297258907
Training Error: 0.06670, Validation Error: 0.16918, Test Error: 0.24
New Best: 0.24 (Hidden: 28, Learning Rate: 0.30)

Elapsed: 0:05:06.761537

So this did worse than a learning rate of 0.5 at 7500 and much worse than 0.4 at 7500, so maybe that is the optimal (0.4 at 7,500) I'm chasing. It seems king of arbitrary, but it works for the assignment.

The submission is timing out for some reason (it only takes 5 minutes to run but the error says it took more than 7 minutes). I might have to try some compromise runtime.

figure, axe = pyplot.subplots(figsize=FIGURE_SIZE)
axe.set_title("Error Over Time (Hidden: {} Learning Rate: {})".format(parameters.hidden_nodes, parameters.learning_rate))
axe.set_ylabel("MSE")
axe.set_xlabel("Repetition")
losses = parameters.losses
axe.plot(range(len(losses["train"])), losses['train'], label='Training loss')
lines = axe.plot(range(len(losses["validation"])), losses['validation'], label='Validation loss')
legend = axe.legend()

fig, ax = pyplot.subplots(figsize=FIGURE_SIZE)

mean, std = scaled_features['cnt']
predictions = parameters.network.run(test_features) * std + mean
expected = (test_targets['cnt'] * std + mean).values
ax.plot(expected, '.', label='Data')
ax.plot(predictions.values, ".", label='Prediction')
ax.set_xlim(right=len(predictions))
legend = ax.legend()

dates = pandas.to_datetime(rides.loc[test_data.index]['dteday'])
dates = dates.apply(lambda d: d.strftime('%b %d'))
ax.set_xticks(numpy.arange(len(dates))[12::24])
_ = ax.set_xticklabels(dates[12::24], rotation=45)