I could only find a link to the current PDF on their page (as of August 4, 2020). The URL embeds the date in it so to work backwards (and maybe later forwards) we need an easy way to set it. Here's what the URL looks like.

EXAMPLE_URL = "https://www.oregon.gov/oha/PH/DISEASESCONDITIONS/DISEASESAZ/Emerging%20Respitory%20Infections/COVID-19-Weekly-Report-2020-07-29-FINAL.pdf"

So to work with it I'll find out what day of the week July 29, 2020 was (the date in the URL).

START = datetime.date(2020, 7, 29)
WEEKDAY = START.weekday()
assert WEEKDAY == calendar.WEDNESDAY
print(WEEKDAY)

2

So, it looks like it came out on a Wednesday, which I'm going to assume is going to be the same every week. Now we can find the most recent Wednesday, which will be the start (or end, depending on how you look at it) of our date-ragen.

TODAY = datetime.date.today()
LAST = TODAY + relativedelta(weekday=WEEKDAY) - relativedelta(weeks=1) 
print(LAST)

2020-08-12

Now I'm going to work backwards one week at a time to create the URLS and file-names. Using the relativedelta makes it easy-peasey to get prior weeks from a given date. Here's the Wednesday before the most recent one.

one_week = relativedelta(weeks = 1)
print(LAST - one_week)

2020-08-05

Now, I don't actually know when these PDFs started so I'm just going to work backwards until I get an error message from my HTTP request and then stop.

FILE_NAME = "COVID-19-Weekly-Report-{}-FINAL.pdf"
BASE_URL = "https://www.oregon.gov/oha/PH/DISEASESCONDITIONS/DISEASESAZ/Emerging%20Respitory%20Infections/{}"

for week in range(20):
    print(f"Checking back {week} weeks from this past Wednesday")
    date = LAST - (one_week * week)
    filename = FILE_NAME.format(date)
    output_path = FOLDER/filename
    if not output_path.is_file():
        url = BASE_URL.format(filename)
        response = requests.get(url)
        if not response.ok:
            print(f"Bad week: {week}\tDate: {date}")
            break
        print(f"Saving {filename}")
        with output_path.open("wb") as writer:            
            writer.write(response.content)

Checking back 0 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-08-12-FINAL.pdf
Checking back 1 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-08-05-FINAL.pdf
Checking back 2 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-07-29-FINAL.pdf
Checking back 3 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-07-22-FINAL.pdf
Checking back 4 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-07-15-FINAL.pdf
Checking back 5 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-07-08-FINAL.pdf
Checking back 6 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-07-01-FINAL.pdf
Checking back 7 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-06-24-FINAL.pdf
Checking back 8 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-06-17-FINAL.pdf
Checking back 9 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-06-10-FINAL.pdf
Checking back 10 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-06-03-FINAL.pdf
Checking back 11 weeks from this past Wednesday
Saving COVID-19-Weekly-Report-2020-05-27-FINAL.pdf
Checking back 12 weeks from this past Wednesday
Bad week: 12    Date: 2020-05-20

So, there are eleven weeks of PDFs going back to May 27, 2020. Which seems a little short, given that Oregon started telling people to stay home in March, but maybe they have the data somewhere else. Anyway, next up is pulling the data from the PDFs from the files using tabula-py.

Nike has hired you as a data science consultant to help them save money on shoe materials. Your first assignment is to review a model one of their employees built to predict how many shoelaces they'll need each month. The features going into the machine learning model include:

• The current month (January, February, etc)
• Advertising expenditures in the previous month
• Various macroeconomic features (like the unemployment rate) as of the beginning of the current month
• The amount of leather they ended up using in the current month

The results show the model is almost perfectly accurate if you include the feature about how much leather they used. But it is only moderately accurate if you leave that feature out. You realize this is because the amount of leather they use is a perfect indicator of how many shoes they produce, which in turn tells you how many shoelaces they need.

Do you think the leather used feature constitutes a source of data leakage? If your answer is "it depends," what does it depend on?

leather_used does seem like a leakage, but it depends on whether the value is known before the shoelace predictions are made. If you won't know it in time for the predictions, then it is a leak. If there's a reason why you would always know the amount used but somehow not know how many shoes were made it might not be a data leak, but it seems odd that it would be useful. It be useable if it is a prediction of the amount of leather that will be needed, but it seems odd to use it then to predict shoelaces.

You have a new idea. You could use the amount of leather Nike ordered (rather than the amount they actually used) leading up to a given month as a predictor in your shoelace model.

Does this change your answer about whether there is a leakage problem? If you answer "it depends," what does it depend on?

Whether it is a leak will depend on whether the leather is always ordered before the shoelaces or not. If they are always ordered before shoelaces then it wouldn't be a leak.

You saved Nike so much money that they gave you a bonus. Congratulations.

Your friend, who is also a data scientist, says he has built a model that will let you turn your bonus into millions of dollars. Specifically, his model predicts the price of a new cryptocurrency (like Bitcoin, but a newer one) one day ahead of the moment of prediction. His plan is to purchase the cryptocurrency whenever the model says the price of the currency (in dollars) is about to go up.

The most important features in his model are:

• Current price of the currency
• Amount of the currency sold in the last 24 hours
• Change in the currency price in the last 24 hours
• Change in the currency price in the last 1 hour
• Number of new tweets in the last 24 hours that mention the currency

The value of the cryptocurrency in dollars has fluctuated up and down by over $100 in the last year, and yet his model's average error is less than $1. He says this is proof his model is accurate, and you should invest with him, buying the currency whenever the model says it is about to go up.

Is he right? If there is a problem with his model, what is it?

The data isn't leaking.

An agency that provides healthcare wants to predict which patients from a rare surgery are at risk of infection, so it can alert the nurses to be especially careful when following up with those patients.

You want to build a model. Each row in the modeling dataset will be a single patient who received the surgery, and the prediction target will be whether they got an infection.

Some surgeons may do the procedure in a manner that raises or lowers the risk of infection. But how can you best incorporate the surgeon information into the model?

You have a clever idea.

1. Take all surgeries by each surgeon and calculate the infection rate among those surgeons.
2. For each patient in the data, find out who the surgeon was and plug in that surgeon's average infection rate as a feature.

Does this pose any target leakage issues? Does it pose any train-test contamination issues?

The infection rate would have a target leak if the calculated value includes the patient whose row it is added to.

You would have train-test contamination if you calculated this value using both the train and test set. You would have to calculate it only on the training set to avoid contamination.

You will build a model to predict housing prices. The model will be deployed on an ongoing basis, to predict the price of a new house when a description is added to a website. Here are four features that could be used as predictors.

1. Size of the house (in square meters)
2. Average sales price of homes in the same neighborhood
3. Latitude and longitude of the house
4. Whether the house has a basement

You have historic data to train and validate the model.

Which of the features is most likely to be a source of leakage?

Average sales price of homes in the same neighborhood. If the home was sold in the past, then it would contribute to the average.

Leakage is a hard and subtle issue. You should be proud if you picked up on the issues in these examples.

Now you have the tools to make highly accurate models, and pick up on the most difficult practical problems that arise with applying these models to solve real problems.

Split up the target and features.

assert not train_data[Data.target].hasnans
y = train_data[Data.target]
X = train_data.drop([Data.target], axis="columns")

X_train, X_validate, y_train, y_validate = train_test_split(
    X, y,
    train_size=Data.train_size, test_size=Data.test_size,
    random_state=Data.random_seed)

In this step, you'll build and train your first model with gradient boosting.

• Begin by setting my_model_1 to an XGBoost model. Use the XGBRegressor class, and set the random seed to 0 (random_state=0). Leave all other parameters as default.
• Then, fit the model to the training data in X_train and y_train.

model = XGBRegressor(random_state=Data.random_seed)

model.fit(X_train, y_train)

predictions_1 = model.predict(X_validate)

Finally, use the mean_absolute_error() function to calculate the mean absolute error (MAE) corresponding to the predictions for the validation set. Recall that the labels for the validation data are stored in y_valid.

mae_1 = mean_absolute_error(predictions_1, y_validate)
print(f"Mean Absolute Error: {mae_1}")

Mean Absolute Error: 17662.736729452055

Now that you've trained a default model as baseline, it's time to tinker with the parameters, to see if you can get better performance.

• Begin by setting my_model_2 to an XGBoost model, using the XGBRegressor class. Use what you learned in the previous tutorial to figure out how to change the default parameters (like n_estimators and learning_rate) to get better results.
• Then, fit the model to the training data in X_train and y_train.
• Set predictions_2 to the model's predictions for the validation data. Recall that the validation features are stored in X_valid.
• Finally, use the mean_absolute_error() function to calculate the mean absolute error (MAE) corresponding to the predictions on the validation set. Recall that the labels for the validation data are stored in y_valid.

estimators = list(range(50, 200, 10))
max_depth = list(range(10, 100, 10)) + [None]
learning_rate = 0.05 * numpy.array(range(1, 10))

grid = dict(n_estimators=estimators,
            max_depth=max_depth)
            #learning_rate=learning_rate)

model = XGBRegressor(random_state=Data.random_seed, learning_rate=0.05)
search = RandomizedSearchCV(estimator=model,
                            param_distributions=grid,
                            n_iter=40,
                            scoring="neg_mean_absolute_error",
                            n_jobs=-1,
                            random_state=1)

X_cv = pandas.concat([X_train, X_validate])
y_cv = pandas.concat([y_train, y_validate])
with TIMER:
    search.fit(X_cv, y_cv)
first_model = search.best_estimator_
print(f"CV Training MAE: {-search.best_score_:0.2f}")
print(search.best_params_)

2020-03-01 21:34:37,418 graeae.timers.timer start: Started: 2020-03-01 21:34:37.418449
2020-03-01 21:36:24,107 graeae.timers.timer end: Ended: 2020-03-01 21:36:24.107671
2020-03-01 21:36:24,109 graeae.timers.timer end: Elapsed: 0:01:46.689222
CV Training MAE: 16048.16
{'n_estimators': 160, 'max_depth': None}

outcome = pandas.DataFrame({"Score": search.cv_results_["mean_test_score"]})

early_stopping_model = XGBRegressor(random_state=Data.random_seed,
                                    learning_rate=0.05,
                                    early_stopping_rounds=5,
                                    n_estimators=500)
early_stopping_model.fit(X_train, y_train, eval_set=[(X_validate, y_validate)],
                         verbose=False)
print(mean_absolute_error(early_stopping_model.predict(X_validate),
                          y_validate))

16728.27523009418

params = model.get_params()
print(f"Trees: {params['n_estimators']}")

Trees: 100

In this step, you will create a model that performs worse than the original model in Step 1. This will help you to develop your intuition for how to set parameters. You might even find that you accidentally get better performance, which is ultimately a nice problem to have and a valuable learning experience!

• Begin by setting my_model_3 to an XGBoost model, using the XGBRegressor class. Use what you learned in the previous tutorial to figure out how to change the default parameters (like n_estimators and learning_rate) to design a model to get high MAE.
• Then, fit the model to the training data in X_train and y_train.
• Set predictions_3 to the model's predictions for the validation data. Recall that the validation features are stored in X_valid.
• Finally, use the mean_absolute_error() function to calculate the mean absolute error (MAE) corresponding to the predictions on the validation set. Recall that the labels for the validation data are stored in y_valid.

parameters = random.choice(search.cv_results_["params"])
print(parameters)
model = XGBRegressor(random_state=Data.random_seed, **parameters)
with TIMER:
    model.fit(X_train, y_train)
print(f"MAE: {mean_absolute_error(model.predict(X_validate), y_validate)}")

2020-02-29 20:55:54,573 graeae.timers.timer start: Started: 2020-02-29 20:55:54.573008
{'n_estimators': 60, 'max_depth': 20, 'learning_rate': 0.7000000000000001}
2020-02-29 20:55:55,019 graeae.timers.timer end: Ended: 2020-02-29 20:55:55.019335
2020-02-29 20:55:55,020 graeae.timers.timer end: Elapsed: 0:00:00.446327
MAE: 25077.38005672089

imputer = IterativeImputer(random_state=Data.random_seed)

final_x_train = pandas.DataFrame(imputer.fit_transform(X_train),
                                 columns=X_train.columns)
early_stopping_model_2 = XGBRegressor(random_state=Data.random_seed,
                                      learning_rate=0.05,
                                      early_stopping_rounds=5,
                                      n_estimators=500)
early_stopping_model_2.fit(final_x_train, y_train, eval_set=[(X_validate, y_validate)],
                         verbose=False)
print(mean_absolute_error(early_stopping_model_2.predict(X_validate),
                          y_validate))

16516.399373929795

predictions = early_stopping_model.predict(X_test)
output = pandas.DataFrame({'Id': X_test.index,
                           'SalePrice': predictions})
output.to_csv(DATA_PATH/'submission.csv', index=False)

This got a score of 14777.96266 on the kaggle submissions page.

predictions = search.predict(X_test)
output = pandas.DataFrame({'Id': X_test.index,
                           'SalePrice': predictions})
output.to_csv(DATA_PATH/'submission.csv', index=False)

This one got a score of 14976.55345, so the early stopping model is the best one so far… It had fewer trees than the model that the RandomSearch CV ended up with, maybe the Random Search overfit the data.

predictions = early_stopping_model_2.predict(X_test)
output = pandas.DataFrame({'Id': X_test.index,
                           'SalePrice': predictions})
output.to_csv(DATA_PATH/'submission.csv', index=False)

This gets a score of 14965.20801 so it looks like the XGBoost model without imputation is the best one.

Split up the target and features.

assert not train_data[Data.target].hasnans
y = train_data[Data.target]
X = train_data.drop([Data.target], axis="columns")

So far, you've learned how to build pipelines with scikit-learn. For instance, the pipeline below will use SimpleImputer() to replace missing values in the data, before using RandomForestRegressor() to train a random forest model to make predictions. We set the number of trees in the random forest model with the n_estimators parameter, and setting random_state ensures reproducibility.

pipeline = Pipeline(steps=[
    ('preprocessor', SimpleImputer()),
    ('model', RandomForestRegressor(n_estimators=50, random_state=Data.random_seed))
])

You have also learned how to use pipelines in cross-validation. The code below uses the cross_val_score() function to obtain the mean absolute error (MAE), averaged across five different folds. Recall we set the number of folds with the cv parameter.

# Multiply by -1 since sklearn calculates *negative* MAE
scores = -1 * cross_val_score(pipeline, X, y,
                              cv=5,
                              scoring='neg_mean_absolute_error')

print("Average MAE score:", scores.mean())

Average MAE score: 18276.410356164386

In this exercise, you'll use cross-validation to select parameters for a machine learning model.

Begin by writing a function get_score() that reports the average (over three cross-validation folds) MAE of a machine learning pipeline that uses:

• the data in X and y to create folds,
• SimpleImputer() (with all parameters left as default) to replace missing values, and
• RandomForestRegressor() (with random_state=0) to fit a random forest model.

The n_estimators parameter supplied to get_score() is used when setting the number of trees in the random forest model.

def get_score(n_estimators):
    """Return the average MAE over 3 CV folds of random forest model.

    Args:
     n_estimators: the number of trees in the forest
    """
    pipeline = Pipeline(steps=[
        ('preprocessor', SimpleImputer()),
        ('model', RandomForestRegressor(n_estimators=n_estimators,
                                        random_state=Data.random_seed))
    ])
    scores = -1 * cross_val_score(pipeline, X, y,
                                  cv=3,
                                  scoring='neg_mean_absolute_error')
    # Replace this body with your own code
    return scores.mean()

Now, you will use the function that you defined in Step 1 to evaluate the model performance corresponding to eight different values for the number of trees in the random forest: 50, 100, 150, …, 300, 350, 400. Store your results in a Python dictionary results, where results[i] is the average MAE returned by get_scores(i).

results = {trees: get_score(trees) for trees in range(50, 450, 50)}

results_frame = pandas.DataFrame.from_dict({"Trees": list(results.keys()), "MAE": list(results.values())})

plot = results_frame.hvplot(x="Trees", y="MAE").opts(
    title="Cross-Validation Mean Absolute Error",
    width=Plot.width,
    height=Plot.height)

source = Embed(plot=plot, file_name="mean_absolute_error")()

print(source)

200 appears to be the best number of trees for our forest.

Split up the target and features.

assert not train_data[Data.target].hasnans
y = train_data[Data.target]
X = train_data.drop([Data.target], axis="columns")

X_train, X_validate, y_train, y_validate = train_test_split(
    X, y,
    train_size=Data.train_size, test_size=Data.test_size,
    random_state=Data.random_seed)

The missing numeric values will be filled in with a simple imputer. When the strategy is set to constant then it will fill missing values with a single value (which is 0 by default).

numerical_transformer = SimpleImputer(strategy='constant')

Now the categorical data transformer. We'll use the most frequent value in any column with missing values to fill them in and the do one-hot encoding.

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

Now we can bundle them together into a single transformer.

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numerical_transformer, numerical_columns),
        ('cat', categorical_transformer, categorical_columns)
    ])

Now, it's your turn! In the code cell below, define your own preprocessing steps and random forest model. Fill in values for the following variables:

• numerical_transformer
• categorical_transformer
• model

numerical_transformer = IterativeImputer(random_state=Data.random_seed)

I'll use the same categorical imputer.

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

And now we bundle them together.

preprocessor = ColumnTransformer(
    transformers=[
        ('numeric', numerical_transformer, numerical_columns),
        ('categorical', categorical_transformer, categorical_columns)
    ])

Now build and train the model.

model = RandomForestRegressor(n_estimators=50, max_depth=60, random_state=0)
pipeline = Pipeline(steps=[('preprocessor', preprocessor),
                           ('model', model)
                           ])
pipeline.fit(X_train, y_train)

predictions = pipeline.predict(X_validate)

print(f"MAE: {mean_absolute_error(y_validate, predictions):,}")

MAE: 17,556.51404109589

So we improved slightly, but we're still not doing as well as with the numeric only dataset.

with TIMER:
    training = X_train[numerical_columns + categorical_columns]
    data = preprocessor.fit_transform(training)
    columns = (numerical_columns
               + list(preprocessor.named_transformers_["categorical"]["onehot"].get_feature_names()))
    data = pandas.DataFrame(
        data,
        columns=columns)
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(data)

2020-03-01 19:25:58,515 graeae.timers.timer start: Started: 2020-03-01 19:25:58.514631
Setting feature_perturbation = "tree_path_dependent" because no background data was given.
2020-03-01 19:26:16,103 graeae.timers.timer end: Ended: 2020-03-01 19:26:16.103820
2020-03-01 19:26:16,104 graeae.timers.timer end: Elapsed: 0:00:17.589189

shap.summary_plot(shap_values, data)
figure = pyplot.gcf()
output = "shap_summary.png"

figure.savefig(OUTPUT_PATH/output)
print(f"[[file:{output}]]")

[[file:shap_summary.png]]
<Figure size 432x288 with 0 Axes>

shap.initjs()
html = shap.force_plot(explainer.expected_value, shap_values, data)
output_file = "force_plot.html"
output = OUTPUT_PATH/output_file
with output.open("w") as writer:
    shap.save_html(writer, html, False)

print(f"""
#+begin_export html
: <object type="text/html" data="{output_file}" style="width:100%" height=800>
:   <p>Figure Missing</p>
: </object>
#+end_export
""")

Split up the target and features.

assert not train_data[Data.target].hasnans
y = train_data[Data.target]
X = train_data.drop([Data.target], axis="columns")

We know that there's missing data, but since this is about handling categorical data, not missing data, we'll just drop the columns that have missing values.

print(X.shape)
X = X[[column for column in X.columns if not X[column].hasnans]]
test_data = test_data[X.columns]
print(X.shape)

(1460, 79)
(1460, 60)

So we lost 19 columns - more than I was expecting.

Now do the train-test split.

X_train, X_validate, y_train, y_validate = train_test_split(
    X, y,
    train_size=Data.train_size, test_size=Data.test_size,
    random_state=Data.random_seed)

print(drop_X_train.info())

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-13-002dd7ea4a19> in <module>
----> 1 print(drop_X_train.info())

NameError: name 'drop_X_train' is not defined

Notice that the dataset contains both numerical and categorical variables. You'll need to encode the categorical data before training a model.

This is the same function used in the missing-values tutorial. It's used to compare different models' Mean Absolute Error (MAE) as we make changes.

def score_dataset(X_train, X_valid, y_train, y_valid):
    model = RandomForestRegressor(n_estimators=100, random_state=0)
    model.fit(X_train, y_train)
    preds = model.predict(X_valid)
    return mean_absolute_error(y_valid, preds)

The first approach is to just drop all the non-numeric columns.

columns = [column for column in X_train.columns if X_train[column].dtype != object]
drop_X_train = X_train[columns]
drop_X_validate = X_validate[columns]

print("MAE from Approach 1 (Drop categorical variables):")
print(f"{score_dataset(drop_X_train, drop_X_validate, y_train, y_validate):,}")

Using all the numeric columns does better than we did with our initial subset of columns (20,928.5), but not as good as we did with imputed values (16,656.3).

Before jumping into label encoding, we'll investigate the dataset. Specifically, we'll look at the 'Condition2' column. The code cell below prints the unique entries in both the training and validation sets.

train_counter = CountPercentage(X_train.Condition2, dropna=False)
validate_counter = CountPercentage(X_validate.Condition2, dropna=False)
train_counter()

validate_counter()

It looks like the validation data has values that aren't in the training data (and vice versa), e.g. RRNn, so encoding the training set won't work with the validation set.

This is a common problem that you'll encounter with real-world data, and there are many approaches to fixing this issue. For instance, you can write a custom label encoder to deal with new categories. The simplest approach, however, is to drop the problematic categorical columns.

Run the code cell below to save the problematic columns to a Python list bad_label_cols. Likewise, columns that can be safely label encoded are stored in good_label_cols.

# All categorical columns
object_columns = [column for column in X_train.columns if X_train[column].dtype == "object"]

# Columns that can be safely label encoded
good_label_columns = [column for column in object_columns if 
                      set(X_train[column]) == set(X_validate[column])]

# Problematic columns that will be dropped from the dataset
bad_label_columns = list(set(object_columns)-set(good_label_columns))

print('Categorical columns that will be label encoded:')
for column in  good_label_columns:
    print(f" - {column}")

print('\nCategorical columns that will be dropped from the dataset:')
for column in bad_label_columns:
    print(f" - {column}")

Categorical columns that will be label encoded:

• MSZoning
• Street
• LotShape
• LandContour
• LotConfig
• BldgType
• HouseStyle
• ExterQual
• CentralAir
• KitchenQual
• PavedDrive
• SaleCondition

Categorical columns that will be dropped from the dataset:

• Condition1
• RoofMatl
• HeatingQC
• ExterCond
• RoofStyle
• SaleType
• Foundation
• Condition2
• Exterior2nd
• Neighborhood
• Heating
• LandSlope
• Utilities
• Functional
• Exterior1st

So far, you've tried two different approaches to dealing with categorical variables. And, you've seen that encoding categorical data yields better results than removing columns from the dataset.

Soon, you'll try one-hot encoding. Before then, there's one additional topic we need to cover. Begin by running the next code cell without changes.

Get number of unique entries in each column with categorical data

object_nunique = [X_train[column].nunique() for column in object_columns]

## Print number of unique entries by column, in descending
cardinality = pandas.DataFrame(dict(Column=object_columns,
                                    Cardinality=object_nunique)
                     ).sort_values(by="Cardinality", ascending=False)
print(TABLE(cardinality))

The output above shows, for each column with categorical data, the number of unique values in the column. For instance, the 'Street' column in the training data has two unique values: 'Grvl' and 'Pave', corresponding to a gravel road and a paved road, respectively.

We refer to the number of unique entries of a categorical variable as the cardinality of that categorical variable. For instance, the 'Street' variable has cardinality 2.

In this step, you'll experiment with one-hot encoding. But, instead of encoding all of the categorical variables in the dataset, you'll only create a one-hot encoding for columns with cardinality less than 10.

Run the code cell below without changes to set low_cardinality_cols to a Python list containing the columns that will be one-hot encoded. Likewise, high_cardinality_cols contains a list of categorical columns that will be dropped from the dataset.

low_cardinality_columns = cardinality[cardinality.Cardinality < 10].Column
high_cardinality_columns = cardinality[~cardinality.Column.isin(low_cardinality_columns)].Column

print("Categorical columns that will be one-hot encoded:")
for column in low_cardinality_columns:
    print(f" - {column}")

Categorical columns that will be one-hot encoded:

• SaleType
• Condition1
• HouseStyle
• RoofMatl
• Functional
• Heating
• Foundation
• RoofStyle
• SaleCondition
• Condition2
• BldgType
• ExterCond
• LotConfig
• HeatingQC
• MSZoning
• ExterQual
• KitchenQual
• LandContour
• LotShape
• LandSlope
• PavedDrive
• Street
• Utilities
• CentralAir

print('Categorical columns that will be dropped from the dataset:')
for column in high_cardinality_columns:
    print(f" - {column}")

Categorical columns that will be dropped from the dataset:

• Neighborhood
• Exterior2nd
• Exterior1st

Use the next code cell to one-hot encode the data in X_train and X_valid. Set the preprocessed DataFrames to OH_X_train and OH_X_valid, respectively.

• The full list of categorical columns in the dataset can be found in the Python list object_cols.
• You should only one-hot encode the categorical columns in low_cardinality_cols. All other categorical columns should be dropped from the dataset.

OH_train = X_train[low_cardinality_columns]
OH_validate = X_validate[low_cardinality_columns]

encoder = OneHotEncoder(sparse=False, handle_unknown="ignore")

OH_train = pandas.DataFrame(encoder.fit_transform(OH_train))
OH_validate = pandas.DataFrame(encoder.fit_transform(OH_validate))

OH_train.index = X_train.index
OH_validate.index = X_validate.index

X_train_numeric = X_train.drop(object_columns, axis="columns")
X_validate_numeric = X_validate.drop(object_columns, axis="columns")

OH_X_train = pandas.concat([X_train_numeric, OH_train], axis="columns")
OH_X_validate = pandas.concat([X_validate_numeric, OH_validate], axis="columns")

print("MAE from Approach 3 (One-Hot Encoding):") 
print(f"{score_dataset(OH_X_train, OH_X_validate, y_train, y_validate):,}")

MAE from Approach 3 (One-Hot Encoding):
17,429.93404109589

So we've improved slightly, but still not as well as the all numeric data with imputed data.

After you complete Step 4, if you'd like to use what you've learned to submit your results to the leaderboard, you'll need to preprocess the test data before generating predictions.

To get the imputation working again we need to re-add the columns with missing values. I'm also going to encode the entire dataset before splitting so that everything is encoded, rather than ignoring the values in the validation set that aren't in the training set.

X_2 = train_data.drop([Data.target], axis="columns")
objects = [column for column in X_2.columns if X_2[column].dtype==object]
missing = [column for column in objects if X_2[column].hasnans]

X_2 = X_2.drop(missing, axis="columns")
OH_X = X_2.drop(high_cardinality_columns, axis="columns").reset_index(drop=True)
for column in low_cardinality_columns:
    encoder = OneHotEncoder(sparse=False)
    encoded = encoder.fit_transform(
        OH_X[column].to_numpy().reshape(-1, 1)
    )
    reencoded = pandas.DataFrame(encoded, columns=encoder.get_feature_names())
    OH_X = pandas.concat([OH_X, reencoded], axis="columns").drop(
        column, axis="columns")

imputer = IterativeImputer(random_state=Data.random_seed)
OH_X = pandas.DataFrame(imputer.fit_transform(OH_X), columns=OH_X.columns)
OH_X_train, OH_X_validate, y_train, y_validate = train_test_split(
    OH_X, y,
    train_size=Data.train_size, test_size=Data.test_size,
    random_state=Data.random_seed)

model = RandomForestRegressor(n_estimators=100, random_state=Data.random_seed)
model.fit(OH_X_train, y_train)

preds_valid = model.predict(OH_X_validate)
error = mean_absolute_error(y_validate, preds_valid)
print(f"MAE: {error:0.2f}")

It seems to have gotten worse… but maybe that's because we tuned the hyperparameters to the numeric-only model.

print(f"{len(train_data):,}")
train_data = train_data.dropna(axis="rows", subset=[Data.target])
print(f"{len(train_data):,}")

1,460
1,460

Doesn't look like there were any missing target values.

y = train_data[Data.target]
x_train = train_data.drop([Data.target], axis="columns")

To keep things simple, we'll use only numerical predictors

print(len(x_train.columns))
X = x_train.select_dtypes(exclude=["object"])
print(len(X.columns))

79
36

print(len(test_data.columns))
x_test = test_data.select_dtypes(exclude=["object"])
print(len(x_test.columns))

79
36

missing_by_column = x_train.isna().sum()
missed = missing_by_column[missing_by_column > 0]
print(TABLE(missed.reset_index().rename(columns={"index": "Feature", 0:"Missing"})))

According to the data description these features are:

LotFrontage: Linear feet of street connected to property MasVnrArea: Masonry veneer area in square feet GarageYrBlt: Year garage was built

Missing = Namespace(
    frontage = "LotFrontage",
    masonry = "MasVnrArea",
    garage = "GarageYrBlt",
    columns = ["LotFrontage", "MasVnrArea", "GarageYrBlt"],
)

This function will help check the Mean Absolute Error (MAE) as we make changes to the dataset.

def score_dataset(X_train, X_valid, y_train, y_valid):
    model = RandomForestRegressor(n_estimators=100, random_state=0)
    model.fit(X_train, y_train)
    preds = model.predict(X_valid)
    return mean_absolute_error(y_valid, preds)

We'll try dropping the columns in the training and validation.

missing_columns = missed.index
keep = [column for column in x_train.columns if not x_train[column].hasnans]
reduced_X_train = x_train[keep]
reduced_X_valid = x_validate[keep]

print("MAE (Drop columns with missing values):")
drop_columns_error = score_dataset(reduced_X_train, reduced_X_valid, y_train, y_validate)
print(f"{drop_columns_error:0.2f}")

MAE (Drop columns with missing values):
17837.83

In this final step, you'll use any approach of your choosing to deal with missing values. Once you've preprocessed the training and validation features, you'll train and evaluate a random forest model. Then, you'll preprocess the test data before generating predictions that can be submitted to the competition.

Select the target variable, which corresponds to the sales price. Save this to a new variable called `y`. You'll need to print a list of the columns to find the name of the column you need.

Our target is SalePrice.

Y = data.SalePrice

Now you will create a DataFrame called `X` holding the predictive features.

Since you want only some columns from the original data, you'll first create a list with the names of the columns you want in `X`.

You'll use just the following columns in the list (you can copy and paste the whole list to save some typing, though you'll still need to add quotes):

• LotArea
• YearBuilt
• 1stFlrSF
• 2ndFlrSF
• FullBath
• BedroomAbvGr
• TotRmsAbvGrd

Training = Namespace(
    features = [
        "1stFlrSF",
        "2ndFlrSF",
        "BedroomAbvGr",
        "FullBath",
        "LotArea",
        "TotRmsAbvGrd",
        "YearBuilt",
    ],
    random_seed=0,
    train_size=0.8,
    test_size=0.2,
)
X = training_data[Training.features]
x_submit = testing_data[Training.features]

Split up the data into training and validation sets.

x_train, x_validate, y_train, y_validate = train_test_split(
    X, Y,
    random_state=Training.random_seed,
    train_size=Training.train_size,
    test_size=Training.test_size)