COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1)
Assignment 1 Brief
Deadline: Tuesday, October 29, 2019 at 14:00 hrs
Number of marks available: 20
Scope: Sessions 1 to 5
How and what to submit
A. Submit a Jupyter Notebook named COM4509-6509_Assignment_1_UCard_XXXXXXXXX.ipynb where
XXXXXXXXX refers to your UCard number.
B. Upload the notebook file to MOLE before the deadline above.
C. NO DATA UPLOAD: Please do not upload the data files used. We have a copy already.
Assessment Criteria
Being able to express an objective function and its gradients in matrix form.
Being able to use numpy and pandas to preprocess a dataset.
Being able to use numpy to build a machine learning pipeline for supervised learning.
Late submissions
We follow Department's guidelines about late submissions, i.e., a deduction of 5% of the mark each working
day the work is late after the deadline. NO late submission will be marked one week after the deadline because
we will release a solution by then. Please read this link
(https://sites.google.com/shef...
pli=1&authuser=1).
Use of unfair means
"Any form of unfair means is treated as a serious academic offence and action may be taken under the
Discipline Regulations." (from the MSc Handbook). Please carefully read this link
(https://sites.google.com/shef...
pli=1&authuser=1) on what constitutes Unfair Means if not sure.
Regularisation for Linear Regression
Regularisation is a technique commonly used in Machine Learning to prevent overfitting. It consists on adding
terms to the objective function such that the optimisation procedure avoids solutions that just learn the training
data. Popular techniques for regularisation in Supervised Learning include Lasso Regression, Ridge
Regression and the Elastic Net.
In this Assignment, you will be looking at Ridge Regression and devising equations to optimise the objective
function in Ridge Regression using two methods: a closed-form derivation and the update rules for stochastic
gradient descent. You will then use those update rules for making predictions on a Air Quaility dataset.
2019/10/27 COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1)
localhost:8888/notebooks/COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1).ipynb 2/7
Ridge Regression
Let us start with a data set for training , where the vector and is the design
matrix from Lab 3, this is,
Our predictive model is going to be a linear model
where .
The objetive function we are going to use has the following form
where is known as the regularisation parameter.
The first term on the right-hand side (rhs) of the expression for is very similar to the least-squares
objective function we have seen before, for example in Lab 3. The only difference is on the term that we use
to normalise the objective with respect to the number of observations in the dataset.
The first term on the rhs is what we call the "fitting" term whereas the second term in the expression is the
regularisation term. Given , the two terms in the expression have different purposes. The first term is looking
for a value of that leads the squared-errors to zero. While doing this, can take any value and lead to a
solution that it is only good for the training data but perhaps not for the test data. The second term is
regularising the behavior of the first term by driving the towards zero. By doing this, it restricts the possible
set of values that might take according to the first term. The value that we use for will allow a compromise
between a value of that exactly fits the data (first term) or a value of that does not grow too much (second
term).
This type of regularisation has different names: ridge regression, Tikhonov regularisation or norm
regularisation.
Question 1: in matrix form (2 marks)
Write the expression for in matrix form. Include ALL the steps necessary to reach the expression.
Question 1 Answer
Write your answer to the question in this box.
Optimising the objective function with respect to
There are two ways we can optimise the objective function with respect to . The first one leads to a closed
form expression for and the second one using an iterative optimisation procedure that updates the value of
at each iteration by using the gradient of the objective function with respect to ,
2019/10/27 COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1)
localhost:8888/notebooks/COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1).ipynb 3/7
at eac te at o by us g t e g ad e t o t e object e u ct o t espect to ,
where is the learning rate parameter and is the gradient of the objective function.
Question 2: Derivative of wrt (2 marks)
Find the closed-form expression for by taking the derivative of with respect to , equating to zero
and solving for . Write the expression in matrix form.
Also, write down the specific update rule for by using the equation above.
Question 2 Answer
Write your answer to the question in this box.
Using ridge regression to predict air quality
Our dataset comes from a popular machine learning repository that hosts open source datasets for educational
and research purposes, the UCI Machine Learning Repository (https://archive.ics.uci.edu/m... We are
going to use ridge regression for predicting air quality. The description of the dataset can be found here
(https://archive.ics.uci.edu/m...
In [ ]:
In [ ]:
We can see some of the rows in the dataset
In [ ]:
The target variable corresponds to the CO(GT) variable of the first column. The following columns correspond
to the variables in the feature vectors, e.g., PT08.S1(CO) is up until AH which is . The original dataset
also has a date and a time columns that we are not going to use in this assignment.
Before designing our predictive model, we need to think about three stages: the preprocessing stage, the
training stage and the validation stage. The three stages are interconnected and it is important to remember
that the testing data that we use for validation has to be set aside before preprocessing. Any preprocessing that
you do has to be done only on the training data and several key statistics need to be saved for the test stage.
𝑥1 𝑥𝐷
import pods
pods.util.download_url('https://archive.ics.uci.edu/m...
import zipfile
zip = zipfile.ZipFile('./AirQualityUCI.zip', 'r')
for name in zip.namelist():
zip.extract(name, '.')
The .csv version of the file has some typing issues, so we use the excel version
import pandas as pd
air_quality = pd.read_excel('./AirQualityUCI.xlsx', usecols=range(2,15))
air_quality.sample(5)
2019/10/27 COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1)
localhost:8888/notebooks/COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1).ipynb 4/7
Separating the dataset into training and test before any preprocessing has happened help us to recreate the
real world scenario where we will deploy our system and for which the data will come without any
preprocessing.
We are going to use hold-out validation for testing our predictive model so we need to separate the dataset into
a training set and a test set.
Question 3: Splitting the dataset (1 mark)
Split the dataset into a training set and a test set. The training set should have 70% of the total observations
and the test set, the 30%. For making the random selection make sure that you use a random seed that
corresponds to the last five digits of your student UCard. Make sure that you comment your code.
Question 3 Answer
In [ ]:
Preprocessing the data
The dataset has missing values tagged with a -200 value. Before doing any work with the training data, we want
to make sure that we deal properly with the missing values.
Question 4: Missing values (3 marks)
Make some exploratory analysis on the number of missing values per column in the training data.
Remove the rows for which the target feature has missing values. We are doing supervised learning so we
need all our data observations to have known target values.
Remove features with more than 20% of missing values. For all the other features with missing values, use
the mean value of the non-missing values for imputation.
Question 4 Answer
In [ ]:
Question 5: Normalising the training data (2 marks)
Now that you have removed the missing data, we need to normalise the input vectors.
Explain in a sentence why do you need to normalise the input features for this dataset.
Normalise the training data by substracting the mean value for each feature and dividing the result by the
standard deviation of each feature. Keep the mean values and standard deviations, you will need them at
test time.
Question 5 Answer
Write your explanation in this box
Write your code here
Write your code here
2019/10/27 COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1)
localhost:8888/notebooks/COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1).ipynb 5/7
In [ ]:
Training and validation stages
We have now curated our training data by removing data observations and features with a large amount of
missing values. We have also normalised the feature vectors. We are now in a good position to work on
developing the prediction model and validating it. We will use both the closed form expression for and
gradient descent for iterative optimisation.
We first organise the dataframe into the vector of targets and the design matrix .
In [ ]:
Question 6: training with closed form expression for (3 marks)
To find the optimal value of using the closed form expression that you derived before, we need to know the
value of the regularisation parameter in advance. We will determine the value by using part of the training
data for finding the parameters and another part of the training data to choose the best from a set of
predefined values.
Use np.logspace(start, stop, num) to create a set of values for in log scale. Use the following
parameters start=-3 , stop=2 and num=20 .
Randomly split the training data into what is properly called the training set and the validation set. As
before, make sure that you use a random seed that corresponds to the last five digits of your student
UCard. Use 70% of the data for the training set and 30% of the data for the validation set.
For each value that you have for from the previous step, use the training set to compute and then
measure the mean-squared error (MSE) over the validation data. After this, you will have num=20 MSE
values. Choose the value of that leads to the lower MSE and save it. You will use it at the test stage.
What was the best value of ? Is there any explanation for that?
Question 6 Answer
In [ ]:
Write your answer to the last question here.
Question 7: validation with the closed form expression for (2 marks)
We are going to deal now with the test data to perform the validation of the model. Remember that the test data
might also contain missing values in the target variable and in the input features.
Remove the rows of the test data for which the labels have missing values.
𝐰
Write your code here
Write your code here to get y and X
y =
X =
Write your code here
2019/10/27 COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1)
localhost:8888/notebooks/COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1).ipynb 6/7
If you remove any feature at the training stage, you also need to remove the same features from the test
stage.
Replace the missing values on each feature variables with the mean value you computed in the training
data.
Normalise the test data using the means and standard deviations computed from the training data
Compute again for the value of that best performed on the validation set using ALL the training data
(not all the training set).
Report the MSE on the preprocessed test data and an histogram with the absolute error.
Does the regularisation have any effect on the model? Explain your answer.
Question 7 Answer
𝐰 𝛼
In [ ]:
Write the explanation to your answer here.
Question 8: training with gradient descent and validation (5
marks)
Use gradient descent to iteratively compute the value of . Instead of using all the training set to compute
the gradient, use a subset of datapoints in the training set. This is sometimes called minibatch gradient
descent where is the size of the minibacth. When using gradient descent with minibatches, you need to find
the best values for three parameters: , the learning rate, , the number of datapoints in the minibatch and ,
the regularisation parameter.
As you did on Question 6, create a grid of values for the parameters and using np.logspace and a
grid of values for using np.linspace. Because you need to find three parameters, start with num=5 and
see if you can increase it.
Use the same training set and validation set that you used in Question 6.
For each value that you have of , and from the previous step, use the training set to compute using
minibatch gradient descent and then measure the MSE over the validation data. For the minibatch gradient
descent choose to stop the iterative procedure after iterations.
Choose the values of , and that lead to the lower MSE and save them. You will use them at the test
stage.
3 marks of out of the 5 marks
Use the test set from Question 7 and provide the MSE obtained by having used minibatch training with the
best values for , and over the WHOLE training data (not only the training set).
Compare the performance of the closed form solution and the minibatch solution. Are the performances
similar? Are the parameters and similar in both approaches? Please comment on both questions.
2 marks of out of the 5 marks
Question 8 Answer
In [ ]:
Write your code here
Write the code for your answer here
2019/10/27 COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1)
localhost:8888/notebooks/COM4509-6509_Assignment_1_UCard_XXXXXXXXX(1).ipynb 7/7
Write the answer to your last question here.
因爲專業，所以值得信賴。如有需要，請加QQ：99515681 或郵箱：[email protected]
微信：codehelp