a-MSE.md

Mean Squared Error (MSE)

Understand the code

The codebase uses a script score.py to define metric functions. Accuracy metric was implemented for you. Run the script to see the result:

./score.py predict.txt solution.txt

Install the necessary packages:

pip install numpy

Implement MSE metric

Now your task as a developer is to implement MSE metric. Recall the formula for MSE is:

MSE = \frac{1}{N} \sum_{i=1}^{N} (predicted_i - actual_i)²

Solution

• Create a feature a_MSE branch:

git pull origin develop
git checkout develop
git checkout -b a_MSE

• Create a mse_metric function and compute its score:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import os
from sys import argv
import numpy as np
    

# ========= Useful functions ==============
def read_array(filename):
    ''' Read array and convert to 2d np arrays '''
    array = np.genfromtxt(filename, dtype=float)
    if len(array.shape) == 1:
        array = array.reshape(-1, 1)
    return array


def accuracy_metric(solution, prediction):
    correct_samples = np.all(solution == prediction, axis=1)
    return correct_samples.mean()


def mse_metric(solution, prediction):
    '''Mean-square error.
    Works even if the target matrix has more than one column'''
    mse = np.sum((solution - prediction)**2, axis=1)
    return np.mean(mse)


def _HERE(*args):
    h = os.path.dirname(os.path.realpath(__file__))
    return os.path.join(h, *args)
    

# =============================== MAIN ========================================
if __name__ == "__main__":

    #### INPUT/OUTPUT: Get input and output directory names

    prediction_file = argv[1]
    solution_file = argv[2]
    score_file = open(_HERE('scores.txt'), 'w')
    
    # # Extract the dataset name from the file name
    prediction_name = os.path.basename(prediction_file)
    
    # Read the solution and prediction values into numpy arrays
    solution = read_array(solution_file)
    prediction = read_array(prediction_file)
    
    # Compute the score prescribed by the metric file 
    accuracy_score = accuracy_metric(solution, prediction)
    mse_score = mse_metric(solution, prediction)
    print(
        "======= (" + prediction_name + "): score(accuracy_metric)=%0.2f =======" % accuracy_score)
    print(
        "======= (" + prediction_name + "): score(mse_metric)=%0.2f =======" % mse_score)
    # Write score corresponding to selected task and metric to the output file
    score_file.write("accuracy_metric: %0.2f\n" % accuracy_score)
    score_file.write("mse_metric: %0.2f\n" % mse_score)
    score_file.close()

• Commit and push changes:

git add score.py
git commit -m "implement MSE metric"
git push origin a_MSE

Implement unit tests for MSE

The tester have to implement unit tests for the accuracy_metric and mse_metric functions in the score.py program. The process involves creating a test class, setting up test data, and then writing test cases for each function. You'll ensure that the tests cover a variety of scenarios, such as correct and incorrect predictions, and edge cases like empty arrays.

Solution

• Install pytest:

pip install pytest

• Create a file tests/test_metrics.py:

import unittest
import numpy as np
import os, sys

sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
from score import accuracy_metric, mse_metric  # Import your metric functions

class TestMetrics(unittest.TestCase):
    
    def setUp(self):
        # Set up test data
        self.solution = np.array([[1, 0, 1], [0, 1, 0], [1, 1, 1]])
        self.correct_prediction = np.array([[1, 0, 1], [0, 1, 0], [1, 1, 1]])
        self.incorrect_prediction = np.array([[0, 1, 0], [1, 0, 1], [0, 0, 1]])
        self.empty_array = np.array([])

    def test_accuracy_metric(self):
        # Test accuracy with correct predictions
        self.assertEqual(accuracy_metric(self.solution, self.correct_prediction), 1.0)
        
        # Test accuracy with incorrect predictions
        self.assertEqual(accuracy_metric(self.solution, self.incorrect_prediction), 0.0)
        
        # Test accuracy with empty arrays
        self.assertEqual(accuracy_metric(self.empty_array, self.empty_array), 0.0)  

    def test_mse_metric(self):
        # Test MSE with correct predictions
        self.assertEqual(mse_metric(self.solution, self.correct_prediction), 0.0)
        
        # Test MSE with incorrect predictions
        self.assertEqual(mse_metric(self.solution, self.incorrect_prediction), 8/3)
        
        # Test MSE with empty arrays
        self.assertEqual(mse_metric(self.empty_array, self.empty_array), 0.0)  

if __name__ == '__main__':
    unittest.main()

• Run the tests:

pytest

• Commit and push changes:

git add tests/test_metrics.py
git commit -m "implement unit tests for metrics"
git push origin a_MSE

Fix MSE metric

The tests failed. The tester tells to the developer that the two metric functions can't handle empty arrays. Now, the developer need to fix this by adding checks to return a default value or raise an error when the input arrays are empty. The appropriate action depends on the expected behavior of your application.

Solution

def accuracy_metric(solution, prediction):
    if len(solution) == 0 or len(prediction) == 0:
        return 0
    correct_samples = np.all(solution == prediction, axis=1)
    return np.mean(correct_samples)


def mse_metric(solution, prediction):
    '''Mean-square error.
    Works even if the target matrix has more than one column'''
    if len(solution) == 0 or len(prediction) == 0:
        return 0
    mse = np.sum((solution - prediction)**2, axis=1)
    return np.mean(mse)

• Run the tests:

pytest

The tests now should pass.

• Commit and push changes:

git add score.py
git commit -m "fix metric functions to handle empty arrays"
git push origin a_MSE

Create a merge request

As the tests passed, the developer creates a merge request from a_MSE to develop.

1. On Gitlab project page, Select Code > Merge requests.
2. In the upper-right corner, select New merge request.
3. Select a source and target branch, then select Compare branches and continue.
4. Complete the fields on the New merge request page,
   • Description: implement MSE metric and unit tests for the two functions.
   • Assignee: Project leader
   • Reviewer: Code reviewer

then select Create merge request.

Approve a merge request

The code reviewer does run all the tests again and ensures that the features MSE and unittest have been accurately implemented. If something goes wrong, the reviewer communicates these findings to the developer and tester for rectification. If everything is OK, click on Approve.

Merge a_MSE into develop

It is the responsibility of the project leader to finalize the integration of changes into the develop. This involves merging the requests that have been created by the developer and subsequently approved by the reviewer by simply clicking Merge button.