plan/06-evaluation.md

Evaluation Module Plan

Overview

A comprehensive evaluation module that computes 4+ metrics and enables comparison between CF methods and between the three approaches.

Evaluation Techniques (6 metrics)

1. RMSE — Root Mean Square Error

Measures prediction accuracy (rating prediction error).

RMSE = sqrt( (1/N) * Σ(r_ui - r̂_ui)² )

Interpretation: Lower is better. Penalizes large errors more heavily.

2. MAE — Mean Absolute Error

Average absolute prediction error.

MAE = (1/N) * Σ |r_ui - r̂_ui|

Interpretation: Lower is better. More interpretable than RMSE.

3. Precision@K

Fraction of recommended items that are relevant (rating ≥ threshold).

Precision@K = |relevant ∩ recommended| / K

Interpretation: Higher is better. How many recommendations were good?

4. Recall@K

Fraction of relevant items that were recommended.

Recall@K = |relevant ∩ recommended| / |relevant|

Interpretation: Higher is better. How many good items did we catch?

5. F1-Score

Harmonic mean of precision and recall.

F1@K = 2 * (Precision@K * Recall@K) / (Precision@K + Recall@K)

Interpretation: Higher is better. Balances precision and recall.

6. Coverage

Percentage of catalog items that can be recommended.

Coverage = |recommendable_items| / |total_items|

Interpretation: Higher is better. Does the system recommend everything or just popular items?

Comparison Dimensions

A) Between CF Methods

Compare all 4-5 CF methods on all metrics.

Method	RMSE	MAE	Prec@5	Rec@5	F1@5	Coverage
User-Based	0.92	0.71	0.60	0.45	0.51	0.40
Item-Based	0.88	0.68	0.65	0.50	0.57	0.55
SVD	0.81	0.62	0.72	0.58	0.64	0.78
KNN	0.95	0.74	0.55	0.40	0.46	0.35
Slope One	0.90	0.70	0.58	0.42	0.49	0.45

B) Between Three Approaches

Compare the best CF method vs Content-Based vs Knowledge-Based.

Metric	CF (Best)	Content-Based	Knowledge-Based
Precision@5	0.72	0.58	0.65
Recall@5	0.58	0.52	0.48
Coverage	0.78	0.60	0.90

C) Under Different Conditions

Condition	Best Approach	Reason
Dense user data	CF	Leverages peer patterns
Cold-start user	Knowledge-Based	No history needed
Cold-start item	Content-Based	Matches item features
Niche categories	Content-Based	Item features override sparsity
Explicit constraints	Knowledge-Based	Precise filtering

Implementation (recommender/evaluation.py)

class Evaluator:
    def __init__(self, ratings_df, predictions_df):
        self.ratings = ratings_df
        self.predictions = predictions_df
    
    def rmse(self): ...
    def mae(self): ...
    def precision_at_k(self, k=5): ...
    def recall_at_k(self, k=5): ...
    def f1_at_k(self, k=5): ...
    def coverage(self): ...
    
    def compare_cf_methods(self): ...
    def compare_approaches(self): ...
    def analyze_conditions(self): ...

Outputs for Analysis

1. Best CF method: Ranked table + recommendation
2. Best approach overall: Comparison chart
3. Condition analysis: When each approach excels
4. Why differences occur: Explanation of algorithmic biases