Project playbook
The order I'd do things in if I started the MLP project again. Works for most tabular Kaggle problems.
Before you write code
- Submit the milestones. Each one is a mark and you need at least two to pass the project; they come from the weekly SEEK forms. Keep the email confirmation.
- Read the allowed-library list. Deep-learning frameworks and AutoML were not allowed in my term; XGBoost, LightGBM, CatBoost and sklearn were.
- The rubric wants at least three different models compared, a proper EDA with written observations, and a score under the cutoff. A great score with a thin EDA loses marks.
- Register and share your notebook exactly as the guideline says. Do it on day one.
1. Understand the metric
RMSLE is RMSE on log(1 + y). It judges errors as percentages, so a 10k miss on a cheap machine costs more than on an expensive one. Train on np.log1p(y), predict, then np.expm1. Your training loss now equals the metric, and the right skew of prices disappears.
2. EDA that earns marks
End every plot with a short written observation. Examiners pick a chart and ask what it shows.
- Shape, dtypes, duplicates, numeric vs categorical count
- Missing fraction per column as a bar chart; decide a drop threshold (I used 75%)
- Target raw vs log histogram with skew before and after
- Numeric columns: histogram + boxplot pairs; spot placeholders like year 1001
- Cardinality of each categorical column; this decides your encoding
- "What moves the price": median target by age, by sale year, by category, by usage bucket
3. Features
| Idea | Why it helps |
|---|---|
| Date parts + days since start | Prices drift over time |
| Age = sale year − manufacture year, clipped | Strongest single driver of price |
| log hours, hours per year, missing and zero flags | Usage is skewed; "not recorded" ≠"zero" |
| Ordinal map for Low / Medium / High | Keeps the order a label encoder loses |
| Category × category combos | Interactions a single split can't express |
| Features from related rows | Same-batch and comparable past sales carried the most signal |
4. Encoding by cardinality
- Few categories: label encoding for trees, one-hot inside a Pipeline for linear models
- Many categories: frequency encoding and smoothed out-of-fold target encoding
- Keep strings like "None or Unspecified" as their own category when they mean "this attachment isn't there"
Smoothing formula: (mean × count + global_mean × k) / (count + k). A category seen twice gets pulled toward the global mean instead of taking an extreme value.
5. Leakage checklist
- Target encoding computed out-of-fold; test uses the full-train version
- Frequencies, medians and scalers fitted on training rows only
- Preprocessing inside a
Pipelineso CV refits it per fold - Time-based "past sales" features only look backwards
- Early stopping on the validation fold; refit on all data with the average best iteration
6. Models
Compare at least: Ridge (linear reference), Random Forest (bagging), XGBoost, LightGBM, CatBoost. On a 40k-row sample my holdout scores were roughly XGBoost 0.221, LightGBM 0.224, CatBoost 0.233, Random Forest 0.237, Ridge 0.414. Boosting wins on tabular data because each tree fixes the previous trees' residuals.
Give the ensemble variety: two XGBoost models of different depth, a wide-leaf LightGBM, and CatBoost with its symmetric trees made errors that overlapped less.
7. Blending
- Collect out-of-fold predictions from each model
- Print their correlation; low correlation means blending will help
- Fit
Ridge(positive=True)on the OOF columns to learn weights - Refit each model on all data, apply the weights to test predictions
- Repeat with several seeds and average in log space
- Clip tiny predictions and reindex to the sample submission order
Keep a results log. A plain text file with version, idea, local CV and real leaderboard score. It stopped me from repeating failed ideas.