MLP Project Notes

The Kaggle journey

Heavy Equipment Selling Price Prediction: about 138k auction sales, 50 columns, predict the price. Metric is RMSLE. The project required a score under 0.20.

improvedmade it worsescore cutoff
Public-leaderboard RMSLE. Up is better on this chart.

A useful discovery: the column layout matched a well-known older Kaggle competition on bulldozer auction prices, just renamed. Reading how people approached that problem gave me a head start on feature ideas. Always search for similar public problems.

Version by version

  1. v1 · 0.20507

    Baseline: label and frequency encoding, one gradient-boosting model, target in log space. Just above the cutoff.

  2. v2–v3 · ~0.208 local

    Plain K-fold target encoding without smoothing. Rare categories got wild values. Adding smoothing brought it back to ~0.205.

  3. v4 · 0.19636

    Smoothed out-of-fold target encoding on every categorical column, interaction features, age and hours buckets, then two XGBoost variants, LightGBM and CatBoost stacked with Ridge. Cleared the cutoff. Local CV said 0.2025, so local was pessimistic.

  4. v5–v8 · no gain

    Parsing numeric-looking strings (mostly empty columns), native CatBoost categoricals, cleaning placeholder years. The trees already handled these through splits. Dead ends.

  5. v9 · 0.20076

    Optuna search found a very deep LightGBM with a better local score (0.2008). On the leaderboard it was worse than v4. The search overfit the 3 folds it searched on. Biggest lesson of the project.

  6. v10 · 0.19431

    Went back to v4 and ran it with 3 different fold seeds, averaging predictions. Boring and effective.

  7. v11 · 0.19831

    Pseudo-labelling: add confident test predictions back into training. Looked fine locally, hurt on the leaderboard.

  8. v12–v19 · ~0.193

    5 seeds, code cleanup and a few ensemble variants. Small steady gains; this is roughly what the notebook on this site scores.

  9. v20 · 0.18681

    The big jump. Machines sold in the same auction batch are related, so features describing the other items in the same sale (how many, what mix, typical prices) added real information. Rank went from ~148 to 12.

  10. v22 · 0.18587

    Comparable-sale anchors: what similar machines sold for earlier, the way a human appraiser prices things. Built strictly from past rows to avoid leakage.

  11. v23–v27 · 0.18475 → 0.18441

    A level-2 stacker on top of more diverse level-1 models. Each step smaller than the last.

  12. v30 → final · 0.18311 → 0.18223

    A cleaner rebuild that was safer on memory and time, then a blend of the best runs.

What I'd tell myself on day one

Worked

  • Log-target training so the loss matches the metric
  • Smoothed, out-of-fold target encoding
  • Seed bagging
  • Features from related rows (same batch, comparable past sales)
  • Diverse models blended with a non-negative Ridge
  • Keeping a log of every version with its real leaderboard score

Didn't

  • Heavy hyperparameter search
  • Pseudo-labelling
  • Rescuing columns that are 70–99% empty
  • "Fixing" outliers that trees already split around
  • Quick CSV uploads without a reproducible notebook behind them

On compute: long ensembles crashed my laptop more than once. Run the heavy versions as Kaggle notebooks, cache out-of-fold predictions to disk, and keep a small smoke-test mode that runs on a sample in a couple of minutes.