Rank 101 in the Amazon ML Challenge: France Was the Final Boss

Final public score: 0.989703. Final rank: 101.

The cut I wanted was the top 50. The leader was sitting around 0.9915, so the entire gap between us and first place was 0.0018. For the last 24 hours of the competition we were fighting over the fourth decimal place, and the fourth decimal place won.

101 is a nice number though. It's the course code of every intro class, which is roughly how I felt at 11 PM.

This is the story of how we got there: a two-stage pipeline, one leak that cost us a submission, a country with zero labels, and a morning where someone else was logged into our server deleting our stuff. Grab a chai, it's a long one.

The Problem

The task sounds boring, which is exactly why it's hard. You get businesses from three sources. For every record in Source 1 (a clean name, an address, a country), find every record in Source 2 and Source 3 that is the same business. Zero, one, or many.

The numbers:

• Test: 1.73M Source-1 businesses against a pool of ~10M Source-2/3 records.
• Train: 2.2M Source-1 businesses, 7.64M ground-truth pairs.
• A pool record belongs to at most one Source-1 business.

And the pool is noisy on purpose. The data generator loves this move: drop one word, append a generic one.

Source 1:  Platinum Harbors LLC
Source 2:  Platinum LLC Service        <- same business (94% of the time in US train)
Source 3:  Platinum Harbors LLC        <- same business, but at house number 412 instead of 214... or is it?

That last line is the real enemy: decoys. Near-copies of a real business with the same name and street but a slightly changed house number, and they're not the same business. Test has about 1.9x more decoys per business than train, which we only figured out by counting.

Now the metric. It's macro F0.5 per Source-1 business, so precision weighs 4x more than recall. It also has a special rule: if a business has no matches and you predict nothing, you score 1. If it has no matches and you predict anything, you score 0. The metric has trust issues, and honestly, fair.

And the plot twist: train has US and India. Test adds France, 15% of it, with zero labels. The organisers basically said "here's how Americans and Indians misspell their businesses, now go guess what the French do." Keep that in mind, because France decides the whole second half of this post.

The Pipeline (It Was Good, Actually)

We didn't start here, obviously. Submission 1 was a single LightGBM over key-based candidates and scored 0.966. The final thing looks like this:

Source-1 record
      │
      ▼
normalize (legal forms, Indic transliteration, French street types, junk removal, ...)
      │
      ▼
block: 12 hashed key types per country  +  bi-encoder top-10        dozens of candidates / business
      │
      ▼
stage 1: LightGBM, 92 pair features  ──►  keep p1 ≥ 0.005               ~5 candidates / business
      │
      ▼
cross-encoder (ModernBERT-base) scores every surviving pair
      │
      ▼
stage 2: LightGBM  (+ competition features, cluster features, cross-encoder score)
      │
      ▼
one owner per pool record  ──►  expected-F0.5 decoding  ──►  matches

A few parts worth talking about:

Blocking recall is the ceiling of everything downstream. If the true pair never becomes a candidate, no model can save you. Plain key blocking got 97.3% of the true pairs. Wider name keys and a phonetic key pushed it to 98.6%. Then a small fine-tuned bi-encoder (bge-small, 33M params) pulling the top-10 nearest records got it to 99.45%, finding 61% of what the keys missed.

Stage 2 knows about the competition. A pair isn't judged alone. Is this the S1's best candidate? Does this pool record have a better claimant? How many candidates share this house number? Decoys come in clusters around the real address, and cluster features catch that.

Decoding is where F0.5 gets respected. Instead of one global threshold, for every business we pick the number of matches k that maximises the expected F0.5, treating the candidates as independent coin flips with the model's probabilities. With precision weighted 4x, this is exactly where "should I add this fourth 60% candidate?" gets answered properly:

import numpy as np

def pbinom(ps):
    """distribution of the number of successes among independent Bernoulli(ps)"""
    d = np.array([1.0])
    for p in ps:
        d = np.append(d * (1 - p), 0) + np.append(0, d * p)
    return d

def best_k(ps):
    ps = np.sort(ps)[::-1]
    best = (np.prod(1 - ps), 0)          # predict nothing: right only if there is nothing
    for k in range(1, len(ps) + 1):
        tp, rest = pbinom(ps[:k]), pbinom(ps[k:])
        # F0.5 = 1.25 * TP / (k + 0.25 * n_true)
        ef = sum(a * b * 1.25 * i / (k + 0.25 * (i + j))
                 for i, a in enumerate(tp) for j, b in enumerate(rest))
        best = max(best, (ef, k))
    return best[1]

It beat the best global threshold, and it adapts per business for free.

Bug 1: The Submission Where Validation Went Up and the Leaderboard Went Down

Submission 3 was supposed to be the big one. Better normalization, IDF-weighted word overlap, decoy features. Validation went from 0.980 to 0.986.

Leaderboard: 0.969 → 0.961. 📉

Nothing makes you question your life choices like a model that improves everywhere except where it counts.

The cause took a while. I had computed the IDF tables (how rare each word is) separately for train and test. Tiny differences, say a word getting 2.31 in train and 2.29 in test. Looks harmless. But a gradient-boosted tree with 127 leaves doesn't see "a rare-ish word", it sees the exact value 2.31, and it memorised those exact values as word identities. On test the values shifted slightly, and the memorised shortcuts broke.

We found it by comparing SHAP contributions for the same pattern on validation and test. "One word replaced by a generic word, same address" had a stage-1 probability of 0.94 on validation and 0.80 on test. Same pattern, different split, different answer. That's a leak with a neon sign on it.

The fix: one IDF table per country computed from train+test together, relative (divided by log N) and quantized in 0.025 steps, so the model can't memorise exact values anymore. The next submission, with the bi-encoder and cross-encoder on top, scored 0.9835.

Lesson I will now tattoo somewhere: no feature may take split-specific exact values. If the same pattern gets different scores on two splits, find out why before you upload.

The Val Said 0.99, the Leaderboard Said "lol"

At this point validation was 0.9908 (US 0.9904, India 0.9915) and the leaderboard was 0.9835. Validation has no France, the leaderboard does. So the gap was France.

To measure how bad "unseen country" is, we faked it: train on US only, test on India. 0.932, against 0.982 when India is in training. So a France score of ~0.95 while the rest sat at ~0.99 was entirely believable.

First fix: self-training. Take the model's confident France predictions (p ≥ 0.97 as matches, p ≤ 0.03 as non-matches), add them to training as pseudo-labels, retrain. On the fake experiment it recovered +0.004, and on the leaderboard +0.0013 (0.98475). A second round with a cross-encoder adapted on French pairs got 0.98504.

Good, but not the 0.991 we wanted. The model was confidently wrong on France, and pseudo-labelling your own confident mistakes mostly makes you more confident.

The Small Tragedy

Before France, a detour I didn't plan for.

On the morning of the final day I logged into the server and my code was not my code anymore. Someone else was logged into our account, and more precisely another AI coding agent, running from a different machine. It had rewritten parts of the pipeline, deleted work/, output/, dataset/, code/ and even the backup folder, and kept sending SIGTERM to every job we started. Two AI agents fighting over one Linux user. 2026 is a strange year.

We survived because of two boring habits: the code lived in a hidden directory with full git history, and the last good submission was already on my laptop. Password changed, jobs rebuilt from the surviving caches, roughly half a day gone. Meanwhile the shared server was so full that the shell couldn't write its own output (1.3 GB free on an 869 GB disk), and LightGBM crawled at a host load of 200 until OMP_WAIT_POLICY=PASSIVE stopped its threads spin-waiting.

There's a longer story there, and it gets its own post. For now: commit everything, tag every submission, keep a copy off the shared machine. Paranoia is a feature.

France: The Final Boss

Evening of the last day. 11 submissions left, a public score just past 0.985, and a lot of France predictions to stare at.

Here's the rule I set before looking: every France fix needs a labelled analogue. France has no labels, so if a change can't be checked against the same pattern in US/India validation data, it doesn't ship. Probing the public leaderboard with 11 submissions and keeping whatever goes up is overfitting the public split, and the private split decides the winner.

Discovery 1: the model doesn't speak French

From US/India the model learned which appended words are generator noise ("services", "center", "inc") and which ones mark decoys ("group", "holdings", "midtown"). France has its own vocabulary, and the model was guessing.

S1:   deleves sportive sarl          6 rue paul doumer tourcoing
pool: developpement deleves sportive sarl   6 rue paul doumer tourcoing
model: p ≈ 0  ("developpement"? never heard of her)

Same address, only a noise word added. On this kind of pair the cross-encoder averaged 0.64 in France against 0.91 for US/India.

The trap: which French words are noise and which mark decoys? Our first attempt (p7) just stripped the "noisy-looking" French words from names. France jumped to 3.99 matches per business with 2.2% empty sets, where the train structure says ~3.4 and ~5.6%. It turns out "holding", "participations", "international" aren't noise, they're decoy markers: "X holding" at the neighbouring house number is a sibling company. Stripping the word made the decoy identical to the real thing. Reverted, and that sanity check (compare predicted matches-per-business and empty rate against train) caught it before a single upload was wasted.

The label-free way to separate them: look at where the words appear. Decoy words show up on near-copies at a changed house number (63% of the time) and almost never at the same address (0.1%). Noise words show up on same-address records. So the rule became: same address, pool name = S1 name plus only noise words (at most one S1 word dropped, and not just a legal form) → accept. On US/India validation, the analogous rule is 99.85% correct.

Leaderboard: 0.98504 → 0.98581.

Discovery 2: French associations live together

This one was the jackpot. I counted pairs of different Source-1 businesses sharing an identical address:

identical-address business pairs per 1000 businesses
US:      56
France: 365

French associations share addresses massively. Think of a city hall with a football club, a union, a hospital association and a choir, all at the same address:

nantes union sas     44 rue romain rolland
nantes hopital sas   44 rue romain rolland     <- different entity, one real word swapped

In US/India, a same-address pair where one real word is swapped for another is rare and mostly a decoy. The model learned "same address + mostly same name = match" and happily merged French siblings. Rejecting accepted pairs with that sibling signature (21.8k pairs):

Leaderboard: 0.98581 → 0.98872. The biggest single jump of the whole competition, +0.0029, from one count and one rule. More sibling signatures (neighbouring numbers, a generic word replaced by an entity word) → 0.98931.

Teaching the Model What the Rules Knew

Rules on top of a model always feel a bit like cheating, and they generalise worse than a model that actually understands the pattern. So the next step: make the model learn it.

We took the post-rule France predictions as labels and re-trained the French cross-encoder and stage 2 on them (cross-fitted, so no pair was scored by a model that trained on it). Round 3:

After that the rules only changed 60 pairs; the model had absorbed them. Leaderboard: 0.989555.

Interesting detail: an earlier attempt that re-trained only stage 2 on the rule labels (submission D) actually lost a little (0.98571 vs 0.98581). Re-training the cross-encoder too is what worked.

Then variance reduction. Self-training rounds each drift a bit randomly, so average them:

• mean(round 2, round 3): 0.989464. Worse than round 3 alone, because round 2 was clearly weaker. Averaging only helps between partners of similar quality.
• mean(round 3, round 4): 0.989652.
• mean(round 3, round 4, round 5): 0.989703. Final.

Down to the last submissions, I went for the version I'd trust on the private split rather than a single latest round, and an average of three rounds is exactly that. It still went up.

The Scoreboard

(not to scale, just vibes)
0.9900 ┤                                                          ● E345 0.98970
0.9895 ┤                                               ● R3  ● E34
0.9890 ┤                                         ● P2
0.9885 ┤                                    ● P1
0.9860 ┤                         ● C  ● D
0.9850 ┤                ● p5c ● B
0.9835 ┤          ● sub5
0.9690 ┤   ● sub2
0.9660 ┤ ● sub1
0.9610 ┤      ● sub3   (the leak)
       └──────────────────────────────────────────────────────────────────────

From 0.9848 to 0.9897 on the last day, almost all of it France.

What I Learned

Your validation set is lying to you if it doesn't contain your test distribution. Our validation said 0.9908 the whole time. It never contained a single French business. Everything that mattered on the last day was invisible to it.

Split-specific exact values are leaks. Quantize, share tables across splits, and compare the same pattern's score on validation vs test before shipping.

Count things before you model things. Decoys per business (1.9x more in test), identical-address pairs (365 vs 56 per 1000). Each count was worth more than any architecture change.

No labels? Find a labelled analogue. Every France rule was checked against the same pattern in US/India data first. That's the difference between finding something and fitting the public leaderboard.

Sanity-check the output's shape. Matches per business and empty-set rate vs the train structure caught the "holding" disaster before it cost a submission.

When a rule works, teach it to the model, then keep the rule as a safety net. 60 pairs changed after round 3; that's how you know the model got it.

Averaging helps only between equals. Weak + strong = worse than strong.

Ops is part of ML. Hidden directory, git, a tag per submission, setsid nohup for long jobs, backups off the shared server. The intruder turned a potential disaster into half a lost day.

And about the AI part, since I wrote a whole post about vibe-coding regret last time. This time my pair was Claude Code on the server: it wrote most of the pipeline stages, babysat the GPU jobs, and argued with me about which submission to spend. What made it work was the opposite of vibe-coding, though: every change measured, every idea checked against labels, every submission tagged in git. The France findings came out of that back-and-forth, and so did the nerve to spend the last submission on the "boring" averaged version.

What's Left (a.k.a. Where the Other 0.0018 Lives)

Honestly, not all of it was reachable. On validation about 82% of the true matches we rejected among our candidates were address-less pool records whose name matches several businesses ("Highland Fund" x4). That's a coin flip for anyone, and it puts the validation ceiling around 0.992.

What we didn't get to:

• Pseudo-labels for US/India test too. Test has a different decoy density everywhere, not just in France.
• A stage-1 fix for French vocabulary. The cascade discards pairs before stage 2 sees them. We measured it, though: only ~300 same-address French noise-word pairs were cut. Small.
• Seed ensembles and a train+validation refit. Probably +0.0003, and we ran out of day.

The End

So yeah, rank 101. Not top 50, but we went from 0.966 to 0.9897, found a leak, learned French (sort of), survived an intruder, and our last submission was our best one. Team Syndicate Smashers did not smash the top 50, but we did smash a lot of French siblings apart.

Thanks for reading, you're cool, weird and nonchalant for getting this far into a post about matching business names. If you find a mistake, congrats, you're better at proofreading than me.

And if you are a recruiter who read this far, HIRE ME PLEASE🥺. I can resolve your entities and your server security incidents.

Checkout my GitHub, LinkedIn, X. bye.