Backport Bot
A backport tool for aws-lc. It works out which supported release branches a security fix still needs to reach, cherry-picks it there, and opens one pull request per branch for a human to review.
1.0 Summary
aws-lc is the cryptography library that sits under a large amount of AWS. Security fixes land on mainline first, then have to be backported to every currently supported LTS and FIPS release branch. That is done by hand with cherry-picks, so the work scales with the number of fixes multiplied by the number of supported branches, and the branch count only grows.
I built a command-line tool that automates the mechanical part of the backport — about 5.7K lines of Python, as a CLI plus a GitHub Actions workflow, covering aws-lc's six supported FIPS release branches. Point it at a mainline fix and it resolves which branches are in scope, decides per-branch whether each one is actually affected, cherry-picks to the ones that are, and opens a pull request on each. The core is a deterministic git engine (Section 4.0). An AI layer runs alongside it to catch the two cases git alone gets wrong (Section 5.0). Nothing is ever auto-merged, so a human reviews every result.
In effect it turns roughly thirty minutes of manual work per fix into about five minutes of review.
2.0 Background & Problem
For a single fix the manual process is manageable. It stops being manageable when several fixes land at once across several supported branches, because the coordination overhead is what actually costs the time, not any individual cherry-pick.
2.1 The Manual Process
When a fix lands on mainline, an engineer has to:
- Work out which branches are currently supported.
- Work out which of those branches contain the affected code.
- Cherry-pick the fix to each affected branch.
- Resolve any merge conflicts.
- Open a pull request per branch.
- Track which branches have been patched.
Steps 1, 3, 5 and 6 are mechanical, which makes them a good target for automation. Step 2 is the one that needs judgment, and step 4 sometimes does.
2.2 Impact Analysis
Not every supported branch is affected by every fix, and there was no formalized way to decide which were. Two methods were in use, and both are manual:
- Commit ancestry. Find the commit that introduced the bug, then check which branches contain it. Quick, but not always conclusive, since a behavioral bug can predate a restructure of the code around it.
- Test-based. Write a test that exercises the bug and run it on every supported branch. More reliable, but much harder to do, since some bugs are difficult to reproduce on demand.
Neither scales when multiple issues arrive at the same time, which is exactly when you least want to be doing it by hand.
3.0 Goals & Scope
3.1 In Scope
- Take a fix as input, as a commit SHA, a commit range for a fix spread over several commits, or a PR number.
- Deterministic git-based impact analysis, using ancestry and patch-id.
- An always-on AI layer that runs alongside the deterministic check on every analyzed branch.
- Running the impact analysis on its own, before any public code change, so an embargoed fix can be assessed per branch without cherry-picks or pull requests.
- Cherry-picking to affected branches and opening one PR per branch.
- Guided, human-in-the-loop conflict resolution (Section 6.0).
- Detecting fixes that are already applied, so no redundant PR is opened.
3.2 Out of Scope
- A model proposing a conflict resolution. Guided resolution is in scope; handing the merge decision to a model is not.
- AI-assisted test generation.
- Reverts and rollbacks of backports.
- Auto-merging any backport PR, under any circumstances.
4.0 Deterministic Engine
Everything with a side effect is deterministic. The AI can move a verdict inside the limits in Section 5.2, but it never writes code, runs commands, or merges anything.
4.1 Resolving In-Scope Branches
In-scope branches come from a machine-readable manifest kept in sync with the library's versioning docs. A branch is in scope when it exists, is actively maintained, and has not passed its end-of-support date. When the manifest is missing, the tool falls back to matching the branch naming convention plus an explicit list for one-off branches, which exist and would otherwise be missed.
4.2 Finding the Introducer
To decide whether a branch is affected, the tool first needs the commit
that introduced the patched lines. The obvious tool is
git blame, and it is the wrong one.
git blame answers "who last touched this line". When a fix
patches lines that were themselves added by an earlier fix, blame
attributes them to that earlier fix, which often only exists on mainline.
The tool would then conclude "not affected" while the underlying bug is
still sitting there. That is a silent false negative, and on a
security backport it is the worst possible failure.
git log -L<range>:<file> --reverse answers a more
useful question: when did these lines first exist. It returns every commit
touching the line range oldest-first, the tool takes the oldest, and it
follows file renames for free. Comment-only and blank hunks are skipped, so
a stale comment does not trace back to an ancient import and over-flag
every branch.
Taking the oldest commit is a heuristic and it has a limit worth being upfront about. It assumes the line was vulnerable from the moment it was written, which is not always true. Vendored third-party code is the clear case: a bulk import predates every branch, so the heuristic flags branches that were never vulnerable. When the heuristic is wrong it over-flags rather than misses, which is the safe direction, and Section 5.1 is how those over-flags get caught.
4.3 Precision Checks
Two checks then trim the over-flags:
- A branch on which none of the fixed files exist is not affected.
- A branch on which the exact lines the fix changes or removes are absent, matched ignoring whitespace and comments, is not affected either. The vulnerable code was rewritten or never existed there, even though an ancestor introduced the surrounding code.
A branch that already carries the fix is skipped as already patched, either
because the fix commit is a direct ancestor or because an equivalent change
matches by git patch-id. That is what catches a fix that was
cherry-picked under a different SHA.
5.0 AI Impact Analysis
5.1 Two Roles
The model runs alongside the deterministic check on every analyzed branch, in one of two roles depending on what git decided:
- False-positive auditor, on branches flagged as affected. This is where the vendored-code over-flag from Section 4.2 gets caught.
- Tie-breaker, on branches the deterministic check cannot resolve. A branch is unresolved when the introducer is neither an ancestor nor a patch-id match but a changed file is still present, so the vulnerable code may be there and history alone cannot say.
Running it on every branch rather than only the unresolved ones was a deliberate change from an earlier design. A fallback that only fires when git is unsure cannot catch git's false positives, because those happen on the confident path, where the engine returns its verdict before any model is consulted.
5.2 Safety Gates
The verdict feeds into the decision, but the two directions are gated by risk so a model can never cause a missed backport:
- As a tie-breaker, "likely affected" upgrades an unresolved branch to a backport. That direction is safe because the only thing it can do is add a pull request for a human to read.
- As an auditor, "likely not affected" can cancel a backport only when it is high-confidence and a deterministic check confirms the exact changed lines are provably absent. If the vulnerable lines are still there, or the fix is a pure addition with nothing to confirm, the PR opens anyway with the caveat attached.
5.3 Results
Measured two ways. Against a 300-cell ground-truth oracle built independently from repository forensics, and replayed over 30 real CVEs, it reached 99% agreement with zero missed backports.
And on a synthetic 7-branch by 7-scenario matrix, which isolates the failure modes rather than sampling real ones, 49 decisions:
| Method | TP | TN | FP | FN |
|---|---|---|---|---|
| Deterministic only | 34 | 9 | 1 | 5 |
| Deterministic + AI | 39 | 10 | 0 | 0 |
The five false negatives were all the same shape: a rename or rewrite obscured a line's history, so ancestry pointed at the wrong introducer and a still-vulnerable branch looked clean. Those are the dangerous ones, and closing them is the whole argument for the AI layer.
6.0 Conflict Handling
Cherry-picks split three ways:
- Clean apply. Opens a normal backport PR.
- Conflict in test or generated files only. The source fix applied cleanly and only a test hunk clashed, so the branch keeps its own tests, the source fix is committed, and it becomes a normal PR with a note in the body.
- Real source conflict. Aborted, so nothing is committed and no half-applied branch is left behind, and reported in the summary. Deciding whether the vulnerability still applies and how to adapt the fix needs human judgment.
Real conflicts are then handled by an interactive resolve
command rather than dumped back on the engineer. It targets exactly the
branches that conflicted and lets them resolve each one in place, in their
own checkout so their editor shows the conflict live, or in a throwaway
worktree. git rerere is enabled, so a resolution recorded on
one branch is applied automatically to an identical conflict on a sibling
branch and surfaced for verification rather than committed silently. That
removes the repetition without taking a human out of any decision.
7.0 Security Model
The tool runs locally under the operating engineer's own git and GitHub credentials rather than as standing automation. That means there is no automation identity to provision and no repository-wide permission to grant for creating pull requests. Every branch and PR it creates is attributed to that engineer and reviewed like their own work, and it can only do what they are already allowed to do.
The AI step is read-only. It has no write access, runs no commands, and applies nothing itself. The most it can do is move the affected decision within the gates in Section 5.2, and that is always realized as a human-reviewed pull request or its absence. Feeding source into a model is a data-egress consideration and prompt injection through that content is the main risk, but it is bounded: a manipulated model cannot cancel a backport whose vulnerable lines git still finds present, cannot write code, and cannot merge.
Wrapping the tool in CI later would reintroduce the standing-credential question, which is why running it locally was the deliberate choice for now.
8.0 Status
The deterministic engine, the CLI, and the AI layer are working, and the interactive conflict-resolution flow is built. Beyond the synthetic matrix above, the tool has been exercised against real backports with a replay harness that reconstructs each branch's pre-backport state for a past fix and compares the tool's verdict against the branches the team actually patched. Across those real fixes it produced no false negatives.
Its remaining false positives were mostly not analysis errors. They were branches that are genuinely affected but had not been backported by hand, either because the work was still in flight or because they were excluded for reasons the git history does not record.