Testing AI Generated Code Before It Ships: A PR Gate

Copilot adds a discount helper that applies 20% twice. It also adds a test that posts /checkout and expects 200. CI is green. Finance sees double discounts at month end. The test never asserted the money.

import { test, expect } from "@playwright/test";

test("discount applies once on checkout total", async ({ request }) => {
  const res = await request.post("/api/checkout", {
    data: { sku: "MUG", qty: 1, discountCode: "SAVE20" },
  });
  expect(res.ok()).toBeTruthy();
  const body = await res.json();
  expect(body.total).toBe(800);
  expect(body.discountApplied).toBe(200);
});

# Cheap mutation: break the helper, tests must go red
# If they stay green, the agent tests are not testing the helper
sed -i 's/applyOnce/applyTwice/' src/pricing/discount.ts
npx playwright test tests/checkout-contract.spec.ts
# expect non-zero exit
git checkout -- src/pricing/discount.ts

• Require a contract test that asserts a business number (total, status enum, email sent), not only HTTP 200.
• Run one targeted mutation on the changed file. If tests stay green, reject the PR.
• Do not let the same agent merge its own tests without a human on the contract file.
• Not sure where your test quality baseline stands? Take the free QA maturity assessment first, then escalate through AI quality engineering services or the contact form.

• Pass: Double-apply mutation turns the contract test red; reverting the mutation turns it green.
• Fail: The only new tests are snapshots of the full HTML or a 200 status.
• Cleanup: Revert the mutation file. Never leave applyTwice in the branch.