mirror of
https://github.com/yusufipk/OpenFrame.git
synced 2026-09-11 09:36:08 +00:00
test(e2e): derive the storage route glob from R2_ENDPOINT
`failure-recovery.spec.ts` hardcoded `http://minio-test:9000/**`, which is the compose hostname. CI publishes MinIO on localhost, so the pattern matched nothing there: the PUT went through, the upload succeeded, and the test sat waiting for an error message that was never going to appear. It passed locally and failed on CI for a reason the diagnostic did not name. The glob now comes from R2_ENDPOINT, and the test counts the PUTs it actually refused and asserts the count is not zero. A pattern that matches nothing is now a failure that says so, rather than a failure that blames the error message. Recorded in AGENTS.md as the third way a test can be worthless, alongside a note to run a new spec under CI conditions and not only locally.
This commit is contained in:
@@ -15,6 +15,9 @@
|
|||||||
`bun run test:db:up` first.
|
`bun run test:db:up` first.
|
||||||
- If you added a batch of tests, hand them to a second reviewer before calling the work
|
- If you added a batch of tests, hand them to a second reviewer before calling the work
|
||||||
done. See "A batch of new tests gets an adversarial review, by somebody else" below.
|
done. See "A batch of new tests gets an adversarial review, by somebody else" below.
|
||||||
|
- If you added an end-to-end spec, run it under CI conditions too, not only locally. Hosts
|
||||||
|
differ between the two, and a `page.route()` glob that matches nothing passes locally and
|
||||||
|
tests nothing anywhere.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
@@ -47,9 +50,9 @@ For an API route, three cases are the minimum: an unauthenticated caller, a call
|
|||||||
signed in but not authorized, and the happy path. Assert the database row, not only the
|
signed in but not authorized, and the happy path. Assert the database row, not only the
|
||||||
status code: a refused DELETE has to leave the row present.
|
status code: a refused DELETE has to leave the row present.
|
||||||
|
|
||||||
### Two ways a test can be worthless
|
### Three ways a test can be worthless
|
||||||
|
|
||||||
Both have been found in this repo, so they are worth naming.
|
All three have been found in this repo, so they are worth naming.
|
||||||
|
|
||||||
1. **A test that cannot fail.** Before you finish, name the specific mutation of the
|
1. **A test that cannot fail.** Before you finish, name the specific mutation of the
|
||||||
production code your test would catch. If you cannot name one, delete the test. When it
|
production code your test would catch. If you cannot name one, delete the test. When it
|
||||||
@@ -58,7 +61,16 @@ Both have been found in this repo, so they are worth naming.
|
|||||||
function looks up means deleting an entry from that constant also deletes its own test
|
function looks up means deleting an entry from that constant also deletes its own test
|
||||||
case. Write expected values by hand as literals.
|
case. Write expected values by hand as literals.
|
||||||
|
|
||||||
A third variant is specific to `tests/api/auth-matrix.test.ts`: a route that refuses a
|
3. **A selector, route pattern or filter that matches nothing.** This one is specific to
|
||||||
|
`tests/e2e/`, and it fails in the worst direction: an injected failure that never gets
|
||||||
|
injected leaves the feature working, and the test then waits for an error message that
|
||||||
|
was never going to appear. Both halves need to be asserted, so count what you
|
||||||
|
intercepted and assert the count, the way
|
||||||
|
`tests/e2e/failure-recovery.spec.ts` does with `refusedPuts`. Never hardcode a host in a
|
||||||
|
`page.route()` glob either: CI publishes MinIO on localhost and the compose stack calls
|
||||||
|
it `minio-test`, and the pattern that matched locally silently matched nothing on CI.
|
||||||
|
|
||||||
|
A fourth variant is specific to `tests/api/auth-matrix.test.ts`: a route that refuses a
|
||||||
malformed request before it reaches its access check produces an entry that passes whether
|
malformed request before it reaches its access check produces an entry that passes whether
|
||||||
or not the guard exists. The suite catches it by requiring a 401 or a 403 rather than
|
or not the guard exists. The suite catches it by requiring a 401 or a 403 rather than
|
||||||
merely a non-2xx, and a 404 counts as suspicious rather than as a refusal, since a fixture
|
merely a non-2xx, and a 404 counts as suspicious rather than as a refusal, since a fixture
|
||||||
|
|||||||
@@ -19,6 +19,20 @@ import { db } from '@/lib/db';
|
|||||||
|
|
||||||
const SAMPLE_VIDEO = path.join(REPO_ROOT, 'tests', 'fixtures', 'sample.mp4');
|
const SAMPLE_VIDEO = path.join(REPO_ROOT, 'tests', 'fixtures', 'sample.mp4');
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Every request to object storage, for `page.route`.
|
||||||
|
*
|
||||||
|
* Derived from R2_ENDPOINT rather than hardcoded, because the host differs by
|
||||||
|
* environment and a pattern that matches nothing fails silently in the worst
|
||||||
|
* possible way: the PUT succeeds, the upload works, and the test that claims to
|
||||||
|
* inject a storage failure just waits for an error message that will never
|
||||||
|
* come. That is exactly what happened on CI, where MinIO is published on
|
||||||
|
* localhost, while locally it is the `minio-test` compose service.
|
||||||
|
*
|
||||||
|
* The fallback is the compose hostname, matching playwright.config.ts.
|
||||||
|
*/
|
||||||
|
const STORAGE_GLOB = `${process.env.R2_ENDPOINT ?? 'http://minio-test:9000'}/**`;
|
||||||
|
|
||||||
test.setTimeout(120_000);
|
test.setTimeout(120_000);
|
||||||
|
|
||||||
/** A video with an active version, cheap enough to make several of. */
|
/** A video with an active version, cheap enough to make several of. */
|
||||||
@@ -54,11 +68,13 @@ test('an upload that fails at the storage PUT leaves the form up and creates not
|
|||||||
// Only the bytes are refused. The presign (`r2-init`) and everything else the
|
// Only the bytes are refused. The presign (`r2-init`) and everything else the
|
||||||
// app serves still work, so the failure is exactly the one this test claims:
|
// app serves still work, so the failure is exactly the one this test claims:
|
||||||
// object storage rejected the upload halfway through.
|
// object storage rejected the upload halfway through.
|
||||||
await page.route('http://minio-test:9000/**', async (route) => {
|
let refusedPuts = 0;
|
||||||
|
await page.route(STORAGE_GLOB, async (route) => {
|
||||||
if (route.request().method() !== 'PUT') {
|
if (route.request().method() !== 'PUT') {
|
||||||
await route.continue();
|
await route.continue();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
refusedPuts += 1;
|
||||||
await route.fulfill({ status: 500, contentType: 'text/plain', body: 'storage is down' });
|
await route.fulfill({ status: 500, contentType: 'text/plain', body: 'storage is down' });
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -77,6 +93,12 @@ test('an upload that fails at the storage PUT leaves the form up and creates not
|
|||||||
timeout: 60_000,
|
timeout: 60_000,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// The interception really happened. Without this, a STORAGE_GLOB that matches
|
||||||
|
// nothing turns this test into an assertion about an error message that some
|
||||||
|
// unrelated failure happened to produce, and the diagnostic points at the
|
||||||
|
// message rather than at the pattern.
|
||||||
|
expect(refusedPuts, `no PUT to ${STORAGE_GLOB} was intercepted`).toBeGreaterThan(0);
|
||||||
|
|
||||||
// Still on the form, so the file list and the title survive for a retry.
|
// Still on the form, so the file list and the title survive for a retry.
|
||||||
await expect(page).toHaveURL(new RegExp(`/projects/${project.id}/videos/new$`));
|
await expect(page).toHaveURL(new RegExp(`/projects/${project.id}/videos/new$`));
|
||||||
await expect(page.getByLabel('Title')).toHaveValue('Doomed Upload');
|
await expect(page.getByLabel('Title')).toHaveValue('Doomed Upload');
|
||||||
@@ -88,7 +110,7 @@ test('an upload that fails at the storage PUT leaves the form up and creates not
|
|||||||
// Positive control: with storage healthy the very same steps succeed, so the
|
// Positive control: with storage healthy the very same steps succeed, so the
|
||||||
// assertions above are about the injected failure and not about the form
|
// assertions above are about the injected failure and not about the form
|
||||||
// being broken.
|
// being broken.
|
||||||
await page.unroute('http://minio-test:9000/**');
|
await page.unroute(STORAGE_GLOB);
|
||||||
await page.getByRole('button', { name: 'Add Video', exact: true }).click();
|
await page.getByRole('button', { name: 'Add Video', exact: true }).click();
|
||||||
await expect(page).toHaveURL(new RegExp(`/projects/${project.id}$`), { timeout: 90_000 });
|
await expect(page).toHaveURL(new RegExp(`/projects/${project.id}$`), { timeout: 90_000 });
|
||||||
await expect(page.getByRole('heading', { name: 'Doomed Upload', level: 3 })).toBeVisible();
|
await expect(page.getByRole('heading', { name: 'Doomed Upload', level: 3 })).toBeVisible();
|
||||||
|
|||||||
Reference in New Issue
Block a user