Skip to content

Quartz Scheduling Integration Tests

Follow-up to PR #2467 (the active-reviewer-tracking overhaul). Implements the scheduler-infrastructure portion of Phase 7 of the review-access-state plan — the piece that was explicitly deferred to a separate PR because establishing the Quartz+MassTransit test harness deserves focused attention rather than being crammed into a UX-heavy change.

The active-reviewer consumer race/idempotency scenario is deliberately split to a later feature-specific PR once both #2467 and this scheduler harness have landed.

Delete this doc when merged.

Why this exists

The active-reviewer-tracking feature (PR #2467, merging into main shortly) relies on MassTransit-scheduled messages — idle timeouts, suspend-and-release grace periods, connection-liveness checks — to release slots that reviewers have abandoned. Every scheduled message must either fire on time or be cancelled when the reviewer resumes; a drift or missed cancellation means slots leak or are released too aggressively.

The scheduling infrastructure is MassTransit-over-Quartz with SQL Server persistence in production (QuartzServiceCollectionExtensions.cs in the syrf-quartz service). The existing consumer tests (SessionTimeoutConsumerTests, CheckConnectionLivenessConsumerTests) are unit-style: they pass a Mock<IMessageScheduler> and verify what the consumer calls on it. That's good coverage for consumer logic, but it exercises zero of the actual scheduling behaviour.

Untested before this PR:

  • Whether SchedulePublish actually fires the message at T + delay.
  • Whether CancelScheduledPublish(tokenId) actually prevents delivery.
  • Whether a late cancel (after the message has already fired) is a safe no-op.
  • Whether reschedule-after-cancel under race conditions leaves orphan triggers.

This PR closes that scheduler-primitive gap. The production-consumer race proof remains separate because it depends on the active-reviewer consumers from #2467 and should live with that feature behaviour.

Scope

In scope

Fast integration suite — in-process Quartz with RAMJobStore wired through MassTransit's ITestHarness. Delays of 1-3 seconds so the full suite runs in under a minute.

Required test cases for this PR (from ADR-008 Decision 3 — docs/decisions/ADR-008-review-access-state-and-timing.md on the feat/signalr-active-reviewer-tracking branch, PR #2467):

  1. Schedule_FiresAfterDelay — schedule a message at T+2s; assert the consumer runs between [T+2s, T+3s]. Establishes that the scheduling primitive works end-to-end at all.
  2. Cancel_BeforeFire_PreventsDelivery — schedule, wait 500ms, cancel, wait past the original fire time; assert consumer was never invoked.
  3. Cancel_AfterFire_IsNoOp — schedule at T+1s, let it fire, then call cancel with the expired token; assert no exception is thrown (the cancellation is simply a no-op against a trigger that's already gone).
  4. Reschedule_OverwritesPrevious — schedule, cancel, schedule again with a new token; assert only the second fires. Covers the join → leave → join scenario where the hub cancels the old token before scheduling a new one.

Follow-up scope

Cancel_RacingDelivery_ConsumerIsIdempotent is intentionally not part of this PR. That test should exercise the production MarkSessionIdleConsumer / RemoveSuspendedSessionConsumer state guards by running schedule/deliver/cancel in a tight loop and asserting the consumers' existing idempotency (reservation is null, already idle, etc.) holds.

That scenario depends on the active-reviewer consumers from #2467 and is feature-specific enough to be clearer as its own follow-up PR, for example test(pm): cover active-reviewer scheduled cancellation race.

Optional smoke tier

2-3 tests using Testcontainers + SQL Server with the same Quartz setup we use in production. Tagged [Trait("Category", "Integration")] and excluded from the fast suite so CI can decide to run them on a cadence. Validates the persistence path specifically — RAMJobStore and the SQL Server ADO store have slightly different misfire semantics and we shouldn't assume parity.

Out of scope

  • Testing the actual consumer implementations (MarkSessionIdleConsumer, RemoveSuspendedSessionConsumer, etc.) against real state — those already have extensive unit coverage and are tested against the mocked scheduler. The gap this PR closes is specifically the scheduler primitive, not its consumers.
  • Testing active-reviewer production consumers under scheduled-delivery races. That belongs in the follow-up PR after #2467 and this scheduler harness are both available on main.
  • Changing any production code. This is a tests-only PR.

Approach

Test harness

MassTransit provides MassTransit.Testing.ITestHarness which gives a first-class testing experience — it integrates with AddMassTransit, captures published/consumed/scheduled messages, and lets you assert on them. Pair it with:

  • Quartz's RAMJobStore (the default, in-memory) instead of the production SQL Server store. No persistence dance, no Testcontainers needed for the fast tier.
  • MassTransit.Quartz for the scheduler implementation — same package the production code uses, so we exercise the real integration.
  • xUnit async lifecycle (IAsyncLifetime) on a test fixture that spins up and tears down the harness per test class.

Typical shape:

public class SchedulerIntegrationTests : IAsyncLifetime
{
    private ServiceProvider _provider = null!;
    private ITestHarness _harness = null!;

    public async Task InitializeAsync()
    {
        var services = new ServiceCollection();
        services.AddQuartz(q =>
        {
            q.SchedulerId = "test";
            q.UseDefaultThreadPool();
            // RAMJobStore is the default when UsePersistentStore is not called.
        });
        services.AddQuartzHostedService();
        services.AddMassTransitTestHarness(x =>
        {
            x.AddPublishMessageScheduler();
            x.AddConsumer<TestConsumer>();
            x.UsingInMemory((ctx, cfg) =>
            {
                cfg.UsePublishMessageScheduler();
                cfg.ConfigureEndpoints(ctx);
            });
        });
        _provider = services.BuildServiceProvider();
        _harness = _provider.GetRequiredService<ITestHarness>();
        await _harness.Start();
    }

    public async Task DisposeAsync() => await _provider.DisposeAsync();

    [Fact]
    public async Task Schedule_FiresAfterDelay() { /* ... */ }
}

Exact layout may vary once I start coding against the actual package versions — treat the above as shape rather than prescription.

Timing tolerances

Use [scheduledAt, scheduledAt + 500ms] as the acceptance window for "fired on time" assertions. Stricter windows make tests flaky on CI under load; wider windows defeat the point. 500ms is defensible and matches how scheduled-message-plugin-backed systems are typically tested.

For "didn't fire" assertions (after cancellation), wait scheduledDelay + 1s before asserting — enough margin that a trigger that was going to fire would already have delivered.

Location

Add to the existing SyRF.ProjectManagement.Endpoint.Tests project as a new file SchedulerIntegrationTests.cs (or a dedicated folder Scheduling/). Rationale: the project already references MassTransit; adding a parallel test project just for scheduling tests would be premature.

If the Testcontainers smoke tier lands, that's an arguably different dependency set (Testcontainers, Microsoft.Data.SqlClient for the Quartz ADO store) and may warrant its own assembly. Decide when implementing.

Task breakdown

Fast tier — DelayedMessageScheduler (in-memory, always runs)

  • Scaffold SchedulerIntegrationTests with MassTransit ITestHarness + DelayedMessageScheduler.
  • Schedule_FiresAfterDelay — scheduler wiring proof (passes).
  • Cancel_AfterFire_IsNoOp — late-cancel API contract (passes).

Cancel-before-fire and reschedule-overwrites cases were moved to the integration tier after investigation showed DelayedMessageScheduler is a simple timer and doesn't support token-tracked cancellation. Those semantics only exist with a real scheduler backend (Quartz).

Integration tier — Testcontainers + production Quartz (opt-in)

  • Add Testcontainers.RabbitMq to the test project.
  • SchedulerQuartzInfrastructureTests fixture: real RabbitMQ container, AddQuartz with RAMJobStore, AddPublishMessageScheduler + AddQuartzConsumers + UsingRabbitMq + UsePublishMessageScheduler — same wiring production uses.
  • Schedule_FiresAfterDelay against the real infrastructure.
  • Cancel_BeforeFire_PreventsDelivery — only meaningful at this tier.
  • Cancel_AfterFire_IsNoOp — end-to-end version of the fast-tier contract test.
  • Reschedule_OverwritesPrevious — only meaningful at this tier.
  • Tag all with [Trait("Category", "Integration")] so the fast-tier CI gate skips them by default; opt in via suite label.
  • Decide: do we also need the SQL Server persistence tier (Testcontainers.MsSql + UsePersistentStore)? Defer until the integration-tier has been in for a while and we see whether RAMJobStore vs. ADO store drift surfaces any bugs.

Three test-infrastructure subtleties surfaced during implementation (all documented on the fixture itself):

  1. Quartz caches the ILoggerFactory in static state via MicrosoftLoggingProvider, so disposing the factory between tests produces ObjectDisposedException. Fixed by IClassFixture<RabbitMqQuartzFixture> — one container + bus + logger for the whole class.
  2. ITestHarness.Consumed.Any<T> returns false once the harness inactivity token fires (~15s by default). In a shared fixture after the first test, subsequent calls return immediately. Fixed with a per-token TaskCompletionSource capture in the consumer (TokenCapturingConsumer), so assertions don't depend on harness observability.
  3. Rapid cancel-after-schedule can race the scheduler's trigger registration. If the cancel command arrives at Quartz before the schedule command has finished registering the trigger, the cancel is a silent no-op. Fixed by settling ~500ms between schedule and cancel in the reschedule test — mirrors the RabbitMQ round-trip timing production gets naturally.

Follow-up after #2467 and this PR

  • Open a follow-up PR, suggested title: test(pm): cover active-reviewer scheduled cancellation race.
  • Implement Cancel_RacingDelivery_ConsumerIsIdempotent at the integration tier. Requires the production MarkSessionIdleConsumer / RemoveSuspendedSessionConsumer from #2467 plus the scheduler harness from this PR.
  • Update docs/planning/review-access-state-overhaul.md (in whichever branch ADR-008 lands on) to mark Phase 7 complete once both the primitive harness and consumer-race coverage are merged.

Progress note (2026-04-18)

Two-tier approach fully landed, 6 tests total, all green:

  • Fast tier (SchedulerIntegrationTests): 2 tests, DelayedMessageScheduler, <5s total. Validates the IMessageScheduler API contract and scheduler wiring. Runs on every CI build.
  • Integration tier (SchedulerQuartzInfrastructureTests): 4 tests, real RabbitMQ container via Testcontainers + production MT.Quartz wiring, ~75s per run (container startup dominates). Covers all 4 scenarios including the two cancellation semantics the fast tier can't meaningfully assert. Tagged [Trait("Category","Integration")] — opt-in via suite label.

The Package Version risk called out in the Risks section below turned out to matter more than expected — the canonical MassTransit test-harness pattern is DelayedMessageScheduler, not Quartz, and that in-memory scheduler doesn't implement cancellation. That's why the integration tier using the real infrastructure is load-bearing, not optional — it's the only tier that can prove cancellation guarantees.

The active-reviewer consumer race/idempotency scenario has been deliberately split into the follow-up PR above. It is not a blocker for this scheduler-harness PR.

Risks / unknowns

  • Package version surprises. The project uses MassTransit 8.4.0 and Quartz 3.x (via QuartzServiceCollectionExtensions). The harness API I've sketched above is based on the public docs for those versions; shape may differ once I wire it up.
  • Host lifecycle. Quartz + MassTransit both register hosted services. The fixture needs to start them in the right order and wait for the scheduler to be ready before calling SchedulePublish. WaitUntilStarted: true (which the production service uses) is likely the right pattern for tests too.
  • Flaky timing on CI runners. Heavily-loaded runners can miss 500ms windows. If that happens, widen to 1s before giving up — but first check whether the delay is consistent (scheduler is slow) or variable (test runner noise).
  • RAMJobStore vs production SQL Server store drift. Covered by the optional smoke tier, but even with that, a misfire edge case unique to the ADO store could slip through. Document any behaviour differences that surface.

References

  • Parent plan: docs/planning/review-access-state-overhaul.md on the feat/signalr-active-reviewer-tracking branch (PR #2467) — Phase 7 "Future work" section.
  • ADR-008 Decision 3 — Integration-test Quartz scheduling directly.
  • Production Quartz setup: src/services/quartz/SyRF.Quartz/QuartzServiceCollectionExtensions.cs.
  • MassTransit scheduler wiring: src/libs/webhostconfig/SyRF.WebHostConfig.Common/Extensions/MassTransitHelpers.cs.
  • MassTransit testing docs — ITestHarness usage.
  • MassTransit + Quartz docs — scheduler configuration.