TDGDC ServiceDEMO CORPORATIONLive major incident: MI000001 · Microsoft 365 authentication failureelapsed 01:08:13
Employee portal www.tdgdc.com

Problems & known errors

Root causes behind repeating incidents, with published workarounds

  • Payroll batch times out during SQL maintenance

    PRB0000521Root cause identified4 incidents

    Priority P2
    ROOT CAUSE
    The nightly payroll job has no retry and its 600s timeout is shorter than the cluster failover window.
    WORKAROUND
    Re-run PAY-NIGHTLY manually after the maintenance window closes; finance confirms output.

  • Cached tokens invalidated by certificate rolls

    PRB0000527Investigation14 incidents

    Priority P1
    ROOT CAUSE
    Signing certificate rotations are applied outside change control with no token cache pre-warm.
    WORKAROUND
    Clear the cached token on affected accounts; users re-authenticate once.

  • VPN sessions drop on dual-stack home networks

    PRB0000509Fix in progress6 incidents

    Priority P3
    ROOT CAUSE
    IPv6 route advertisement conflicts with the tunnel on concentrator firmware 9.4.2.
    WORKAROUND
    Disable IPv6 on the client adapter until firmware 9.5 ships in CHG0000433.

  • Floor 3 print devices lose server registration

    PRB0000498Resolved9 incidents

    Priority P4
    ROOT CAUSE
    A firmware push reset the print server registration on three devices.
    WORKAROUND
    Power cycle and re-register. Permanent fix deployed 12 September.