Incidents

What happens when something goes wrong, on managed hosting and on your own.

Severity

Level	Means	Example
SEV1	Down, or data at risk	The deployment is unreachable. A suspected breach
SEV2	Badly degraded	Email is not sending. A connector the desk depends on is failing
SEV3	Working, with a problem	A report is wrong. A screen is slow

Severity is about impact, not about how interesting the cause is.

The first fifteen minutes

The order matters, and it is deliberately not “find the cause first”.

1. Confirm the scope. One tenant or all of them. One feature or everything. /api/health/ready and /api/health/version answer the basic questions without a database.
2. Say something. An acknowledgement with what is known beats a polished statement twenty minutes later. On managed hosting the affected customers are told; self-hosted, tell your own school.
3. Stop it getting worse. Roll back if a change is implicated. The update script prints its own rollback command.
4. Preserve evidence. Take a copy of the logs before restarting anything. Restarting is often the fix and always destroys the state you would want to look at.
5. Then investigate.

Step four is the one people skip and regret.

Communication

Tell people what you know, not a holding statement. “The API is returning 500 on every request and we are rolling back the release from this morning” is more useful than “we are investigating an issue”.

Use more than one channel. The status page is one, and it is no use if the outage is Plugboard itself. Email or a chat channel as well.

Say when you will next update, and then do, even if the update is that you know no more.

Data breaches

A suspected breach is SEV1 regardless of how it looks operationally.

Notification obligations depend on jurisdiction and are generally measured in hours rather than days. GDPR is 72 hours to the supervisory authority. Australia’s notifiable data breaches scheme requires an assessment within 30 days and notification as soon as practicable once it is established. Others differ.

Know your obligation before you need it. Whoever owns privacy at your school should be part of the response, not told afterwards.

Plugboard’s audit log is the primary evidence source: it records who accessed what and when, and it cannot be edited or deleted through the application by anybody.

Managed hosting

You are told about anything affecting your deployment. With what we know rather than with a holding statement.

Your audit log is yours during an incident as at any other time, including vendor access entries.

Restores are requested as described in support, restores and exits. We confirm what will be lost before doing anything, because a restore is one of the few operations where confirmation matters more than speed.

Self-hosted

The response is yours. Worth writing down before you need it:

• Who decides to roll back, and who they call at 7am.
• Where the backups are, and who can restore one.
• Where SECRETS_MASTER_KEY is, and who has it. Not on the server.
• How you tell your community, if the desk is down during a school day.

A page in your ICT runbook, reviewed once a year, is enough.

Afterwards

Write down what happened. Not to assign blame, but because the same class of thing will happen again and the note is what makes it shorter the second time.

Include the timeline. When it started, when it was noticed, when it was fixed. The gap between the first two is usually the actionable part, and it is usually a monitoring problem rather than a fix problem.

Change one thing. A retrospective that produces twelve actions produces zero.

Reporting a vulnerability

If you have found a security issue in Plugboard rather than an outage, see the project’s SECURITY.md for the disclosure process.