Get notified about events
Read your inbox, see which events notify whom, and send them to Slack or PagerDuty. Also shows how to test a channel and what each delivered message contains.
Approvals wait for a decision and production deployments fail while nobody watches. A notification puts the event in the inbox of the members who need to see it, and a channel delivers it to the organization's Slack or PagerDuty.
Before you begin
- Every signed-in member has an inbox in each organization. An API token has none.
- Seeing the channels on Settings > Integrations needs the
viewerrole. Adding, changing, testing and deleting a channel needsadminorowner. An API token limited to specific projects cannot manage channels. - For Slack, an incoming webhook URL that starts with
https://. For PagerDuty, the Events API v2 routing key (integration key) of the PagerDuty service to page.
Read your inbox
Select Inbox in the top bar. A badge shows the unread count, up to 99+. The popover lists your newest 50 notifications, each with a colored dot for its kind, a title, the start of its body and its age. Older history is in Activity.
Switch between All and Unread. When nothing is left, you see "You're all caught up."
Select a notification. It is marked read and the console opens its target: the approval, the deployment, the service's tab, or the cluster.
Select Mark all read to clear the badge.
The inbox updates without a reload. Through the API, listNotifications (GET /api/v1/notifications, with unread=true for unread only) and markNotificationsRead (POST /api/v1/notifications/read, with {"all": true} or a list of ids) do the same.
See who is notified
Every notification has one of four kinds. A channel subscribes to kinds, not to single events. The console labels them Approvals, Failures, Warnings and Deploys.
"Admins" below means members with the admin or owner role. "Production" means an environment marked as production.
| Event | Kind | Inbox | Slack |
|---|---|---|---|
| An approval is requested | approval | Admins, except the requester | Yes |
| A deployment fails | failure | The requester. In production, also admins. | Production only |
| A production deployment turns healthy | deploy | The requester | Yes |
| A build fails, or builds but cannot deploy | failure | The person who started it | Production only |
| A volume backup fails | failure | The requester, or admins for a scheduled backup | Yes |
| A cluster loses contact with its agent | warning | Admins | Yes |
| An agent update is available | warning | Admins, once per cluster and image | No |
| A public domain's certificate expires within 14 days | warning | Admins, once per certificate expiry | Yes |
| A cluster reconnects | warning | Nobody | No |
PagerDuty acts on two kinds only. A production deployment failure and a cluster losing contact each open an incident. The next healthy production deployment of that service, or the cluster reconnecting, resolves it. A PagerDuty channel subscribed to Approvals or Deploys receives nothing from them.
You cannot turn single events off in your own inbox. There is no per-member preference. Channels choose kinds with checkboxes.
Connect Slack
Select Settings, then Integrations. In the Notifications group, select Connect on the Slack row. The dialog Add a notification channel opens with Slack chosen under Channel kind.
Enter a Label, for example #deploys, and paste the webhook URL into Incoming webhook URL. The field stays hidden and is stored encrypted. It is never shown again.
Under Events, tick the kinds to deliver: Approvals, Failures, Warnings, Deploys. All four start ticked for Slack.
Select Add channel. A toast reads "LABEL added" and suggests sending a test. The row shows the label, the webhook's host, the chosen kinds and the state Connected.
The webhook URL must be absolute, start with https:// and carry no user name. A host that resolves only to private, loopback or link-local addresses is refused on every delivery.
Connect PagerDuty
Select Settings, then Integrations, and select Connect on the PagerDuty row.
Enter a Label, for example Production on-call, and the Events API v2 routing key. The key is 20 to 64 letters and digits, 32 as PagerDuty issues it.
Leave Failures and Warnings ticked. Those two start ticked for PagerDuty. Select Add channel.
Over the API, use the call above with "kind": "pagerduty" and the routing key as secret. The hint is then the key's last four characters, with a leading ….
To deliver to a second Slack channel or PagerDuty service, select Add channel in the group header. Each channel has its own events.
Send a test
On the channel's row, select Manage, then Send test.
A toast reads "Test delivered to LABEL". If the provider refused it, the toast reads "The test did not reach LABEL" and gives the reason.
Slack receives the title Test from NebulaCtrl: this channel is connected and a body such as Notifications for approval, failure, warning, deploy will arrive here.. PagerDuty receives a change event, which appears on the service's timeline and pages nobody. The API call is testNotificationChannel (POST /api/v1/notification-channels/{id}/test); delivered says whether it arrived.
Change, turn off or delete a channel
Select Manage on the channel's row.
- Turn Deliver to this channel off to pause it. The row shows Off, and deliveries not yet sent are dropped. Turning it on again does not replay them.
- Change Label or Events, or paste a new secret. A blank secret keeps the current one. Select Save changes. A new secret clears the last error.
- Select Delete, then Delete channel.
Deleting a channel removes its stored webhook URL or routing key and everything still queued for it. Members' inboxes are unaffected. To deliver there again, add a new channel with a new secret.
Creating, changing and deleting a channel is recorded in the audit log as notification_channel.created, notification_channel.updated and notification_channel.deleted.
Read a delivered message
Each message carries the notification's title, its body and a link that opens the target in the console. The control plane builds the link from its public URL.
| Kind of message | Title |
|---|---|
| Approval | Approval needed: deploy SERVICE to ENVIRONMENT. The verb is deploy, roll back, promote or apply a …. A change set has no service: Approval needed in ENVIRONMENT. |
| Deployment failed | Release #N of SERVICE failed to deploy to ENVIRONMENT |
| Deployment healthy | Release #N of SERVICE is live in ENVIRONMENT |
| Build failed | Build #N of SERVICE for ENVIRONMENT failed, or … was built but not deployed |
| Backup failed | Backup of VOLUME (SERVICE, ENVIRONMENT) failed |
| Cluster | Cluster NAME lost contact with its agent |
| Certificate | Certificate for HOSTNAME expires in N days, … expires in 1 day, … expires within a day, … has expired |
A failure body states the reason, or The agent reported no reason., followed by what to do next, for example "Open the deployment to see which stage stopped, then fix and redeploy."
In Slack the title is bold and the body follows, together cut at 3,000 characters, and a line below links Open in NebulaCtrl. In PagerDuty a trigger carries the title as the summary (cut at 1,024 characters), the source NebulaCtrl, a severity of error, warning or info, the body as details and the same link. A resolve carries only the keys that tie it to its incident.
If a send fails, the control plane retries with a wait that starts at 30 seconds, doubles and never exceeds an hour, for up to 24 attempts, about 16 hours. An HTTP 4xx answer other than 408 or 429 stops at once. Each attempt times out after 10 seconds, and a redirect counts as a failure. While it retries, the channel shows Failing and Last delivery failed: with the attempt number and the next wait. When it gives up it says gave up after N attempt(s). Fix the secret, then send a test.
Verify
- Request an approval from another account. The admins' Inbox badge increments, and a channel subscribed to Approvals posts the message.
- On the channel's row, the description ends with "delivered" and a relative time, and the state is Connected.
Next steps
- Approvals and freezes: the requests that raise Approvals.
- Roll back or promote a release: act on a failed deployment.
- Members and teams: who holds the
adminandownerroles. - Audit log: the record of channel changes.
Read logs and metrics
Tail and search a service's logs, read build and deploy logs, and read the Metrics tab and the numbers on each canvas card. Also covers opting a worker's own metrics in.
Manage members and teams
Invite people to an organization, set their roles, group them into teams, require two-factor sign-in and provision members from your identity provider with SCIM.