Broadcasts Design

I wanted to put together a quick write-up of Broadcasts: a protocol I designed for Noko’s API to send quasi-realtime updates. It combines the simplicity of webhooks with efficient message passing and Hypermedia URLs.

The problem

We’ve been adding webhooks to Noko’s API v2 so that developers can respond to events in real-time and build even better integrations. Particular parts of Noko’s API don’t fit into traditional webhooks, such as: Timers, and Budgets & Goals in the future.

Rapidly changing resources

These resources can change rapidily. For example, in a matter of seconds, someone can:

1. Pause their running Timer
2. Add some notes to the description for that Timer
3. Discard a paused Timer that is no longer relevant
4. Start a Timer for a wholly different project, because their client starting DMing them on Slack 10 minutes ago
5. Add the 10 minutes to their newly running Timer

Developer simplicity and API reliability

A core design philosophy for Noko is to be low-maintenance, including our APIs. A core tenant of our API is to make it simple to build integrations and debug issues.

We’ve baked this into our design for webhooks, which are sent serially, in-order, and halt on any failed payload. That way, someone can fix a bug and not have to clean up data because they continued to receive updates while encountering an error.

Generating and sending individual, verbose Webhook payloads for each update on a Timer would slow down the integration’s ability to respond in real-time, when the user only cares about the latest state of their Timers.

The solve

Broadcasts are a hybrid protocol that use the foundation of webhooks, with a few specialized rules to make them easy for developers to implement. Once I wrapped my head around the problem, the structure came together quickly, and it’s remarkably simple!

Structure

• Developers can set up a Broadcast Receiver, which will receive Broadcasts for the Subjects they have signed up for (such as a Timer).
• Noko tracks the latest state of Subjects, and sends out the latest state at regular intervals (the Broadcast).
• Developers receive the Broadcast, and use the attached Hypermedia URL to get the full details of the resource.

Here’s an example of a Broadcast message, from our API docs:

{
  "broadcast_receiver": {
    "id": 77885522,
    "name": "Da Best Timer Sync"
  },
  "sent_at": "2020-09-30T15:19:28Z",
  "last_event": "timer.running",
  "subject": {
    "type": "timer",
    "url": "https://api.nokotime.com/v2/projects/37396/timer"
  }
}

The design is based on combining three tools to create a protocol that’s efficient and straightforward for everyone:

• webhooks
• UDP-style messaging
• Hypermedia URLs

webhooks

The beauty of webhooks is their simplicity: They’re straightforward HTTP requests. This makes them easy for webapps to process, since they’re already running a server listening for requests. Using webhooks to send Broadcast messages means that a developer can add real-time updates to their integration without adding any more dependencies or third-party libraires.

UDP-style messaging

UDP-style messaging means that we don’t care if a message is successfully received. Noko will continue to send messages, even if a particular one fails, and messages are never retried.

This works for Broadcasts because their payload is so efficient, and the developer will be making a Hypermedia request to get the current state of the resource anyways. Retrying a payload wouldn’t make any sense, because by the time it’s retried, the resource will have already changed.

Hypermedia URLs

Noko has Hypermedia URLs throughout, and using them here makes Broadcasts easy to implement. The API docs specify that Hypermedia URLs should be used for requests, so that a client can navigate to actions & resources without needing to generate URLs.

With the Hypermedia URL in a Broadcast, developers can reuse their existing Noko API Client to get the details for the resource. This means code-reuse, and no additional dependencies.

It also means that our Broadcast messages can be remarkably small. Compare the following Webhook Payload to a Broadcast message:

{
  "webhook": {
    "id": 123456,
    "name": "Da Best Noko Webhook"
  },
  "type": "entry.updated",
  "created_at": "2012-01-09T08:33:29Z",
  "changes": {
    "minutes": [
      30,
      60
    ],
    "invoiced_at": [
      null,
      "2012-01-10T08:33:29Z"
    ],
    "invoice": [
      null,
      {
        "id": 12345678,
        "reference": "AA001",
        "invoice_date": "2013-07-09",
        "state": "unpaid",
        "total_amount": 189.33,
        "url": "https://api.nokotime.com/v2/invoices/12345678"
      }
    ]
  },
  "object": {
    "id": 1,
    "date": "2012-01-09",
    "user": {
      "id": 5538,
      "email": "john.test@test.com",
      "first_name": "John",
      "last_name": "Test",
      "profile_image_url": "https://api.nokotime.com/images/avatars/0000/0001/avatar.jpg",
      "url": "https://api.nokotime.com/v2/users/5538"
    },
    "billable": true,
    "minutes": 60,
    "description": "noko",
    "project": {
      "id": 37396,
      "name": "Gear GmbH",
      "billing_increment": 10,
      "enabled": true,
      "billable": true,
      "color": "#ff9898",
      "url": "https://api.nokotime.com/v2/projects/37396"
    },
    "tags": [
      {
        "id": 249397,
        "name": "noko",
        "billable": true,
        "formatted_name": "#noko",
        "url": "https://api.nokotime.com/v2/tags/249397"
      }
    ],
    "source_url": "http://someapp.com/special/url/",
    "invoiced_at": "2012-01-10T08:33:29Z",
    "invoice": {
      "id": 12345678,
      "reference": "AA001",
      "invoice_date": "2013-07-09",
      "state": "unpaid",
      "total_amount": 189.33,
      "url": "https://api.nokotime.com/v2/invoices/12345678"
    },
    "import": {
      "id": 8910,
      "url": "https://api.nokotime.com/v2/imports/8910"
    },
    "approved_at": "2012-01-10T08:33:29Z",
    "approved_by": {
      "id": 5538,
      "email": "john.test@test.com",
      "first_name": "John",
      "last_name": "Test",
      "profile_image_url": "https://api.nokotime.com/images/avatars/0000/0001/avatar.jpg",
      "url": "https://api.nokotime.com/v2/users/5538"
    },
    "url": "https://api.nokotime.com/v2/entries/1711626",
    "invoiced_outside_of_noko_url": "https://api.nokotime.com/v2/entries/1711626/marked_as_invoiced",
    "approved_url": "https://api.nokotime.com/v2/entries/1711626/approved",
    "unapproved_url": "https://api.nokotime.com/v2/entries/1711626/unapproved",
    "created_at": "2012-01-09T08:33:29Z",
    "updated_at": "2012-01-09T08:33:29Z"
  }
}

{
  "broadcast_receiver": {
    "id": 77885522,
    "name": "Da Best Timer Sync"
  },
  "sent_at": "2020-09-30T15:19:28Z",
  "last_event": "timer.running",
  "subject": {
    "type": "timer",
    "url": "https://api.nokotime.com/v2/projects/37396/timer"
  }
}

Why Broadcasts and webhooks are different

Using different names & rules (despite their similarities) makes the API clear and keeps an integration’s infrastructure reliable. A bug with Broadcasts will not break their webhooks, which are business critical.

It also helps demystify real-time updates for developers (which is something I’ve struggled with on my own projects). Webhooks are a rock-solid, reliable way for a developer to be notified of any events in Noko. Broadcasts allow them to add a bit of extra real-time sugar on top, without polling for data endlessly.

Try them out!

Designing Noko’s Broadcasts was an incredibly fun and satisfying challenge! It was a great reminder to always review the tools you know about when building a new piece of infrastructure. I’ve used each of these extensively, but combining them to create a single, unified protocol took some brainstorming and mental meandering.

If you’re working on adding realtime updates to your project, I’d seriously suggest trying out this Broadcasts protocol! I think you (and the developers using your project!) will be pleasantly surprised. 😄

And if you’re looking for fantastic, painless time-tracking, try out Noko! 😉