Client Specifications

Foreword

For years, the sole and unique client was Gecko (the platform behind Firefox, Thunderbird..). In order to reach out new platforms and products, a fully-featured Rust client was built, using the application-services components (Viaduct, NSS, …). Despite our efforts to provide this fully featured, cross-platform client, new clients still emerged that obliged us to admit that our idea of having a single client of reference is dead. Instead, we are going to provide specifications for Remote Settings clients, to at least mitigate the consequences of clients fragmentation.

That being said, we still strongly discourage the implementation of new ad-hoc clients.

We distinguish two major use-cases:

• authenticated write operations, ie. publish data;

• anonymous read operations, ie. fetch data from within our products.

Since the former does not take place on clients, it matters less than the latter, which has a major impact on traffic and our servers load.

Before launching your own implementation, please keep in consideration that:

• client fragmentation hinders our cohesive view of the service (from server to clients)

• heterogenous implementations slow down the roll out of API changes

• having multiple code bases for one service raises the cost of code review, maintenance, support, feature parity, security assurance

• the Remote Settings team has to be consulted to validate the implementation

Existing Clients

As of April 2024:

	Gecko	Rust Client	application-services/remote-settings	kinto-http.py	kinto.js
Write Operations	❌	✅	❌	✅	✅
Add/Remove Attachments	❌	✅	❌	❌	✅
Approve/Reject reviews	❌	✅	❌	❌	❌
Changesets Endpoints	✅	✅	❌	✅	❌
Cache Busting	✅	✅	❌	✅	❌
Env Switching	✅	~ Partial	✅	❌	❌
Signature Verification	✅	✅ Optional	❌	❌	❌
Local State	✅	✅ Optional	❌	❌	✅
Fetch Attachments	✅	✅	✅ Without integrity check	❌	✅
Backoff	✅	✅	✅	✅	✅
Deprecation	❌	❌	❌	✅ Minimal	✅

Specifications

Remote Settings is a layer on top of the Kinto API. Although every read-only operation offered by the Kinto API is available on our Remote Settings server, clients must restrict the amount of distinct interactions. Millions of devices sending arbitrary requests could have a significant impact on infrastructure.

That’s why clients developers MUST keep their implementation as close as possible to the existing ones, or at least get in touch with us if there is a solid reason to derive from it.

Endpoints

Clients MUST set their User-Agent request header, mentioning application name and version.

Clients SHOULD leverage Gzip transport using the Accept-Encoding: gzip request header.

The following two endpoints MUST be used to retrieve data. Clients MUST NOT use other endpoints.

Fetch collection:

GET /v1/buckets/{bid}/collections/{cid}/changeset?_expected={timestamp}.

Returns the following response for the collection {cid} in the bucket {bid} (likely main):

• changes: list of records, optionally filtered with ?_since="{timestamp}"

• metadata: collection attributes

• timestamp: records timestamp

Note

The _expected={} querystring parameter is mandatory. Either you pass the current collection timestamp value obtained when polling for changes in order to bust the CDN cache, or you use a hard-coded value (eg. 0) and rely on the cache TTL. See section below about cache busting.

Examples:

• get_changeset() in remote-settings-client

• fetch_changeset() in Gecko

Clients SHOULD NOT rely on arbitrary server side filtering. In Remote Settings, collections are quite small anyway, and can usually be fetched entirely to be filtered on the client side. This helps us reduce our CDN cache cardinality.

Client MAY filter the list of changes to only obtain the changes since the last polling, using the ?_since= querystring parameter. The value is a timestamp obtained from a previous changeset response.

For each collection, the amount of possible values for the timestamps is finite. Indeed, each timestamp value corresponds to the date and time of the data publication (reviewer approving changes on the server). Some collections change several times a day, but that still allows a lot of caching. The push notifications are debounced too, in case several users approve changes in a short amount of time.

Poll for changes:

GET /v1/buckets/monitor/collections/changes/changeset?_expected={timestamp}.

Returns the list of collections and their current timestamp.

• changes: list of collections and their timestamp, optionally filtered with ?_since="{timestamp}"

• timestamp: highest collections timestamp

Note

The _expected={} querystring parameter is mandatory. Either you receive a Push notification from the server, and pass the timestamp value in order to bust the CDN cache, or you use a hard-coded value (eg. 0) and rely on the cache TTL. See section below about cache busting.

{
  "metadata": {},
  "timestamp": 1713532462683,
  "changes": [
    {
      "id": "19e79f22-62cf-92e1-c12c-a3b4b9cf51be",
      "last_modified": 1603126502200,
      "bucket": "blocklists",
      "collection": "plugins",
      "host": "firefox.settings.services.mozilla.com"
    },
    {
      "id": "b7f595f9-5fc5-d863-b5dd-e5425dcf427a",
      "last_modified": 1604940558744,
      "bucket": "blocklists",
      "collection": "addons",
      "host": "firefox.settings.services.mozilla.com"
    }
  ]
}

Examples:

• get_latest_change_timestamp() in remote-settings-client

• fetchLatestChanges() in Gecko

Cache Busting

Using push notifications

With push notification, we want the first requests to bust the CDN cache of the polling endpoint with the received value.

• The push notification payload contains the highest of all collections

• This timestamp is passed to the ?_expected={} querystring param when polling for changes

• The polling endpoint will return the list of collections with their respective timestamps (last_modified field):

• Each collection can now be fetched using the timestamp obtained from the polling endpoint (eg. using the above example: /buckets/blocklists/plugins/changeset?_expected=1603126502200)

Without push notifications (cached polling)

Without push notification, we use hard-coded value (?_expected=0) and rely on the cache TTL of the polling endpoint. And use the timestamps obtained in the polling endpoint response as described above with push notifications.

Without push notifications nor polling for changes (cached fetching)

With this approach, we skip the step that poll for changes, and rely on the cache TTL for the collection data.

Note

As the service owners, we don’t guarantee that we will keep the collection TTL under X hours.

Environment Switching

Clients MAY offer a convenient way to switch between DEV, STAGE, or PROD environments, in order to facilitate the work of QA teams.

Clients SHOULD use PROD by default. And for security reasons, there must be some protection in place to prevent users to switch environments.

Signature Verification

Clients SHOULD verify the integrity of the downloaded data.

Note

Although Gecko on desktop is not exposed to the same risks as on mobile where applications and data are jailed, verifying signatures is a keystone in the chain of trust for data. It is the only way to guarantee the authenticity (and/or integrity) of the data.

Signature validation steps are:

• Download the certificates chain provided in metadata

• Verify the certificates chain: each certificate must be valid, and the SHA-256 root hash of the root certificate should match one of the hardcoded values at build time.

• Serialize the downloaded data using Canonical JSON

• Verification that the signature provided in metadata matches the one computed on downloaded data

Examples:

• In Rust from scratch using Ring

• In Python, using cryptography

Clients embedded in products SHOULD use NSS (true in ~2023), and its high level API for signature verification.

Examples:

• _validateCollectionSignature() in Gecko client

• Verification Trait in Remote Settings client

Local State

Clients MAY have a local state and copy of the data, in order to limit the amount of data to fetch from the server.

The local state SHOULD contain the timestamp of the last successful fetch, to be provided in the ?_since= filter on the next call. The deleted records are then returned in the form of tombstones ({"id": "xyz", "deleted": true}), which MUST be removed from local copy. Created and updated records are returned in the same form and MUST be upserted in local copy.

Examples:

• importChanges() in Gecko

• merge_changes() in Remote Settings client

Attachments

The attachments base URL is obtained on the root URL of the server:

GET /v1/

Returns the metadata of the server.

• capabilities.attachments.base_url: the base URL for attachments with a trailing /

Records with an attachment have the necessary metadata to download and verify it.

• attachment.location: path to the attachment, to be concatenated with the base_url

• attachment.hash: SHA-256 of the file

• attachment.size: size of the file in bytes

Clients SHOULD verify the size and hash of their downloaded copy in order to implement our security model and guarantee integrity and authenticity of CDN content.

Examples:

• fetch_attachment() in remote-settings-client

• fetchAttachment() in Gecko

Backoff Headers

As owners of the backend, we want to be able to tell clients to gently delay their hits on the server.

Client MUST honour the wait interval in seconds set in the Backoff response headers.

Examples:

• ensure_no_backoff() in application-services/remote-settings

• pollChanges() in Gecko

Deprecation Headers

Client SHOULD react on deprecation headers. Ideally make it visible to the final users that the version of their product is relying on a service that is going away.

When enabled, the server sends a Alert header with a JSON serialized value, that contains extra-information (eg. message, url).

Examples:

• _checkForDeprecationHeader() in kinto.js

Documentation:

• API Docs