Offline-first sync

No signal in a basement, yet a payment must be recorded. An offline-first app saves each change on the phone first, and sends it once a network returns.

Assembling the machine…

Listing

  1. // app · delivery.ts
  2. async function collect(order, euros) {
  3. await db.transaction(async (tx) => {
  4. await tx.payments.add(order, euros);
  5. await tx.outbox.add({ id: uuid(),
  6. method: "POST", order, euros });
  7. });
  8. }
  9. // sync worker · runs once online
  10. async function sync() {
  11. const changes = await db.outbox.all();
  12. for (const c of changes) {
  13. const res = await api.send(c, {
  14. "Idempotency-Key": c.id,
  15. "If-Match": c.version }); // edits
  16. if (res.status === 412) pull(c);
  17. await db.outbox.delete(c.id);
  18. }
  19. }

Step 1 / 8

The list opens with no signal

No signal in the basement, yet the deliveries open at once: they are on the phone (local database).

Event log · 1
  1. 1.00 msREADtoday's deliveries → 12, from the phone · no network

The concept

No signal in a basement, yet a payment must be recorded. An offline-first app saves each change on the phone first, and sends it once a network returns.

Offline-first is a way to build an app so that it works without a network: it reads and writes its own copy of the data on the device, and syncs that copy with a server when a network is there. Android's guide to offline-first apps makes the local database the app's source of truth: screens read only from it, and the network only ever updates it.

A change made offline is written to the local database first and queued to be sent later, what the guide calls a lazy write. Here the queue is an outbox: a table in the same database, written in the same transaction as the change, so the two never disagree, and kept on disk, because the system can stop a background app's process at any time to reclaim memory. The system's scheduler, WorkManager on Android, runs the sync once a network is connected, even after the app was closed, and tries again with exponential backoff when it fails.

A network can drop after the server has applied a change but before its answer arrives. The phone cannot tell, so it sends the change again: a change may arrive more than once. A unique ID made on the phone and sent with each change, as in the IETF's Idempotency-Key draft, lets the server recognise a repeat and answer it without applying it twice.

While the phone is offline, someone else can change the same data on the server. Android's guide names last write wins as a common approach on mobile: each change carries the time it was made, and the server keeps the newer, silently discarding the other. Applying changes in the order they arrive is a different rule, and a worse one: a change made offline arrives late and erases newer work. Sending the version a change started from, with HTTP's If-Match, lets the server refuse a change made against an old version, so the conflict is seen and settled instead of lost.

Local-first software, as Kleppmann and his co-authors described it in 2019, goes further: the copy on the device is the primary one, and concurrent changes merge by themselves with CRDTs. This machine shows the simpler and more common form, where the server holds the copy every phone shares. The 60 ms each way and the time spent offline are modelled.

The run, step by step

Every scenario, in words.

What the machine plays, step by step. Beside a step, what a what-if switch plays in its place, or adds right after it.

Collect cash with no signal

8 steps
  1. Play

    The list opens with no signal

    The courier opens today's deliveries. The app reads them from the phone's own database, filled at the depot this morning, and never waits for the network: every screen reads that database, with or without a signal.

  2. Play

    Order 42 paid, saved on the phone

    The courier records order 42 as paid, 30 € in cash. In one transaction the app writes the payment to the phone's database and adds it to the outbox, the changes the server has not confirmed yet, with an ID made on the phone: 7f3a. Both are saved or neither, and the screen shows paid at once.

    • With Keep the outbox in memory

      Play

      Order 42 paid, the note in memory

      The courier records order 42 as paid, 30 € in cash. The app writes the payment to the phone's database, but the change to send goes into a list held in the app's memory, not on disk. The screen shows paid at once; nothing has failed yet.

  3. Play

    No network: the sync waits

    The app asks the phone's system to run the sync worker once a network is connected: WorkManager, on Android. Until then nothing is sent and nothing fails. The courier walks on to the next delivery.

  4. Play

    The phone closes the app

    In the background, the system stops the app's process to free memory, as phones do. Nothing is lost: the payment and its outbox change are both in the phone's database, on disk, and the scheduled sync does not need the app open.

    • With Keep the outbox in memory

      Play

      The phone closes the app, memory and all

      The system stops the app's process to free memory. The payment is still on disk, but the queued change lived in the app's memory and went with it. Nothing on the phone remembers that the server was never told.

  5. Play

    Signal back: the sync starts

    Twenty minutes later there is a network. The system starts the sync worker by itself, the app still closed, and the worker reads the outbox: one change, ID 7f3a.

    • With Keep the outbox in memory

      Play

      Signal back: nothing to send

      Twenty minutes later the system starts the sync worker in a fresh process. Its list is empty. The phone says order 42 is paid, the server says it is not, and nothing will ever send the payment.

  6. Play

    Send it, with its ID

    The worker sends POST /payments with the change, and its ID in an Idempotency-Key header. It sends one change at a time, in the order they were made.

    • With Retry without a change ID

      Play

      Send it, without its ID

      The worker sends POST /payments with the change, but not its ID. Nothing goes wrong this time: the answer will arrive.

  7. Play

    The server records 30 €

    The API has never seen change 7f3a. It records the payment in the server database and keeps the ID with the answer it gives.

    • With Retry without a change ID

      Play

      The server records 30 €

      The API records the payment in the server database. With no ID, nothing tells this payment from another of the same amount.

  8. Play

    Confirmed: the note is crossed out

    201 Created comes back. Only then does the worker delete the change from the outbox. Phone and server now both say order 42 is paid, 30 €.

Lose the answer

6 steps
  1. Play

    Send 30 €, with its ID

    The courier collected 30 € for order 42 offline; now there is a network. The worker sends POST /payments with the change and its ID, 7f3a, in an Idempotency-Key header.

    • With Retry without a change ID

      Play

      Send 30 €, without its ID

      The courier collected 30 € for order 42 offline; now there is a network. The worker sends POST /payments with the change, but not its ID.

  2. Play

    The server records it

    The API has never seen change 7f3a: it records the payment, keeps the ID with its answer, and answers 201 Created.

    • With Retry without a change ID

      Play

      The server records it

      The API records the payment in the server database and answers 201 Created.

  3. Play

    The answer is lost in a tunnel

    The van enters a tunnel and the 201 is lost on its way back. The worker cannot tell a request that never arrived from an answer that never came back, so it keeps the change and asks the system to run it again later, waiting longer after each failure (backoff).

  4. Play

    Send it again

    Thirty seconds later the worker sends the change again, as it must: it never got an answer. The ID is the same, 7f3a.

    • With Retry without a change ID

      Play

      Send it again

      Thirty seconds later the worker sends the change again, as it must: it never got an answer. Without its ID, the request looks like a new payment.

  5. Play

    Same ID: already done

    Change 7f3a is already recorded. The API writes nothing and answers as it did the first time: 201, with the same payment. The customer paid 30 € once, and the books say 30 €.

    • With Retry without a change ID

      Play

      Recorded twice: 60 €

      The API sees a new payment and inserts a second row. The customer handed over 30 €; the books say 60 €, and tonight's cash count will come up 30 € short.

  6. Play

    The OK arrives

    201 Created comes back, the same answer as the first time, and the worker deletes the change from the outbox. The payment was sent twice and recorded once.

    • With Retry without a change ID

      Play

      The OK arrives

      201 Created comes back for the second request, and the worker deletes the change from the outbox. Nothing on the phone or in its log says the server holds two payments.

Change it on both sides

5 steps
  1. Play

    Nobody home: try tomorrow

    At 10:02 nobody answers the door for order 43. The courier, still without a signal, moves the delivery to tomorrow. The phone saves it and queues the change, noting the version of the order it started from: version 2.

  2. Play

    The office moves it to 18:00

    At 10:05 the customer phones: they will be home tonight, at 18:00. Staff change the slot through the API, and order 43 goes from version 2 to version 3. The courier's phone, still offline, knows nothing of it.

  3. Play

    Signal back: the phone sends its change

    At 10:22 the worker sends PATCH /orders/43 with the new slot and If-Match: "v2", the version the courier saw: change it only if it is still that one.

    • With Let the last change to arrive win

      Play

      Signal back: the phone sends its change

      At 10:22 the worker sends PATCH /orders/43 with the new slot, and nothing about the version the courier saw.

  4. Play

    Version 3, not 2: refused

    Order 43 changed after the courier saw it. The API writes nothing and answers 412 Precondition Failed: the office's 18:00 slot stays.

    • With Let the last change to arrive win

      Play

      The last to arrive wins

      With no version to check, the API applies the change. It was made at 10:02, before the office's, but arrived at 10:22, after it: order 43 goes back to tomorrow, version 4.

  5. Play

    The phone takes version 3

    The worker fetches order 43, writes version 3 into the phone's database and drops its own change. The app tells the courier what changed and who changed it. Here the app keeps the office's slot; another app could ask the courier to choose. Either way, the conflict is seen.

    • With Let the last change to arrive win

      Play

      The customer waits at 18:00

      The phone gets 200 OK and deletes the change. The office's screen now says tomorrow, with no trace of the call. The customer stays in at 18:00; the parcel comes the next day.