Parcel Post

The verifier. THIS is the file to copy — constant-time comparison, the raw bytes rather than the parsed body, and the replay window. It knows nothing about parcels.

"""Verify a SipsPOS webhook delivery. THIS is the file to copy.

It knows nothing about parcels, or about what you do with an event. Its whole
job is to answer one question honestly: *did SipsPOS send this exact body, and
recently?* Everything else in this example is an application built on top of
that answer.

Standard library only. Drop it beside your own code and call `verify()` from
whatever framework you run.

    from sipspos_webhook import verify, Rejected

    body = request.get_data()                  # THE RAW BYTES. See below.
    try:
        verify(SECRET, request.headers["X-SipsPOS-Signature"], body)
    except Rejected as e:
        return "", 401 if e.reason != "malformed" else 400
    event = json.loads(body)                   # only now is it safe to parse

THE FIVE THINGS PEOPLE GET WRONG, and what this file does about each:

1. COMPARING WITH `==`. String comparison returns early on the first differing
   byte, so how long it takes leaks how much of a guess was right. Given
   enough attempts that is a forgery. `hmac.compare_digest` takes the same
   time either way.

2. VERIFYING THE PARSED BODY. The signature covers the bytes that arrived. If
   you `json.loads` and then re-serialise to check, you are checking a
   DIFFERENT string — key order, spacing and unicode escaping are all free to
   change — and you will either reject everything or, worse, write a
   "normalise first" step that an attacker can aim at. This function takes
   `bytes` and never decodes them.

3. IGNORING `t`. The timestamp is inside the signed string precisely so a
   captured delivery stops being usable. Verify the HMAC and skip the clock
   and every delivery you ever received is replayable against you forever.

4. ASSUMING EXACTLY-ONCE. It is at-least-once — see `parcels.py`. Verification
   is not where that is solved, but it is where people assume it away.

5. DOING THE WORK BEFORE ANSWERING. Also not this file's job, also assumed
   away here first. See `parcels.py`.
"""
import hashlib
import hmac
import time

#: How old a delivery may be, in seconds. Stripe's default, and this is
#: Stripe's scheme.
#:
#: A FIVE-MINUTE WINDOW DOES NOT DROP RETRIES, and this is the part worth
#: reading twice. SipsPOS retries on a 1m / 5m / 30m / 2h / 12h ladder, so the
#: obvious conclusion is that a delivery could arrive twelve hours late and a
#: 300-second tolerance would throw it away. It will not: **every attempt is
#: signed afresh with the current time**, so the twelve-hour retry carries a
#: `t` from twelve hours later, not from when the event happened.
#:
#: This matters because the alternative — widening the window to a day so
#: retries "work" — is exactly what makes the timestamp worthless. If you find
#: yourself needing a large tolerance, the problem is a clock, not the ladder.
DEFAULT_TOLERANCE = 300


class Rejected(Exception):
    """A delivery that must not be acted on.

    `reason` is one of:

      `malformed`      the header is not the shape this scheme defines. A bug
                       or a probe, not a forgery attempt worth alarming about.
      `bad_signature`  the HMAC does not match. Somebody sent you a body
                       SipsPOS did not sign.
      `stale`          the HMAC matches but `t` is outside the tolerance — a
                       genuine SipsPOS delivery being replayed at you, or a
                       clock that has drifted far enough to look like one.

    Kept apart because they mean different things operationally: a burst of
    `bad_signature` is worth an alert, a burst of `stale` is usually NTP.
    """

    def __init__(self, reason, message):
        super().__init__(message)
        self.reason = reason


def _parse(header):
    """`t=<int>,v1=<hex>` -> `(t, v1)`. Order-independent, extra parts
    ignored — a `v2=` may be added one day and must not break a v1 consumer."""
    if not header:
        raise Rejected("malformed", "no X-SipsPOS-Signature header")
    parts = {}
    for chunk in header.split(","):
        key, _, value = chunk.partition("=")
        if value:
            parts.setdefault(key.strip(), value.strip())
    try:
        timestamp = int(parts["t"])
    except (KeyError, ValueError):
        raise Rejected("malformed", f"no usable t= in {header!r}")
    if "v1" not in parts:
        raise Rejected("malformed", f"no v1= in {header!r}")
    return timestamp, parts["v1"]


def expected(secret, timestamp, body):
    """The v1 digest for this body at this timestamp.

    Exposed because a receiver that cannot verify is worth debugging, and the
    first question is always "what did you expect instead". Do NOT use it to
    compare by hand — see `verify`.
    """
    if isinstance(body, str):                     # be forgiving at the door,
        body = body.encode()                      # strict everywhere after it
    signed = str(timestamp).encode() + b"." + body
    return hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()


def verify(secret, signature_header, body, *, now=None,
           tolerance=DEFAULT_TOLERANCE):
    """Return the signed timestamp, or raise `Rejected`.

    `body` MUST be the raw request bytes, exactly as they arrived. Not a dict,
    not a re-serialised string — see the module docstring.

    `now` is a parameter rather than a call to the clock so that the staleness
    behaviour can be tested without waiting five minutes for it.

    The order matters: shape, then signature, then age. Checking the clock
    first would tell an attacker whether a timestamp was acceptable without
    them having to sign anything.
    """
    timestamp, sent = _parse(signature_header)

    if not hmac.compare_digest(expected(secret, timestamp, body), sent):
        raise Rejected("bad_signature",
                       "signature does not match the body — this was not "
                       "signed by the secret you configured")

    now = int(time.time() if now is None else now)
    if abs(now - timestamp) > tolerance:
        raise Rejected(
            "stale",
            f"delivery is {now - timestamp}s old (tolerance {tolerance}s). "
            f"Every SipsPOS retry is re-signed with the current time, so a "
            f"late retry is NOT why this happens — check your clock.")
    return timestamp

What this application does with a verified event: the log, the dedupe on the delivery id, and answering before doing the work.

"""Parcel Post — a delivery log for the pack bench.

What this application is: it listens for the fulfillment and order events and
keeps an append-only record of what happened to each parcel, so the bench has
a list that is current the moment something changes rather than on the next
sweep. `services/webhooks.py` names exactly this case as the reason webhooks
exist — a 3PL polling `GET /v1/fulfillments` learns about a parcel on its next
poll, not when it appears.

Everything about VERIFYING a delivery lives in `sipspos_webhook.py`, which is
the file to copy. This file is what one particular application does with an
event once it is known to be genuine.

    export SIPSPOS_WEBHOOK_SECRET=whsec_…
    python parcels.py                       # listens on :8400

TWO PROPERTIES THAT ARE NOT ABOUT CRYPTOGRAPHY, and are the ones that bite in
production:

DELIVERY IS AT-LEAST-ONCE. `WebhookDelivery` is one row per (event, endpoint)
with an attempt counter, and anything that is not a 2xx — including a response
that was merely slow — is rescheduled. So you WILL see the same delivery
twice. `X-SipsPOS-Delivery` is the idempotency key, and here it is the PRIMARY
KEY of the log: a repeat cannot be double-recorded because the database will
not have it, rather than because a branch remembered to check. A receiver that
treats delivery as exactly-once ships the parcel twice.

ANSWER FIRST, WORK AFTER. The sender applies a timeout and treats a non-2xx as
a failure to retry, escalating to disabling the endpoint after 1m/5m/30m/2h/12h
of them. A handler that does its work inline is one slow query away from being
retried, and a few of those from being switched off. So this one records the
fact and answers; the work happens off the response path.
"""
import json
import os
import sqlite3
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer

from sipspos_webhook import Rejected, verify

#: The events this receiver acts on. Every one of these is in
#: `services.webhooks.EVENT_TYPES` — `scripts/parcel_post_test.py` asserts it,
#: because a receiver subscribed to an event the platform does not send waits
#: forever and looks, from the inside, exactly like one that works.
HANDLED = {
    "fulfillment.created": "waiting to pack",
    "fulfillment.shipped": "shipped",
    "order.paid": "paid",
    "order.fulfilled": "fulfilled",
    # The one worth having a bench see. A refund AFTER the parcel left is the
    # case where somebody has to go and stop it, and it is invisible on a
    # pick list that only ever grows.
    "order.refunded": "REFUNDED — check before it leaves",
}

DEFAULT_PORT = 8400


class Store:
    """The log. SQLite because it is in the standard library and because the
    dedupe wants a real UNIQUE constraint, not a dictionary that dies with the
    process."""

    def __init__(self, path=":memory:"):
        # `check_same_thread=False` because the work happens on a worker
        # thread (see `handle`); every write below is a single statement in
        # its own transaction, which SQLite serialises for us.
        self.db = sqlite3.connect(path, check_same_thread=False)
        self.db.execute("""
            CREATE TABLE IF NOT EXISTS parcel_log (
                delivery_id TEXT PRIMARY KEY,
                event       TEXT NOT NULL,
                status      TEXT NOT NULL,
                subject     TEXT,
                received_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
            )""")
        self.db.commit()

    def record(self, delivery_id, event, subject):
        """True if this is new, False if it is a redelivery.

        The INSERT is what decides, not a preceding SELECT: two copies of the
        same retry arriving together would both pass a check-then-insert, and
        this is a receiver whose entire job is to be safe under retry.
        """
        try:
            self.db.execute(
                "INSERT INTO parcel_log (delivery_id, event, status, subject)"
                " VALUES (?, ?, ?, ?)",
                (delivery_id, event, HANDLED.get(event, event), subject))
            self.db.commit()
            return True
        except sqlite3.IntegrityError:
            return False

    def queue(self):
        """Everything logged, newest first — what the bench reads."""
        return self.db.execute(
            "SELECT delivery_id, event, status, subject, received_at"
            "  FROM parcel_log ORDER BY received_at DESC, rowid DESC"
        ).fetchall()

    def count(self):
        return self.db.execute("SELECT COUNT(*) FROM parcel_log").fetchone()[0]


def subject_of(payload):
    """A human label for the thing the event is about.

    Deliberately forgiving: an event's `data` is whatever the serializer sends,
    and a receiver that raises on an unexpected shape turns a new field into
    an outage. Unknown shapes get a blank label and are still logged — the
    delivery id is what matters.
    """
    data = (payload or {}).get("data") or {}
    for key in ("public_id", "id", "order_number", "tracking_number"):
        if data.get(key):
            return str(data[key])
    return ""


def handle(store, event, delivery_id, payload):
    """The application logic, with no HTTP in it. `"recorded"` or
    `"duplicate"`."""
    if event not in HANDLED:
        # Not an error: an endpoint may be subscribed to more than this
        # receiver cares about, and refusing would earn a retry forever.
        return "ignored"
    return "recorded" if store.record(delivery_id, event, subject_of(payload)) \
        else "duplicate"


def make_handler(secret, store, on_work=None):
    """A `BaseHTTPRequestHandler` for `POST /webhooks`.

    `on_work` is the seam the test uses to run the work synchronously; in
    production it is a thread (see below).
    """

    class Handler(BaseHTTPRequestHandler):
        protocol_version = "HTTP/1.1"

        def log_message(self, fmt, *args):        # quiet by default
            pass

        def _reply(self, code, payload):
            body = json.dumps(payload).encode()
            self.send_response(code)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)

        def do_POST(self):                        # noqa: N802 (stdlib name)
            if self.path.rstrip("/") != "/webhooks":
                return self._reply(404, {"error": "not found"})

            length = int(self.headers.get("Content-Length") or 0)
            body = self.rfile.read(length)        # RAW BYTES, kept as bytes

            # VERIFY BEFORE PARSING. Not a style preference: until this
            # returns, the body is a string a stranger sent us, and
            # `json.loads` on it is the first place we would have trusted it.
            try:
                verify(secret, self.headers.get("X-SipsPOS-Signature"), body)
            except Rejected as e:
                code = 400 if e.reason == "malformed" else 401
                return self._reply(code, {"error": e.reason})

            try:
                payload = json.loads(body)
            except ValueError:
                # Signed by us and still not JSON: our bug, not theirs. 400
                # rather than 500 so it is not retried into a loop.
                return self._reply(400, {"error": "not json"})

            event = self.headers.get("X-SipsPOS-Event") or payload.get("event")
            delivery_id = self.headers.get("X-SipsPOS-Delivery")
            if not delivery_id:
                return self._reply(400, {"error": "no delivery id"})

            # ANSWER FIRST. The record itself is one INSERT and is done here so
            # the reply can honestly say what happened; anything SLOW belongs
            # in `on_work`. A real deployment puts that on a queue — the
            # thread below is the smallest thing that shows the shape without
            # dragging a broker into an example.
            result = handle(store, event, delivery_id, payload)
            self._reply(200, {"ok": True, "result": result})

            if result == "recorded" and on_work is not None:
                on_work(event, delivery_id, payload)

    return Handler


def serve(secret, store, port=DEFAULT_PORT):
    def work(event, delivery_id, payload):
        threading.Thread(target=lambda: None, daemon=True).start()

    server = HTTPServer(("127.0.0.1", port), make_handler(secret, store, work))
    print(f"parcel-post: listening on http://127.0.0.1:{port}/webhooks")
    print(f"parcel-post: logging {len(HANDLED)} event types")
    server.serve_forever()


def main(argv=None):
    secret = os.environ.get("SIPSPOS_WEBHOOK_SECRET")
    if not secret:
        print("parcel-post: SIPSPOS_WEBHOOK_SECRET is not set. The winery's "
              "admin shows it beside the endpoint in Admin -> Setup -> "
              "Webhooks.")
        return 2
    serve(secret, Store(os.environ.get("PARCEL_POST_DB", "parcel-post.db")))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The five things receivers get wrong, and why a five-minute replay window does not drop a twelve-hour retry.

# Parcel Post — a SipsPOS webhook receiver

A delivery log for the pack bench. It listens for the fulfillment and order
events and keeps a record of what happened to each parcel, so the bench sees a
change when it happens rather than on the next poll.

It is also the reference **receiver**: the half of the integration that arrives
uninvited. If you only take one file, take `sipspos_webhook.py`.

```
sipspos_webhook.py   verify a delivery. THIS is the file to copy.
parcels.py           what this application does with a verified event.
```

Standard library only — `http.server`, `hmac`, `sqlite3`. No framework,
because yours will be different.

## Run it

1. The winery adds your URL in **Admin → Setup → Webhooks** and picks the
   events. They are shown a **signing secret** — that is what you need. It is
   not an API key: it proves a body came from SipsPOS and grants nothing.
2. Then:

```sh
export SIPSPOS_WEBHOOK_SECRET=whsec_…
python parcels.py                      # listens on 127.0.0.1:8400/webhooks
```

To receive real deliveries while developing, put a tunnel in front of it
(`cloudflared tunnel --url http://localhost:8400`, `ngrok http 8400`) and give
the winery the tunnel's URL. Point it at their **sandbox** first — a test
winery emits the same events with invented parcels.

## The five things receivers get wrong

Each one passes a casual test and fails in production. This is the whole
reason the example exists.

**1. Comparing with `==`.** String comparison returns on the first differing
byte, so how long it takes leaks how much of a guess was right. Use
`hmac.compare_digest`.

**2. Verifying the parsed body.** The signature covers *the bytes that
arrived*. If you `json.loads` and re-serialise to check, you are checking a
different string — key order, spacing and unicode escaping are all free to
change. Keep the raw bytes; verify those; parse afterwards.

**3. Ignoring the timestamp.** `t` is inside the signed string precisely so a
captured delivery stops being usable. Verify the HMAC and skip the clock, and
every delivery you have ever received is replayable against you forever.

**4. Assuming exactly-once.** It is at-least-once. `X-SipsPOS-Delivery` is the
idempotency key. Here it is the *primary key* of the log, so a repeat is
refused by the database rather than by a branch somebody might forget. A
receiver that treats delivery as exactly-once ships the parcel twice.

**5. Doing the work before answering.** The sender applies a timeout and
retries anything that is not a 2xx, escalating to **disabling your endpoint**
after 1m / 5m / 30m / 2h / 12h of failures. Record the fact, answer, then work.

## The bit people get wrong about the replay window

**A 300-second tolerance does not drop retries.** Reading the 1m/5m/30m/2h/12h
ladder, the obvious conclusion is that a twelve-hour retry would fall outside
any sane window, so the window has to be a day — at which point the timestamp
stops protecting anything.

It does not, because **every attempt is signed afresh with the current time**.
The twelve-hour retry carries a `t` from twelve hours later, not from when the
event happened. If you find yourself needing a large tolerance, the problem is
a clock rather than the ladder.

## What it answers

| Situation | Status |
|---|---|
| Verified, recorded | `200` |
| Verified, already seen | `200` — and nothing is written twice |
| Verified, an event it does not handle | `200` — a refusal would be retried forever |
| Signature header malformed | `400` |
| Signature wrong, or `t` outside the window | `401` |

Anything that is not a 2xx will be retried, so answer 2xx to everything you
have safely absorbed — including duplicates and events you ignore.

## What this example does not do

**It does not re-GET the changed resource.** `customer.updated` means "a
watched field changed", and the honest response is to re-read the customer
rather than treat the payload as a diff. That needs an API key as well as a
signing secret, which is the *other* half of the integration — see the Club
Pulse example for the client. Parcel Post consumes the payload it is given.

**The work is a thread, not a queue.** "Answer first, work after" is shown with
the smallest thing that has the right shape. A real deployment puts the work on
a queue; a broker in an example would swamp the thing being demonstrated.

**No TLS and no hosting story.** It listens on localhost. Put a tunnel or a
reverse proxy in front of it.

## Tested against the real sender

`scripts/parcel_post_test.py` signs with `services/webhooks.py`'s own
`signature()` — the function production uses — and drives a real
`WebhookEndpoint` through `deliver_due()` into this handler. An example that
verified its own fake signatures would only prove it agrees with itself.

That is not theoretical. SipsPOS shipped a bug where deliveries were signed
with a timestamp hours in the past; a receiver applying the tolerance
documented above would have rejected every one of them. This example, run
against the old code, answers `401` and the delivery goes unacknowledged —
which is exactly how it was caught.