Email API benchmarks

All metrics reflect a trailing 90-day window. Point-in-time load tests don't capture the variability of a production system handling thousands to millions of requests daily, so we use a longer window to surface the effects of system congestion and public incidents without penalizing providers that are actively improving.

Volume represents the total number of sampled email API requests routed through Knock during the period. Because Knock is an orchestration layer, we cannot influence the sending patterns of our customers, so volume varies by provider. To normalize for this, we group providers into volume bands (e.g. <1M, 1M–10M, 10M–25M) that indicate how statistically confident we are in the observed metrics. Providers with low volume are flagged so you can weigh the results accordingly.

Response time is measured as the elapsed time from when Knock sends an API request to when the provider returns a 200 status indicating message acceptance (a "sent" event in Knock). We compute percentiles (p50, p90, p95, p99) using the quantile function in ClickHouse, which uses an approximate algorithm with up to 2% error tolerance. In practice, this produces values within 0–3ms of exact percentiles for typical latency distributions, well within the margin of meaningful difference when comparing providers. The measurement includes HTTP connection pooling time and network latency, applied consistently across all providers. Each API request is measured individually — if Knock's automatic retry logic triggers a new request, that retry is recorded as a separate measurement with its own response time rather than being added to the original request's duration. This measures API acceptance latency, not end-to-end delivery to the recipient's inbox.

Error rate is the ratio of requests that received a 5xx status code (500, 502, 503, etc.) to total requests for the period. We treat all 5xx responses equally — whether a 503 Service Unavailable indicating an overloaded server or a generic 500 Internal Server Error — because they either indicate provider-side issues or represent cases where input could be rejected with a more instructive 4xx code. Within Knock's retry logic, most 5xx codes are retryable, so a single request can produce multiple error entries if each retry also fails. We believe this accurately represents the provider's availability during those periods.

In Knock, a connection to a provider API is modeled as a channel. Growth rank (1st through 10th) reflects how many new channels were created for each provider during the 90-day window. To detect directional trends, we split the window into two 45-day halves and compare them, producing the trend indicators (up, down, or stable) shown alongside each rank.

On provider detail pages, we integrate with each provider's public status page via RSS or API and overlay reported incidents on the performance data we collect. Our view is that a severe degradation in measured performance should correspond to a status page entry, and the degree of correlation between observed metrics and public incidents is a useful signal for how a provider handles incident management. Incidents are not inherently negative — status page updates paired with observed degradation are indicators of engineering discipline.

When modeling pricing curves and cost-per-email estimates, we select the lowest-cost plan that covers a given sending volume. This means we optimize purely for volume — we do not factor in premium features like dedicated IPs, advanced analytics, or priority support that may only be available on higher-tier plans. If a provider offers multiple plans at the same volume level, we use the one with the lower base price. Overage rates are applied on top of the included email allowance for each plan. The result is a floor estimate of what each provider costs at a given volume, making it straightforward to compare providers on price alone.