Designing Idempotent APIs: Retries Without Duplicate Side Effects

Every backend eventually gets the same bug report: “the customer was charged twice”, “the order was created twice”, “the email went out three times”. Usually nothing is wrong with the business logic. A client retried a request that had actually succeeded, and the server treated the retry as new work. This post covers how to design POST endpoints so they’re safe to retry: idempotency keys, a deduplication store, the right status codes, and an ASP.NET Core implementation you can adapt.

Why retries happen

Retries aren’t an edge case. They’re how distributed systems stay reliable. A few common sources:

• Timeouts. The client gives up after N seconds, but the server keeps going and commits the work. From the client’s point of view the request failed; from the database’s point of view it succeeded.
• Resilience policies. HTTP clients configured with automatic retries (Polly, Microsoft.Extensions.Http.Resilience, service meshes, API gateways) will resend a request on transient errors, sometimes without your code knowing.
• At-least-once delivery. Most message brokers guarantee a message is delivered at least once, not exactly once. Consumers must expect duplicates.
• Humans. Users double-click “Pay”, refresh a page that resubmits a form, or tap again on a slow mobile connection.

You can’t stop retries from happening, so the server has to make them harmless.

Idempotency in HTTP: which methods are idempotent by definition

HTTP semantics (RFC 9110) define a method as idempotent if sending the same request several times has the same intended effect on the server as sending it once. By that definition PUT, DELETE and the safe methods (GET, HEAD, OPTIONS, TRACE) are idempotent. POST and PATCH are not.

Two practical notes:

• Idempotent is a contract, not an automatic property. A PUT handler that appends to a list or sends an email on every call isn’t idempotent, whatever the RFC says. Design the handler to honour the contract.
• Idempotency is about side effects, not identical responses. A second DELETE can return 404 instead of 204 and still be idempotent, because the resource is gone either way.

Where you can, model operations so they’re naturally idempotent, for example PUT /orders/{clientGeneratedId}. For operations that really are “create something new” or “do this action”, you need an explicit mechanism. That’s what idempotency keys are for.

The idempotency-key pattern: request flow end to end

The client generates a unique key (a UUID is fine) for each logical operation and sends it in a header. The same key is reused on every retry of that operation. The IETF HTTP APIs working group drafted a standard header for this, Idempotency-Key (draft-ietf-httpapi-idempotency-key-header). The draft has since expired without becoming an RFC, but its conventions are widely used and a sensible default.

POST /payments HTTP/1.1
Content-Type: application/json
Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"

{ "amount": 2500, "currency": "USD", "recipientId": "r_123" }

On the server, each request with a key falls into one of four cases:

1. New key: process the request normally and store the key, a fingerprint of the request and the response.
2. Seen key, same fingerprint, finished: don’t execute again. Return the stored response.
3. Seen key, still in progress: another attempt is running. The draft recommends 409 Conflict, and the client can retry later.
4. Seen key, different payload: the client is misusing the key. The draft recommends 422 Unprocessable Content.

If an endpoint requires the header and it’s missing, the draft recommends 400 Bad Request. The draft also says the server should define and document how long keys stay valid. Make that part of your API contract.

Storing keys: schema, TTL and concurrent duplicates

The store needs to answer one question atomically: “has anyone claimed this key yet?” A relational table with a primary key on the idempotency key does this well, because the database enforces uniqueness for you, even under concurrent requests.

CREATE TABLE IdempotencyRecords (
    IdempotencyKey   NVARCHAR(100)  NOT NULL,
    Scope            NVARCHAR(100)  NOT NULL,  -- e.g. client id + endpoint
    RequestHash      CHAR(64)       NOT NULL,  -- SHA-256 hex of the body
    Status           TINYINT        NOT NULL,  -- 0 = in progress, 1 = completed
    ResponseCode     INT            NULL,
    ResponseBody     NVARCHAR(MAX)  NULL,
    CreatedAtUtc     DATETIME2      NOT NULL,
    ExpiresAtUtc     DATETIME2      NOT NULL,
    CONSTRAINT PK_IdempotencyRecords PRIMARY KEY (Scope, IdempotencyKey)
);

CREATE INDEX IX_IdempotencyRecords_Expires ON IdempotencyRecords (ExpiresAtUtc);

A few design decisions matter here:

• Scope the key. Include the client or tenant identifier (and usually the endpoint) in the key, so two different clients that happen to send the same UUID don’t collide.
• Claim first, then work. Insert an in progress row before doing the business operation. If the insert fails with a unique-constraint violation, someone else got there first.
• Expire old keys. Run a scheduled job that deletes rows past ExpiresAtUtc. Choose the window based on how long clients realistically retry (hours to days, not forever).
• Same database, same transaction. If you can, store the record in the same database as the business data, and mark it complete in the same transaction as the business write. Otherwise a crash between the two leaves them inconsistent.

Redis with SET key value NX EX ttl is another option for the claim step, and it’s fast. But you lose the transactional link with your business data, so I prefer the relational approach for operations that move money or create orders.

Edge cases: different payload, in-flight requests, partial failures

• Same key, different body. Compare a hash of the canonical request body with the stored hash, and return 422 on a mismatch. Never silently return the old response for a different request.
• In-flight duplicates. If the record exists with status in progress, return 409. Don’t block the second request waiting for the first, because that ties up threads and connections.
• Failure before commit. If processing throws, delete the in-progress record (or let the transaction roll it back) so the client can retry with the same key.
• Crashed workers. If a process dies mid-request, an in-progress row can be left behind. Give in-progress records a short lease and treat stale ones as retryable.
• Which responses to cache. Store successful results and deterministic client errors. Don’t store 5xx responses: the client should be able to retry those.

Implementing it in ASP.NET Core

Here’s a compact EF Core implementation. The service claims the key, runs the operation, and stores the response, all inside one database transaction with the business change.

using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Microsoft.EntityFrameworkCore;

public enum IdempotencyStatus : byte { InProgress = 0, Completed = 1 }

public class IdempotencyRecord
{
    public required string Scope { get; set; }
    public required string IdempotencyKey { get; set; }
    public required string RequestHash { get; set; }
    public IdempotencyStatus Status { get; set; }
    public int? ResponseCode { get; set; }
    public string? ResponseBody { get; set; }
    public DateTime CreatedAtUtc { get; set; }
    public DateTime ExpiresAtUtc { get; set; }
}

public class Payment
{
    public Guid Id { get; set; }
    public decimal Amount { get; set; }
    public required string Currency { get; set; }
}

public class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(options)
{
    public DbSet<IdempotencyRecord> IdempotencyRecords => Set<IdempotencyRecord>();
    public DbSet<Payment> Payments => Set<Payment>();

    protected override void OnModelCreating(ModelBuilder b) =>
        b.Entity<IdempotencyRecord>().HasKey(r => new { r.Scope, r.IdempotencyKey });
}

public sealed class IdempotencyService(AppDbContext db)
{
    private static readonly TimeSpan Retention = TimeSpan.FromHours(24);
    private static readonly JsonSerializerOptions Web = new(JsonSerializerDefaults.Web);

    public static string Hash(object request) =>
        Convert.ToHexString(SHA256.HashData(
            Encoding.UTF8.GetBytes(JsonSerializer.Serialize(request, Web))));

    public async Task<IResult> ExecuteAsync<TResponse>(
        string scope, string key, string requestHash,
        Func<Task<(int StatusCode, TResponse Body)>> operation,
        CancellationToken ct)
    {
        var existing = await db.IdempotencyRecords.AsNoTracking()
            .SingleOrDefaultAsync(r => r.Scope == scope && r.IdempotencyKey == key, ct);

        if (existing is not null)
            return Replay(existing, requestHash);

        await using var tx = await db.Database.BeginTransactionAsync(ct);

        var record = new IdempotencyRecord
        {
            Scope = scope,
            IdempotencyKey = key,
            RequestHash = requestHash,
            Status = IdempotencyStatus.InProgress,
            CreatedAtUtc = DateTime.UtcNow,
            ExpiresAtUtc = DateTime.UtcNow.Add(Retention)
        };
        db.IdempotencyRecords.Add(record);

        try
        {
            await db.SaveChangesAsync(ct); // claims the key (PK enforces uniqueness)
        }
        catch (DbUpdateException)
        {
            // A concurrent request claimed the key first.
            return Results.Problem(statusCode: StatusCodes.Status409Conflict,
                title: "A request with this Idempotency-Key is already being processed.");
        }

        var (statusCode, body) = await operation(); // business changes on the same db

        record.Status = IdempotencyStatus.Completed;
        record.ResponseCode = statusCode;
        record.ResponseBody = JsonSerializer.Serialize(body, Web); // same casing as Results.Json
        await db.SaveChangesAsync(ct);
        await tx.CommitAsync(ct);

        return Results.Json(body, statusCode: statusCode);
    }

    private static IResult Replay(IdempotencyRecord r, string requestHash)
    {
        if (r.RequestHash != requestHash)
            return Results.Problem(statusCode: StatusCodes.Status422UnprocessableEntity,
                title: "This Idempotency-Key was already used with a different request.");

        if (r.Status == IdempotencyStatus.InProgress)
            return Results.Problem(statusCode: StatusCodes.Status409Conflict,
                title: "A request with this Idempotency-Key is still in progress.");

        return Results.Content(r.ResponseBody ?? "null", "application/json",
            Encoding.UTF8, r.ResponseCode);
    }
}

And the endpoint that uses it:

app.MapPost("/payments", async (
    CreatePayment request,
    HttpRequest http,
    AppDbContext db,
    IdempotencyService idempotency,
    CancellationToken ct) =>
{
    var key = http.Headers["Idempotency-Key"].ToString().Trim('"');
    if (string.IsNullOrWhiteSpace(key) || key.Length > 100)
        return Results.Problem(statusCode: StatusCodes.Status400BadRequest,
            title: "A valid Idempotency-Key header is required.");

    var scope = "payments:" + (http.HttpContext.User.Identity?.Name ?? "anonymous");

    return await idempotency.ExecuteAsync(scope, key, IdempotencyService.Hash(request),
        async () =>
        {
            var payment = new Payment
            {
                Id = Guid.NewGuid(), Amount = request.Amount, Currency = request.Currency
            };
            db.Payments.Add(payment);
            await db.SaveChangesAsync(ct);
            return (StatusCodes.Status201Created, new { payment.Id, payment.Amount, payment.Currency });
        }, ct);
});

public record CreatePayment(decimal Amount, string Currency);

Because the claim, the business write and the stored response are committed together, a failure anywhere rolls everything back and the key becomes reusable. The primary key on (Scope, IdempotencyKey) guarantees only one request can claim a key. To use this for more than one endpoint, the same logic lifts cleanly into an endpoint filter.

One caveat: with the transaction held open, a concurrent duplicate may wait on the row lock and then fail its insert, rather than seeing the in-progress row straight away. Either way it gets a 409. If your operations are long-running, commit the claim in a short transaction of its own and use a lease, as described above.

Testing and observability: proving it works under retries

Idempotency code is only as good as the tests that attack it. The tests I’d write:

• Replay test: send the same request and key twice. Assert one row in the business table and identical responses.
• Mismatch test: same key with a different body. Assert 422 and no new row.
• Concurrency test: fire the same request in parallel (for example with Task.WhenAll on 10 calls). Assert exactly one success and one row.
• Failure test: make the operation throw. Assert the key can be used again afterwards.

[Fact]
public async Task Parallel_retries_create_exactly_one_payment()
{
    var client = _factory.CreateClient();
    var key = Guid.NewGuid().ToString();

    var tasks = Enumerable.Range(0, 10).Select(_ =>
    {
        var msg = new HttpRequestMessage(HttpMethod.Post, "/payments")
        {
            Content = JsonContent.Create(new { amount = 25.00m, currency = "USD" })
        };
        msg.Headers.Add("Idempotency-Key", key);
        return client.SendAsync(msg);
    });

    var responses = await Task.WhenAll(tasks);

    Assert.Contains(responses, r => r.StatusCode == HttpStatusCode.Created);
    Assert.Equal(1, await CountPaymentsAsync());
}

In production, log the idempotency key on every request as a structured property, and count replays, 409s and 422s as metrics. A sudden jump in replays usually points to a client timeout set too aggressively. A stream of 422s means a client is reusing keys incorrectly. Both are worth knowing about before they become incidents.

Retries are inevitable. With idempotency keys, a deduplication record committed together with your business data, and clear status codes, they become harmless.