In today's distributed systems, ensuring reliable and consistent API behavior is crucial. One of the key challenges developers face is handling duplicate requests gracefully. This is where idempotency comes into play.
This post will guide serverless developers and AWS users through implementing idempotent APIs using AWS Lambda Powertools. By the end, you'll understand how to build resilient APIs that can handle network issues and client retries without producing duplicate operations.
The code for this is inside the spike-idempotent-requests repo. It uses the serverless framework to deploy the DDB table and the backend lambda function and then the frontend is using react/vite.
See the demo https://demo-idempotent-requests.netlify.app
Idempotency is a property of an API endpoint that ensures multiple identical requests produce the same result as a single request. This is particularly important in scenarios where network issues or client retries might cause the same request to be sent multiple times.
TLDR; Only do things once.
Implementing idempotent APIs is crucial for:
Understanding when to use idempotency is vital. Here are common scenarios where idempotent APIs make a difference:
Here's how our idempotent API system works:
Idempotency-Key header that is the hashed contents of the payload┌─────────────┐ ┌─────────────┐ ┌───────────────────┐ ┌──────────────┐
│ Client │───────▶│ API │───────▶│ Lambda Function │───────▶│ Business │
│ (with Key │ │ Gateway │ │ with Idempotency │ │ Logic │
│ Key) │ │ │ │ Wrapper │ │ │
└─────────────┘ └─────────────┘ └───────────────────┘ └──────────────┘
│
│
▼
┌──────────────────┐
│ DynamoDB │
│ (Idempotency │
│ Records) │
└──────────────────┘Let's break down how to implement idempotent APIs with AWS Lambda Powertools:
In our backend code we are wrapping our business logic with makeIdempotent from @aws-lambda-powertools/idempotency
This will check in memory (if possible), then also check in a DynamoDB table if our request has already been made.
// https://github.com/DavidWells/spike-idempotent-requests/blob/master/src/backend/src/idempotent.js
import { createHash } from 'crypto'
import { makeIdempotent, IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb'
import { Logger } from '@aws-lambda-powertools/logger'
const logger = new Logger()
if (!process.env.IDEMPOTENCY_TABLE) {
throw new Error('IDEMPOTENCY_TABLE is not set in environment variables')
}
const persistenceStore = new DynamoDBPersistenceLayer({
tableName: process.env.IDEMPOTENCY_TABLE,
keyAttr: 'id',
expiryAttr: 'expiration',
})
// Helper function to generate ETag from response body
function generateETag(responseBody) {
return `"${createHash('md5').update(responseBody).digest('hex')}"`
}
// Add cache headers to the response
function addCacheHeaders(response, etag) {
// Get cache TTL from env or use default (3600 seconds = 1 hour)
const cacheTtl = process.env.CACHE_TIME_IN_SECONDS || 3600
// Create new headers object - don't mutate corsHeaders directly
response.headers = {
...response.headers,
'ETag': etag,
'Cache-Control': `max-age=${cacheTtl}`,
'Vary': 'Accept-Encoding, Accept'
}
return response
}
function responseHook(response, record) {
console.log('responseHook response')
console.log(response)
console.log('responseHook record')
console.log(record)
const currentBody = JSON.parse(response.body)
// Add serverCacheHit flag to indicate this is from the idempotency cache
const newBody = JSON.stringify(Object.assign({}, currentBody, {
serverCacheHit: true
}))
response.body = newBody
// Return modified response
return response
}
const idempotencyConfig = new IdempotencyConfig({
eventKeyJmesPath: 'headers."Idempotency-Key" || headers."idempotency-key"',
expiresAfterSeconds: parseInt(process.env.CACHE_TIME_IN_SECONDS), // 24 * 60 * 60, // 24 hours
throwOnNoIdempotencyKey: true,
useLocalCache: true,
maxLocalCacheSize: 512,
responseHook,
})
const corsHeaders = {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Headers': '*',
'Access-Control-Allow-Methods': 'OPTIONS,POST,GET'
}
/**
* Lambda handler for idempotent requests
* @param {import('aws-lambda').APIGatewayProxyEvent} event - The request event
* @returns {Promise<Response>} The response
*/
export const handler = makeIdempotent(
async (event) => {
try {
logger.info('Processing request', { event })
// Check for idempotency key
const idempotencyKey = event.headers?.['Idempotency-Key'] || event.headers?.['idempotency-key']
if (!idempotencyKey) {
return {
statusCode: 400,
headers: corsHeaders,
body: JSON.stringify({ error: 'Idempotency key is required' })
}
}
// Check if the request includes an If-None-Match header
const ifNoneMatch = event.headers?.['If-None-Match'] || event.headers?.['if-none-match']
// Create the response body
const responseBody = JSON.stringify({
message: 'Processed',
requestId: idempotencyKey
})
// Generate an ETag for this response
const etag = generateETag(responseBody)
console.log('etag', etag)
// If client sent If-None-Match header that matches our ETag, return 304
if (ifNoneMatch && ifNoneMatch === etag) {
logger.info('ETag match - returning 304', { etag, ifNoneMatch })
return {
statusCode: 304, // Not Modified
headers: addCacheHeaders({ headers: corsHeaders }, etag)
}
}
// Otherwise, return full response with ETag
logger.info('Returning full response with ETag', { etag })
return addCacheHeaders({
statusCode: 200,
headers: corsHeaders,
body: responseBody
}, etag)
} catch (err) {
logger.error('Handler error', { error: err })
// Handle missing idempotency key error
if (err.message?.includes('No idempotency key found')) {
return {
statusCode: 400,
headers: corsHeaders,
body: JSON.stringify({ error: 'Idempotency key is required' })
}
}
return {
statusCode: 500,
headers: corsHeaders,
body: JSON.stringify({ error: err.message })
}
}
},
{
persistenceStore,
config: idempotencyConfig
}
)When implementing idempotent APIs, thorough testing is essential. Here's how to test your endpoints:
serverCacheHit flag in responsesHere's a simple test script example:
async function testIdempotency() {
const idempotencyKey = `test-${Date.now()}`
// First request should succeed normally
const response1 = await fetch('https://your-api.com/endpoint', {
method: 'POST',
headers: {
'Idempotency-Key': idempotencyKey,
'Content-Type': 'application/json'
},
body: JSON.stringify({ test: 'data' })
})
// Second request with same key should return cached result
const response2 = await fetch('https://your-api.com/endpoint', {
method: 'POST',
headers: {
'Idempotency-Key': idempotencyKey,
'Content-Type': 'application/json'
},
body: JSON.stringify({ test: 'data' })
})
const data1 = await response1.json()
const data2 = await response2.json()
console.log('First response:', data1)
console.log('Second response:', data2)
console.log('Server cache hit:', data2.serverCacheHit)
}A demonstration project showcasing how to implement idempotent requests in AWS Lambda functions using the AWS Lambda Powertools library. This project includes both backend and frontend components to demonstrate the complete flow of idempotent request handling.
.
├── src/ # Source code for both frontend and backend
│ ├── backend/ # AWS Lambda functions and infrastructure
│ └── frontend/ # React frontend application
├── scripts/ # Utility scripts
└── docs/ # Project documentationIn the root dir run:
## Install dependencies
npm run setupConfigure AWS credentials:
~/.aws/credentialsDeploy the backend:
cd src/backend
npm run deployThe backend consists of AWS Lambda functions that demonstrate idempotent request handling using AWS Lambda Powertools. Key features:
To start the development server:
npm startTo run tests:
cd src/backend
pnpm run testThe frontend is a React application that demonstrates how to interact with the idempotent API endpoints.
To start the development server:
cd src/frontend
npm startMake sure to deploy the backend first to setup the API endpoint.
Pull in the backend api endpoint automatically with
cd src/frontend
npm run syncThe project uses:
@aws-lambda-powertools/idempotency for idempotencyISC
This project serves as a practical example of implementing idempotent APIs in a serverless environment. By using AWS Lambda Powertools, we can implement these patterns with minimal boilerplate code while maintaining best practices for observability and reliability.
The complete source code is available in the repository, along with detailed documentation on setup and usage. Whether you're building a new API or improving an existing one, this project provides valuable insights into implementing idempotent request handling in a production environment.
See the code in https://github.com/DavidWells/spike-idempotent-requests/
Enjoy ✌️