Token bucket

Token Bucket

The token bucket classes that power Choked’s dual rate limiting functionality.

Overview

Choked uses two different token bucket implementations depending on your backend configuration:

• RedisTokenBucket: For distributed rate limiting using Redis
• ProxyTokenBucket: For managed rate limiting via the Choked proxy service

The appropriate implementation is automatically selected based on your Choked class configuration. You typically don’t need to use these classes directly.

RedisTokenBucket

Used when a Choked instance is created with redis_url. Provides distributed coordination using Redis with atomic Lua scripts.

Constructor

RedisTokenBucket(
    key: str, 
    request_capacity: int, 
    request_refill_rate: float,
    token_capacity: int, 
    token_refill_rate: float,
    redis_url: Optional[str] = None
)

key · str · required

Unique identifier for the rate limit bucket

request_capacity · int · required

Maximum number of request tokens in the bucket (0 = no request limiting)

request_refill_rate · float · required

Rate at which request tokens are added to the bucket (tokens per second, 0.0 = no request limiting)

token_capacity · int · required

Maximum number of content tokens in the bucket (0 = no token limiting)

token_refill_rate · float · required

Rate at which content tokens are added to the bucket (tokens per second, 0.0 = no token limiting)

redis_url · str

Redis connection URL. If not provided, uses the redis_url from the parent Choked instance.

Methods

acquire()

async def acquire(requests_needed: int = 1, tokens_needed: int = 0) -> bool

Attempts to acquire the specified number of request and content tokens from their respective buckets.

Parameters:

• requests_needed (int): Number of request tokens to acquire (typically 1)
• tokens_needed (int): Number of content tokens to acquire (estimated from function args)

Returns: True if both types of tokens were successfully acquired, False otherwise.

Behavior:

• Both request and token buckets are checked atomically
• If either bucket lacks sufficient tokens, no tokens are consumed
• Buckets are refilled based on elapsed time since last access
• Uses Redis Lua scripts for atomic operations

Configuration

Requires Redis connection via the parent Choked instance:

choke = Choked(redis_url="redis://localhost:6379/0")
# RedisTokenBucket automatically uses this redis_url

Redis Storage

The Redis implementation stores:

• Request bucket state: {key}:requests (current tokens, last refill time)
• Token bucket state: {key}:tokens (current tokens, last refill time)
• Uses Redis Hash data type for efficient storage
• Atomic operations via Lua scripts prevent race conditions

ProxyTokenBucket

Used when a Choked instance is created with api_token. Provides managed rate limiting through the Choked proxy service.

Constructor

ProxyTokenBucket(
    api_token: str,
    key: str, 
    request_capacity: int, 
    request_refill_rate: float,
    token_capacity: int, 
    token_refill_rate: float
)

api_token · str · required

API token for the Choked proxy service

key · str · required

Unique identifier for the rate limit bucket

request_capacity · int · required

Maximum number of request tokens in the bucket (0 = no request limiting)

request_refill_rate · float · required

Rate at which request tokens are added to the bucket (tokens per second, 0.0 = no request limiting)

token_capacity · int · required

Maximum number of content tokens in the bucket (0 = no token limiting)

token_refill_rate · float · required

Rate at which content tokens are added to the bucket (tokens per second, 0.0 = no token limiting)

Methods

acquire()

async def acquire(requests_needed: int = 1, tokens_needed: int = 0) -> bool

Attempts to acquire tokens via the proxy service.

Parameters:

• requests_needed (int): Number of request tokens to acquire
• tokens_needed (int): Number of content tokens to acquire

Returns: True if tokens were successfully acquired, False otherwise.

Behavior:

• Makes HTTP request to proxy service
• Service handles bucket logic server-side
• Automatic retry on network failures
• No local state storage required

Configuration

Requires API token for the managed service. Contact us for access.

Network Behavior

• HTTP-based communication with proxy service
• Automatic retries on network failures
• Respects standard HTTP timeouts
• Falls back gracefully on service unavailability

Usage Notes

Automatic Instantiation

Token buckets are automatically created by the Choked decorator:

from choked import Choked

choke = Choked(redis_url="redis://localhost:6379/0")

@choke(key="my_api", request_limit="10/s", token_limit="1000/m", token_estimator="openai")
def my_function():
    # RedisTokenBucket automatically created with:
    # - key="my_api"  
    # - request_capacity=10, request_refill_rate=10.0
    # - token_capacity=1000, token_refill_rate=16.67
    pass

Rate Limit Conversion

The Choked class automatically converts rate limit strings to bucket parameters:

# "10/s" becomes: capacity=10, refill_rate=10.0 tokens/second
# "100/m" becomes: capacity=100, refill_rate=1.67 tokens/second (100/60)
# None becomes: capacity=0, refill_rate=0.0 (no limiting)

Dual Bucket Coordination

Both implementations coordinate request and token buckets:

# Request-only limiting
await bucket.acquire(requests_needed=1, tokens_needed=0)

# Token-only limiting  
await bucket.acquire(requests_needed=0, tokens_needed=estimated_tokens)

# Dual limiting
await bucket.acquire(requests_needed=1, tokens_needed=estimated_tokens)

Error Handling

Both implementations handle errors gracefully:

• Network failures: Automatic retry with exponential backoff
• Redis connection issues: Connection pooling and retry logic
• Service unavailability: Graceful degradation (may allow requests through)
• Invalid parameters: Immediate validation errors

Performance Characteristics

RedisTokenBucket Performance

• Latency: ~1-2ms per acquire() call (local Redis)
• Throughput: 10,000+ operations/second per bucket
• Memory: ~100 bytes per bucket in Redis
• Network: 1 round-trip per acquire() call
• Scalability: Horizontal scaling via Redis clustering

ProxyTokenBucket Performance

• Latency: ~10-50ms per acquire() call (network dependent)
• Throughput: 1,000+ operations/second per bucket
• Memory: Zero local memory usage
• Network: 1 HTTP request per acquire() call
• Scalability: Managed by proxy service infrastructure

Performance Optimization

• Both implementations cache connection objects
• Redis implementation uses connection pooling
• Proxy implementation reuses HTTP connections
• No performance penalty for unused buckets (request_capacity=0 or token_capacity=0)

Advanced Usage

Direct Instantiation (Not Recommended)

While possible, direct instantiation bypasses the Choked class conveniences:

from choked.token_bucket import RedisTokenBucket

# Manual instantiation (not recommended)
bucket = RedisTokenBucket(
    key="manual_bucket",
    request_capacity=10,
    request_refill_rate=10.0,
    token_capacity=1000, 
    token_refill_rate=16.67,
    redis_url="redis://localhost:6379/0"
)

# Manual usage
success = await bucket.acquire(requests_needed=1, tokens_needed=50)
if success:
    # Proceed with rate-limited operation
    pass

Custom Integration

For custom rate limiting logic:

import asyncio
from choked.token_bucket import RedisTokenBucket

async def custom_rate_limiter():
    bucket = RedisTokenBucket("custom", 5, 1.0, 0, 0.0)  # 5 requests/5 seconds
    
    while True:
        if await bucket.acquire(1, 0):
            print("Request allowed")
        else:
            print("Rate limited, waiting...")
            await asyncio.sleep(1)

Monitoring and Observability

Both implementations support monitoring:

# Redis backend: Monitor via Redis commands
# HGETALL bucket_key:requests  # Current request bucket state
# HGETALL bucket_key:tokens    # Current token bucket state

# Proxy backend: Monitor via service dashboard
# (Contact us for monitoring access)

Thread Safety

• Both implementations are fully thread-safe and async-safe
• Redis implementation uses atomic Lua scripts
• Proxy implementation uses thread-safe HTTP clients
• Safe for concurrent use across multiple threads/processes
• No race conditions in bucket state management

Migration Between Backends

Switching between Redis and proxy backends:

# From Redis to Proxy
# OLD: choke = Choked(redis_url="redis://localhost:6379/0")
# NEW: choke = Choked(api_token="your-token")

# From Proxy to Redis  
# OLD: choke = Choked(api_token="your-token")
# NEW: choke = Choked(redis_url="redis://localhost:6379/0")

# Bucket state is not preserved across backend changes
# Rate limiting behavior remains identical

The token bucket implementations provide the foundation for Choked’s powerful dual rate limiting system, handling the complex coordination between request and token limits while maintaining high performance and reliability.