Serverless WASM Functions in Kubernetes: FaaS with WebAssembly Performance

The serverless paradigm reaches new heights when combined with WebAssembly’s performance characteristics. By deploying WASM functions in Kubernetes, we achieve the best of both worlds: serverless simplicity with near-native performance and sub-millisecond cold starts. This comprehensive guide explores building production-ready serverless WASM platforms on Kubernetes.

Table of Contents

1. Serverless WASM Architecture
2. Platform Comparison and Selection
3. Knative with WASM Runtime
4. OpenFaaS WASM Implementation
5. Custom Serverless Platform
6. Function Development Patterns
7. Event-Driven Architecture
8. Auto-Scaling and Performance
9. Security and Isolation
10. Monitoring and Observability
11. CI/CD for Serverless WASM
12. Production Operations

Serverless WASM Architecture

Traditional vs WASM Serverless Comparison

Traditional Serverless Architecture:
┌─────────────────────────────────────────────────┐
│                  API Gateway                    │
├─────────────────────────────────────────────────┤
│           Function Runtime (Node.js)            │
│  ┌─────────────┐  ┌─────────────┐  ┌──────────┐ │
│  │   Cold      │  │    Warm     │  │  Scale   │ │
│  │ Start: 2s   │  │ Start: 50ms │  │ Up: 30s  │ │
│  │ Mem: 128MB  │  │ Mem: 128MB  │  │          │ │
│  └─────────────┘  └─────────────┘  └──────────┘ │
├─────────────────────────────────────────────────┤
│            Container Runtime (Docker)           │
├─────────────────────────────────────────────────┤
│               Kubernetes/VMs                    │
└─────────────────────────────────────────────────┘

WASM Serverless Architecture:
┌─────────────────────────────────────────────────┐
│                  API Gateway                    │
├─────────────────────────────────────────────────┤
│             WASM Function Runtime               │
│  ┌─────────────┐  ┌─────────────┐  ┌──────────┐ │
│  │   Cold      │  │    Warm     │  │  Scale   │ │
│  │ Start: 1ms  │  │ Start: <1ms │  │ Up: <1s  │ │
│  │ Mem: 2MB    │  │ Mem: 2MB    │  │          │ │
│  └─────────────┘  └─────────────┘  └──────────┘ │
├─────────────────────────────────────────────────┤
│           WASM Runtime (Wasmtime)               │
├─────────────────────────────────────────────────┤
│               Kubernetes Nodes                  │
└─────────────────────────────────────────────────┘

Function Execution Model

# WASM Function Execution Lifecycle
execution_lifecycle:
  trigger_received:
    - event: 'HTTP request, queue message, timer'
    - latency: '0ms'

  function_lookup:
    - action: 'Find function definition'
    - latency: '<1ms'

  runtime_initialization:
    - cold_start: '1-5ms (WASM instantiation)'
    - warm_start: '<1ms (existing instance)'

  function_execution:
    - wasm_module_load: 'Pre-compiled, instant'
    - memory_allocation: 'Linear memory, isolated'
    - execution: 'Near-native performance'

  response_handling:
    - serialization: 'Minimal overhead'
    - cleanup: 'Automatic memory reclamation'

  scaling_decisions:
    - scale_up_trigger: 'Queue depth > 0'
    - scale_down_trigger: 'Idle for 30s'
    - scale_speed: 'Instantaneous'

Resource Allocation Model

WASM Function Resource Allocation:
┌─────────────────────────────────────────────────┐
│                 Node Resources                  │
│  ┌─────────────────────────────────────────────┐│
│  │            Available: 32GB RAM              ││
│  │  ┌────┐┌────┐┌────┐┌────┐┌────┐┌────┐       ││
│  │  │2MB ││2MB ││2MB ││2MB ││2MB ││2MB │ ...   ││
│  │  │F1  ││F2  ││F3  ││F4  ││F5  ││F6  │       ││
│  │  └────┘└────┘└────┘└────┘└────┘└────┘       ││
│  │                                             ││
│  │  Theoretical Max: ~16,000 concurrent        ││
│  │  Practical Max: ~8,000 concurrent           ││
│  └─────────────────────────────────────────────┘│
└─────────────────────────────────────────────────┘

Traditional Container Functions:
┌─────────────────────────────────────────────────┐
│                 Node Resources                  │
│  ┌─────────────────────────────────────────────┐│
│  │            Available: 32GB RAM              ││
│  │  ┌────────┐┌────────┐┌────────┐             ││
│  │  │ 128MB  ││ 128MB  ││ 128MB  │             ││
│  │  │   F1   ││   F2   ││   F3   │             ││
│  │  └────────┘└────────┘└────────┘             ││
│  │                                             ││
│  │  Theoretical Max: ~256 concurrent           ││
│  │  Practical Max: ~100 concurrent             ││
│  └─────────────────────────────────────────────┘│
└─────────────────────────────────────────────────┘

Platform Comparison and Selection

Feature Matrix

Decision Framework

platform_selection:
  choose_knative_when:
    - need_enterprise_features: true
    - team_size: '> 10 developers'
    - compliance_requirements: 'strict'
    - event_sources: 'diverse (Kafka, Cloud Events, etc.)'
    - budget: 'high'

  choose_openfaas_when:
    - simplicity_priority: true
    - team_size: '< 10 developers'
    - quick_deployment: true
    - event_sources: 'basic (HTTP, queues)'
    - budget: 'medium'

  choose_custom_when:
    - unique_requirements: true
    - performance_critical: true
    - full_control_needed: true
    - team_expertise: 'high'
    - budget: 'flexible'

Knative with WASM Runtime

Installation and Setup

# @filename: models.py
# Install Knative Serving
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.0/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/download/knative-v1.12.0/serving-core.yaml

# Install Knative Eventing
kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.12.0/eventing-crds.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/knative-v1.12.0/eventing-core.yaml

# Install networking layer (Kourier)
kubectl apply -f https://github.com/knative/net-kourier/releases/download/knative-v1.12.0/kourier.yaml

# Configure Knative to use Kourier
kubectl patch configmap/config-network \
  --namespace knative-serving \
  --type merge \
  --patch '{"data":{"ingress-class":"kourier.ingress.networking.knative.dev"}}'

# Install WASM runtime class
kubectl apply -f https://github.com/spinkube/spin-operator/releases/latest/download/spin-operator.runtime-class.yaml

WASM Runtime Configuration

# wasm-runtime-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-features
  namespace: knative-serving
data:
  # Enable WASM runtime support
  kubernetes.podspec-runtimeclassname: 'enabled'
  kubernetes.podspec-securitycontext: 'enabled'

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-deployment
  namespace: knative-serving
data:
  # WASM-specific deployment settings
  progressDeadline: '120s'
  registriesSkippingTagResolving: 'ko.local,dev.local'

  # Queue proxy settings for WASM
  queueSidecarImage: 'gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:...'

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-autoscaler
  namespace: knative-serving
data:
  # Optimized for WASM cold starts
  enable-scale-to-zero: 'true'
  scale-to-zero-grace-period: '30s'
  stable-window: '60s'

  # WASM-optimized scaling
  max-scale-up-rate: '1000.0' # WASM can scale very quickly
  max-scale-down-rate: '2.0'
  target-burst-capacity: '200'

  # Concurrency settings
  container-concurrency-target-default: '100'
  container-concurrency-target-percentage: '70'

  # Panic thresholds (lower for WASM due to fast startup)
  panic-window-percentage: '10.0'
  panic-threshold-percentage: '200.0'

WASM Function Service

# wasm-function.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: image-processor
  namespace: default
spec:
  template:
    metadata:
      annotations:
        # Use WASM runtime
        autoscaling.knative.dev/class: 'kpa.autoscaling.knative.dev'
        autoscaling.knative.dev/target: '100'
        autoscaling.knative.dev/minScale: '0'
        autoscaling.knative.dev/maxScale: '1000'

        # WASM-specific optimizations
        run.googleapis.com/cpu-throttling: 'false'
        run.googleapis.com/execution-environment: 'gen2'
    spec:
      # Use WASM runtime class
      runtimeClassName: wasmtime

      containers:
        - name: function
          image: registry.example.com/image-processor-wasm:v1.0.0

          ports:
            - containerPort: 8080
              protocol: TCP

          env:
            - name: RUST_LOG
              value: 'info'
            - name: MAX_IMAGE_SIZE
              value: '10MB'
            - name: SUPPORTED_FORMATS
              value: 'jpeg,png,webp'

          resources:
            requests:
              cpu: '50m'
              memory: '32Mi'
            limits:
              cpu: '1000m'
              memory: '128Mi'

          # Health checks optimized for WASM
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 10

          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 1
            periodSeconds: 1

          # Security context for WASM
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL

---
# Traffic routing
apiVersion: serving.knative.dev/v1
kind: Route
metadata:
  name: image-processor-route
  namespace: default
spec:
  traffic:
    - revisionName: image-processor-v1
      percent: 90
    - revisionName: image-processor-v2
      percent: 10
      tag: canary

Event-Driven Function Triggers

# event-triggers.yaml
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: image-upload-trigger
  namespace: default
spec:
  broker: default
  filter:
    attributes:
      type: com.example.storage.upload
      source: /storage/bucket/images
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: image-processor

---
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: scheduled-cleanup-trigger
  namespace: default
spec:
  broker: default
  filter:
    attributes:
      type: dev.knative.sources.ping
      source: /cleanup/scheduler
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: cleanup-function

---
# Kafka event source
apiVersion: sources.knative.dev/v1beta1
kind: KafkaSource
metadata:
  name: user-events-source
  namespace: default
spec:
  consumerGroup: image-processor-group
  bootstrapServers:
    - kafka-broker-1.kafka.svc.cluster.local:9092
    - kafka-broker-2.kafka.svc.cluster.local:9092
  topics:
    - user-uploads
    - user-deletions
  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: image-processor

OpenFaaS WASM Implementation

OpenFaaS Installation with WASM Support

# @filename: script.sh
# Install OpenFaaS
curl -sL https://cli.openfaas.com | sudo sh

# Create OpenFaaS namespace
kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml

# Add OpenFaaS Helm repository
helm repo add openfaas https://openfaas.github.io/faas-netes/
helm repo update

# Install OpenFaaS with WASM support
helm upgrade openfaas \
  --install openfaas/openfaas \
  --namespace openfaas \
  --set functionNamespace=openfaas-fn \
  --set generateBasicAuth=true \
  --set serviceType=LoadBalancer \
  --set faasIdler.dryRun=false \
  --set prometheus.create=true \
  --set alertmanager.create=true

WASM Function Template

# template/wasm-rust/template.yml
language: wasm-rust
fprocess: ./handler
welcome_message: |
  You have successfully created a new function which uses WASM runtime.

  Languages supported: Rust -> WASM

  Features:
  - Fast cold starts (<5ms)
  - Memory efficient (<10MB)
  - Secure sandboxing
  - Near-native performance

# @filename: Dockerfile
# template/wasm-rust/Dockerfile
FROM --platform=${BUILDPLATFORM:-linux/amd64} rust:1.75 as builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM

WORKDIR /usr/src/app
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo "fn main() {}" > src/main.rs
RUN rustup target add wasm32-wasi
RUN cargo build --target wasm32-wasi --release

COPY src ./src
RUN touch src/main.rs
RUN cargo build --target wasm32-wasi --release

# Optimize WASM binary
RUN curl -L https://github.com/WebAssembly/binaryen/releases/download/version_116/binaryen-version_116-x86_64-linux.tar.gz | tar xz
RUN ./binaryen-version_116/bin/wasm-opt -Os target/wasm32-wasi/release/handler.wasm -o handler.wasm

FROM scratch
COPY --from=builder /usr/src/app/handler.wasm ./handler
CMD ["./handler"]

WASM Function Implementation

// @filename: main.rs
// src/main.rs - Image processing function
use std::io::{self, Read};
use image::{ImageFormat, DynamicImage, ImageBuffer, Rgba};
use serde::{Deserialize, Serialize};
use base64::{Engine as _, engine::general_purpose};

#[derive(Deserialize)]
struct FunctionInput {
    image_data: String,      // Base64 encoded image
    operation: String,       // resize, rotate, blur, etc.
    parameters: Parameters,
}

#[derive(Deserialize)]
struct Parameters {
    width: Option<u32>,
    height: Option<u32>,
    angle: Option<f32>,
    blur_radius: Option<f32>,
    quality: Option<u8>,
}

#[derive(Serialize)]
struct FunctionOutput {
    success: bool,
    image_data: Option<String>,
    error: Option<String>,
    metadata: ImageMetadata,
}

#[derive(Serialize)]
struct ImageMetadata {
    original_size: (u32, u32),
    processed_size: (u32, u32),
    format: String,
    processing_time_ms: u64,
}

fn main() {
    let mut input = String::new();
    io::stdin().read_to_string(&mut input).expect("Failed to read input");

    let start_time = std::time::Instant::now();

    let result = match serde_json::from_str::<FunctionInput>(&input) {
        Ok(request) => process_image(request, start_time),
        Err(e) => FunctionOutput {
            success: false,
            image_data: None,
            error: Some(format!("Invalid input: {}", e)),
            metadata: ImageMetadata {
                original_size: (0, 0),
                processed_size: (0, 0),
                format: "unknown".to_string(),
                processing_time_ms: start_time.elapsed().as_millis() as u64,
            },
        },
    };

    println!("{}", serde_json::to_string(&result).unwrap());
}

fn process_image(input: FunctionInput, start_time: std::time::Instant) -> FunctionOutput {
    // Decode base64 image
    let image_bytes = match general_purpose::STANDARD.decode(&input.image_data) {
        Ok(bytes) => bytes,
        Err(e) => return error_response(format!("Base64 decode error: {}", e), start_time),
    };

    // Load image
    let mut image = match image::load_from_memory(&image_bytes) {
        Ok(img) => img,
        Err(e) => return error_response(format!("Image load error: {}", e), start_time),
    };

    let original_size = (image.width(), image.height());

    // Apply requested operation
    image = match input.operation.as_str() {
        "resize" => resize_image(image, &input.parameters),
        "rotate" => rotate_image(image, &input.parameters),
        "blur" => blur_image(image, &input.parameters),
        "grayscale" => Ok(image.grayscale()),
        "flip_horizontal" => Ok(image.fliph()),
        "flip_vertical" => Ok(image.flipv()),
        "brighten" => brighten_image(image, &input.parameters),
        "contrast" => adjust_contrast(image, &input.parameters),
        _ => Err(format!("Unsupported operation: {}", input.operation)),
    }.unwrap_or_else(|e| {
        eprintln!("Processing error: {}", e);
        image // Return original on error
    });

    let processed_size = (image.width(), image.height());

    // Encode result
    let mut output_bytes = Vec::new();
    let format = ImageFormat::Png; // Default to PNG for lossless

    if let Err(e) = image.write_to(&mut std::io::Cursor::new(&mut output_bytes), format) {
        return error_response(format!("Image encode error: {}", e), start_time);
    }

    let encoded_image = general_purpose::STANDARD.encode(&output_bytes);

    FunctionOutput {
        success: true,
        image_data: Some(encoded_image),
        error: None,
        metadata: ImageMetadata {
            original_size,
            processed_size,
            format: "png".to_string(),
            processing_time_ms: start_time.elapsed().as_millis() as u64,
        },
    }
}

fn resize_image(image: DynamicImage, params: &Parameters) -> Result<DynamicImage, String> {
    let width = params.width.ok_or("Width parameter required for resize")?;
    let height = params.height.ok_or("Height parameter required for resize")?;

    Ok(image.resize(width, height, image::imageops::FilterType::Lanczos3))
}

fn rotate_image(image: DynamicImage, params: &Parameters) -> Result<DynamicImage, String> {
    let angle = params.angle.ok_or("Angle parameter required for rotate")?;

    match angle as i32 {
        90 | -270 => Ok(image.rotate90()),
        180 | -180 => Ok(image.rotate180()),
        270 | -90 => Ok(image.rotate270()),
        _ => Err(format!("Unsupported rotation angle: {}. Use 90, 180, or 270", angle)),
    }
}

fn blur_image(image: DynamicImage, params: &Parameters) -> Result<DynamicImage, String> {
    let radius = params.blur_radius.ok_or("Blur radius parameter required")?;

    if radius < 0.0 || radius > 100.0 {
        return Err("Blur radius must be between 0 and 100".to_string());
    }

    Ok(image.blur(radius))
}

fn brighten_image(image: DynamicImage, params: &Parameters) -> Result<DynamicImage, String> {
    // Use blur_radius as brightness value for simplicity
    let brightness = params.blur_radius.unwrap_or(0.0) as i32;

    Ok(image.brighten(brightness))
}

fn adjust_contrast(image: DynamicImage, params: &Parameters) -> Result<DynamicImage, String> {
    // Use blur_radius as contrast value for simplicity
    let contrast = params.blur_radius.unwrap_or(1.0);

    if contrast < 0.0 {
        return Err("Contrast must be positive".to_string());
    }

    Ok(image.adjust_contrast(contrast))
}

fn error_response(error: String, start_time: std::time::Instant) -> FunctionOutput {
    FunctionOutput {
        success: false,
        image_data: None,
        error: Some(error),
        metadata: ImageMetadata {
            original_size: (0, 0),
            processed_size: (0, 0),
            format: "unknown".to_string(),
            processing_time_ms: start_time.elapsed().as_millis() as u64,
        },
    }
}

Function Configuration

# stack.yml
provider:
  name: openfaas
  gateway: http://127.0.0.1:8080

functions:
  image-processor:
    lang: wasm-rust
    handler: ./image-processor
    image: registry.example.com/image-processor-wasm:latest

    # WASM-specific configuration
    constraints:
      - 'kubernetes.io/arch=amd64'

    # Resource limits optimized for WASM
    limits:
      memory: '128Mi'
      cpu: '1000m'
    requests:
      memory: '32Mi'
      cpu: '50m'

    # Environment variables
    environment:
      RUST_LOG: 'info'
      MAX_IMAGE_SIZE: '10485760' # 10MB
      ALLOWED_FORMATS: 'jpeg,png,gif,webp'

    # Annotations for WASM runtime
    annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path: '/metrics'
      prometheus.io/port: '8080'

    # Labels
    labels:
      runtime: 'wasm'
      language: 'rust'
      category: 'image-processing'

  # Additional functions
  thumbnail-generator:
    lang: wasm-rust
    handler: ./thumbnail-generator
    image: registry.example.com/thumbnail-generator-wasm:latest

    environment:
      THUMBNAIL_SIZES: '150x150,300x300,600x600'

  image-metadata-extractor:
    lang: wasm-rust
    handler: ./metadata-extractor
    image: registry.example.com/metadata-extractor-wasm:latest

    environment:
      EXTRACT_GPS: 'true'
      EXTRACT_CAMERA_INFO: 'true'

Deployment and Management

# @filename: script.sh
# Build and deploy functions
faas-cli template pull https://github.com/openfaas/templates
faas-cli build -f stack.yml
faas-cli push -f stack.yml
faas-cli deploy -f stack.yml

# Test function
echo '{"image_data":"iVBORw0KGgoAAAANSUhEUgAA...","operation":"resize","parameters":{"width":300,"height":200}}' | \
faas-cli invoke image-processor

# Scale function
faas-cli scale image-processor --replicas=10

# Monitor function
faas-cli logs image-processor --since=5m --follow

# Update function
faas-cli deploy -f stack.yml --update=true

Custom Serverless Platform

Platform Architecture

// @filename: main.rs
// src/main.rs - Custom serverless platform controller
use k8s_openapi::api::apps::v1::{Deployment, DeploymentSpec};
use k8s_openapi::api::core::v1::{
    Pod, PodSpec, Container, Service, ServiceSpec, ServicePort,
    EnvVar, ResourceRequirements, ResourceList,
};
use kube::{
    Api, Client,
    api::{Patch, PatchParams, ListParams, PostParams, DeleteParams},
    runtime::{Controller, controller::Action, watcher::Config},
    CustomResource, Resource, ResourceExt,
};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
use std::collections::BTreeMap;
use tokio::time::Duration;
use tracing::*;

#[derive(CustomResource, Deserialize, Serialize, Clone, Debug)]
#[kube(group = "serverless.example.com", version = "v1", kind = "WasmFunction")]
#[kube(namespaced)]
pub struct WasmFunctionSpec {
    pub image: String,
    pub runtime: String,
    pub min_replicas: Option<i32>,
    pub max_replicas: Option<i32>,
    pub target_cpu: Option<i32>,
    pub target_memory: Option<i32>,
    pub timeout: Option<String>,
    pub environment: Option<Vec<EnvVar>>,
    pub triggers: Vec<FunctionTrigger>,
    pub resources: Option<ResourceRequirements>,
}

#[derive(Deserialize, Serialize, Clone, Debug)]
pub struct FunctionTrigger {
    pub trigger_type: String,  // http, kafka, cron, etc.
    pub config: BTreeMap<String, String>,
}

#[derive(Deserialize, Serialize, Clone, Debug)]
pub struct WasmFunctionStatus {
    pub replicas: Option<i32>,
    pub ready_replicas: Option<i32>,
    pub conditions: Vec<FunctionCondition>,
    pub endpoint: Option<String>,
}

#[derive(Deserialize, Serialize, Clone, Debug)]
pub struct FunctionCondition {
    pub condition_type: String,
    pub status: String,
    pub reason: Option<String>,
    pub message: Option<String>,
}

impl WasmFunction {
    pub fn to_deployment(&self) -> Deployment {
        let name = self.name_any();
        let namespace = self.namespace().unwrap_or_default();

        let mut labels = BTreeMap::new();
        labels.insert("app".to_string(), name.clone());
        labels.insert("runtime".to_string(), "wasm".to_string());
        labels.insert("managed-by".to_string(), "wasm-serverless".to_string());

        let container = Container {
            name: "function".to_string(),
            image: Some(self.spec.image.clone()),
            ports: Some(vec![k8s_openapi::api::core::v1::ContainerPort {
                container_port: 8080,
                protocol: Some("TCP".to_string()),
                ..Default::default()
            }]),
            env: self.spec.environment.clone(),
            resources: self.spec.resources.clone(),
            security_context: Some(k8s_openapi::api::core::v1::SecurityContext {
                run_as_non_root: Some(true),
                run_as_user: Some(65534),
                read_only_root_filesystem: Some(true),
                allow_privilege_escalation: Some(false),
                capabilities: Some(k8s_openapi::api::core::v1::Capabilities {
                    drop: Some(vec!["ALL".to_string()]),
                    ..Default::default()
                }),
                ..Default::default()
            }),
            liveness_probe: Some(k8s_openapi::api::core::v1::Probe {
                http_get: Some(k8s_openapi::api::core::v1::HTTPGetAction {
                    path: Some("/health".to_string()),
                    port: k8s_openapi::apimachinery::pkg::util::intstr::IntOrString::Int(8080),
                    ..Default::default()
                }),
                initial_delay_seconds: Some(1),
                period_seconds: Some(10),
                ..Default::default()
            }),
            readiness_probe: Some(k8s_openapi::api::core::v1::Probe {
                http_get: Some(k8s_openapi::api::core::v1::HTTPGetAction {
                    path: Some("/ready".to_string()),
                    port: k8s_openapi::apimachinery::pkg::util::intstr::IntOrString::Int(8080),
                    ..Default::default()
                }),
                initial_delay_seconds: Some(1),
                period_seconds: Some(1),
                ..Default::default()
            }),
            ..Default::default()
        };

        Deployment {
            metadata: k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta {
                name: Some(name.clone()),
                namespace: Some(namespace.clone()),
                labels: Some(labels.clone()),
                owner_references: Some(vec![self.controller_owner_ref(&()).unwrap()]),
                ..Default::default()
            },
            spec: Some(DeploymentSpec {
                replicas: Some(self.spec.min_replicas.unwrap_or(1)),
                selector: k8s_openapi::apimachinery::pkg::apis::meta::v1::LabelSelector {
                    match_labels: Some(labels.clone()),
                    ..Default::default()
                },
                template: k8s_openapi::api::core::v1::PodTemplateSpec {
                    metadata: Some(k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta {
                        labels: Some(labels),
                        ..Default::default()
                    }),
                    spec: Some(PodSpec {
                        runtime_class_name: Some(self.spec.runtime.clone()),
                        containers: vec![container],
                        restart_policy: Some("Always".to_string()),
                        ..Default::default()
                    }),
                },
                ..Default::default()
            }),
            ..Default::default()
        }
    }

    pub fn to_service(&self) -> Service {
        let name = self.name_any();
        let namespace = self.namespace().unwrap_or_default();

        let mut labels = BTreeMap::new();
        labels.insert("app".to_string(), name.clone());

        Service {
            metadata: k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta {
                name: Some(format!("{}-service", name)),
                namespace: Some(namespace),
                owner_references: Some(vec![self.controller_owner_ref(&()).unwrap()]),
                ..Default::default()
            },
            spec: Some(ServiceSpec {
                selector: Some(labels),
                ports: Some(vec![ServicePort {
                    name: Some("http".to_string()),
                    port: 80,
                    target_port: Some(k8s_openapi::apimachinery::pkg::util::intstr::IntOrString::Int(8080)),
                    protocol: Some("TCP".to_string()),
                    ..Default::default()
                }]),
                type_: Some("ClusterIP".to_string()),
                ..Default::default()
            }),
            ..Default::default()
        }
    }
}

#[derive(Clone)]
pub struct Context {
    pub client: Client,
}

pub async fn reconcile(wasm_function: Arc<WasmFunction>, ctx: Arc<Context>) -> Result<Action, Box<dyn std::error::Error + Send + Sync>> {
    let name = wasm_function.name_any();
    let namespace = wasm_function.namespace().unwrap_or_default();

    info!("Reconciling WasmFunction {}/{}", namespace, name);

    let deployments: Api<Deployment> = Api::namespaced(ctx.client.clone(), &namespace);
    let services: Api<Service> = Api::namespaced(ctx.client.clone(), &namespace);
    let functions: Api<WasmFunction> = Api::namespaced(ctx.client.clone(), &namespace);

    // Create or update deployment
    let deployment = wasm_function.to_deployment();
    let deployment_name = deployment.metadata.name.as_ref().unwrap();

    match deployments.get_opt(deployment_name).await? {
        Some(_existing) => {
            info!("Updating deployment {}/{}", namespace, deployment_name);
            deployments.patch(
                deployment_name,
                &PatchParams::apply("wasm-serverless-controller"),
                &Patch::Apply(&deployment),
            ).await?;
        },
        None => {
            info!("Creating deployment {}/{}", namespace, deployment_name);
            deployments.create(&PostParams::default(), &deployment).await?;
        }
    }

    // Create or update service
    let service = wasm_function.to_service();
    let service_name = service.metadata.name.as_ref().unwrap();

    match services.get_opt(service_name).await? {
        Some(_existing) => {
            info!("Updating service {}/{}", namespace, service_name);
            services.patch(
                service_name,
                &PatchParams::apply("wasm-serverless-controller"),
                &Patch::Apply(&service),
            ).await?;
        },
        None => {
            info!("Creating service {}/{}", namespace, service_name);
            services.create(&PostParams::default(), &service).await?;
        }
    }

    // Update status
    let status = WasmFunctionStatus {
        replicas: Some(1), // This should be read from actual deployment
        ready_replicas: Some(1),
        conditions: vec![FunctionCondition {
            condition_type: "Ready".to_string(),
            status: "True".to_string(),
            reason: Some("Deployed".to_string()),
            message: Some("Function deployed successfully".to_string()),
        }],
        endpoint: Some(format!("http://{}.{}.svc.cluster.local", service_name, namespace)),
    };

    let mut updated_function = (*wasm_function).clone();
    updated_function.status = Some(status);

    functions.patch_status(
        &name,
        &PatchParams::default(),
        &Patch::Merge(&updated_function),
    ).await?;

    Ok(Action::requeue(Duration::from_secs(60)))
}

pub fn error_policy(_wasm_function: Arc<WasmFunction>, error: &Box<dyn std::error::Error + Send + Sync>, _ctx: Arc<Context>) -> Action {
    error!("Reconciliation error: {:?}", error);
    Action::requeue(Duration::from_secs(30))
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    tracing_subscriber::init();

    let client = Client::try_default().await?;
    let context = Arc::new(Context { client: client.clone() });

    let functions = Api::<WasmFunction>::all(client);

    Controller::new(functions, Config::default())
        .run(reconcile, error_policy, context)
        .for_each(|_| async {})
        .await;

    Ok(())
}

Function CRD Definition

# crd/wasmfunction.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: wasmfunctions.serverless.example.com
spec:
  group: serverless.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                  description: 'WASM function container image'
                runtime:
                  type: string
                  description: 'WASM runtime class name'
                  default: 'wasmtime'
                minReplicas:
                  type: integer
                  minimum: 0
                  default: 0
                maxReplicas:
                  type: integer
                  minimum: 1
                  default: 100
                targetCpu:
                  type: integer
                  minimum: 1
                  maximum: 100
                  default: 70
                targetMemory:
                  type: integer
                  minimum: 1
                  maximum: 100
                  default: 80
                timeout:
                  type: string
                  default: '30s'
                environment:
                  type: array
                  items:
                    type: object
                    properties:
                      name:
                        type: string
                      value:
                        type: string
                triggers:
                  type: array
                  items:
                    type: object
                    properties:
                      triggerType:
                        type: string
                        enum: ['http', 'kafka', 'cron', 'storage']
                      config:
                        type: object
                        additionalProperties:
                          type: string
            status:
              type: object
              properties:
                replicas:
                  type: integer
                readyReplicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      conditionType:
                        type: string
                      status:
                        type: string
                      reason:
                        type: string
                      message:
                        type: string
                endpoint:
                  type: string
      subresources:
        status: {}
        scale:
          specReplicasPath: .spec.minReplicas
          statusReplicasPath: .status.replicas
  scope: Namespaced
  names:
    plural: wasmfunctions
    singular: wasmfunction
    kind: WasmFunction
    shortNames:
      - wfn
      - wfunc

Function Deployment Example

# example-function.yaml
apiVersion: serverless.example.com/v1
kind: WasmFunction
metadata:
  name: image-resizer
  namespace: default
spec:
  image: 'registry.example.com/image-resizer:v1.0.0'
  runtime: 'wasmtime'

  # Scaling configuration
  minReplicas: 0
  maxReplicas: 1000
  targetCpu: 70
  targetMemory: 80

  # Function configuration
  timeout: '30s'
  environment:
    - name: MAX_IMAGE_SIZE
      value: '10485760'
    - name: ALLOWED_FORMATS
      value: 'jpeg,png,webp'
    - name: RUST_LOG
      value: 'info'

  # Event triggers
  triggers:
    - triggerType: 'http'
      config:
        path: '/resize'
        methods: 'POST'
    - triggerType: 'kafka'
      config:
        topic: 'image-upload-events'
        consumerGroup: 'image-resizer-group'
    - triggerType: 'storage'
      config:
        bucket: 'user-uploads'
        events: 'object.created'

  # Resource requirements
  resources:
    requests:
      cpu: '50m'
      memory: '32Mi'
    limits:
      cpu: '2000m'
      memory: '256Mi'

---
apiVersion: serverless.example.com/v1
kind: WasmFunction
metadata:
  name: log-analyzer
  namespace: monitoring
spec:
  image: 'registry.example.com/log-analyzer:v1.2.0'
  runtime: 'wasmtime'

  minReplicas: 1 # Always running for log processing
  maxReplicas: 50

  triggers:
    - triggerType: 'kafka'
      config:
        topic: 'application-logs'
        consumerGroup: 'log-analyzer'
    - triggerType: 'cron'
      config:
        schedule: '0 */5 * * * *' # Every 5 minutes

  environment:
    - name: LOG_LEVEL
      value: 'warn'
    - name: ALERT_THRESHOLD
      value: '100'
    - name: ELASTICSEARCH_URL
      valueFrom:
        secretKeyRef:
          name: elasticsearch-credentials
          key: url

Function Development Patterns

Multi-Format Handler Pattern

// @filename: main.rs
// src/lib.rs - Multi-format function handler
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::io::{self, Read};

#[derive(Debug, Deserialize)]
#[serde(untagged)]
pub enum FunctionInput {
    // HTTP request format
    HttpRequest {
        method: String,
        path: String,
        headers: HashMap<String, String>,
        body: String,
        query_parameters: Option<HashMap<String, String>>,
    },

    // Event format (Kafka, SQS, etc.)
    Event {
        source: String,
        event_type: String,
        data: serde_json::Value,
        timestamp: String,
        metadata: Option<HashMap<String, String>>,
    },

    // Simple function call
    DirectCall {
        function: String,
        parameters: serde_json::Value,
    },

    // Batch processing
    Batch {
        items: Vec<serde_json::Value>,
        batch_id: String,
        configuration: Option<HashMap<String, String>>,
    },
}

#[derive(Debug, Serialize)]
pub struct FunctionOutput {
    pub success: bool,
    pub data: Option<serde_json::Value>,
    pub error: Option<String>,
    pub metadata: OutputMetadata,
}

#[derive(Debug, Serialize)]
pub struct OutputMetadata {
    pub processing_time_ms: u64,
    pub memory_used_bytes: usize,
    pub items_processed: usize,
    pub function_version: String,
}

pub trait FunctionHandler {
    fn handle_http(&self, request: HttpRequest) -> Result<serde_json::Value, String>;
    fn handle_event(&self, event: Event) -> Result<serde_json::Value, String>;
    fn handle_direct_call(&self, call: DirectCall) -> Result<serde_json::Value, String>;
    fn handle_batch(&self, batch: Batch) -> Result<serde_json::Value, String>;
}

pub struct ImageProcessor {
    max_size: usize,
    allowed_formats: Vec<String>,
}

impl ImageProcessor {
    pub fn new() -> Self {
        let max_size = std::env::var("MAX_IMAGE_SIZE")
            .unwrap_or_else(|_| "10485760".to_string())
            .parse()
            .unwrap_or(10485760);

        let allowed_formats = std::env::var("ALLOWED_FORMATS")
            .unwrap_or_else(|_| "jpeg,png,webp".to_string())
            .split(',')
            .map(|s| s.trim().to_string())
            .collect();

        Self { max_size, allowed_formats }
    }
}

impl FunctionHandler for ImageProcessor {
    fn handle_http(&self, request: HttpRequest) -> Result<serde_json::Value, String> {
        match request.method.as_str() {
            "POST" => {
                if request.path == "/resize" {
                    self.process_resize_request(&request.body)
                } else if request.path == "/thumbnail" {
                    self.process_thumbnail_request(&request.body)
                } else {
                    Err(format!("Unsupported path: {}", request.path))
                }
            },
            "GET" => {
                if request.path == "/health" {
                    Ok(serde_json::json!({"status": "healthy", "uptime": get_uptime()}))
                } else if request.path == "/metrics" {
                    Ok(serde_json::json!(get_metrics()))
                } else {
                    Err(format!("Unsupported GET path: {}", request.path))
                }
            },
            _ => Err(format!("Unsupported method: {}", request.method)),
        }
    }

    fn handle_event(&self, event: Event) -> Result<serde_json::Value, String> {
        match event.event_type.as_str() {
            "image.uploaded" => {
                let image_data = event.data.get("image_data")
                    .and_then(|v| v.as_str())
                    .ok_or("Missing image_data in event")?;

                self.process_image_from_base64(image_data)
            },
            "batch.process" => {
                let images = event.data.get("images")
                    .and_then(|v| v.as_array())
                    .ok_or("Missing images array in event")?;

                self.process_image_batch(images)
            },
            _ => Err(format!("Unsupported event type: {}", event.event_type)),
        }
    }

    fn handle_direct_call(&self, call: DirectCall) -> Result<serde_json::Value, String> {
        match call.function.as_str() {
            "resize_image" => {
                let params = call.parameters.as_object()
                    .ok_or("Parameters must be an object")?;

                let image_data = params.get("image_data")
                    .and_then(|v| v.as_str())
                    .ok_or("Missing image_data parameter")?;

                let width = params.get("width")
                    .and_then(|v| v.as_u64())
                    .ok_or("Missing width parameter")? as u32;

                let height = params.get("height")
                    .and_then(|v| v.as_u64())
                    .ok_or("Missing height parameter")? as u32;

                self.resize_image_base64(image_data, width, height)
            },
            _ => Err(format!("Unsupported function: {}", call.function)),
        }
    }

    fn handle_batch(&self, batch: Batch) -> Result<serde_json::Value, String> {
        let mut results = Vec::new();
        let mut errors = Vec::new();

        for (index, item) in batch.items.iter().enumerate() {
            match self.process_batch_item(item) {
                Ok(result) => results.push(serde_json::json!({
                    "index": index,
                    "result": result
                })),
                Err(error) => errors.push(serde_json::json!({
                    "index": index,
                    "error": error
                })),
            }
        }

        Ok(serde_json::json!({
            "batch_id": batch.batch_id,
            "total_items": batch.items.len(),
            "successful_items": results.len(),
            "failed_items": errors.len(),
            "results": results,
            "errors": errors
        }))
    }
}

impl ImageProcessor {
    fn process_resize_request(&self, body: &str) -> Result<serde_json::Value, String> {
        // Implementation for resize request
        let request: serde_json::Value = serde_json::from_str(body)
            .map_err(|e| format!("Invalid JSON: {}", e))?;

        // Extract parameters and process
        Ok(serde_json::json!({"processed": true}))
    }

    fn process_thumbnail_request(&self, body: &str) -> Result<serde_json::Value, String> {
        // Implementation for thumbnail generation
        Ok(serde_json::json!({"thumbnail_generated": true}))
    }

    fn process_image_from_base64(&self, image_data: &str) -> Result<serde_json::Value, String> {
        // Implementation for processing base64 image
        Ok(serde_json::json!({"image_processed": true}))
    }

    fn process_image_batch(&self, images: &[serde_json::Value]) -> Result<serde_json::Value, String> {
        // Implementation for batch processing
        Ok(serde_json::json!({"batch_processed": images.len()}))
    }

    fn resize_image_base64(&self, image_data: &str, width: u32, height: u32) -> Result<serde_json::Value, String> {
        // Implementation for resizing image
        Ok(serde_json::json!({
            "resized": true,
            "width": width,
            "height": height
        }))
    }

    fn process_batch_item(&self, item: &serde_json::Value) -> Result<serde_json::Value, String> {
        // Implementation for processing individual batch item
        Ok(serde_json::json!({"item_processed": true}))
    }
}

// Helper functions
fn get_uptime() -> u64 {
    // Return uptime in seconds
    std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap()
        .as_secs()
}

fn get_metrics() -> serde_json::Value {
    serde_json::json!({
        "memory_usage": get_memory_usage(),
        "cpu_usage": get_cpu_usage(),
        "requests_processed": get_request_count(),
        "errors": get_error_count()
    })
}

fn get_memory_usage() -> usize {
    // Return current memory usage in bytes
    0 // Placeholder
}

fn get_cpu_usage() -> f64 {
    // Return current CPU usage percentage
    0.0 // Placeholder
}

fn get_request_count() -> u64 {
    // Return total requests processed
    0 // Placeholder
}

fn get_error_count() -> u64 {
    // Return total errors encountered
    0 // Placeholder
}

// Main function entry point
fn main() {
    let start_time = std::time::Instant::now();

    // Read input from stdin
    let mut input = String::new();
    io::stdin().read_to_string(&mut input)
        .expect("Failed to read input");

    // Parse input
    let function_input: FunctionInput = serde_json::from_str(&input)
        .unwrap_or_else(|_| FunctionInput::DirectCall {
            function: "unknown".to_string(),
            parameters: serde_json::json!({}),
        });

    // Create handler
    let handler = ImageProcessor::new();

    // Process request
    let result = match function_input {
        FunctionInput::HttpRequest { method, path, headers, body, query_parameters } => {
            handler.handle_http(HttpRequest { method, path, headers, body, query_parameters })
        },
        FunctionInput::Event { source, event_type, data, timestamp, metadata } => {
            handler.handle_event(Event { source, event_type, data, timestamp, metadata })
        },
        FunctionInput::DirectCall { function, parameters } => {
            handler.handle_direct_call(DirectCall { function, parameters })
        },
        FunctionInput::Batch { items, batch_id, configuration } => {
            handler.handle_batch(Batch { items, batch_id, configuration })
        },
    };

    // Create output
    let output = match result {
        Ok(data) => FunctionOutput {
            success: true,
            data: Some(data),
            error: None,
            metadata: OutputMetadata {
                processing_time_ms: start_time.elapsed().as_millis() as u64,
                memory_used_bytes: get_memory_usage(),
                items_processed: 1,
                function_version: env!("CARGO_PKG_VERSION").to_string(),
            },
        },
        Err(error) => FunctionOutput {
            success: false,
            data: None,
            error: Some(error),
            metadata: OutputMetadata {
                processing_time_ms: start_time.elapsed().as_millis() as u64,
                memory_used_bytes: get_memory_usage(),
                items_processed: 0,
                function_version: env!("CARGO_PKG_VERSION").to_string(),
            },
        },
    };

    // Output result
    println!("{}", serde_json::to_string(&output).unwrap());
}

// Type aliases for pattern matching
type HttpRequest = FunctionInput;
type Event = FunctionInput;
type DirectCall = FunctionInput;
type Batch = FunctionInput;

Testing Framework

// @filename: lib.rs
// tests/integration_test.rs
use serde_json::json;
use std::process::{Command, Stdio};
use std::io::Write;

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_http_resize_request() {
        let input = json!({
            "method": "POST",
            "path": "/resize",
            "headers": {
                "content-type": "application/json"
            },
            "body": json!({
                "image_data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8/5+hHgAHggJ/PchI7wAAAABJRU5ErkJggg==",
                "width": 300,
                "height": 200
            }).to_string(),
            "query_parameters": {}
        });

        let result = run_function(input);
        assert!(result["success"].as_bool().unwrap());
        assert!(result["metadata"]["processing_time_ms"].as_u64().unwrap() > 0);
    }

    #[test]
    fn test_event_processing() {
        let input = json!({
            "source": "storage.bucket",
            "event_type": "image.uploaded",
            "data": {
                "image_data": "base64-encoded-image",
                "bucket": "user-uploads",
                "key": "images/photo.jpg"
            },
            "timestamp": "2024-01-24T10:00:00Z",
            "metadata": {
                "user_id": "12345"
            }
        });

        let result = run_function(input);
        assert!(result["success"].as_bool().unwrap());
    }

    #[test]
    fn test_batch_processing() {
        let input = json!({
            "items": [
                {"image_data": "image1", "operation": "resize"},
                {"image_data": "image2", "operation": "thumbnail"},
                {"image_data": "image3", "operation": "rotate"}
            ],
            "batch_id": "batch_123",
            "configuration": {
                "parallel": "true",
                "timeout": "30s"
            }
        });

        let result = run_function(input);
        assert!(result["success"].as_bool().unwrap());
        assert_eq!(result["data"]["total_items"].as_u64().unwrap(), 3);
    }

    #[test]
    fn test_error_handling() {
        let input = json!({
            "method": "POST",
            "path": "/invalid",
            "headers": {},
            "body": "",
            "query_parameters": {}
        });

        let result = run_function(input);
        assert!(!result["success"].as_bool().unwrap());
        assert!(result["error"].as_str().unwrap().contains("Unsupported path"));
    }

    #[test]
    fn test_performance_requirements() {
        let input = json!({
            "function": "resize_image",
            "parameters": {
                "image_data": "small-test-image",
                "width": 100,
                "height": 100
            }
        });

        let start = std::time::Instant::now();
        let result = run_function(input);
        let duration = start.elapsed();

        assert!(result["success"].as_bool().unwrap());
        assert!(duration.as_millis() < 100, "Function took too long: {:?}", duration);
        assert!(result["metadata"]["processing_time_ms"].as_u64().unwrap() < 100);
    }

    fn run_function(input: serde_json::Value) -> serde_json::Value {
        let mut child = Command::new("wasmtime")
            .arg("run")
            .arg("target/wasm32-wasi/release/image_processor.wasm")
            .stdin(Stdio::piped())
            .stdout(Stdio::piped())
            .stderr(Stdio::piped())
            .spawn()
            .expect("Failed to start WASM function");

        let stdin = child.stdin.as_mut().expect("Failed to get stdin");
        stdin.write_all(input.to_string().as_bytes())
            .expect("Failed to write to stdin");
        stdin.flush().expect("Failed to flush stdin");

        let output = child.wait_with_output()
            .expect("Failed to read output");

        if !output.status.success() {
            panic!("Function failed: {}", String::from_utf8_lossy(&output.stderr));
        }

        let stdout = String::from_utf8(output.stdout)
            .expect("Invalid UTF-8 output");

        serde_json::from_str(&stdout)
            .expect("Invalid JSON output")
    }
}

// Load testing
#[cfg(test)]
mod load_tests {
    use super::*;
    use std::sync::Arc;
    use std::sync::atomic::{AtomicUsize, Ordering};
    use tokio::time::{Duration, timeout};

    #[tokio::test]
    async fn test_concurrent_requests() {
        let request_count = Arc::new(AtomicUsize::new(0));
        let success_count = Arc::new(AtomicUsize::new(0));
        let error_count = Arc::new(AtomicUsize::new(0));

        let mut handles = Vec::new();

        // Spawn 100 concurrent requests
        for i in 0..100 {
            let request_count = Arc::clone(&request_count);
            let success_count = Arc::clone(&success_count);
            let error_count = Arc::clone(&error_count);

            let handle = tokio::spawn(async move {
                let input = json!({
                    "function": "resize_image",
                    "parameters": {
                        "image_data": format!("test-image-{}", i),
                        "width": 200,
                        "height": 200
                    }
                });

                request_count.fetch_add(1, Ordering::SeqCst);

                // Set timeout for each request
                let result = timeout(Duration::from_secs(5), async {
                    run_async_function(input).await
                }).await;

                match result {
                    Ok(Ok(response)) => {
                        if response["success"].as_bool().unwrap_or(false) {
                            success_count.fetch_add(1, Ordering::SeqCst);
                        } else {
                            error_count.fetch_add(1, Ordering::SeqCst);
                        }
                    },
                    Ok(Err(_)) | Err(_) => {
                        error_count.fetch_add(1, Ordering::SeqCst);
                    }
                }
            });
            handles.push(handle);
        }

        // Wait for all requests to complete
        for handle in handles {
            handle.await.expect("Task panicked");
        }

        let total_requests = request_count.load(Ordering::SeqCst);
        let successful_requests = success_count.load(Ordering::SeqCst);
        let failed_requests = error_count.load(Ordering::SeqCst);

        println!("Load test results:");
        println!("  Total requests: {}", total_requests);
        println!("  Successful: {}", successful_requests);
        println!("  Failed: {}", failed_requests);
        println!("  Success rate: {:.2}%", (successful_requests as f64 / total_requests as f64) * 100.0);

        // Assert at least 95% success rate
        assert!(successful_requests as f64 / total_requests as f64 >= 0.95);
        assert_eq!(total_requests, 100);
    }

    async fn run_async_function(input: serde_json::Value) -> Result<serde_json::Value, Box<dyn std::error::Error + Send + Sync>> {
        // Async version of run_function
        tokio::task::spawn_blocking(move || {
            run_function(input)
        }).await.map_err(|e| e.into())
    }
}