Compute Sizing Guide AI Prompt

Determine the right compute configuration for this data engineering workload.

Workload: {{workload_description}}
Data volume: {{data_volume}}
Runtime requirement: {{runtime_sla}}
Budget constraint: {{budget}}

1. Spark cluster sizing:
   - Driver: 1 node with 4–8 cores and 16–32GB RAM (driver is a coordinator, not a worker)
   - Executor memory rule: executor_memory = (node_memory × 0.75) / executors_per_node
   - Executor cores: 4–5 per executor (sweet spot — too many causes context switching, too few underutilizes memory parallelism)
   - Number of executors: total_data_size_GB / (executor_memory × compression_ratio) as a starting point
   - For shuffle-heavy jobs: more executors with less memory each (shuffle writes to local disk)
   - For memory-heavy joins: fewer executors with more memory each

2. Scaling strategy:
   - Start with a cluster that fits the data comfortably in memory
   - Profile first: identify if job is CPU-bound, memory-bound, or I/O-bound before scaling
   - CPU-bound: add more cores (more executors)
   - Memory-bound: add more RAM per executor (increase executor memory)
   - I/O-bound: add more storage bandwidth (use instance storage types like i3 on AWS)

3. Spot/preemptible instances:
   - Use spot for worker nodes (can tolerate eviction + checkpoint recovery)
   - Use on-demand for driver (eviction kills the entire job)
   - Savings: 60–80% cost reduction vs on-demand

4. Autoscaling:
   - Enable autoscaling for interactive and variable workloads
   - Disable for scheduled batch jobs with predictable volume (autoscaling overhead not worth it)

5. Benchmark procedure:
   - Run the job at 1×, 2×, 4× the baseline cluster size
   - Plot runtime vs cost: find the point of diminishing returns

Return: sizing recommendation, benchmark procedure, spot instance configuration, and cost estimate.