Price Report

v1 from Modal Cost for Alex

Object Cost from Modal

The cost breakdown for the searching service is categorized into the following components:

1. GPU Running Time: Costs vary depending on the GPU type used (e.g., Nvidia H100, L4, L40S).
2. CPU Running Time: Costs are based on the number of CPU cores reserved and their usage duration.
3. Memory Allocation: Costs are calculated per GiB per second.
4. Cloud Storage & Data Transfer: Includes volume usage, CloudBucketMount, and data egress costs.
5. Idle Container Time: Costs are incurred for the duration a container remains active, even if idle.

The Methodology Plan to Measure the Cost

1. GPU Tasks: Measure the cost per second for each GPU type used in the Catalog and Lookup classes.
2. CPU Tasks: Calculate the cost based on the number of CPU cores reserved and their usage time.
3. Memory Usage: Track memory allocation and usage to determine the cost per GiB per second.
4. Idle Container Time: Monitor the container idle timeout to calculate the cost of keeping containers active.
5. Testing Data: Use real-world testing data (e.g., API calls, image processing) to validate cost calculations.

Key Points of Cost

1. GPU Costs:

   • Nvidia H100: $0.001267/sec
   • Nvidia L4: $0.000222/sec
   • Nvidia L40S: $0.000542/sec

2. CPU Costs:

   • $0.000038/core/sec

3. Memory Costs:

   • $0.00000667/GiB/sec

4. Idle Container Costs:

   • Containers remain active for 720 seconds by default, incurring costs even when idle.

5. Optimization Strategies:

   • Increase memory and concurrency for Lookup to handle more requests per container.
   • Reduce container idle timeout for API services to minimize idle costs.

Cost Log Report

Testing Data

1. get_image:

   • Cost: $0.40
   • Time: 347.9s + 17.06s

2. get_image_urns:

   • Memory: 1.06GB/16GB
   • CPU: 0.11 core/4 core
   • GPU Memory: 16.40GB
   • Cost: $1.42 ($0.92 GPU + $0.27 CPU + $0.19 Memory)
   • Time: 23mins 17s

3. First API Call:

   • Time: 32s
   • Cost: $0.24 ($0.18 GPU + $0.06 CPU + $0.01 Memory)

4. Frequent Request Test (40 requests/min):

   • Total Requests: 160
   • Cost: $0.88 ($0.64 GPU + $0.22 Memory + $0.02 CPU)
   • Time: 12mins
   • Cost per Call: $0.0055

5. Optimized Configuration Test (200 requests in 5 mins):

   • Cost: $0.31 ($0.21 GPU + $0.07 Memory + $0.03 CPU)
   • Cost per Call: $0.0015

Cost Analysis

1. GPU Costs:

   • The Nvidia L40S GPU is the most cost-effective for frequent requests, with a cost of $0.000542/sec.
   • The Nvidia H100 is more expensive but may be necessary for specific tasks.

2. CPU Costs:

   • CPU costs are minimal compared to GPU costs but should still be optimized by reducing reserved cores.

3. Memory Costs:

   • Memory costs are relatively low but can add up with high concurrency and large datasets.

4. Idle Container Costs:

   • Reducing the container idle timeout significantly lowers costs, especially for API services.

5. Optimization Impact:

   • Increasing memory and concurrency for Lookup reduced the cost per call from $0.0055 to $0.0015.
   • Shortening the container idle timeout for API services further minimized idle costs.

Price Table

Optimization Plan

1. Lookup Class (GPU Model)

• Goal: Handle more requests per container to reduce the number of containers needed.
• Actions:
  • Increase memory from 1024 MB → 4096 MB (4 GB).
  • Set container_idle_timeout=300s (5 minutes) to avoid cold starts.
  • Set concurrency_limit=4 to allow handling 4 requests simultaneously.
• Impact:
  • Fewer containers are created, reducing GPU and memory costs.
  • Cost per call reduced from $0.0055 to $0.0015.

2. API Service (FastAPI Web Handler)

• Goal: Make the API service lightweight and disposable to minimize idle costs.
• Actions:
  • Reduce container_idle_timeout to 30 seconds.
  • Set concurrency_limit=2 to limit the number of active containers.
  • Disable keep_warm to avoid pre-warmed containers.
• Impact:
  • API containers shut down quickly when idle, reducing idle costs.
  • Startup time is fast (3s), so new requests are handled efficiently.

Cost Projections

Current Configuration

• Cost per Call: $0.0015 (optimized configuration).
• Hourly Cost: $1.08 (based on 200 requests in 5 minutes).
• Monthly Cost: ~$777.60 (assuming 24/7 operation).

Future Improvements

6. Further Increase Concurrency:
   • If concurrency_limit is increased to 8, the cost per call could drop further.
7. Dynamic Scaling:
   • Implement dynamic scaling based on request load to reduce idle time.
8. GPU Selection:
   • Evaluate if a lower-cost GPU (e.g., Nvidia T4) can handle the workload without performance degradation.

Key Takeaways

9. GPU Costs Dominate:
   • The choice of GPU (e.g., L40S vs. H100) has the most significant impact on overall costs.
10. Idle Time is Expensive:

• Reducing container_idle_timeout for API services significantly lowers costs.

11. Optimization Works:

• Increasing memory and concurrency for Lookup reduced costs by 72% ($0.0055 → $0.0015 per call).

12. Testing is Crucial:

• Real-world testing (e.g., 200 requests in 5 minutes) validated the cost savings.

Next Steps

13. Monitor Costs:

• Continuously track costs after implementing the optimized configuration.

14. Evaluate GPU Options:

• Test if a lower-cost GPU (e.g., Nvidia T4) can handle the workload.