Assuming serverless is out if the question for your use case, have you tried spending a couple of days investigating a managed Kubernetes cluster with node autoscaling enabled? EKS, AKS, GKE...
Honestly it sounds like you could be at the point where K8s is worthwhile.