I have a EMR spark cluster, on which I have enabled EMR managed auto scaling as auto scaling configuration.
and primary - c5a.xlarge
Core - c5a.xlarge
Task - c5a.xlarge
With these cluster configuration, I submitted a set up spark-submit job as step with concurrency as 2. with this I observed resizing started from 0 to 6 instances. But it got stuck in same phase for 3+hrs as 0 (6 requested) but not scaling up and not starting new instances.
And after almost 3hrs and 30 mins the cluster got auto terminated with internal error without running any steps. And also there is no logs generated for steps execution.
getting no clue on this EMR behavior. And How to solve this issue?
Thanks in advance