After deploying the DEMAND ES Azure Web App in PaaS, we started seeing:

• Slow responses from ElasticSearch.

• ElasticSearch node overload.

• Search thread pool queue saturation.

• 429 errors and intermittent downtime.

However, this issue never happened when the same workload ran in VMs. Previously this was an IIS hosted APIs in 4 different VMs. We migrated this API in Azure Web App.

Why This Never Happened in VMs (IaaS):

Now Recommendation APIs run in these four servers:

·       VW800ABC

·       VW800EFG

·       VW800HIJ

·       VW800LMN

Each VM has:

• Its own dedicated outbound IP address.

• Its own full range of network ports.

• Direct connections to ElasticSearch.

• No shared outbound networking.

Load Balancer Role (Azure ALB):

• The Common Elastic URL points to an Azure Load Balancer in front of the three ElasticSearch nodes.

• ALB distributes requests evenly across all three nodes based on the source IP and source port.

• Since each VM has a unique IP and many available ports, requests naturally spread across all nodes.

Result: No single ElasticSearch node becomes overloaded, even during heavy traffic.
Outcome: IAAS-based apps never experienced slowness or 429 errors.

Why the Problem Appeared Only After Migration to PaaS (Web Apps):

PaaS Setup:
Web App (5 instances) ---> Common Elastic URL ---> Azure Load Balancer ---> ES Cluster (3 nodes)

alb does not support round robin, so one particular node went slow/down. Node hotspotting occurs only in PaaS because of shared IP + limited ports, not because of ElasticSearch or code issues.

When I check with chatgpt, it seems Azure Application Gateway is not the recommended approach.

public class ElasticSearchManager : IElasticSearchManager
    {
        private readonly IConfiguration _configuration;
        private readonly IKeyVaultHelper _keyVaultHelper;

        private readonly Lazy> _lazyClient;

        public ElasticSearchManager(IConfiguration configuration, IKeyVaultHelper keyVaultHelper)
        {
            _configuration = configuration;
            _keyVaultHelper = keyVaultHelper;

            _lazyClient = new Lazy>(InitializeClientAsync);
        }

        private async Task InitializeClientAsync()
        {
            var password = await _keyVaultHelper.GetSecretAsync(
                _configuration["Values:Password"],
                _configuration["Values:AzureKeyVaultURL"]);

            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;

            var pool = new SingleNodeConnectionPool(new Uri(_configuration["Values:ElasticURL"]));

            var settings = new ConnectionSettings(pool)
                .DefaultIndex(_configuration["Values:ResourceIndex"])
                .BasicAuthentication(_configuration["Values:SuperUser"], password)
                .EnableHttpCompression()
                .DisableAutomaticProxyDetection()
                .SniffOnStartup(false)
                .SniffOnConnectionFault(false)
                .SniffLifeSpan(TimeSpan.Zero)
                .DisableDirectStreaming(false)
                .RequestTimeout(TimeSpan.FromSeconds(60))
                .PingTimeout(TimeSpan.FromSeconds(3))
                .EnableTcpKeepAlive(TimeSpan.FromMinutes(2), TimeSpan.FromSeconds(20))
                .MaximumRetries(0)
                .MaxRetryTimeout(TimeSpan.FromSeconds(40))
                .ConnectionLimit(300)
                .ServerCertificateValidationCallback((o, cert, chain, errors) => true);

            return new ElasticClient(settings);
        }

        public async Task GetClientAsync()
        {
            return await _lazyClient.Value;
        }
    }

Nodes:
10.2xx.xxx.89
10.2xx.xxx.90
10.2xx.xxx.91