Getting a 403.501 error response from IIS? Check your 'Dynamic IP Restriction Settings'

One of the applications I inherited is composed of multiple applications and uses one of them as a central authentication solution (issuing tokens on login, refreshing them, revoking them, etc,..) that's exposed in IIS as a web application that's only available internally. Unfortunately we were seeing requests to this fail, sometimes returning a 403.501 response code and sometimes failing to respond at all, when under load. This was particularly noticeable when using an automated test tool to place the system under load as an attempt to make 20 simultaneous requests would see roughly 15 of them fail - sometimes a little less than 15, sometimes a little more. The variance was likely due to the rather naïve way the testing tool was implemented, being a quick and dirty solution it was a simple C# console app that spun up a number of threads, started them and then waited for them to all finish so that the result could be reported.

After much head scratching it turns out that the application had 'Dynamic IP Restriction Settings' configured and enabled in IIS. This throttles/blocks a high number of requests from the same IP address which is a useful tool to have when you're dealing with requests from the outside world which will likely be coming from a wide variety of different IP addresses.

If you've got a web application running that's only accessed internally from a very small number of IP addresses (or you've got a WAF or similar in front of your web servers that doesn't forward on the source IP), such as those of the two web servers that host the applications that talk to it,... Not so much! Don't use Dynamic IP Restriction settings on web applications / sites that aren't surfaced to the outside world!

The above all should be obvious (though apparently to some it wasn't!), hopefully this post will provide a bit of search juice to help someone else track this down much more quickly in the future.