DNS Server Reserves 2500 Ports.

After applying the patch for MS08-037 - KB 953230 (the multi-OS DNS flaw found by Dan Kaminski), you may notice your Windows Server 2003 machine gets a little greedy. At least, mine sucks up 2500 - yes, that's two thousand five hundred - UDP sockets sitting there apparently waiting for incoming packets.

Output of 'netstat -bona -p udp' command, showing ports bound by DNS.EXE

This is, apparently, one of those behaviours sure to be listed in the knowledge base as "this behavior is by design" - a description that graces some of the more entertaining elements of the Microsoft KB.

Why does this happen? I can only guess. But here's my best guess.

The fix to DNS, implemented across multiple platforms, was to decrease the chance of an attacker faking a DNS response, by increasing the randomness in the DNS requests that has to be copied back in a response.

I don't know how this was implemented on other platforms, but I do know that it's already been reported that BIND's implementation is slower than it used to be (hardly a surprise, making random numbers is always slower than simply counting up) - and maybe that's what Microsoft tried to forestall in the way that they create the random sockets.

Instead of creating a socket and binding it to a random source port at the time of the request, Microsoft's patched DNS creates 2500 sockets, each bound to a random source port, at the time that the DNS service is started up. This way, perhaps they're avoiding the performance hit that BIND has been criticised for.

There are, of course, other services that also use a UDP port. ActiveSync's connection to Exchange, IPsec, IAS, etc, etc. Are they affected?

Sometimes.

Randomly, and without warning or predictability. Because hey, the DNS server is picking ports randomly and unpredictably.

[Workaround: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ReservedPorts is a registry setting that lists multiple port ranges that will not be used when binding an ephemeral socket. The DNS server will obey these reservations, and not bind a socket to ports specified in this list. More explanation in the blog linked above, or at http://support.microsoft.com/kb/812873]

DNS, you see, is a fundamental underpinning of TCP/IP services, and as such needs to start up before most other TCP/IP based services. So if it picks the port you want, it gets first pick, and it holds onto that port, preventing your application from binding to it.

This just doesn't seem like a fix written by someone who 'gets' TCP/IP. Perhaps I'm missing something that explains why the DNS server in Windows Server 2003 works this way, but I would be inclined to take the performance hit of binding and rebinding in order to find an unused random port number, rather than binding before everyone else in an attempt to pre-empt other applications' need for a port.

There are a couple of reasons I say this:

  1. Seriously, how many Windows Server 2003 users out there have such a high-capacity DNS server that they will notice the performance hit?
  2. Most Windows Server 2003-based DNS servers are small caching servers for businesses, rather than Internet infrastructure servers responsible for huge numbers of requests per second - even if you implement this port-stealing method, it shouldn't be the default, because the majority of users just don't need that performance.
  3. If you do need the performance, get another server to handle incoming requests. Because the cost of having your DNS server's cache poisoned is considerably greater than the cost of increasing the number of servers in your pool, if you're providing major DNS service to that many customers.
  4. A major DNS service provider will be running fewer services that would pre-empt a DNS server request to bind to a random port, whereas systems running several UDP-based services are going to need less performance on their outgoing DNS requests.

I'd love to know if I'm missing something here, but I really hope that Microsoft produces a new version of the DNS patch soon, that doesn't fill your netstat -a output with so many bound and idle sockets, each of which takes up a small piece of nonpaged pool memory (that means real memory, not virtual memory).

Published Sat, Jul 19 2008 0:02 by Alun Jones

Comments

# re: DNS Server Reserves 2500 Ports.

agreed. This may cause issues to a lot of users.

Sunday, July 20, 2008 3:33 PM by Yoav

# re: DNS Server Reserves 2500 Ports.

It's not much better if you're using BIND as your DNS server, apparently (mostly in the Linux world).

BIND can't tell when it's selecting a port that's already bound by another application, so it randomly, unpredictably, and without warning will send out a request on a port that causes the response to come back to an application other than BIND's "named".

[For discussion of this problem, see: this Usenet thread]

That's all very well for applications that are used to being exposed to the Internet - after all, they receive random junk packets all the time, so they should be resilient.

But what about apps that aren't used to Internet exposure? Remember - the DNS server sends out a packet first, so it will cause most firewalls to open up a hole for the return packet.

Any UDP-based service on your DNS server will receive random UDP data, whether you've blocked it at the firewall or not.

Just like in Windows with the ReservedPorts option, there's a way to work around this - there's a new setting for named called "avoid-v4-udp-ports" that you can use to list ports that named shouldn't bind to for its outgoing requests.

Sunday, July 20, 2008 6:32 PM by Alun Jones

# re: DNS Server Reserves 2500 Ports.

You're absolutely correct.  Sadly.

Remember that MSDNS still needs to pick random numbers at some point here, we still need to randomize requests among the various available ports, so we're not saving any CPU time at all by doing the randomization first, although we are probably reducing DNS query time by a couple MS.  We could achieve the same performance improvement by building a list, but not pre-binding.

Worse, by pre-binding to 2500 ports, we might even be helping an attacker map out which ports are and are not used by DNS.  Sure, sending out 2500 times more spoofed packets then before sounds like a lot, but with zombie farms ranging in the millions, getting an important DNS poison (PayPal anyone?) cached is probably not outside the scope.

If we build a random map out in advance, but don't bind until needed, an attacker would still need to flood some ~65000 ports.  Oh well, 2500 is better then 1.

The other performance improvement is that by pre-binding, rather then binding on-demand, we don't need to confirm that a binding is successful before sending out a DNS query.  Good, also saves a ms or two.  A better fix might be to have the server pre-bind a small number of ports in advance, only using a single port once (or at least only sending it to a single destination IP once, you can probably reuse a remote-DNS-IP:local-port pair pretty safely, since this whole random port scheme does nothing for MITM attacks anyway.

Anyway, an effective pre-binding algorithm could avoid clobbering ports other applications need, plus avoid the performance impact of binding on demand.  The numbers may need to be tweaked, but start with this: For the first three minutes of operation, you'd only pre-bind five ports.  These ports could be anywhere in a large range, maintaining some level of randomness, but don't forget a comprehensive list of known-services to avoid, maybe Windows' own "%systemroot%\system32\drivers\etc\services" list would be a good place to start.

After three minutes, larger numbers of ports can be safely allocated without difficulty, since services have now had an opportunity to start and do their initial bindings.  At this stage, MSDNS could decide how many ports to pre-bind allowing for MSDNS to stay 30-60 seconds ahead -- Enough time to allow for a bump in activity, but not enough time for an attacker to build a map.

Thoughts?

Sunday, July 20, 2008 11:49 PM by The Dave

# re: DNS Server Reserves 2500 Ports.

What about some active evasion, as well - detect when you might be being flooded with requests, and either refuse to serve such requests (discarding any responses that you might already have cached from forged responses), or serve them using a more protected connection, such as a TCP connection to the remote DNS server?

I'm sure that the DNS folks at Microsoft have spent a lot of time talking with other DNS server authors and coming to some consensus as to what's a good approach - and it's always possible that I'm missing something huge and important in my analysis.

What I'd like to see is a new Winsock socket option, to say "when I call bind() on this socket, I want you to give me a pseudo-random (not N+1) port assignment, that isn't currently in use by another application" - FTP would benefit from this, because PASV responses are supposed to bind to an unpredictable port to improve security.

I do like the idea of leaving DNS less responsive (or even non-responsive to external queries) until a significant time after booting, though.

Another angle that I don't think has been considered is to tailor the approach to two scenarios - one of a high-traffic DNS server, which is probably almost exclusively running DNS as its only service; the other of a DNS server that runs multiple other services. I think the threats can be handled differently, because the level of service to be provided is different.

Monday, July 21, 2008 12:39 AM by Alun Jones

# re: DNS Server Reserves 2500 Ports.

There seems to be a Registry option to control the number of ports reserved.  It's

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DNS\Parameters\SocketPoolSize

See support.microsoft.com/.../953230

As I understand it, Vista and Server 2008 now use the original BSD intended behaviour of assigning ephemeral ports from above 5000, support.microsoft.com/.../929851  I wonder if the current DNS Server code assigns ports only from that range on those platforms, that would at least solve the conflicts described in the SBS blog (1024<p<5000 vs emphemeral >=5000).

I agree with your points however; I particularly like the Winsock-give-me-a-randomized-port-assignment option.

Alan

Blast from the past... 

Alan, yours is a name I remember from relatively early on in the development of Winsock - as another developer trying to use and make sense of the darn thing, if I remember correctly.

Yeah, I like the idea of providing for other protocols which also need to bind to random ports (DNS isn't alone, because FTP needs it - and where two protocols need a behaviour, you can bet that there's another half dozen at least) - I was going to suggest that ephemeral ports' behaviour should change and become random assignment, because "of course" developers knew that they couldn't guarantee any particular assignment sequence, let alone one after the other.

But then sanity reined me in, and I realised that of course developers - particularly the cheap ones that get the job done quickly - tend to rely on observed behaviour more than documentation or training. So I thought it'd better be an option. And a socket option makes sense - call it between socket() and bind() / connect(), and you have an answer.

Sunday, July 20, 2008 4:32 PM by Alun Jones

Tuesday, July 22, 2008 10:10 AM by Alan McFarlane

# re: DNS Server Reserves 2500 Ports.

There is another big problem!

Then ms dns running on server that have NAT role (in my case 2003 + ISA 2006 sp1 + ms dns for caching and forwading to ISP DNS and PPPoE connection to ISP)

Then users send dns request to local dns server -> forwad to ISP DNS, but NAT cant create UDP sockets, becouse sockets from previous reguests still alive.

In my case im set SocketPoolSize=16 and now all work fine.

Not really solving the problem 

Sure, that fixes the problem of DNS causing your system to run out of ephemeral ports - but it does so by undoing almost all of the work of patching your DNS system.

Far more useful than reducing the size of socket pool used by DNS would be to increase the number of ports assigned for ephemeral ports. I believe the registry setting that you'll want to investigate is MaxUserPorts - on pre-Vista systems, this is set to 5000, so that ephemeral sockets exist at ports between 1024 and 5000.

As you've seen, take a few thousand out of that range, especially if you have to double the number through NAT, and you're hurting. So extend the range, and you'll find DNS is still patched against this exploit. [Don't forget to put the pool size back]

Friday, July 25, 2008 8:42 PM by Alun Jones

Friday, July 25, 2008 6:47 AM by Ivan_83

# SOLVING! :)

MaxUserPort - was 65535.

And problem with dns + nat (on ISA) start before that path (i detect them year ago or early).

Sunday, July 27, 2008 3:22 PM by Ivan_83

# re: DNS Server Reserves 2500 Ports.

Alun, is dns on your server really binding to the registered ports range (1024 through 49151)?  On my server it's only binding to the dynamic/private range (49152 through 65535).

Monday, July 28, 2008 4:14 PM by Harry Johnston

# re: DNS Server Reserves 2500 Ports.

That behaviour depends on whether you're running a pre-Vista or post-Vista operating system, as Microsoft's KB notes.

Monday, July 28, 2008 4:55 PM by Alun Jones

# re: DNS Server Reserves 2500 Ports.

According to KB953230, update 953230 changes Windows 2003 to have the behaviour I observed (and hence shouldn't have issues with most other software) unless MaxUserPort is set.

Are there situations in which MaxUserPort is likely to be set?

Monday, July 28, 2008 8:09 PM by Harry Johnston

# re: DNS Server Reserves 2500 Ports.

Yes - if you're running on Windows SBS 2003, or have other applications that already set MaxUserPort, or if your system administrator determined that the default ephemeral port range of 1024-5000 was not appropriate (either too small or too large).

Tuesday, July 29, 2008 7:54 AM by Alun Jones

# thousands of dns.exe UDP connections, what to do? | keyongtech

Pingback from  thousands of dns.exe UDP connections, what to do? | keyongtech

# re: DNS Server Reserves 2500 Ports.

blogs.technet.com/sseshad/archive/2008/12/03/windows-dns-and-the-kaminsky-bug.aspx

How do you control the value of this setting?

Ø  Dnscmd /Info /SocketPoolSize will tell you the current size of the socket pool

Ø  Dnscmd /Config /SocketPoolSize <val> sets the socket pool size to #val

Ø  Once you reset the size, you must restart the DNS service.  Use net stop dns to stop the service, and net start dns to restart the service.  Once the service has restarted, the new socket pool will come into effect.

Friday, March 20, 2009 9:37 AM by stopka2top

# re: DNS Server Reserves 2500 Ports.

what should i set the socketpoolsize for optimal performance?

Wednesday, March 25, 2009 12:02 PM by hanna

Leave a Comment

(required) 
(required) 
(optional)
(required) 
If you can't read this number refresh your screen
Enter the numbers above: