Tales from the Crypto : FTP

My take on the SSL MITM Attacks – part 3 – the FTPS attacks

Alun Jones — Thu, 19 Nov 2009 05:02:00 GMT

[Note - for previous parts in this series, see Part 1 and Part 2.]

FTP, and FTP over SSL, are my specialist subject, having written one of the first FTP servers for Windows to support FTP over SSL (and the first standalone FTP server for Windows!)

Rescorla and others have concentrated on the SSL MITM attacks and their effects on HTTPS, declining to discuss other protocols about which they know relatively far less. OK, time to step up and assume the mantle of expert, so that someone with more imagination can shoot me down.

FTPS is not vulnerable to this attack.

No, that’s plainly rubbish. If you start thinking along those lines in the security world, you’ve lost it. You might as well throw in the security towel and go into a job where you can assume everybody loves you and will do nothing to harm you. Be a developer of web-based applications, say. :-)

FTPS has a number of possible vulnerabilities

And they are all dependent on the features, design and implementation of your individual FTPS server and/or client. That’s why I say “possible”.

Attack 1 – renegotiation with client certificates

The obvious attack – renegotiation for client certificates – is likely to fail, because FTPS starts its TLS sessions in a different way from HTTPS.

In HTTPS, you open an unauthenticated SSL session, request a protected resource, and the server prompts for your client certificate.

In FTPS, when you connect to the control channel, you provide your credentials at the first SSL negotiation or not at all. There’s no need to renegotiate, and certainly there’s no language in the FTPS standard that allows the server to query for more credentials part way into the transaction. The best the server can do is refuse a request and say you need different or better credentials.

Attack 2 – unsolicited renegotiation without credentials

A renegotiation attack on the control channel that doesn’t rely on making the server ask for client credentials is similarly unlikely to succeed – when the TLS session is started with an AUTH TLS command, the server puts the connection into the ‘reinitialised’ state, waiting for a USER and PASS command to supply credentials. Request splitting across the renegotiation boundary might get the user name, but the password wouldn’t be put into anywhere the attacker could get to.

Attack 3 – renegotiating the data connection

At first sight, the data connection, too, is difficult or impossible to attack – an attacker would have to guess which transaction was an upload in order to be able to prepend his own content to the upload.

But that’s betting without the effect that NATs had on the FTP protocol.

Because the PORT and PASV commands involve sending an IP address across the control channel, and because NAT devices have to modify these commands and their responses, in many implementations of FTPS, after credentials have been negotiated on the control channel, the client issues a “CCC” command, to drop the control channel back into clear-text mode.

Yes, that’s right, after negotiating SSL with the server, the client may throw away the protection on the control channel, so the MitM attacker can easily see what files are going to be accessed over what ports and IP addresses, and if the server supports SSL renegotiation, the attacker can put his data in at the start of the upload before renegotiating to hand off to the legitimate client. Because the client thinks everything is fine, and the server just assumes a renegotiation is fine, there’s no reason for either one to doubt the quality of the file that’s been uploaded.

How could this be abused? Imagine that you are uploading an EXE file, and the hacker prepends it with his own code. That’s how I wrote code for a ‘dongle’ check in a program I worked on over twenty years ago, and the same trick could still work easily today. Instant Trojan.

There are many formats of file that would allow abuse by prepending data. CSV files, most exploitable buffer overflow graphic formats, etc.

Attack 3.5 – truncation attacks

While I’m on FTP over SSL implementations and the data connection, there’s also the issue that most clients don’t properly terminate the SSL connection in FTPS data transfers.

As a result, the server can’t afford to report as an error when a MitM closes the TCP connection underneath them with an unexpected TCP FIN.

That’s bad – but combine it with FTP’s ability to resume a transfer from part-way into a file, and you realize that an MitM could actually stuff data into the middle of a file by allowing the upload to start, interrupting it after a few segments, and then when the client resumed, interjecting the data using the renegotiation attack.

The attacker wouldn’t even need to be able to insert the FIN at exactly the byte mark he wanted – after all, the client will be sending the REST command in clear-text thanks to the CCC command. That means the attacker can modify it, to pick where his data is going to sit.

Not as earth-shattering as the HTTPS attacks, but worth considering if you rely on FTPS for data security.

How does WFTPD Pro get around these attacks?

1. I never bothered implementing SSL / TLS renegotiation – didn’t see it as necessary; never had the feature requested. Implementing unnecessary complexity is often cause for a security failure.

2. I didn’t like the CCC command, and so I didn’t implement that, either. I prefer to push people towards using Block instead of Stream mode to get around NAT restrictions.

I know, it’s merely fortunate that I made those decisions, rather than that I had any particular foresight, but it’s nice to be able to say that my software is not vulnerable to the obvious attacks.

I’ve yet to run this by other SSL and FTP experts to see whether I’m still vulnerable to something I haven’t thought of, but my thinking so far makes me happy – and makes me wonder what other FTPS developers have done.

I wanted to contact one or two to see if they’ve thought of attacks that I haven’t considered, or that I haven’t covered. So far, however, I’ve either received no response, or I’ve discovered that they are no longer working on their FTPS software.

Let me know if you have any input of your own on this issue.

How FTP Data Connections Work Part 2 (OR: Fun With Port 20)

Alun Jones — Tue, 14 Jul 2009 06:48:26 GMT

As we mentioned in the 1st part of this series, FTP is a more complex protocol than many, using one control connection and one data connection.

A recap of the first post…

In typical Stream Mode operation, a new data connection is opened and closed for each data transfer, whether that’s an upload, a download, or a directory listing. To avoid confusion between different data connections, and as a recognition of the fact that networks may have old packets shuttling around for some time, these connections need to be distinguishable from one another.

In the previous article, we noted that two network sockets are distinguished by the five elements of “Local Address”, “Local Port”, “Protocol”, “Remote Address”, and “Remote Port”. For a data connection associated with any particular request, the local and remote addresses are fixed, as the addresses of the client and server. The protocol is TCP, and only the two ports are variable.

For a PASV, or passive data connection, the client-side port is chosen randomly by the client, and the server-side port is similarly chosen randomly by the server. The client connects to the server.

For a PORT, or active data connection, the client-side port is chosen randomly by the client, and the server-side port is set to port 20. The server connects to the client.

All of these work through firewalls and NAT routers, because firewalls and NAT routers contain an Application Layer Gateway (ALG) that watches for PORT and PASV commands, and modifies the control (in the case of a NAT) and/or uses the values provided to open up a firewall hole.

Isn’t there a totally predictable data connection?

For the default data connection (what happens if no PORT or PASV command is sent before the first data transfer command), the client-side port is predictable (it’s the same as the source port the client used when connecting the control channel), and the server-side port is 20. Again, the server connects to the client.

Because firewalls and NATs open up a ‘reverse’ hole for TCP sockets, the default data port works with firewalls and NATs that aren’t running an ALG, or whose ALG cannot scan for PORT and PASV commands.

Why would an ALG stop scanning for PORT and PASV commands?

There are a couple of reasons – the first is that it doesn’t know that the service connected to is running the FTP protocol. This is common if the server is running on a port other than the usual port 21.

The second reason is that the FTP control connection doesn’t look like it contains FTP commands – usually because the connection is encrypted. This can happen because you’re tunneling the FTP control connection through an encrypted tunnel such as SSH (don’t laugh – it does happen!), or hopefully it’s because you’re running FTP over SSL, so that the control and data connections can be encrypted, and you can authenticate the identity of the FTP server.

So how do you get FTP over SSL to work through a firewall?

In the words of Deep Thought: “Hmm… tricky”.

There are a couple of classic solutions:

Allow PASV data connections, select a wide range of ports, and open that range for incoming traffic from all external addresses in your firewall configuration; hope that your FTP server can be configured to use only that range of ports (WFTPD Pro can), and that it has protections against traffic stealing attacks (again, WFTPD Pro has). Still, this option seems really risky.
Block all PASV connections, and make the clients responsible for opening up holes in their firewalls. If you’re convinced the risk is too great to do this on your server, how does it look to convince your users that they should accept that risk?
After you’ve authenticated the server and provided your username and password in the encrypted control connection, issue the “CCC” (Clear Control Channel) command, to switch the control connection back into clear-text. I dislike this as a solution, because it requires the ALG pay attention to a lot of SSL traffic in the hope that there might be clear-text coming up, and because you may want the control channel to remain encrypted.

Awright, clever clogs, you solve the problem.

The astute reader can probably see where I’m going with this.

The default data port is predictable – if the client connects from port U to port L at the server (L is usually 21), then the default data port will be opened from port L-1 at the server to port U at the client.

The default data port doesn’t need the firewall to do anything other than allow reverse connections back along the port that initiated the connection. You don’t need to open huge ranges at the server’s firewall (in fact you should be able to simply open port 21 inbound to your server).

The default data port is required to be supported by FTP servers going back a long way- at least a couple of decades. Yes, really, that long.

If it’s that simple, why isn’t everyone doing it?

Good point, that, and a great sentence to use whenever you wish to halt innovation in its tracks.

Okay, it’s obvious that there are some drawbacks:

In stream mode, the data transfer is ended by closing the stream. This means that you have to open a new control connection. Not good, given the number of round-trips you need for a logon, and the work needed to start an SSL connection.
Most FTP clients view the default data connection as, at best, a fail-over in case the PORT or PASV commands fail to work. Obviously, that means it’s not likely to be a well-tested or favoured solution on these clients.

Even with those drawbacks, there are still further solutions to apply – the first being to use Block-mode instead of Stream-mode. In Stream-mode, each data transfer requires opening and closing the data connection; in Block-mode, which is a little like HTTP’s chunked mode, blocks of data are sent, and followed by an “EOF” marker (End of File), so that the data connection doesn’t need to be closed. If you can convince your FTP client to request Block-mode with the default data connection, and your FTP server supports it (WFTPD Pro has done so for several years), you can achieve FTP over SSL through NATs and firewalls simply by opening port 21.

For the second problem, it’s worth noting that many FTP client authors implemented default data connections out of a sense of robustness, so default data connections will often work if you can convince the PORT and PASV commands to fail – by, for instance, putting restrictive firewalls or NATs in the way, or perhaps by preventing the FTP server from accepting PORT or PASV commands in some way.

Clearly, since Microsoft’s IIS 7.5 downloadable FTP Server supports FTPS in block mode with the default data port, there has been some consideration given to my whispers to them that this could solve the FTP over SSL through firewall problem.

Other than my own WFTPD Explorer, I am not aware of any particular clients that support the explicit use of FTP over SSL with Block-mode on the default data connection – I’d love to hear of your experiments with this mode of operation, to see if it works as well for you as it does for me.

How FTP Data Connections Work Part 1 (OR: Don’t Open Port 20 in your Firewall!)

Alun Jones — Thu, 09 Jul 2009 06:18:42 GMT

This will be the first of a couple of articles on FTP, as I’ve been asked to post this information in an easy-to-read format in a public place where it can be referred to. I think my expertise in developing and supporting WFTPD and WFTPD Pro allow me to be reliable on this topic. Oh, that and the fact that I’ve contributed to a number of RFCs on the subject.

Enough TCP to be dangerous

First, a quick refresher on TCP – every TCP connection can be thought of as being associated with a “socket” at each device along the way – from one computer, through routers, to the other computer. The socket is identified by five individual items – the local IP address, the local port, the remote IP address, the remote port, and the protocol (in this case, the protocol is TCP).

Firewalls are essentially a special kind of router, with rules not only for how to forward data, but also rules on connection requests to drop or allow. Once a connection request is allowed, the entire flow of traffic associated with that connection request is allowed, also – any traffic flow not associated with a previously allowed connection request is discarded.

When you set up a firewall to allow access to a server, you have to consider the first segment – the “SYN”, or connection request from the TCP client to the TCP server. The rule can refer to any data that would identify the socket to be created, such as “allow any connection request where the source IP address is 10.1.1.something, and the destination port is 54321”.

Typically, an external-facing firewall will allow all outbound connections, and have rules only for inbound connections. As a result, firewall administrators are used to saying things like “to enable access to the web server, simply open port 80”, whereas what they truly mean is to add a rule that applies to incoming TCP connection requests whose source address and source port could be anything, but whose destination port is 80, and whose destination address is that of the web server.” This is usually written in some short hand, such as “allow tcp 0.0.0.0:0 10.1.2.3:80”, where “0.0.0.0” stands for “any address” and “:0” stands for “any port”.

Firewall rules for FTP

For an FTP server, firewall rules are known to be a little trickier than for most other servers.

Sure, you can set up the rule “allow tcp 0.0.0.0:0 10.1.2.3:21”, because the default port for the control connection of FTP is 21. That only allows the control connection, though.

What other connections are there?

In the default transfer mode of “Stream”, every file transfer gets its own data connection. Of course, it’d be lovely if this data connection was made on port 21 as well, but that’s not the way the protocol was built. Instead, Stream mode data connections are opened either as “Active” or “Passive” connections.

Active and Passive Data Connections

The terms "Active" and "Passive" refer to how the FTP server connects. The choice of connection method is initiated by the client, although the server can choose to refuse whatever the client asked for, at which point the client should fail over to using the other method.

In the Active method, the FTP server connects to the client (the server is the “active” participant, the client just lies back and thinks of England), on a random port chosen by the client. Obviously, that will work if the client's firewall is configured to allow the connection to that port, and doesn't depend on the firewall at the server to do anything but allow connections outbound. The Active method is chosen by the client sending a “PORT” command, containing the IP address and port to which the server should connect.

In the Passive method, the FTP client connects to the server (the server is now the “passive” participant), on a random port chosen by the server. This requires the server's firewall to allow the incoming connection, and depends on the client's firewall only to allow outbound connections. The Passive method is chosen by the client sending a “PASV” command, to which the server responds with a message containing the IP address and port at the server that the client should connect to.

The ALG comes to the rescue!

So in theory, your firewall now needs to know what ports are going to be requested by the PORT and PASV commands. For some situations, this is true, and you need to consider this – we’ll talk about that in part 2. For now, let’s assume everything is “normal”, and talk about how the firewall helps the FTP user or administrator.

If you use port 21 for your FTP server, and the firewall is able to read the control connection, just about every firewall in existence will recognise the PORT and PASV commands, and open up the appropriate holes. This is because those firewalls have an Application Level Gateway, or ALG, which monitors port 21 traffic for FTP commands, and opens up the appropriate holes in the firewall. We’ve discussed the FTP ALG in the Windows Vista firewall before.

So why port 20?

Where does port 20 come in? A rather simplistic view is that administrators read the “Services” file, and see the line that tells them that port 20 is “ftp-data”. They assume that this means that opening port 20 as a destination port on the firewall will allow FTP data connections to flow. By the “elephant repellant” theory, this is proved “true” when their firewalls allow FTP data connections after they open ports 21 and 20. Nobody bothers to check that it also works if they only open port 21, because of the ALG.

OK, so if port 20 isn’t needed, why is it associated with “ftp-data”? For that, you’ll have to remember what I said early on in the article – that every socket has five values associated with it – two addresses, two ports, and a protocol. When the data connection is made from the server to the client (remember, that’s an Active data connection, in response to a PORT command), the source port at the server is port 20. It’s totally that simple, and since nobody makes firewall rules that look at source port values, it’s relatively unimportant. That “ftp-data” in the Services file is simply so that the output from “netstat” has a meaningful service name instead of “:20” as a source port.

Coming up in part 2…

Next time, we’ll expand on this topic, to go into the inability of the ALG to process encrypted FTP control traffic, and the resultant issues and solutions that face encrypted FTP.

FTP - Untrustworthy? I Don't Think So!

Alun Jones — Wed, 30 Jul 2008 04:53:15 GMT

Lately, as if writers all draw from the same shrinking paddling-pool of ideas, I've noticed a batch of stories about how unsafe, unsecure and untrustworthy is FTP.

SC Magazine says so.

First it was an article in the print version of SC Magazine, sadly not repeated online, titled "2 Minutes On... FTP integrity challenged", by Jim Carr. I tried to reach Jim by email, but his bounce message tells me he doesn't work for SC Magazine any more.

This article was full of interesting quotes.

"8,700 FTP server credentials were being used to access and infect more than 2,000 legitimate websites in the US". The article goes on to quote Finjan's director of security research who says they were "most likely hijacked by malware" - since most malware can do keystroke logging for passwords, there's not much can be done at the protocol level to protect against this, so this isn't really an indictment of FTP so much as it is an indication of the value and ubiquity of FTP.

Then we get to a solid criticism of FTP: "The problem with FTP is it transfers data, including authorization credentials, in plain text rather than in encrypted form, says Jeff Debrosse, senior research analyst at security vendor ESET". Okay, that's true - but in much the same vein as saying that the same problems all apply to HTTP.

Towards the end of the article, we return to Finjan's assertion that malware can steal credentials for FTP sites - and as I've mentioned before, malware can get pretty much any user secret, so again, that's not a problem that a protocol such as FTP - or SFTP, HTTP, SSH, SCP, etc - can fix. There's a password or a secret key, and once malware is inside the system, it can get those credentials.

Fortunately, the article closes with a quote from Trent Henry, who says "That means FTP is not the real issue as much as it is a server-protection issue."

OK, But a ZDNet blogger says so, too.

Well, yeah, an article in a recent ZDNet blog entry - on storage, not networking or security (rather like getting security advice from Steve Gibson, a hard-drive expert) - rants on about how his web site got hacked into (through WordPress, not FTP), and as a result, he's taken to heart a suggestion not to use FTP.

Such a non-sequitur just leaves me breathless. So here's my take:

FTP Has Been Secure for Years

But some people have just been too busy, or too devoted to other solutions, to take notice.

FTP first gained secure credentials with the addition of support for SASL and SKey. These are mechanisms for authenticating users without passing a password or password-equivalent (and by "password-equivalent", I'm including schemes where the hash is passed as proof that you have the password - an attacker can simply copy the hash instead of the password). These additional authentication methods give FTP the ability to check identity without jeopardising the security of the identified party. [Of course, prior to this, there were IPsec and SOCKS solutions that work outside of the protocol.]

OK, you might say, but that only protects the authentication - what about the data?

FTP under GSSAPI was defined in RFC 2228, which was published in October 1997 (the earliest draft copy I can find is from March 1995), from a draft developed over the preceding couple of years. What's GSSAPI? As far as anyone really needs to know, it's Kerberos.

This inspired the development of FTP over SSL in 1996, which became FTP over TLS, and which finally became RFC 4217. From 1997 to 2003, those of us in the FTPExt Working Group were wondering why the standard wasn't yet an RFC, as draft after draft were submitted with small changes, and then apparently sat on by the RFC editor - during this time, several compatible FTP clients, servers and proxies were produced that compatibly supported FTP over TLS (and/or SSL).

Why so long from draft to publication?

One theory that was raised is that the IETF were trying to get SSH-based protocols such as SFTP out before FTP over TLS (which has become known as "FTPS", for FTP over SSL).

SFTP was abandoned after draft 13, which was made available in July 2006; RFC 4217 was published in October 2005. So it seems a little unlikely that this is the case.

The more likely theory is simply that the RFC Editor was overworked - the former RFC Editor, Jon Postel, died in 1998, and it's likely that it took some time for the new RFC Editor to sort all the competing drafts out, and give them his attention.

What did the FTPExt Working Group do while waiting?

While we were waiting for the RFC, we all built compatible implementations of the FTP over TLS standard.

One or two of us even tried to implement SFTP, but with the draft mutating rapidly, and internal discussion on the SFTP mailing list indicating that no-one yet knew quite what they wanted SFTP to be when it grew up, it was like nailing the proverbial jelly to a tree. Then the SFTP standardisation process ground to a halt, as everyone lost interest. This is why getting SFTP implementations to interoperate is sometimes so frustrating an experience.

FTPS, however - that was solidly defined, and remains a very compatible protocol with few relevant drawbacks. Sadly, even FTP under GSSAPI turned out to have some reliability issues (the data transfer and the control connection, though over different asynchronous channels, share the same encryption context, which means that the receiver must synchronise the two asynchronous channels exactly as the sender did, or face a loss of connection) - but FTP over TLS remains strong and reliable.

So, why does no-one know about FTPS?

Actually, there's lots of people that do - and many clients and servers, proxies and tunnels, exist in real life implementations. Compatibility issues are few, and generally revolve around how strict servers are about observing the niceties of the secure transaction.

Even a ZDNet blogger or two has come across FTPS, and recommends it, although of course he recommends the wrong server.

My recommendation?

WFTPD Pro. Unequivocally. Because I know who wrote it, and I know what went into it. It's all good stuff.

Vistafy Me.

Alun Jones — Fri, 11 Jul 2008 05:07:25 GMT

I have a little time over the next couple of weeks to devote to developing WFTPD a little further.

This is a good thing, as it's way past time that I brought it into Vista's world.

I've been very proud that over the last several years, I have never had to re-write my code in order to make it work on a new version of Windows. Unlike other developers, when a new version of Windows comes along, I can run my software on that new version without changes, and get the same functionality.

The same is not true of developers who like to use undocumented features, because those are generally the features that die in new releases and service packs. After all, since they're undocumented, nobody should be using them, right? No, seriously, you shouldn't be using those undocumented features.

So, WFTPD and WFTPD Pro run in Windows Vista and Windows Server 2008.

But that's not enough. With each new version of Windows, there are better ways of doing things and new features to exploit. With Windows Vista and Windows Server 2008, there are also a few deprecated older behaviours that I can see are holding WFTPD and WFTPD Pro down.

I'm creating a plan to "Vistafy" these programs, so that they'll continue to be relevant and current.

Here's my list of significant changes to make over the next couple of weeks:

Convert the Help file from WinHelp to HTML Help.

WinHelp is not supported in Vista - you can download a WinHelp version, but it's far better to support the one format of Help file that Windows uses. So, I'm converting from WinHelp to HTML Help.

Changing the Control Panel Applet for WFTPD Pro.

CPL files still work in Windows Vista, but they're considered 'old', and there's an ugly user experience when it comes to making them elevate - run as administrator.
There are two or three ways to go here -

one is to create an EXE wrapper that calls the old CPL file. That's fairly cheap, and will probably be the first version.
Another is to write an MMC plugin. That's a fair amount of work, and requires some thought and design. That's going to take more than a couple of weeks.
A third option is to create some form of web-based interface. I don't want to go that way, because I don't want to require my users to install IIS or some other web server.

So, first blush it seems will be to wrap the existing interface, and secondly I'll be investigating what an MMC should look like.

Support for IPv6.

I already have this implemented in a trial version, but have yet to fully wire it up to a user interface that I'm willing to unleash on the world. So that's on the cards for the next release.

Multiple languages

There are two elements to support for multiple languages in FTP:

File names in non-Latin character sets
Text messages in languages other than English

The first, file names in different character sets, will be achieved sooner than the second. If the second ever occurs, it will be because customers are sufficiently interested to ask me specifically to do it.

SSL Client Certificate authentication

SSL Client Certificate Auth has been in place for years - it's a secret feature. The IIS guys warned me off developing it, saying "that's really hard, don't try and do anything with client certs".
I didn't have the heart to tell them I had the feature working already (but without an interface), and that it simply required a little patience.

Install under Local Service and Network Service accounts
Build in Visual Studio 2008, to get maximum protection using new compiler features.

/analyze, Address Space Layout Randomisation, SAL - all designed to catch my occasional mistakes.

As I work on each of these items, I'll be sure to document any interesting behaviours I find along the way. My first article will be on converting your WinHelp-using MFC project to using HTML Help, with minimal changes to your code, and in such a way that you can back-pedal if you have to.

Of course, I also have a couple of side projects - because I've been downloading a lot from BBC 7, I've been writing a program to store the program titles and descriptions with the MP3 files, so that they show up properly on the MP3 player. ID3Edit - an inspired name - allows me to add descriptions to these files.

Another side-project of mine is an EFS tool. I may use some time to work on that.

Vista's Secret Windows Firewall hole

Alun Jones — Fri, 25 Jan 2008 05:19:00 GMT

First, the good news - it's not a flaw in the operation of Windows Firewall on Windows Vista. It's a design feature, it makes sense, and it fits in with the principle that the firewall should keep out unsolicited traffic. It's not really a hole, but I thought I'd grab your attention.

The symptom first came up in a Usenet posting (thanks, Jesper, for bringing me in) about Vista and a third-party FTP client:

When I do a directory listing, and a PORT command is issued, and the
server attempts to connect, it works, but at the same time a dialogue
appears telling me it's blocked, and I can keep blocking or unblock.
I choose keep blocking but it doesn't actually block it once.

Here's how it looks.

First, if you haven't got a third-party FTP client let's fake it, by copying Microsoft's command-line FTP client from the Windows System32 directory to another directory:

C:\users\MyMe> copy %windir%\system32\ftp.exe
1 file(s) copied.

The FTP client will not display prompts to you, but that's a minor issue - if it upsets you, try downloading a third-party client and trying it.

Anyway, here we go - let's try the issue in question:

Type ftp ftp.microsoft.com
After you see the "220" greeting message, enter ftp as the user - press enter.
Now you're prompted for a password - enter anything and press enter.
Once you're logged on, enter dir - again, press enter.
You'll see the directory listing succeed, but you'll also see a warning that a connection is being blocked:

Wow - that's freaky - at the same time you're being told that the connection used for the file listing will be blocked, it allows the connection through!

What's more, even if you specify Keep Blocking, and then go issue another dir command, that one succeeds.

Huh? And why on earth did you make me use a copy of FTP?

Let's go look at the Windows Advanced Firewall Rules for Inbound, and see if this sheds any light:

[That means click the Start button, type Firewall into the search box, and right-click on Windows Firewall with Advanced Security - select Run as Administrator

and accept the elevation prompt from UAC. If you don't have an elevation prompt, then you should really re-enable UAC. Now select Inbound Rules in the left-hand pane]

Me, I've got a few rules labeled File Transfer Program:

That first (and fourth) rule is set to block any listening ports opened by the File Transfer Program in C:\users\myme\ftp.exe, the second two seem to be allowing any listening ports created by the one in C:\windows\system32\ftp.exe.

Obviously, that's why I asked you to copy ftp.exe to a new directory, so that any previous allowance by the firewall rules wouldn't get in the way.

So what's happening here? Is the "Allow" rule somehow overriding the "Block" rule, even though it's not dealing with the same executable?

We can test that simply by deleting both sets of rules - go ahead and do that, I'll wait for you.

Didn't make a bit of difference, did it? It still allowed the traffic, then prompted you if you wanted to block it. Even if you selected to "Keep Blocking", the next and subsequent transfers still worked, right?

Okay - let's consult the Big Book of Knowledge (alright, what I can vaguely remember after mumbleteen years in the networking world). Some routers and firewalls use an Application Layer Gateway (ALG) to translate FTP commands, and open ports. Is that what's going on here?

Let's take a peek at the services on this machine (as an administrator, run services.msc):

Bingo - there it is, the Application Layer Gateway Service. And when you have Internet Connection Sharing running, that's what translates IP addresses in FTP commands for you, and what opens up port mappings and holes in the NAT that ICS hosts.

Oh, but wait a moment - what's that in the "Status" column?

That's right, nothing. This service isn't running.

Something must be happening to open this port up - it's not just a case of "port left open", nor is it an outbound port. Those ports are closed tight until the FTP client starts listening for incoming data connections, and then they're opened up.

Here's where I go into MVP-mode, and start searching in all the nooks and crannies of the web and whatever documentation it holds.

Net result - Windows Firewall in Windows Vista includes something called a "connection inspection engine".

Sounds like something from "Schoolhouse Rock".

No, seriously, there's a "connection inspection engine" for FTP - if you connect to port 21, the firewall monitors your communications on that channel, looking for PORT commands. When it finds one, it opens up a hole in the firewall for the incoming data connection.

So why the scary dialog warning that something's going to block traffic?

Probably because the dialog pops up whenever an application starts listening, whereas the connection inspection engine only opens a hole when it sees a PORT command. And an FTP client can't actually give the PORT command until it's started listening.

So, the process goes something like this:

Start the FTP client.
Connect to the FTP server on port 21, waking up the connection inspection engine.
Log on, then type dir
The FTP client knows that it needs to open a data connection.
To start the data connection, the FTP client binds to port 0, and starts listening.
The firewall says "Oh no, an unknown program has started listening - better warn them that they won't get any traffic."
The FTP client checks what port it actually got, and sends a matching PORT command.
The connection inspection engine says "PORT command? That's my cue!" and opens a hole in the firewall to incoming data connections.

Well, that's easy, but what if I don't ever want to do an FTP connection? How do I stop this from becoming a potential hacker tool?

Okay, apart from the obvious - that if a hacker could connect out to a server on port 21, nothing's stopping that hacker from transferring data in - you might want to cripple this functionality.

No problem - just set the following DWORD registry value to 1:

HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ SharedAccess \ Parameters \ FirewallPolicy \ DisableStatefulFTP

The default setting for this value on Windows Vista is 0. [It remains to be seen what value will be the default on Windows Server 2008]

How could Microsoft make this better?

I'd really like to see this documented. Just so that it's not a surprise to anyone.
I'd like to know how many other connection inspection engines there are (at least one, judging from the DisableStatefulPPTP value - but I don't know enough about PPTP to know how that affects operation).
I'd like to know if I can add my own connection inspection engine to the firewall.
Above all, I'd like to do away with the rather confusing and clumsy "We're going to block your incoming ... wait, what just happened?" dialog. If the connection inspection engine is monitoring a command channel, and the process that owns the socket for that command channel starts listening, perhaps we could wait a quarter of a second for a PORT command before calling this a blocked connection?

Finally, is this a vulnerability, a hole, or anything outside the correct operation of a firewall?

No, because the firewall is documented as blocking unsolicited incoming connections - and by any reasonable definition, the data connection requested by a PORT command is solicited.

What should I do now I can compete?

Alun Jones — Mon, 18 Sep 2006 04:37:01 GMT

My departure from Microsoft is very nearly reaching its first anniversary.

As befits someone approaching that milestone, my thoughts drift to ... the non-compete clause.

That's the niggling part of the contract every Microsoft employee signs, and which restricts them, to lesser or greater extent, from engaging in any activity considered to be competitive to Microsoft, using knowledge gained while working at Microsoft.

Now, in my case, the non-compete clause is weak to begin with - a condition I made of my employment was that I could continue my WFTPD work, which was, in some ways, directly competitive to Microsoft's FTP server in IIS.

It's further weakened, I'd say, by the fact that Microsoft sent me to exactly one class while I was there - the mandatory coding security class - and bought me one book - the mandatory "Writing Secure Code".

But, weakened or not, I have chosen to observe it steadfastly - I have not added a single feature to WFTPD or WFTPD Pro or WFTPD Explorer that was based on anything I learned at Microsoft, or even on anything I had hoped to add to Microsoft's FTP server during my stay there. I have even steered clear of adding features that I was planning to add to WFTPD and WFTPD Pro before Microsoft offered me the job, just to avoid the appearance of competing with my ex-employer.

So now, I'm giddy with anticipation, as this mostly self-imposed deadline is about to expire.

What should I start to code into my software?

Well, there's an open comments section below - what do you think I should do, now that I can compete?

I wish Larry hadn't written that...

Alun Jones — Tue, 27 Jun 2006 03:52:00 GMT

Oh, Larry, Larry, Larry...

Articles 1 and 2 were great - really necessary reading to a lot of would-be network programmers.

But article 3... where to start with the corrections?

I'm not going to. It's an article you shouldn't read, because you're not going to use the right terms for the right things, and when you go asking for help from networking experts, they'll look at you in much the same way as security experts look at Steve Gibson. [The look is "how the hell do you get anything done, knowing so little about the field?"]

I've met Larry, and he's a nice guy, so I really thought twice about making this post - and I apologise if I hurt Larry's feelings by saying it... but I have been on a fifteen-year march to persuade people to stop writing crap networking apps, by getting them to understand what they're doing, and I can't stop now.

At the risk of opening myself up to abuse similar to that which I'm heaping on Larry now, I'll point you to my earlier article, where I describe the interaction between delayed ACKs and Nagle - it's much, much simpler than Larry stated, and I think I've got it more accurately described.

Finally, in the case that perhaps I don't have it correct, I'm going to retro-edit it if mistakes are pointed out, because the worst thing you can do is have someone search for the answer, and the first text they come back with is wrong.

DELAY or NODELAY - Riffing on Larry, who's riffing on Raymond...

Alun Jones — Tue, 09 May 2006 00:01:00 GMT

[Why is this under "Programmer Hubris"? Because it's about developers who find "an easy fix" and apply it, without trying to figure out why it made things appear to work better.]

I like to read Larry Osterman and Raymond Chen's blogs, because they've seen most things, and learned most of the lessons right. Today, Larry posted on network optimisation - reminding me once again:

Trying to fix network speed problems by enabling TCP_NODELAY is almost always wrong. Setting TCP_NODELAY disables the Nagle algorithm

When you do this, it's an indication that either your program is broken, or your protocol is broken, or quite possibly both.

"But what about FTP?" people say.

Yes, the protocol is broken. It requires a send / send / recv exchange, and you have to enable TCP_NODELAY (disabling the Nagle algorithm in the process) to make it work properly, so that you can have more than five files transfer per second.

So what does Nagle do, exactly? Well, John Nagle spends his time pushing dead bodies down staircases. The Nagle algorithm, named after John despite his humble protestations, does a couple of simple things every time you try to send data:

If there is unacknowledged data in the queue, then don't send until:

all data is acknowledged, or
you have a segment's worth to send.
Uh... that's it.

The idea is to take a program that, rather stupidly, sends one character at a time, resulting in a 1:40 data:framing ratio, and turn it into one that sends several characters at a time, using the network bandwidth more efficiently and not becoming a network hog.

Way back when, this was a perfect idea, and could have remained perfect, if it weren't for the delayed ACK algorithm, whose author is not remembered so fondly as to name the algorithm after him. Delayed Ack states that you should not send an unaccompanied ack until either:

you receive two segments
200ms expires since the first piece of data was received.

[An ACK is included in every TCP segment, so it's not an overhead of any kind except when you're sending nothing but an ACK.]

In most environments, this is still good, because most protocols are "client sends command, server sends response" over and over again, so each side is doing one send, then one recv. Model this behaviour in your head, and you'll see that the Nagle algorithm won't stop any outgoing traffic, and nor will delayed ACK hold up any acknowledgements.

If one side hiccups and calls send() twice (on short data) before receiving, however, things come to (in networking terms) a screeching halt. The second send queues up because the Nagle algorithm doesn't get an acknowledgement from the first send until the delayed ACK algorithm has exhausted its timer.

The answer is always "don't send / send / recv". Always group logically-associated data together in one call to send, unless you're sending large amounts of data, in which case you can happily send / send / send until you've exhausted your data.

Setting TCP_NODELAY will look like it makes your program perform at top speed, but it's now become the network hog that you programmed it to be, and that Nagle was helpfully preventing you from being. Fixing the program will make your program perform faster than it would with TCP_NODELAY alone, and you will find that the TCP_NODELAY setting will have no further effect, on or off. Your program is now working smoothly, a good network citizen, and the Nagle net-cop allows it to go about its business unimpeded.

So, my final piece of advice - if TCP_NODELAY looks like it makes your program perform better, fix your damn program! There's too much crappy networking software out there already, and you don't want to add to it.

"FTPS" document finally makes it to RFC status.

Alun Jones — Thu, 03 Nov 2005 23:05:00 GMT

News I've been waiting for for years - the document formally known as draft-murray-auth-ftp-ssl-16.txt has finally been released by the RFC editor as RFC 4217 - “Securing FTP with TLS”

What exactly does this mean? Technically, not very much - FTPS has been implemented by several FTP clients, servers and wrappers for several years. I added FTPS support to WFTPD Pro back in 2001, after first expressing interest in doing so in 1997, but being held back by the lack of crypto support in Windows.

I nearly had it ready in 2000, but spent some time trying to debug an issue that turned out to be caused by a corrupted certificate issued by the Windows 2000 Server CA that I was testing against. Let that be a lesson to you crypto developers - sometimes the code is right, and it's the certs that are wrong!

A few minor things have changed since then in the document that is now RFC 4217, but almost nothing significant to the compatibility of FTPS offerings.

I will end with a brief FAQ for you - please let me know if there are any other questions you'd like to see answered:

1. What's TLS, and what is its relation to SSL?

TLS is Transport Layer Security, and is the name of the protocol that grew from Netscape's SSL and Microsoft's PCT. Most people still use the term “SSL”, but TLS is where all ongoing work is carried out by the IETF.

2. Is FTPS the official term?

No - the RFC is “Securing FTP with TLS”, and perhaps the official term should be “AUTH TLS”. However, with the general public already familiar with the concept of “https” being the secured equivalent of “http”, the term “ftps” has sprung up in general use to describe an FTP transfer, or session, encrypted and/or authenticated with SSL or TLS.

3. How different is FTPS from HTTPS?

Quite significantly - HTTPS uses a separate port for incoming SSL connections (usually port 443), compared to the port for unprotected HTTP connections (usually port 80). Because FTP is (and has always been) a session-based protocol, it allows the client to “negotiate up” to SSL or TLS security through the use of the AUTH command described in RFC 2228.

Note also that FTP uses two channels - a control channel and a data channel, and that these channels can be secured - or left unsecured - almost independently. HTTPS is secured from the moment you connect to the HTTPS port, until you close down the connection. FTP is secured on the control channel from the moment you send an “AUTH TLS” or “AUTH SSL” command, until you log out; the data channel is not necessarily secured by default, and security on the data channel can be turned on or off using the PROT command, with parameters “C” for “Clear” or “P” for “Private”.

FTPS always authenticates the server through its certificate, and can be configured to authenticate the client by certificate, or by USER / PASS commands supplying username and password. HTTP and HTTPS have several other methods of authentication (none of which bear much examination at the moment) - NTLM CHAP, Basic, Digest, etc, etc.

4. What about SFTP? What's that?

I get to answer this question a lot. With all these acronyms getting thrown around, it's easy to get confused. Many people automatically assume that any acronym including the letters “FTP” refer to protocols based on FTP. Obviously, that's why “FTPS” was chosen as an informal description of “Securing FTP with TLS”. Unfortunately, others may create confusing acronyms by including the FTP letters, either by accident or on purpose. One such confusion was always “TFTP - Trivial File Transfer Protocol”. This is about as far from FTP as you can get, and still be associated with transferring files from one machine to the other.

The same is true of “SFTP” - it's a file transfer extension to “SSH”. As that sentence implies, to do an SFTP file transfer, you need to have an SSH connection in place. This isn't always practical.