Monday, October 17, 2005

SSH port probes and port knocking

On the topic of people receiving unwanted SSH port probes, I have previously posted a script to catch requests and blacklist the source IP's. Though this strategy is effective, it lacks a certain finesse and it also doesn't prevent what I saw as the major problem about such probes - log flooding.

An average Internet-facing server has room for hundreds of megs of logs, an admin capable and willing to go through the logs en-masse and blacklist troublesome IP's. Unfortunately for myself the only real factor in these potential attacks is in the annoyance that a large log or repeated visible attempts generates.

I am always security conscious and this hasn't changed, however I think it's time for my home machine to stop storing logs of every little script-kiddie that pings my port 22 with their tools and instead just ignore them. That was previously much more difficult for me, especially when you see the same IP try and try again. It's just so tempting to blacklist them and imagine their reaction on the other end.

Previous to this article, my port 22 was open to the world (but no other ports). The control of who got in was left in the more-than-capable hands of a religiously updated OpenSSH, with only one working logon allowed to actually come in, passwordless logins denied and a large private-key that's kept secure, protected also by a long random passphrase. That allows access to an extremely limited account whose only purpose is to allow me to su with the right password and become root.

I use SSH to log in from machines when I am at work (where it's incredibly useful to bypass over-zealous school internet filters, with a little help from a proxy on my home machine) and when I need to fix other people's machines. All the schools I work for are NAT'd across the internet through a single IP. Thus, it would be simple to allow SSH through just that IP. However, when I am fixing someone's machine (which could be at a moment's notice), I need SSH to still let me in. This means that I would have a basically random IP trying to connect to my home machine and it would have to deal with it accordingly. This is why my port 22 was left open up until now.

Previously, I just watched the world and his brother bounce standard logins and passwords off of my home machines in a vain attempt to see if I had left some stupid login or password on the machine that might give them some sort of platform to springboard onto higher access. I was secure in the knowledge that not only would they never get the only working account name by bruteforce, nor would they have the publickey and neither would they have the passphrase which went with it. The worst case was that they exploited some kind of flaw in OpenSSH itself which I consider highly unlikely and easily fixable. In that case, it wouldn't be just me who'd be in trouble but half the servers on the Internet and someone would come up with a fix VERY quickly.

These dumb brute-force attacks left a little bit of a bad taste, though. My logs were constantly flooded by "invalid user", "potential break-in attempt" and so on, all through the day. The machine that controls port 22 is a desktop and toy machine, selectively exposed to the internet for convenience and for trying out new systems and therefore it's quite annoying to have logs flooded by such futile attempts at breaking in.

I have put on this blog a script that I used to use, it basically watches the logs via a regular cron job and blocks any IP that tries too often. This works wonderfully and is the situation I would use if I needed random people from random IP's (e.g. employees etc.) to be able to access the SSH on that machine.

Fortunately, I don't need to. The only person to ever use the machine will be myself and therefore it's much easier to me to use port-knocking.

Port-knocking is the Internet equivalent of a secret handshake. As far as anyone on the Internet is concerned the machine is not responding to any connection requests whatsoever. What it is secretly doing, though, is monitoring all such requests for a particular, pre-determined pattern. When it sees that pattern of requests, it modifies the firewall to open a particular port to that particular IP.

For example, it's possible to set it up so that when it sees connection requests on ports 1000, 2000 and then 3000, all from the same IP address, in that order and without any other port requests in between, it opens up port 22 for the IP that tried to connect. Thus, from any computer in the world, I could take a small utility or even just a batch file and a telnet command, hit those secret-handshake ports in the right order and as if by magic SSH would suddenly be available for me to use from that computer.

A similar "knock" of some different ports would also close the SSH port behind me, too. Using this, it's perfectly possible to rap on the internet port of my home computer, get it to let me in, SSH and do whatever I need to do, then rap a different tattoo and it would close SSH. All the time this is happening, every other computer in the world could be trying to get in and all they would see would be a closed or stealthed port (depending on whatever firewall config you have). Being closed or stealthed would also mean that SSH doesn't even have to wake up to deal with such requests, so bypassing any possibility of having some SSH exploit being randomly tried out on the machine. It also means that people can't even GET to the stage where they can try to log in, so instantly cutting out quite a lot of log-spam.

The utility that can knock like this can be as simple as batch or shell script with certain parameters to telnet, to a custom-made tiny C program or a generic port-knocking utility which you supply the port list to. The ports can be UDP or TCP, they can be any numbers in any order, they could even be encoded versions of a particular passphrase or password. Depending on the port-knock listen server in use, the knocks can also be one-time passwords chosen from a predefined list (so that even if someone sniffs your secret-handshake going across the internet, they wouldn't be able to replay it).

I have now switched over to a port-knock system (a good one seems to be Knock from the author of Arch Linux), if only to make my logs easier to read, and have noticed a myriad of advantages from this system.

1) It doesn't make my system any less secure - the only required software is a port-knock daemon that's trivial in implementation and quite secure by design (all it does is listen to connection attempts at the IP level and then extracts the source IP and destination port. It then executes a shell script of your choice which does all the opening, closing of ports etc.)

2) It makes my system more painful for someone unscrupulous to try to access - ports appear closed or non-existent and there is absolutely no indication that anything "strange" is going on. Knock-opened ports are only opened to the IP that knocked correctly, thus keeping everyone else in the world in the dark.

3) It means that the software behind the port-knock is protected from random, "dumb" brute-force attacks and new exploits. Though this isn't 100% effective (as every security analyst has argued when confronted with port-knocking) it's made another step harder, to the point where you really need to be able to sniff all traffic in and out to be able to defeat the simplest configuration of port-knocking. If someone can do that, you're already in trouble.

4) The knock acts as a magic key. It can be a simple numeric list, it can be TCP or UDP or even (theoretically) ICMP. Equally, however, it can be a one-time password that expires after use (so combatting sniffing/replay techniques to open the port), an encrypted sequence, a time-dependant sequence, an IP-dependent sequence, or a combination of those and many other possible sequences (one based on the last time the admin logged in, for example?). Until the magic key is used, nobody can access any port or tell that anything screwy is going on. As far as the casual "hacker" is concerned, there's no way to tell if the machine is switched on and looking for a portknock or switched off entirely.

5) Anybody trying knocks can be blacklisted in a similar way to we had before, but using the failure logs of the portknocker rather than millions of attempts at usernames.

6) The knocks are immune against every-port scans, like an nmap scan, as the knock has to be done in the right sequence. At no point in the sequence is any clue given to a potential brute-forcer about how far through the sequence, which individual knocks were successful etc. because of the multi-stage nature of portknocking.

Knocking port 1000 in the example above makes everything still look closed down. You knock port 1001 afterwards (or indeed any port but 2000) and you have to start the sequence again to be allowed in. However, you get absolutely no feedback about whether or not each stage was successful.

Therefore the chances of brute-forcing the knock have decreased dramatically. Say you have a three-knock TCP-only rule. Then you would have to try EVERY single combination of three numbers all in the range 0-65535. That's 281,462,092,005,375 combinations, over 281 million million, each consisting of a single packet. And in between each one you would have to scan every port to see if it opened anything that wasn't open before. That's going to need a VERY long time and be more than a little obvious, even on the fastest connection in the world.

But now imagine you have mixtures of TCP, UDP and ICMP, four-knock, five-knock etc. rules, time-based rules and other clever trickery to determine what the secret handshake is. It would be all-but impossible, with a decent well-thought out port-knock system, to get anything to open up at all, thus stopping any sort of brute-force attempt before a login dialog had even been found.

7) The minimal overhead of such a system is valuable too. Possible brute-force attackers take no more bandwidth than random port hitters. No resources are required to track hundreds of parallel connections other than standard TCP/IP. The CPU strain is kept to an absolute minimum and the worst case would be a DoS attack requiring substantial resources behind the attackers to initiate. Again, you're already in trouble but at least they're bouncing off your TCP/IP stack rather than trying to guess your passwords.

Multiple parallel attempts from different IP's would be no more help, either. The knock would have to be completed from a single IP correctly. Additionally, it may well be that the knock is IP dependent or that some IP's are blacklisted from any sort of port-knocking anyway!

Additionally, it's one piece of software on the server and one trivial piece of software on your USB key or in your normal remote kit. Currently I always have PuTTY, my private key and TightVNC as my standard remote kit anyway, so adding a tiny portknock utility is a pittance. The system is simple and completely follows TCP/IP rules and standards. To people who are allowed access, the overhead is just one more command, one more password that may even be linked to the main SSH password.

8) The system is completely customisable, from what the port-knock actually consists of down to what it does. In the end, each knock sequence runs a shell script and so can do anything that the user it's running as can do.

9) At the end of the day, even if someone manages to sniff your network, steal the time-dependence equations or even completely guess your port-knocks, you're then left in no less secure a position that you were before. You can still blacklist, you're still password/public-key protected and no changes have to be made to any of the application software either on the client end or server. Everything is just as it was before.


This system is for the ultra-paranoid as well. As an example, consider trying to attack a system with these layers:

1) No ports are opened to anyone without the special knock. The entire world sees nothing but a blank reply, identical to one which a dead connection or turned-off machine would give.
2) The knock is a hash of source IP, time, secret password and other variables. Nobody would see the knock without sniffing every bit of traffic into the computer.
3) Even when the knock is sniffed, replaying it does not open any ports. With no ports open, nobody can attack.
4) Say someone steals the sheet with all the necessary portknock passwords and equations to open the port (an extremely unlikely but yet still not very dangerous scenario).

This is where any open SSH server on the Internet currently is... SSH is considered uncrackable even when all communcation is sniffed if implemented properly.

5) Now they have to find a login that is allowed in via SSH.
6) There are no passwords on any logins, only publickeys, so they also need the publickey of that user (which cannot be sniffed in any way).
7) The publickey is also protected by a passphrase. They now need that as well.
8) They have all the above and managed to get into SSH and login as that user. Now they could (possibly) do some damage by exploiting vulnerabilities in the underlying system.

What would you do if you were faced with such a system? Personally, I think it would be at least 281 million million times easier to just break into the actual building that held that server rather than even try to access it.

But port-knocking is not just for SSH. As hinted above, because the result of a successful port-knock is the execution of a specific shell script, opening holes in a firewall for a particular IP is the most basic of uses. With the right software or hardware it could do anything you wanted. The right port-knock could do anything from turn your machine off to waking up every machine on the LAN, initiating backups to rebooting, putting the kettle on to sending out alerts to every technician.