My ugly mug

Hi, I'm Thomas Cannon!

I work on Freckle with Amy Hoy and Thomas Fuchs, and I teach people how to add an extra layer of polish to their work. Previously I worked for BMW and Clemson University, and have a bunch of smaller apps I tinker with on the side. Follow me on Twitter!

As a developer, it’s a good idea to use multiple SSH keys. If you work for multiple companies or use multiple source control services, you’ll want to use a different public/private key pair for each of them. That way, if the SSH keys are somehow compromised, the damage is isolated. By the same logic, you’ll also want to use different SSH keys for each server you connect to on a regular basis.

It’s important to generate a strong passphrase for your SSH keys, but it’s also a pain in the ass to retype the passphrase every time you want to use a key. Thankfully, OS X comes with “Keychain Access”, which saves the passphrases for your keys so you don’t have to remember them and automatically loads them into ssh-agent (fun fact: Keychain Access is your central credential storage manager on OS X. It’s where you can find your saved Wi-Fi passwords, SSL certificates, application passwords, and other security-related information).

There are a lot of tutorials about how to generate and add multiple SSH keys (such as this one), but none of them address the inevitable problem you’ll face when you add one too many keys:

Received disconnect from 123.456.789.012: 2: Too many authentication failures for tcannon

To understand why this error occurs, we need to take a look at how SSH handles keys.

SSH Key Management (Or: How SSH blindly throws keys at the authentication gate by default)

If you’re working with less than 5 SSH keys, you probably won’t notice any problem at all. To understand why, we need to know a bit about how SSH authenticates and SSH server configuration.

The SSH Authentication Process

Connecting to a server via SSH involves the following steps:

  1. Start connection to the server
  2. The SSH server asks for the authentication
  3. If the ssh-agent has any keys, it passes them to the SSH connection. Each one is tried until the server accepts one.
  4. If you have run out of keys before the SSH server kicks you out for too many authentication attempts, then you are prompted for a password.

Normally, SSH servers have a limit of 5-6 authentication attempts. This is why if you have less than 5 SSH keys, you never run into a problem connecting. However, if you have more than 5 keys, you’ll eventually run into the dreaded “Too many authentication failures”.

This is really easy to see if you run SSH in verbose mode

tcannon@localhost:~$ ssh awesome-app.com -v
OpenSSH_5.9p1, OpenSSL 0.9.8x 10 May 2012
debug1: Reading configuration data /Users/tcannon/.ssh/config
....
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/tcannon/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering RSA public key: /Users/tcannon/test/1
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering RSA public key: /Users/tcannon/test/2
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering RSA public key: /Users/tcannon/test/3
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering RSA public key: /Users/tcannon/test/4
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Offering RSA public key: /Users/tcannon/test/5
Received disconnect from 123.456.789.012: 2: Too many authentication failures for tcannon

SSH tried 6 public keys, and ran out of authentication attempts before I could authenticate with a password. We can avoid this problem by configuring SSH to use specific public keys for specific hosts.

The SSH Configuration File

You can actually configure SSH to do things like port fowarding and setting the username on a host-by-host basis. If you’ve never played around with the SSH configuration file, I’d highly recommend reading Joël Perras’ SSH Configuration File Tutorial.

There are two Configuration options we will be focusing: IdentitiesOnly and IdentityFile. Reading the man page for the SSH config file helps us understand what each options does:

IdentitiesOnly Specifies that ssh(1) should only use the authentication identity files configured in the ssh_config files, even if ssh-agent(1) offers more identities. The argument to this keyword must be “yes” or “no”. This option is intended for situations where ssh-agent offers many different identities. The default is “no”.

IdentityFile Specifies a file from which the user’s RSA or DSA authentication identity is read. The default is ~/.ssh/identity for protocol version 1, and ~/.ssh/id_rsa and ~/.ssh/id_dsa for protocol version 2. Additionally, any identities represented by the authentication agent will be used for authentication.

The file name may use the tilde syntax to refer to a user’s home directory or one of the following escape characters: ‘%d’ (local user’s home directory), ‘%u’ (local user name), ‘%l’ (local host name), ‘%h’ (remote host name) or ‘%r’ (remote user name).

It is possible to have multiple identity files specified in configuration files; all these identities will be tried in sequence.

So, to recap:

That means if we have the following configuration:

host awesome-app.com
  IdentitiesOnly yes
  IdentityFile ~/public_keys/tcannon@awesome-app.com

Then we will only use the tcannon@awesome-app.com key when connecting to awesome-app.com:

tcannon@localhost:~$ ssh awesome-app.com -v
OpenSSH_5.9p1, OpenSSL 0.9.8x 10 May 2012
debug1: Reading configuration data /Users/tcannon/.ssh/config
....
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/tcannon/public_keys/tcannon@awesome-app.com
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to awesome-app.com ([123.456.789.012]:22).
....

Okay, but what about unconfigured hosts?

Unfortunately, most tutorials on adding multiple keys to SSH or configuring SSH don’t address the issue of unconfigured hosts. If you create more than 5 keys, you will run into the “Too many authentication failures” error (even if you’ve specified that each key is only to used with a specific host).

This is because the keys still exist in your ssh-agent, because you added them to your SSH key manager (which automatically adds them to ssh-agent). Unfortunately, most of the advice surrounding this problem isn’t very helpful (or actually makes things harder for you). You could delete all of your extra ssh-keys and only use a few, but that defeats your security measures! You could use some funky command line argument to ignore any public-key files or tell SSH which specific one to use, but you have to remember that every time you want to connect to the server. Some people recommend an SSH wrapper, but why throw another piece of software into the mix? Plus, SSH doesn’t really need anything added to it.

No, the real fix lies in SSH configuration file and a clever use of wildcards:

Host *
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa

The reason this works

The reason this configuration works is because * becomes a catch-all for any host not explicitly specified in your configuration file. This configuration tells SSH that you only want to use a specific few SSH keys (in this case, the default SSH key). So if the server doesn’t accept your default SSH key, you can still use password authentication!

Note that this won’t collide with any of your existing SSH configurations:

Host *
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa

host awesome-app
  IdentitiesOnly yes
  user tcannon
  hostname awesome-app.com
  IdentityFile ~/public_keys/tcannon@awesome-app.com

Note: Filezilla will still be broken

Unfortunately, this fix doesn’t seem to work with Filezilla (which is really sad, because I like Filezilla as a cross-platform open-source FTP client). Thankfully, there are a bunch of other alternatives out there: