If you have more than about 5 SSH keys, you need to setup your
~/.ssh/config file so that SSH to github.com (or other SSH server) uses the correct SSH Public Key the first time.
and add the lines
Host github.com Hostname github.com User git Port 22 IdentitiesOnly yes PubKeyAuthentication yes IdentityFile ~/.ssh/MyGithubKey
where MyGitHubKey is the key you created following the process (without ssh-agent)