set PasswordAuthentication No and only publickey authentication will work. and it will ask for passphrase only if it was entered then you did created a key. P.S. Make sure you put your public key in .ssh/authorized_keys file