How to fetch ssh keys from servers?

Broman asked:

I need to login to a host via ssh, get a keyfile, close the connection and then use the key to login without password on several machines.

The underlying problem is that I need to perform tests on a couple of servers via ssh. I have already successfully written code using paramiko that does that with regular login. But I just cannot figure out how ssh works.

This is what I have done to set it all up. I create keys on the client and copy them to a server:

[[email protected] ~]$ ssh 192.168.122.33
[email protected]'s password: 
Last login: Mon Apr 29 09:26:15 2019 from gateway
[[email protected] ~]$ exit
logout
Connection to 192.168.122.33 closed.
[[email protected] ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/me/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/me/.ssh/id_rsa.
Your public key has been saved in /home/me/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:CV4HoJfwABDD15AoGDtZdTgAlEWZJfP8cFW+18lk9Gg [email protected]
The key's randomart image is:
+---[RSA 2048]----+
 Removed
+----[SHA256]-----+

[[email protected] ~]$ ssh-copy-id 192.168.122.33
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 2 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password: 

Number of key(s) added: 2

Now try logging into the machine, with:   "ssh '192.168.122.33'"
and check to make sure that only the key(s) you wanted were added.

[[email protected] ~]$ ssh 192.168.122.33
Last login: Mon Apr 29 09:27:26 2019 from gateway
[[email protected] ~]$ 

Works as expected so far. Now I can login without being prompted for password. But now there are things I don’t understand. I started with deleting the files id_rsa and id_rsa.pub from the client:

[[email protected] ~]$ exit
logout
Connection to 192.168.122.33 closed.
[[email protected] ~]$ cd .ssh/
[[email protected] .ssh]$ ls
id_rsa  id_rsa.pub  known_hosts
[[email protected] .ssh]$ mkdir ../backup
[[email protected] .ssh]$ mv id* ../backup

Now, I expected to NOT being able to login without a password, but that’s obviously wrong.

[[email protected] .ssh]$ ssh 192.168.122.33
Last login: Mon Apr 29 09:29:20 2019 from gateway
[[email protected] ~]$ 

So I suspected that the keys are cached somewhere, so I tried to restart the ssh daemon.

[[email protected] .ssh]$ sudo service sshd restart
[sudo] password for me: 
Redirecting to /bin/systemctl restart sshd.service
[[email protected] .ssh]$ ssh 192.168.122.33
Last login: Mon Apr 29 09:38:23 2019 from gateway
[[email protected] ~]$ 

Did not work either. Sorry for being unclear, but there is probably something conceptual I just don’t understand. What I want to have done right now is a procedure to configure one server in a way so that I can login to it with username and password and fetch a keyfile. Then I want to copy that configuration to a number of other servers. Lastly, I want to use my client to fetch the keyfile from one of the servers and use it to login and run various commands on the servers. For security reasons I also need to store the keyfile in ram, and probably encrypt it somehow, but that’s a later issue.

I will write everything in Python later, but I need to solve this to start coding. This pseudopython describes what I’m looking for:

hosts=['192.168.122.' + str(i) for i in range(1,10)]
key=fetch_key(hosts[0]) # Or any of the servers.
for host in hosts: # This part is already done. I just need to get the key.
    execute_test(host, key)

The servers are running CentOS 7

I posted this at SO first, but it was OT, but by request from a commenter, here is the output from ssh -vv

[[email protected] ~]$ ssh -vv 192.168.122.33
OpenSSH_7.8p1, OpenSSL 1.1.1 FIPS  11 Sep 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.122.33 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.122.33 [192.168.122.33] port 22.
debug1: Connection established.
debug1: identity file /home/me/.ssh/id_rsa type -1
debug1: identity file /home/me/.ssh/id_rsa-cert type -1
debug1: identity file /home/me/.ssh/id_dsa type -1
debug1: identity file /home/me/.ssh/id_dsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa type -1
debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519 type -1
debug1: identity file /home/me/.ssh/id_ed25519-cert type -1
debug1: identity file /home/me/.ssh/id_xmss type -1
debug1: identity file /home/me/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 6 setting O_NONBLOCK
debug1: Authenticating to 192.168.122.33:22 as 'me'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: [email protected] need=32 dh_need=32
debug1: kex: [email protected] need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:tDHtYxD9gjd6WxS82Y5unZRfSTA42S0BWPQB244fP2s
debug1: Host '192.168.122.33' is known and matches the ECDSA host key.
debug1: Found key in /home/me/.ssh/known_hosts:1
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: [email protected] (0x5620eac46f80), agent
debug2: key: [email protected] (0x5620eac4a830), agent
debug2: key: /home/me/.ssh/id_rsa ((nil))
debug2: key: /home/me/.ssh/id_dsa ((nil))
debug2: key: /home/me/.ssh/id_ecdsa ((nil))
debug2: key: /home/me/.ssh/id_ed25519 ((nil))
debug2: key: /home/me/.ssh/id_xmss ((nil))
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:a3sjstTdNHguzTkUQ7u/1ExNPPijAkXbNYl7CbrwCOQ [email protected]
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg rsa-sha2-256 blen 279
debug2: input_userauth_pk_ok: fp SHA256:a3sjstTdNHguzTkUQ7u/1ExNPPijAkXbNYl7CbrwCOQ
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.122.33 ([192.168.122.33]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 6 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug1: Sending env XMODIFIERS = @im=ibus
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Apr 29 10:24:40 2019 from gateway
[[email protected] ~]$ 

My answer:


Your ssh key was held in memory by your ssh agent. Deleting it won’t prevent the agent from continuing to use it. You would need to stop the agent (usually by logging out of your workstation) first.


View the full question and any other answers on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.