NixOps locks itself out when deploying to a node for the first time with --force-reboot and the SSH key for provisioning isn't in the nixops configuration

[grahamc]

So this is a bit weird, but...

1. Have an ssh key just for provisioning and have it be the only key accepted by the server, not sure this is significant
2. do a nixops deploy for the first time to a server using that key and use --force-reboot
3. the ssh key is no longer acceptable by the target server, and nixops doesn't dare try to use the SSH key it generated and placed
4. nixops just prompts for the password, which doesn't exist

Workaround: don't --force-reboot the first deploy, or: add the provisioning ssh key to your ssh agent and do a regular deploy once

[kimburgess]

I'm seeing the same behavior here too.

Using a configuration that has a secondary user which can be used to SSH in and inspect the post-deploy state, the public key for the root user is definitely inserted:

[root@cotag-5-01:/]# cat /etc/ssh/authorized_keys.d/root
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILNnvPApjYRx/9glo5FEeJ5r2LviY72SvSh0MnqiB+7i NixOps client key for cotag-5-01

So it looks like this may be something on the nixops connection side.

As a side note: in my environment only the second workaround above worked. Even when doing a live switch (no --force-reboot) the public key generated by nixops is still inserted into the target machine, but it's private counterpart does not appear to be used when trying to connect.

[selaux]

I ran into the same issue. I narrowed it down to this condition: https://github.com/NixOS/nixops/blob/master/nixops/backends/none.py#L82. self.cur_toplevel is None in this case as the configuration has not been applied yet. I think the same happens if the first deployment fails.

As far as I can see the check for self.cur_toplevel is unneccessary in this case as self._ssh_public_key_deployed already handles the case when the ssh key has not yet been deployed. Should I open a PR for this?

[asymmetric]

@selaux how did you debug this?

[selaux]

I noticed that the ssh key is not written into the temporary directory during deployment, so it seemed that nixops was not using it. I looked into the state file where sshPublicKeyDeployed was set to 1, which seemed right to me. Then I basically used a local version of nixops and just put in a print statement...

Not sure how this change will influence other things, but since sshPublicKeyDeployed is set on first deployment it should be fine.

[asymmetric]

I would definitely open a PR so it can be tested more easily :)