Day 74: A Key Each Way and a Schedule

Day 74 of DevOps, Day 24 of Azure. Both tasks today were about SSH keys, and specifically about which way round they point.

A Jenkins job needed one server to be able to reach another. An Azure task started by checking whether a key already existed, and the .ssh directory turned out to be full of a file that pointed the other way. Same two files, opposite directions, and getting them mixed up produces a setup that looks finished and cannot connect.

One Jenkins task, one Azure task. Schedule a job that copies Apache logs between servers, then build a VM that only accepts key-based SSH. The tasks come from the KodeKloud Engineer platform.

Task 1: A job on a six-minute schedule

The task: a freestyle job named copy-logs, running every six minutes, copying the Apache logs from stapp03 into /usr/src/finance/ on the storage server.

Step 1: Check the logs are actually there

[banner@stapp03 ~]$ cat /var/log/httpd/access_log
127.0.0.1 - - [-] "GET / HTTP/1.1" 200 -

Boring step, and it is the one that makes everything after it mean something. A scheduled job that copies nothing succeeds, on time, forever. The build goes green, the destination stays empty, and nothing in Jenkins tells you why, because from Jenkins' point of view nothing went wrong.

Step 2: Key the connection between the two servers

[banner@stapp03 ~]$ ssh-keygen -t rsa -b 4096
Enter file in which to save the key (/home/banner/.ssh/id_rsa):

[banner@stapp03 ~]$ ssh-copy-id natasha@ststor01
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
natasha@ststor01's password:
Number of key(s) added: 1

Generated as banner, not as root, which is why ssh-keygen offered /home/banner/.ssh/id_rsa. The identity has to belong to whichever account opens the connection.

Then one detail in the login banner on the way in, which is worth reading rather than scrolling past:

Last failed login: Sun Apr 12 09:41:57 UTC 2026 from 10.244.81.53 on ssh:notty
There were 2 failed login attempts since the last successful login.

Those two failures are ssh-copy-id probing with the new key before it was installed, which is exactly what its "1 key(s) remain to be installed" message was telling me. Benign here. On a server where you had not just run ssh-copy-id, two failed logins from another host is something to go and look at.

Step 3: Let the schedule fire

Six minutes later the build ran on its own, and:

[natasha@ststor01 ~]$ ls /usr/src/finance/
access_log  error_log

What the docs recommend

A six-minute schedule written the obvious way is */6 * * * *. Jenkins documents a better form, and the reason is load.

The H symbol "should be used wherever possible". The example Jenkins gives is a dozen jobs all set to 0 0 * * *, which produces "a large spike at midnight", against H H * * *, which still runs each job once a day but at a time of its own.

The part worth knowing is that H is not random. It is "a hash of the job name, not a random function", so the value "remains stable for any given project". You get spreading without losing reproducibility. For short cycles the documented example is "every fifteen minutes (perhaps at :07, :22, :37, :52)" written as H/15 * * * *.

So H/6 * * * * is the form for this job. Still every six minutes, but offset within each window by a hash of this job's name, so that this job and the next one somebody writes do not fire in the same instant for the rest of time.

Jenkins also warns about exactly this kind of schedule: "short cycles such as */3 or H/3 will not work consistently near the end of most months", because "Hashes are always chosen in the 1-28 range". Worth knowing before a six-minute job becomes load-bearing.

One link in the chain I cannot account for

This job needs two hops. The logs are on stapp03. The destination is ststor01. The build runs as jenkins on the Jenkins controller, which is neither machine.

What I set up and verified is the second hop, stapp03 to ststor01. What I did not set up anywhere in this task is the first, from the controller to stapp03. And the job ran on schedule and the files arrived.

Two explanations fit. The controller may already have been trusted by the app servers from earlier work, since Day 71 keyed jenkins@jenkins to the storage server and the same thing could have been done to stapp03 on a day my notes do not cover. Or the build did not run on the controller.

I cannot tell from what I have, so I am recording it as an open question rather than writing in the step that would make the story tidy. Same gap as Day 71, where the build command in my notes could not have produced the result my notes also recorded. Same cause, too: a screenshot shows you a green build, not the chain of trust that let it go green.

Task 2: A VM that only accepts keys

The task: create an SSH key if one does not already exist, build a VM that trusts it, and prove password-less access works.

Step 1: A non-empty .ssh directory is not a key pair

-rw------- 1 root root  562  authorized_keys

/root/.ssh/ existed and already had a file in it. No id_rsa, no id_rsa.pub.

Those two serve opposite directions:

The lab seeds authorized_keys so that I can reach the client machine. It does not give the client machine an identity of its own.

So "check if a key already exists" has to mean checking for the file the next step needs:

if [ -f /root/.ssh/id_rsa ]; then echo "key exists, reusing"; else ssh-keygen ... ; fi

Written as "does ~/.ssh exist", or "is it empty", the check finds files, concludes a key is present, and leaves the VM creation with no public key to install.

Which is the Jenkins half of today from the other end. There, stapp03 needed an outbound identity and ststor01 needed that public key in its inbound list. Two files, two directions. Confuse them and you get a setup that looks complete and cannot connect.

Step 2: The key size was a downgrade and I did not notice

ssh-keygen -t rsa -b 2048 -f /root/.ssh/id_rsa -N '' -q

-b 2048 reads like a sensible conservative default. It is not a default, and it is below the real one.

The ssh-keygen manual gives ed25519 as the default key type, and for RSA, "the minimum size is 1024 bits and the default is 3072 bits". So this command overrode the type to get RSA and then overrode the size downwards, from 3072 to 2048. Omitting -b entirely would have produced a stronger key than asking for 2048 did.

Worth putting beside Day 71, three days ago, where I used -t rsa -b 4096 and called 4096 an upgrade from 3072. Same series, same flag, used in opposite directions, and in neither case because I had checked what the default was. The honest conclusion is to omit -b and take 3072, or omit both flags and take ed25519.

-N '' is the flag that makes this password-less: it sets an empty passphrase, and a passphrase-protected key prompts on every connection unless an agent holds it. Real tradeoff rather than a convenience, though. The private key on disk is now the only thing between anyone who can read that file and the VM.

Step 3: Name the key rather than generating one

--ssh-key-values /root/.ssh/id_rsa.pub

Earlier tasks used --generate-ssh-keys, which is documented to "Generate SSH public and private key files if missing" and store them in ~/.ssh. Convenient, and it hides which of the two branches ran.

Here the first step had already settled the key, so naming it explicitly means the VM gets the key I just checked, with no conditional behaviour in between. Two steps that each do one thing.

Step 4: Password authentication came out disabled

"PasswordAuthDisabled": true

That is the observation that makes the task's title accurate, and it is worth separating from my explanation of it. I had written that creating a Linux VM with a key and no --admin-password makes Azure set disablePasswordAuthentication: true.

The observation is real. The mechanism is not documented in those terms. The ARM reference describes the property as specifying "whether password authentication should be disabled" and gives no default for it, where neighbouring properties do state theirs. What the CLI does document is --authentication-type, which "Defaults to password for Windows and SSH public key for Linux".

So the chain from "Linux VM, key supplied, no password" to "password authentication off" is consistent with the documentation rather than stated by it. Which I would rather say than present an inference as a fact.

The consequence holds regardless: there is no password to guess, so the VM is not exposed to credential stuffing against port 22 even with the NSG allowing SSH from anywhere. Passing --admin-password instead would have left password authentication on with port 22 open to the internet, which is the combination behind most compromised cloud VMs. It also means losing the private key locks you out, and recovery runs through az vm user update or Run Command.

What the docs recommend

The verification is the part I would reuse. Configuration and behaviour are different claims and need separate checks:

PasswordAuthDisabled: true          <- configuration
KeyPaths: [.../authorized_keys]     <- configuration

ssh -o BatchMode=yes ... 'whoami'   -> azureuser     <- behaviour

az vm show proves Azure was asked for the right thing. The SSH proves the guest accepted the key. And BatchMode=yes is what makes the second one mean "password-less" rather than just "worked": without it, a successful login is ambiguous, because ssh could have fallen back to a prompt that got satisfied somehow. With it, a prompt is impossible, so the key is the only explanation.

One last distinction, because it is easy to assume otherwise. This key is not an Azure resource. Day 1 created a Microsoft.Compute/sshPublicKeys object, reusable across VMs by name. Today's key exists only on the client machine and as a copy inside the VM's authorized_keys, and az sshkey list would return nothing. Both give working SSH; only one gives you something you can refer to later.

What I learned

Two files, two directions, and almost every SSH problem I have had is one of them in the wrong place.

The Azure task made it explicit by seeding authorized_keys and nothing else, so that a naive "is there a key here" check would say yes and be wrong. The Jenkins task made the same point by needing an outbound identity on one server and an inbound entry on another, and my own notes then failed to account for one of the two hops the job must have used.

The key size is the smaller finding and the more embarrassing one. I used -b 4096 three days ago and -b 2048 here, in both cases without checking that the RSA default is 3072. One was an upgrade, one was a downgrade, and neither was a decision.

So here is the Day 74 question. In the SSH setups you maintain, which machine holds an identity it no longer needs, and which one is still trusting a key whose owner left?

Day 74 down. Twenty-six to go.