Every engineer knows the feeling. You finally fix the bug that's been blocking your ticket, you push it, and you sit there waiting for that beautiful green tick.
That was me during my internship. My Terraform kept failing to create a database server because of a disk error. With some help from a senior engineer, I found the problem: my AMI filter was grabbing the wrong image, one that needed a bigger disk than I'd asked for. I fixed the filter, pushed the change and waited.
The pipeline didn't go green. It didn't really go red either. It didn't run at all.
Sometimes your pipeline is broken and your code is completely fine.
Here's what happened, how it got fixed, and what I'd do so it never catches you off guard.
1. Five Failed Jobs, Zero Logs
The pipeline page showed the whole thing as Failed. 5 out of 7 jobs, all red.
Normally that's fine. You click into a job, open the log, scroll to the red text and find your mistake.
This time there was nothing to read. The failure reason just said:
No more compute minutes available
No job log
No stack trace. No Terraform error. Nothing. The jobs never even started, so there was nothing to log.
If you ever see a pipeline full of failed jobs with no logs at all, check this first.
2. Retry Made It Look Even Worse
Of course I hit retry. (We all do.) Instead of running, GitLab threw this at the top of the page:
An error occurred while making the request.
Status cannot transition via "drop"
That one looks properly scary, like something is broken inside GitLab itself. As far as I can tell, it's just GitLab failing to update jobs that never got the chance to start. Same problem, more confusing message.
Retrying won't help until the minutes are back. You'll just get the same error every time. Once the minutes were topped up, one retry was all it took.
3. Compute Minutes Belong to the Account, Not You
Here's the part nobody tells you as a junior: you probably can't fix this yourself.
I told the senior engineer reviewing my change. She couldn't check the monthly quota either, so she pointed me to the colleague who manages our GitLab account. I found out later why: viewing a group's usage needs the Owner role, so a lot of developers can't even see how close they are to the limit.
So there's no point digging through your code. The fix is a message to the right person. Here's roughly what I sent:
Hi, our GitLab pipeline on the Terraform project has stopped running. It says we've run out of CI compute minutes for the month. Would you be able to check the quota when you get a chance? I've got a change ready, it just can't run until there's compute available.
He confirmed we'd run out and topped up the account.
The annoying part was the timing. Between waiting on a review and this, my ticket barely moved for days, and I had to explain at standup that my work was ready but nothing could run. If you're ever in that spot, keep it simple: what's done, what's blocking it, and who's sorting it out.
4. How to Check Before It Happens
If you're an Owner on the group, go to the group (not a subgroup), then Settings > Usage quotas. On a personal account, it's Usage quotas in the left sidebar. (GitLab docs)
A few things I wish I'd known:
- Usage resets every month, so this usually bites near the end of the month.
- GitLab renamed "CI/CD minutes" to "compute minutes" in version 16.1, so older Stack Overflow answers use the old name.
- If you're not an Owner, ask whoever is to keep an eye on it. Way easier than finding out mid-ticket like I did.
5. How to Make Your Minutes Last Longer
If I wanted to stop this from happening again, this is where I'd start.
Only run jobs when their files change. If your Terraform is split into components, there's no reason to plan the database when you only touched the load balancer. rules: changes: sorts that out:
plan:db:
script:
- terraform -chdir=db plan
rules:
- changes:
- db/**/*
Stop duplicate pipelines. A push to a branch with an open merge request can start two pipelines, one for the branch and one for the MR. A workflow: rules block at the top of .gitlab-ci.yml can make sure only one runs.
Cancel old pipelines when you push again. Pushed three fixes in five minutes? You don't need all three pipelines to finish. Mark jobs as interruptible: true and turn on auto-cancel for redundant pipelines in your CI/CD settings.
Cache what you download every time. Dependencies and Terraform providers get downloaded on every single run unless you cache them.
Make expensive jobs manual. If a job doesn't need to run on every push, when: manual is your friend.
Use your own runners for heavy work. According to GitLab's docs, project and group runners aren't affected by the compute quota. If your team already has a server, a self-managed runner can take the heavy jobs off your quota completely.
Final Thoughts
The funny thing is, my fix was right the whole time. The pipeline just had nothing to run on.
So I'm curious: what's the most annoying reason your pipeline has failed that had nothing to do with your code? Drop it in the comments. I want to collect them all 😅
Top comments (16)
The message you sent to the account owner is the most underrated part of this post. Clear, says what's broken, what you already checked, and that your change is ready. A lot of people take years to learn to escalate like that.
Another "my code is fine, the pipeline isn't" classic in GitLab: protected CI/CD variables. If a secret is marked protected, it's only exposed to pipelines on protected branches and tags. Push the same commit to a feature branch and the job fails with an empty variable, often as a confusing auth error from Terraform or the cloud CLI rather than "variable missing". Worth checking whenever a job works on main and fails everywhere else.
And +1 on self-managed runners for the heavy jobs. That also keeps your Terraform credentials on infrastructure you control.
Thank you Kiell! Honestly, that message came from being too nervous to send anything vague 😅 I wanted whoever read it to know straight away that my change was ready and the only thing missing was compute.
The protected variables one is such a good catch, and it’s sneaky because the error doesn’t point you there at all.
Nervous turned out to be the right instinct here 😄 A message that says "the change is ready, the only thing missing is compute" is the one people can act on straight away. And yes, that protected-variables error sends you looking everywhere except the actual cause. Great write-up, Maame!
I've lived this exact shape of failure on GitLab — the pipeline that fails with zero logs while your code is fine. In our case it was a stale group runner token, so no runner ever picked the job up and GitLab gave us nothing to grep, which taught me the same lesson you landed on: once there are no logs, the question stops being about code. Your point about usage quotas living on the account rather than the user is the one I'd tattoo on every junior's forehead: CI minutes are a shared resource, and only the group Owner can see them, so treat the quota like a budget you check in standup. One addition from my side: caching Terraform providers on the runner won't save you from an exhausted quota, but it makes every minute go further when you're re-running plans all day. Has your team turned on auto-cancel for redundant pipelines yet?
A stale runner token sounds even worse, Robert. At least I got a message. You just get a job that never starts. Treating the quota like a budget in standup is a great habit, and good point on caching providers. We haven’t set up auto-cancel yet, but it’s next on my list.
You are right that a silent never-start is worse than an error message, because there is nothing to search for and nothing to retry against. The check that saved me in GitLab was a small pre-flight job that validated the runner token and the quota before the real job queued, so a stale token failed loudly at the start instead of vanishing. When you add auto-cancel, will you cancel the whole pipeline or only the superseded job, so a slow but valid run does not get killed by its own retry?
Great write-up, Maame!
I really appreciate how you turned a frustrating internship experience into something other developers can learn from. One thing I’ve learned from managing CI infrastructure is that a failed pipeline doesn’t always mean failed code. Sometimes the problem is simply that the infrastructure never gave the job a chance to run. 😄
I’m particularly glad you mentioned self-managed runners. I’ve found them really useful for keeping CI workloads under my own control, although they do come with their own maintenance responsibilities.One small consideration with rules:changes: shared Terraform modules and dependencies need to be included in the rules too, otherwise important changes might accidentally skip validation. As for your question, one of the most frustrating situations is when a runner becomes unavailable while the code is perfectly fine. Everything is ready, but the pipeline just sits there waiting. 😂
Thanks for sharing your experience so openly. Keep learning and building!
Hi Mustafa, thank you so much! 😂😂 I was so stressed out in the beginning because I honestly thought I broke something. Seeing 5 red jobs and no logs at all, my first thought was “what did I just do?” Then I hit retry and got that “Status cannot transition via drop” error and it got even worse 😅
And thanks for the add up on rules:changes! That makes a lot of sense. If someone updates a shared module and the job only watches its own folder, the change could slip through without being checked. Will definitely keep that in mind and include the shared modules in the rules.
The runner one sounds just as frustrating, especially because nothing tells you it’s not your code. You just sit there wondering if it’s going to start or not 😂
Haha, I can imagine that moment! 😂 Five red jobs, no logs, and then GitLab throws an even scarier error at you. That’s enough to make anyone question their entire career for a few seconds! 😅
The funny thing about DevOps is that sometimes we spend more time figuring out why something didn’t run than debugging what actually ran. 😂
But you handled it well, Maame. You investigated, asked the right people, and turned the experience into something useful for others. That’s what matters.
Keep sharing these stories! I genuinely enjoy reading about real engineering experiences, especially the messy ones we all learn from. 😄
Thank you so much, Mustafa. Your comments really mean a lot and I learned something new from you too. I’ll definitely keep sharing the messy stories, there will be plenty more
Thanks again for reading and taking the time to share your experience! :)
Ayoub's line sums it up: once there are no logs, the question stops being what is wrong with my code and becomes who owns the thing that never started. Most people learn that much later than an internship, so writing it up this clearly is useful.
Two small things have helped on teams I have worked with. First, put the CI quota on the same alert list as disk space and certificate expiry, so the person with the Owner role gets a warning at 80 percent instead of a pipeline full of red with no logs. Second, keep a short line in the project README that says who owns the runners, the quota and the billing. It turns the message you had to write into a two minute job.
Your standup summary, what is done, what is blocking it and who is sorting it out, is also a good template for any blocked ticket. Did your team add a quota alert afterwards?
I think adding who owns the runners is the README is a brilliant idea. We did leave a quota alert afterwards so that hasn’t happened in a while
A failure with no logs is a different class of problem. The question stops being what's wrong with my code and becomes who owns the thing that never started. Finding that person was the real fix here, and nobody teaches it.
Exactly 💯
The "code is fine, pipeline is broken" case is so disorienting because there are no logs to start from. CI minutes running out silently is a sneaky one to catch.
Right?! That total silence is the absolute worst. When you're so used to scrolling through a wall of red logs to find your mistake, seeing nothing makes you question your whole setup (and your sanity for a second). Glad to know I'm not the only one who's gotten tripped up by the silent compute limit trap