All posts
Technology

Your Hobby Project Cost 20 Cents a Month for Half a Year and $2,700 Last Weekend

Stop the runaway before you read the invoice, then the billing case. Plus the guardrails that terminate resources instead of emailing you after the money is gone.

7 views
Photo: Artem Podrez / Pexels
A conversation with a TrueTalk advisor about: Your Hobby Project Cost 20 Cents a Month for Half a Year and $2,700 Last Weekend

Don't read the bill yet

$2,743.11. On the personal card. From an account that has cost you about twenty cents a month since February.

Your instinct right now is to open the invoice and understand what happened. Wrong order. Whatever caused this is, in all likelihood, still running, and billing data is refreshed roughly daily — so the number you're staring at is history, not the current position. Understanding comes second. Stopping comes first.

The first hour

Check every region. Not the one you use. The console shows you one region at a time and a runaway rarely respects that. Go to the Billing console, open Cost Explorer, and group by service, then by region, for the last seven days. That's two clicks and it usually names the culprit immediately.

If a region you've never used has compute in it, stop reading and treat this as a credential compromise. Mining on someone else's card is a common cause of a personal account going from cents to thousands overnight, and it has a recognisable shape: instances in regions you don't recognise, often several of them, often the largest type your account can launch. Rotate immediately. Deactivate and delete every access key on the account, including ones you think are unused. Rotate the root password, turn on MFA if it isn't on. If a key ever touched a public repo, a CI log, a Docker image layer, or a pasted config, assume it's out. AWS does automatically apply a quarantine policy to keys it detects as exposed, but detection isn't a guarantee and it isn't instant.

If it's your own code, kill the resource, not the symptom. And know that deleting the obvious thing often leaves the expensive thing behind. Terminating an EC2 instance can leave EBS volumes and snapshots. Deleting an RDS instance can leave a final snapshot you'll keep paying for. Removing a Lambda doesn't remove what it wrote to CloudWatch Logs, and log ingestion and retention are billed separately from the function.

Check the usual suspects while you're in there. NAT gateways bill per hour and per gigabyte processed, so a chatty container behind one adds up quietly. Data transfer out is charged. Public IPv4 addresses now carry an hourly charge whether or not anything is attached to them. Managed control planes for container services bill by the hour regardless of whether you deployed anything. A DynamoDB table on on-demand pricing with a runaway loop against it is a genuinely fast way to spend money.

A recursive function deserves its own paragraph. Lambda has loop detection that will stop certain recursion patterns after a set number of invocations, but it doesn't cover every service combination, and the list of what it covers changes. If your architecture has a function that can, directly or otherwise, trigger itself, don't rely on the platform to catch it. Set the function's reserved concurrency to zero right now. That's an instant, reversible off switch that doesn't require you to delete anything or work out what went wrong first.

One piece of good news that used to catch people: unauthorised requests to your S3 buckets that fail with an access-denied error aren't billed to you as the bucket owner. AWS changed that. If you'd been told a public bucket getting scanned could bankrupt you, that specific fear is out of date.

The refund conversation

Now you can look at the invoice.

Billing and account support is included on the free Basic support plan. You do not need a paid support plan to open a billing case, and if anyone tells you otherwise they're wrong.

What's true, and what I'd rather say plainly than let you find out the hard way: a one-time billing adjustment is discretionary. It is a goodwill decision, not an entitlement, and there's no policy you can point to that obliges anyone to grant one. Plenty of people get one. Plenty of people don't. The variables that appear to matter are whether it's a first occurrence, whether it was a genuine mistake rather than usage you benefited from, and whether you've already fixed it.

So write the case accordingly. Short, factual, no drama:

State what happened in one sentence. Name the exact service, region, resource and date range. Say what you've already done to stop it — the specific actions, not "I've fixed it." Say what you've put in place to prevent a recurrence, and name the actual controls. Then ask, once, for consideration of a one-time adjustment on a personal account. Don't threaten to leave. Don't mention what you earn. Don't send five follow-ups in a day.

If a credential compromise was involved, say so directly and include the timeline of when you rotated. Fraudulent usage sits in a different category from usage you caused, and they'll want to know which one they're looking at.

TrueTalk has an advisor called Cloud Architect Maya, an AI persona working in cloud computing, system architecture and DevOps. Not an AWS employee, no ability to see your account or influence a support case. It's useful for the bit at 2am where you've found something running in a region you've never touched and need a second read on what it is before you delete it, or want to sanity-check the wording of a case before you send it. It's available instantly.

Guardrails that actually fail closed

Budget alerts email you after the money is spent. Treat them as reporting, not protection.

Budget actions. A budget can be configured to do something at a threshold rather than just tell you — apply a restrictive IAM policy or service control policy, or stop EC2 and RDS instances. It needs a role with permission to act. This is the closest thing to a kill switch that exists on AWS and setting one up takes about fifteen minutes.

Cost Anomaly Detection. Free, and it flags unusual spend patterns rather than waiting for a threshold, which means it can catch a spike before your monthly budget number is anywhere near breached.

A separate account per project, under an Organization. Then a service control policy that denies every region you don't use and every service you don't need. This is the strongest control available to an individual, and it's the one that would have prevented the crypto-mining scenario entirely, because instances in an unavailable region simply fail to launch.

Ask support to lower limits you'll never need. Some service quotas can be reduced on request. A hard ceiling you can't accidentally exceed beats an alert you read the next morning.

Worth knowing if you also use Google Cloud: it has a documented pattern for programmatically disabling billing on a project when a budget threshold is hit, using a Pub/Sub message and a function. It's a real off switch, and it will take down everything on that project, which is the point. AWS has no equivalent single button. Budget actions are the nearest thing and they're narrower.

The last piece is the least technical. Set a monthly budget on a personal account at a number that would genuinely upset you — twenty dollars, forty, whatever your actual line is — with actions attached, not just alerts. The instinct after a bill like this is to build elaborate monitoring. What you need is one thing that turns the tap off while you're asleep.

awscloud costsside projectsdevopsbilling