I once woke up to a $65k invoice from AWS and almost puked in my pants.

2 years ago I was cycling through freelancers on upwork to find a decent candidate. I guess i hired the wrong guy because even after revoking access, they got into my AWS account and spun up 1000's of ec2 instances in multiple locations.

I was a newly wed and feared for my life. How do I tell my wife I gota pay $65k with barely any savings.

I contacted AWS. And they started off by saying "we see some suspicious activity on your account, don't worry, we will handle it". I was like "yehhh cuz I've been paying pennies for the last couple of years. No way I can rake up a bill like that."

Since I was going through so many devs, it was impossible to figure out who. But even if I did find out, what am I gona do. I'm just a code monkey.

yep, that's a big risk in our industry, you never know how revengeful,sneaky,scammy they might be...

If you don't trust 'em ( and you shouldn't when they are fresh ) no root access or anything similar for them... thrust is building slowly...

bruh, cmon man

Precisely why I avoid AWS. I was a big fan at the start but the first time I had a large unintended bill from them I said “No more!”

We've been using Azure Service Bus for several years, monthly bill is usually between 50 cents and a few dollars. Few months ago our network admin sent an email concerned about our use of Service Bus because the Azure bill was over $50,000. Grumbles of moving Service Bus back in-house were being heard (been there, done that, no thank you)

After I saw the billing details, it was a machine learning subscription, not service bus. One of our business analyst created a bunch models and been running various scenarios for months (not realizing each time cost a lot of $$). He used the 'not my fault, the computer let me' excuse and Microsoft was willing to renegotiate some of the costs. I wasn't part of that discussion, so I don't know the details.

Flip side, we're not allowed around our AWS subscription(s) because Amazon does not negotiate when something goes haywire.

@PaperTrail i think every medium to big company should have atleast some kind of a system administrator that would look after such cases.

@Areg It was an expensive lesson for sure. Our admin didn't have threshold alerts for our dev side, which the BA was using. My guess is the BA knew that and decided to go under-the-cover and got caught. Nothing nefarious or anything, BA was using it for legit business purposes.

Bonus was the internal audit caught dozens of random Azure 'things' folks had created and forgot about that had wide-open/admin credentials.

I was guilty for a lot of those. :)

Just for reference, OPs case probably had to do with

1) them/the org removing the offender's user permissions from IAM, but failing to invalidate programmatic access keys.

2) the offender having valid ssh keys to a ec2 instance with a role with unnecessary permissions that allowed the attack.

@Nanos

Programmatic access keys are stupidly easy to compromise.

They *must* be rotated frequently and kept to minimum permissions.

AWS allows you to set billing alarm. You only need to set the threshold on how much you are willing to spend each month.

@CoreFusionX that's probably exactly what happened. Because they spun up thousands of instances in a matter of minutes. So they probably ran a script.

Just introduce 2FA into launching new services in AWS and that will sort out 90% issues like this.

I mean initiating any billable launch event, by a staff account, should be mandated to provide a one time code sent to the account owner's phone.

@Sid2006

That's a terrible solution which will bog down any deployment or provisioning process terribly.

Besides, such processes should be automated, and therefore use a programmatic key, which is not subject to MFA anyway.

The only acceptable solution is taking up responsibility for your resources and handle IAM properly.

@CoreFusionX he said staff account. If you are using staff accounts for deployment and provisioning the you are doing it wrong

This talk of data and the idea that data could be worth millions of dollars. One delete and fuck... I don't even work in that area and this gives me anxiety.