Here I go...automating myself out again.

Wanna talk? What happened?

But someone has to maintain the automation system ;)

@alexbrooklyn nothing "happened". I actually enjoy automating anything I can. That's my real joy. It usually takes me a few months to dial in the repetitive tasks and then I start to replace myself. It's becoming a neat hobby. :-)

@endor this is True. I have connections with my previous employer and the automation eventually breaks down when not maintained.

Oh I know that feeling! It's amazing, isn't it? :) building a script/pipeline that is going to replace some people!

@netikras well yes and no. The automation I write replaces myself. I try not to mess with anyone else. The motivation had always come from co-workers/supervisors waiting until the last minute to request something. Like, deadline tomorrow? Let's tell Chen today!

So, I was like...."it's the same everywhere. This isn't going to change. F*** it."

I started automating whatever I could. Anything and everything. Installs went from an hour down to 10 minutes (with configuration). That was before I got into DevOps. My 8-hour work day eventually turned into 2 hours of answering emails and initiating scripts.

Then, I left for this job...and now found my dialed in points of automation.

Yesterday. I was asked "why can't we automate that". This morning I came up with a good solution, and hoped it could be spread accross clients. I spent all of today testing every component of it and picking holes in my own solution. Probably not the best thing to do when you're proposing a solution. Luckily it was just internal though.

It was lucky I did that though because I found the limits of what was asked.

I work in big data and we are expected to handle any amount of data so knowing that it can take 200 seconds to report 200,000 rows is not comforting. That's low for us.

Id love to hear any other ideas on how to report the completion of an ETL and the resultng table.

Notes: the ETL is a custom EMR cluster, my solution is to catch the end event and use s3 select to count the final table.

Automating is beautiful. What language do you use to automate?

@Bubbles I use a combination of Linux bash scripting and Python. I guess that's pretty standard? I used to get shit for being a big Ruby guy. Whatever gets the job done.

Automation IS beautiful! I think it's one of the aspects of computing that really highlight the reason we have them.

@chenb0x dude I don’t know Ruby (yet) but you gotta show that shit off like a badge Ruby is too underrated and overshadowed by Python. I’ve used it before and loved it. It has some really unique concepts and everything is just great. I wish it got more credit. Is it any good for automating like python??

@Bubbles I agree. Ruby has great aesthetics to its code structure. You can definitely automate with it the way you can with any other scripting language.

I think I started shying away from it because the libraries in comparison to Python just weren't there. But is absolutely a solid language.

I was doing security research that I presented at defcon and just needed to code quickly...with library support. Maybe I just didn't research it well enough. I'm comfortable with both.

@chenb0x I’m eventually gonna pick it up, not for a job or anything, more or less cause I just want to. I love learning programming languages. Scripting languages are also exceptionally easier to learn luckily so I can stock pile them!