AboutPassionate and adaptable systems engineer with deep experience in automation, site reliability, performance, monitoring & Linux platforms. Presently preoccupied with Python, Kubernetes & Prometheus.
SkillsLinux, Python, Kubernetes, JS, Scala
Joined devRant on 7/21/2018
Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Of course the shouting episodes all happened during the era I was doing WordPress dev.
So we were a team of consultants working on this elephant-traffic website. There were a couple of systems for managing content on a more modular level, the "best" being one dubbed MF, a spaghettified monstrosity that the 2 people who joined before me had developed.
We were about to launch that shit into production, so I was watching their AWS account, being the only dev who had operational experience (and not afraid to wipe out that macos piece of shit and dev on a real os).
Anyhow, we enable the thing, and the average number of queries per page load instantly jumps from ~30 (even vanilla WP is horrible) to 1000+. Instances are overloaded and the ASG group goes up from 4 to 22. That just moves the problem elsewhere as now the database server is overwhelmed.
Me: we have to enable database caching for this thing *NOW*
Shitty authors of the monstrosity (SAM): no, our code cannot be responsible for that, it's the platform that can't handle the transition.
Me: we literally flipped a single switch here and look at the jump in all these graphs.
SAM: nono, it's fine, just add more instances
Me: ARE YOU FUCKIN SERIOUS?
Me: - goes and enables database caching without any approvals to do so, explaining to mgmt. that failure to do so would impair business revenue due to huge loading times, so they have to live with some data staleness -
SAM: Noooo, we'll show you it's not our code.
SAM: - pushes a new release of the monstrosity that makes DB queries go above 2k / page load -
Tho on the bright side, from that point on I focused exclusively on performance, was building a nice fragment caching framework which made the site fly regardless of what shitty code was powering it, tuned the stack to no end and learned a ton of stuff in the process which allowed me to graduate from the tar pit of WP development.5
It's gotta be the Linux kernel.
It's so good at managing base resources on all platforms that it allows hundreds of thousands of hipsta-ass devs to write shitty code and still get decent speed.1
*** writes unit tests with great zeal and fervor ***
IDE is reporting 85% line coverage, woohoo!
*** grins like a bastard, thinks it's perhaps time to stop and celebrate ***
*** runs branch/conditional coverage... ***
*** facepalm ***3
By far the stupidest thing I've done while drunk is setting up a layered LVM-over-RAID1 + striped LVM storage system *and* managing not to lose any data.
Next time I ran `lsblk` and realized how that structure came to be I kinda turned white.
I just couldn't let this one go.
See if you can figure out JO's message to facecunt
(made with gimp/gap)4
Any 'A Perfect Circle' fans out there? Was wondering if you could recommend some similar acts?
Got this tune stuck in my head resonating deeply:
"Time to put the silicon obsession down
Take a look around
Find a way in the silence
Lie supine away with your back to the ground
Dis- and re-connect to the resonance now
You were never an island"2
Like most people I needed some extra cash during uni, so I proceeded to learn CSS + Photoshop (yeah, I know). Followed by PHP and WordPress.
It can be a very shitty platform until you realize that you can stop combining plug-ins from all over the place with dubious code quality and roll your own.
Anyhow I kept at it until I was able to join a niche company doing a quite popular caching plug-in for WP (yeah, W3 Total) when I suddenly became *very* interested in anything and everything performance.
This landed me a very cozy consulting gig in the Nordics - they were using WP for an elephant-traffic website and had run into a myriad of perf issues.
Fixing them and breaking the monolith awarded me with skills in nodejs, linux, asynchronous caching among others.
I was soon in charge with managing the dev boxes for the entire team, and when the main operations dude left, I was promoted to owning the entire platform. (!) Tinkering with Linux for most of my life really came in handy here. (remember Debian potato?)
Used saltstack + aws cloudformation to achieve full parity between all environments. Learned myself some python and all various tips and tricks which in the end amounted to 90% reduction in time-to-first-byte and considerable cost savings.
By the end of the 2yr contract I had turned myself into a fullstack systems engineer and never looked back.
Lawyers not getting along resulted in us having to abandon NewRelic, so I got to learn and deploy the ELK stack as a homegrown replacement, which was super-fun.
Now I work in the engineering effectiveness department of a Swedish fintech unicorn where all languages under the Sun are an option (tho we prefer Python), so the tech stack is unlimited. Infinite tools and technologies, but with strong governing principles and with performance always in mind so as to pick the right tool for the job.
It's like that childhood feeling when you've just dumped a ton of Lego on the floor and are about to build something massive.
I guess the morale here is however disappointed you feel by your current stack - don't. Always strive to make things better, faster, more decoupled, easier to test, etc. and always challenge yourself to go outside the comfort zone.6
Hello devRant world, from Stockholm!
I've been lurking around for a while now and feel time has come to open up and start contributing :)
Thank you for being there and for lifting our spirits up, through good and dark times, equally.
(photo: mad light at midnight)13