I think I've shown in my past rants and comments that I'm pretty experienced. Looking back though, I was really fucking stupid. Since I haven't posted a rant yet on the weekly topics, I figure I would share this humbling little gem.

Way back in the ancient era known as 2009, I was working my first desk job as a "web designer". Apparently the owner of this company didn't know the difference between "designer", which I'm not, and "developer", which I am, nor the responsibilities of each role.

It was a shitty job paying $12/hour. It was such a nightmare to work at. I guess the silver lining is that this company now no longer exists as it was because of my mistake, but it was definitely a learning experience I hold in high regard even today. Okay, enough filler...

I was told to wipe the Dev server in order to start fresh and set up an entirely new distro of Linux. I was to swap out the drives with whatever was available from the non-production machines, set up the RAID 5 array and route it through the router and firewall, as we needed to bring this Dev server online to allow clients to monitor the work. I had no idea what any of this meant, but I was expected to learn it that day because the next day I would be commencing with the task.

Astonishingly, I managed to set up the server and everything worked great! I got a pat on the back and the boss offered me a 4 day weekend with pay to get some R&R. I decided to take the time to go camping. I let him know I would be out of town and possibly unreachable because of cell service, to which he said no problem.

Tuesday afternoon I walked into work and noticed two of the field techs messing with the Dev server I built. One was holding a drive while the other was holding a clipboard. I was immediately called into the boss's office.

He told me the drives on the production server failed during the weekend, resulting in the loss of the data. He then asked me where I got the drives from for the Dev server upgrade. I told him that they came from one of the inactive systems on the shelf. What he told me next through the deafening screams rendered me speechless.

I had gutted the drives from our backup server that was just set up the week prior. Every Friday at midnight, it would turn on through a remote power switch on a schedule, then the system would boot and proceed to copy over the production server's files into an archive for that night and shutdown when it completed. Well, that last Friday night/Saturday morning, the machine kicked on, but guess what didn't happen? The files weren't copied. Not only were they not copied, but the existing files that got backed up previously we're gone. Why? Because I wiped those drives when I put them into the Dev server.

I would up quitting because the conversation was very hostile and I couldn't deal with it. The next week, I was served with a suit for damages to this company. Long story short, the employer was found in the wrong from emails I saved of him giving me the task and not once stating that machine was excluded in the inactive machines I could salvage drives from. The company sued me because they were being sued by a client, whose entire company presence was hosted by us and we lost the data. In total just shy of 1TB of data was lost, all because of my mistake. The company filed for bankruptcy as a result of the lawsuit against them and someone bought the company name and location, putting my boss and its employees out of a job.

If there's one lesson I have learned that I take with the utmost respect to even this day, it's this: Know your infrastructure front to back before you change it, especially when it comes to data.

I've caught the efficiency bug.

I recently started a minimum wage job to get my life back in order after a failed 2 year project (post mortem: next time bring more cash for a longer runway)

I've noticed this thing I do at every job, where I see inefficiency and I think "how can I use technology to automate myself out of this job?"

My first ever application was in C++ for college (a BASIC interpreter) and it's been so long I've since forgotten the language.

But after a while every language starts to look like every other language, and you start to wonder if maybe the reason you never seriously went anywhere as a programmer was because you never really were cut out for it.

Code monkey, sure. Programmer? Dunno, maybe I just suffer from imposter syndrome.

So a few years back I worked at a retail chain. Nothing as big as walmart, but they have well over 10k store locations. They had two IBM handscanners per store, old grungy ugly things, and one of these machines would inevitably be broken, lost or in need of upgrade/replacement about once a year, per location. District manager, who I hit it off with, and made a point of building report with, told me they were paying something like $1500 a piece.

After a programming dry spell, I picked up 'coding' with MIT app inventor. Built a 'mostly complete' inventory management app over the course of a month, and waited for the right time.

The day of a big store audit, (and the day before a multi-regional meeting), I made sure I was in-store at the same time as my district manager, so he could 'stumble upon' me working, scanning in and pricing items into the app.

Naturally he asked about it, and I had the numbers, the print outs, and the app itself to show him. He seemed impressed by what amounted to a code monkeys 'non-code' solution for a problem they had.

Long story short, he does what I expected, runs it by the other regionals and middle executives at the meeting, and six months later they had invested in a full blown in house app, cutting IBM out of the mix I presume.

From what I understand they now use the app throughout the entire store chain.

So if you work at IBM, sorry, that contract you lost for handscanners at 10k+ stores? Yeah that was my fault (and MIT app inventor).

They say software is 'eating the world' but it really goes to show, for a lot of 'almost coders' and 'code monkeys' half our problem is dealing with setup and platform boilerplate. I think in the future that a lot of jobs are either going to be created or destroyed thanks to better 'low code' solutions, and it seems to be a big potential future market.

In the mean while I've realized, while working on side projects, that maybe I can do this after all, and taken up Kotlin. I want to do a couple of apps for efficiency and store tracking at my current employer to see if I'm capable and not just an mit app-inventor codemonkey after all.

I'm hoping, by demonstrating what I can do, I can use that as a springboard into an internal programming position at my current gig (which seems to be a company thats moving towards a more tech oriented approach to efficiency and management). Also watching money walk out the door due to inefficiency kinda pisses me off, and the thought of fixing those issues sounds really interesting. At the end of the day I just like learning new technologies, and maybe this is all just an excuse to pick up something new after spending so long on less serious work.

I still have a ways to go, but the prospect of working on B2B, and being able to offer technological solutions to common and recurring business needs excites the hell out of me..as cringy and over-repeated as that may sound.

Coming back here after years to rant about... myself.
TLDR: I fucked up and now have to call a thousand people as a dev, I'm not even getting paid for it and they all get crazy about a random ID that got assigned to them, so now I want to throw away all my electronics and become a skilift operator.

Stupid me deployed a project shortly before we have the largest amount of orders in the year. (Like 90% of yearly orders in a couple minutes cause they are sold out fast and people wait to order first)

I got this horrible legacy "plain self written framework php" project which I tried to upgrade state of the art.
There was one piece missing to upgrade everything and nicely deploy it to some fresh new servers which can handle the high load which peaks at the time orders open.

So I did it the day before orders open and... everything worked well! Nothing crashed.
I wrote my client to wait a little before he confirms the orders, since after confirmation each of the people who ordered will receive an email where they can choose a unique number which they'll receive as a sticker with the order.

Since it's an event my client is promoting, people will meet each other wearing those unique stickers and being able to identify each other online and in person with this number.

Suddenly my clients call me that "customers are complaining about that there is something wrong"
Turned out he confirmed all orders straight away and that part of the application which makes the number unique was broken on the update.
So everyone could chose any number (also taken ones) as his "unique" number.

In my panic, I told my client "It's my mistake, I'll deal with it of course and call the affected people in my free time, since it's my mistake you don't have to pay for it". (it's my largest client by far, am a freelancer)
Realizing when people can chose any number it'll not be a few ones who have the same, it's like almost everyone did chose "69", "1", "420", "88 (a scary amount of people)",... (with 69 being the number being chosen by most people btw, even more then "1")

So now I have to call about a thousand people telling them a new random ID will be assigned to them. I thought of course about mailing them, wrote a script that deals with the issue automatically, and FUCKED IT UP TOO so everyone is confused and the only way to deal with it is by a call basically.
And while I'm sitting here now for 2 days straight calling people in my free time about their random ID will have to change, I realized that some people are quite crazy about random ID's.
I'm talking about yelling and threatening because "is it too much to ask for a working website when ordering this expensive product".

I hate my life right now and am getting quite serious about throwing all my electronic devices away and become a skilift operator instead. Fuck the higher pay, it's not worth the shit, I wanna have only responsibility about one button to press while watching people fall on their face.

The last software I worked on in my previous company (a few months back), was a temporary replacement because they were switching techs. It was meant to be replaced within 2 years.

So, before I left, I added a kill. 2 years and 2 months into the future. First it spams the devs with emails "how is the tech upgrade going?" with no further clues. 6 months later it will start throwing random exceptions at random intervals. 6 months after that it just terminates the application immediately upon startup. Snuck it in between large commits, and since they stopped code reviews when I left, doubt they found it.

There is a setting in configuration with an obscure name to disable it all.

I marked the dates in my calendar. Would love to be a fly on the wall then.

Ok, here goes...

I was once asked to evaluate upgrade options for an online shop platform.

The thing was built on Zend 1, but that's not the problem.

The geniuses that worked on it before didn't have any clue about best practices, framework convention, modular thinking, testing, security issues...nothing!

There were some instances when querying was done using a rudimentary excuse for a model layer. Other times, they would just use raw queries and just ignore the previous method. Sometimes the database calls were made in strange function calls inside randomly loaded PHP files from different folders from all over the place. Sometimes they used JOINs to get the data from multiple tables, sometimes they would do a bunch of single table queries and just loop every data set to format it using multiple for loops.

And, best of all, there were some parts of the app that would just ignore any ideea of frameworks, conventions and all that and would be just a huge PHP file full of spagetti code just spalshed around, sometimes with no apparent logic to it. Queries, processing, HTML...everything crammed in one file...

The most amazing thing was that this code base somehow managed to function in production for more than 5 years and people actualy used it...

Imagine the reaction I got from the client the moment I said we should burn it to the ground and rebuild the whole thing from scratch...

Good thing my boss trusted me and backed me up (he is a great guy by the way) and we never had to go along with that Frankenstein monster...

Finally finished the longest ticket I've ever worked on in my life. The ticket title and description was a pretty simple and straightforward one: "Upgrade from PHP 7.4 to 8".

If it was only so simple in real life. Our application is mostly done with API Platform framework, which is based on top of Symfony framework which is based on top of PHP language.

Once I did PHP 7 => 8 upgrade I needed to upgrade API Platform 2 => 3. But of-course that couldn't have been done as before that I needed to upgrade from Symfony 5 => 6.

This all was literally an equivalent of touching into a wasp nest - it took me a bit over 5 months and 800 hours of work and there was literally not a single source file left untouched.

In the process of all of this I've ran into literally dozen undocumented feature-breaking changes, broken backwards-compatibility promises and inside out architectural changes - from both the frameworks and the language itself.

Upgrading just one major version of anything SHOULD NOT be so hard. And to top it all up just to think I will need to do this again in a year or two..

Experiences like these really set my hate for time-based model of releases and the state of today's development in general.

The year was 2006. During the first half of my career, I use to work in the NOC. This was before I made my transition to software engineer. I worked on the third shift for a bank services company. The company was on a down turn. Just years earlier they just went public, and secured a deal with a huge well known bank. Eventually they entered a really bad contract with the bank and was put into a deal they couldn't deliver on. The partnership collapse and their stock plummeted. The CEO was dismissed, and a new CEO came in who wanted to "clean things up".

Anyway I entered the company about a year after this whole thing went down. The NOC was a good stepping stone for my career. They let me work as many hours as I liked. And I took advantage of it, clocking in 80 hours a week on average. They gave me the nick name "Iron Man".

Things started to turn around for the company when we were able to secure a support contract with a huge bank in the Alabama area. As the NOC we were told to handle the migration and facilitate the onboarding.

The onboarding was a mess with terrible instructions that didn't work. A bunch of software packages that crashed. And the network engineers were tips off, as they tunnel between our network and the banks was too narrow, creating an unstable connection between us and them. Oh, and there were all sorts of database corruption issues.

There was also another bank that was using an old version of our software. The sells team had been trying to get them off our old software for over a year. They refuse to move. This bank was the last one using this version, and our organization wanted to completely cut support.

One of the issue we would have is that they had an overnight batch job that had an ETA to be done by 7 AM. The job would often get stuck because this version of the software didn't know how to fail when it was caught in an undesired state. So the job hung, and since the job didn't have logging, no one could tell if it failed unless the logs stopped moving for an hour. It was a heavily manually process that was annoying to deal with. So we would kill the JVM to "speed" the job up. One day I killed the JVM but the job was still late. They told me that they appreciated the effort, but that my job was only to report the problem and not fix it.

This got me caught up in a major scandal. Basically they wanted the job to always have issues everyday. Since this was critical for them, all we needed to do was keep reporting it, and then eventually this would cause the client to have to upgrade to our new software. It was our sales team trying to play dirty. It immediately made me a menace in the company.

For the next 6 months I was constantly harassed and bullied by management. My work was nitpicked. They asked me to come into work nearly everyday, and there was a point I worked 7 days with no off days. They were trying to run me so dry that I would quit. But I never did.

On my last day at the company, I was on a critical call with a customer, and my supervisor was also on the line. My supervisor made a request that made no sense, and was impossible. I told her it wasn't possible. She then scalded me on the call in front of customers. She said "I'm your supervisor, you're just a NOC technician, you do what I say and don't talk back". It was embarrassing to be reprimanded on a call with customers. I never quite recovered from that. I could fill myself steaming with anger. It was one of the first times in my adult life that I felt I really wanted to be violent towards someone. It was such a negative feeling I quit that day at the end of my shift with no job lined up.

I walked away from the job feeling very uncertain about my future, but VERY relieved. I paid the price, basically unable to find a job until a year and a half later. And even was forced to move back in with my mother. After I left, the company still gave my a severance. Probably because of the supervisor's unprofessional conduct in front of customers, and the company probably needed to save face. The 2008 crash kept me out of work until 2009. It did give me time to work on myself, and I swore to never let a job stress me out to that degree. That job was also my last NOC job and the last job where did shift work. My next few jobs was Application Support and I eventually moved into development full time, which is what I always wanted to do.

Anyway sorry if it's a bit long, but that's my burnout story.

Want to waste some time? Tell duplicity to

- - include /home/user/A/
- - include /home/user/B/
- - exclude /

and watch it backup only *part of* B.

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.

Was watching OITNB at home when boss called sounded urgent about SSL not working on one of our subdomains. We use a paid cloud app for some of our reports which. So the subdomain is a CNAME to the providers app subdomain. Recently there was an upgrade at our hosting but it shouldn't be related.

Boss: Hey, there is an error prompt when I visit our reporting site with https
Me: That's cos we never installed any SSL cert for that subdomain.
Boss: Well it worked before and you will need to get it fixed.
Me: Wait.. It worked before? How is that possible? We've never set it up and the subdomain is a CNAME pointing to another site which we don't own. The cert will have to load from their server and we have not done any setup with them.
Boss: I'm very sure it worked before the hosting upgrades. All along our customers has been accessing with https.
Me: Okay.... That's something new because and I am pretty SURE the last I checked, the app provider doesn't allow that yet.
* meanwhile I when to search the app provider docs and it says not able to support multiple SSL yet for CNAME
Me: Look, it says so here in the docs.
Boss: Ok, can you try to fix it as its important for the users to not see that error. It has been working all along.
Me: Hmmmm... I'll get back to you.

How do I fix something that didn't exist / broken?? How did it work before??

I know it can be possible to install the cert on the cloud provider end but we haven't done this before. And their support docs says feature not available yet.

Was it magic?? Am I missing something?? Anyway, I've sent an email to the provider's support team and telling them "it worked before"

Fucking hate to explain basic shit to computer illiterate. Usually I don't mind, but right know I working on the project, want to automate one thing I need to do every morning, put two numbers to web page(I will explain details maybe in next rant). So I am only one who fix, buys computers, printer(for some problems I call for other repair man.). Generally speaking working as IT guy. Firm has like 50 computers, some of them has SCADA software. Some computers have Win 7, some win 8 and others win 10, can't upgrade those computers, not enough money(I can deal with this problem). And yes, computer buying is not the fastest, easiest thing too. Because is public firm, I need to do public buying(I don't know how to translate to english), and most of the time wins the lowest price, I am ok with that. But I can't on item specification write I want that model pc or it components. Example: I can't write I want intel processor, however I can write number of cores, frequency. But it's not that bad, usually i have template for all things I buy. One of the worst thing is this, our firm bought new bookkeeping software version, old version was using visual foxpro framework. Good thing I didn't initiate the purchase, because right know I would be jobless, not because I would be fired, but because our senior accountant would drive me crazy. In fact accountants drive me crazy, but I can handle it for now. As I wrote before our form has about 120 workers, major part of workers are old, like my parents age. (I am 28 btw. Mom is 55.). As you all know what happens if you say you work with computers. So our accountants are like 60 years old, got new program, don't know how to work with it, and they ask me how to do certain things. if I don't know how to I ask program's support, every question is like 90 Eur. So in short accountants expect I should know their work and how program works. If I try say something they don't like, they try to make my day hard. Next thing is our billing program. Man that worked before me done some payments import. And when I came everyone expect me to do that. Ok I did that because that people working with billing program would probably fuck it up. And I semi automated that, so I don't mind that much. Sometimes that program fucks up, like it happened yesterday, it send email invoices attachment without filename. Example: people got this attachment ".pdf"(no filename, only extension), And if you save it you need do OPEN WITH command and then select pdf reader or rename file (I don't know what easier). And surprise surprise our firm, customer support redirects all phone calls, emails to me. But I did explain to customer support what to say to people. Still they redirect it to me.
PS: This is my first job after school. I work as part time.
TL;DR Thinking my life, carrier choices. accountants are not the nicest people.