Is this the code life
Another scrum meeting
Caught in the the Node life
No escape from reality
Open your eyes
Look up to the screens and see..
I'm just a dev boy
Doing some debugging
Because there's warnings here
Errors there
Segment faults
Everywhere
Anytime you distract
Takes another hour from me
From me

*piano starts

Mama. Just committed a bug
Merge the branch to production
Did it fast for milestones
Mama. The repo has just begun
But now they going to throw the stack away.
Mama. U u u uu
Didn't mean to code in LAMP
But it's the only stack i know how to setup
In Ubuntu. Without docker
I really don't get vagrant

*piano

It's too late
My team is done
Some dev is working in Nepal
A UX dev. Now what is that?
Goodbye everybody
I've got to go
Gotta leave this lame meeting
And face the truth
Oh nooooo. I i interns
(they have questions)
I want to debug
I don't want to stay till 3 in the morning

*epic guitar

I see a litlle dev over there
Let's code review, let's code review
Did he do the last commit?
Coding in the white board
Very very frightening me
That's bug(that's a bug)
That's a bug (that's a bug)
What the f*ck did you do that?
Magnificcooooooo

I was just coding and nobody liked it
He was coding and nobody liked it, spare his some time to do his debugging

Easy man. Here go. Will you let me code?
A meeting. No,we will not let you code. ( let me code)
A meeting. we will not let you code. ( let me code)
A meeting. we will not let you code. ( let me code)
We will not let you code
Never never let you go
Never let you code, oh
No no no no no no no
Oh mama mia, mama mia ( dude, you've gotta let me code)
Screw you guys, I'm gonna code and commit. Commit. Comiiiiitt!

*epic guitar

So you think you can review me and spit in my eye?
So you think you can dump me and erase my branch?
Oh baby, cant do this to me baby
I've just have to log out.
I've just have to log outta here

*epic guitar solo

Nothing really matters
The users will not care
Nothing really matters
To them

Any way this code blows

So I was hired about 4 months or so in this companty, we will name it 'Derp & Co.'

The first task they want me to do was to 'clean' an android app that, for what they told me:

- Previous dev fired. said that tasks have been done but totally a lie.
- Took a fully week of 2 fellows coworkers to 'undo' the mess.
- And for the last but not least, zero documentation, like ZERO.

So, I clone the repo, install android studio, blah blah blah, get hands to the pile of code and jesus...

- The whole app was working with a gargantuan json, there was no use of POJOs at all. Objects are for normies.
- A masive copy/paste code, like 'I will need this here, crtl-c... ctrl-v, DONE!'
- Threads are free, isn't it? let's just put a thread whenever I desire to make an HTTP request and not reuse code at all.

So... with this on mind, my first task is to make proper objects:
- Coworker: 'Sorry dev, we don't have documentation for this, you must debug the code to se what the server will send to you'.
- Me: 'Real?'

Shit... ok. So I first try to figure out how the hell is made my gargantuan json. A month was entirely lost to unravel this data and implement Objects, improve their code, reuse code, etc. but at the very end:

- coworker: 'Good job dev, when the POJOs are done, we can focus on the next task, whe have to define a new DATA MODEL because the one we are using now is not good at all'.

*note: the app is on production and working with all the previous 'features' and today it still on use on some enviroments.

- Me: 'Wait... this is a joke, now you want to define new data models? This should have been done in first place!' <WTF face>
- Coworker: 'I don't think so dev, Mr. boss have this list with things to improve on the app an this is the order of do the tasks'.

Mr. boss is on vacations, two days after he came back:

- Mr boss: 'Coworker said that you have been working with POJOs, is that right?'
- Me: 'Yes'
- Mr boss: 'Why? Did not see the need of a new data model?'
- Me: 'I told that to him, but he insist on "the order" of the list.
- Mr. boss <facepalm>

This is one of the few tales i have from 'Derp & Co.'

PS: Sorry if i made a mistake on writing, english is not my first language and maybe I have done some mistakes.

A while back a co-worker of mine fucked up by leaving some debug code and pushing to production.
He quickly repaired it, redeployed and everything was good again before the customer experienced any issues.

Later that day, management showed up by his desk to ask what happened, how it happened and stating that he was not "angry enough" about his fuck up, long after it has been repaired.

Up to this day i regret not asking in what unit of measurement we could determine if we were angry enough; decibels? gray hairs? grams of shit in my underwear?

I was only seventeen back then and I was a Java Developer Intern, not knowing much about enterprise oriented coding.

The project leader in our dev team saw a lot of potential and passion in my work, but was convinced I wasn't taught enough to do the right thing.

I was mainly doing shitty mappers and services back then, which were somewhat used but never lasted long and were ditched a few months later, which always bummed me out. I wanted to make an impact on REAL projects that would deploy into production.

So Mister Mentor (GDPR forbid to use the actual name), who was always first to come and last to leave the office, taught me what it means to code for real.

We stayed after 5pm until 7-8pm multiple times a week and he taught me in a deeply understanding and calm way how to:
- Git (SVN)
- Refactor
- SOA
- Annotate
- Deploy
- Unit Test

And most importantly:
- How to debug like an absolute BOSS
(We even debugged native Java Libraries just for fun to see if we could break them)

Fast-forward a month later and little intern me made his first commit on production.

Without Mister Mentor, I wouldn't be half as good of a developer as I am today.

Advice to all new programmers, take this one from personal experience. DO NOT PUT SWEAR WORDS IN DEBUG STATEMENTS.
You will miss one, it will go to production and it will get picked up by your log monitoring...

So, a few years ago I was working at a small state government department. After we has suffered a major development infrastructure outage (another story), I was so outspoken about what a shitty job the infrastructure vendor was doing, the IT Director put me in charge of managing the environment and the vendor, even though I was actually a software architect.
Anyway, a year later, we get a new project manager, and she decides that she needs to bring in a new team of contract developers because she doesn't trust us incumbents.
They develop a new application, but won't use our test team, insisting that their "BA" can do the testing themselves.
Finally it goes into production.
And crashes on Day 1. And keeps crashing.
Its the infrastructure goes out the cry from her office, do something about it!
I check the logs, can find nothing wrong, just this application keeps crashing.
I and another dev ask for the source code so that we can see if we can help find their bug, but we are told in no uncertain terms that there is no bug, they don't need any help, and we must focus on fixing the hardware issue.
After a couple of days of this, she called a meeting, all the PMs, the whole of the other project team, and me and my mate. And she starts laying into us about how we are letting them all down.
We insist that they have a bug, they insist that they can't have a bug because "it's been tested".
This ends up in a shouting match when my mate lost his cool with her.
So, we went back to our desks, got the exe and the pdb files (yes, they had published debug info to production), and reverse engineered it back to C# source, and then started looking through it.
Around midnight, we spotted the bug.
We took it to them the next morning, and it was like "Oh". When we asked how they could have tested it, they said, ah, well, we didn't actually test that function as we didn't think it would be used much....
What happened after that?
Not a happy ending. Six months later the IT Director retires and she gets shoed in as the new IT Director and then starts a bullying campaign against the two of us until we quit.

After months of development, testing, testing and even more testing the app was ready for deployment to production. Happy days, the end was in sight!

I had a week's leave so I handed over the preparation for deployment to my Senior Developer and left it in his capable hands while I enjoyed the sun and many beers.

I came back on the day of deployment and proudly pressed the deploy button. Hurrah!

Not long after I got loads of phone calls from around the country as the app wasn't working. What madness is this?! We tested this for months!

Turns out my Senior didn't like the way I'd written the SQL queries so he changed them. Which is obviously both annoying and unprofessional, but even worse he got a join wrong so the memory usage was a billion times more and it drained the network bandwidth for the whole site when I tried to debug it.

I got all the grief for the app not working and for causing many other incidents by running queries that killed the network.

So...much... rage!!!

Me when I debug in production

People don't seem to know how to properly do print-debugging, so here's a simple guide:

1. A log of "aaaaaa" or "got here" isn't as helpful as you think when ALL OF THEM ARE THE FUCKING SAME. You put a descriptive label or copy verbatim the conditional statement. This saves time matching statements, allows one to watch multiple branches at once, and allows others to understand and help faster when dragged in to help.
2. When trying to see where code fucks up, before each line, paste said line into a proper print statement for your language. If there's, say, a function call or some shit, have it output something like "functionCall(varA=<varA contents>,varB=<varB contents);" Most normal lines should be like this too, but it's especially helpful for calls and comparisons.
If need be, add return values after if they're not shown in another print statement later.
This allows for a trail of execution AND the line that fucks up will be the last in the log, making finding it easier when dealing with hangs and such.
3. Putting something unique like "DEBUG: " or something in front of all statements ensures you can just search for them to ensure you're not rolling one out to production. It also separates debug output from normal output at a glance, making digging through logs faster.

Discovered pro tip of my life :
Never trust your code

Achievements unlocked :
Successfully running C++ GPU accelerated offscreen rendering engine with texture loading code having faulty validation bug over a year on production for more than 1.5M daily Android active users without any issues.

History : Recently I was writing a new rendering engineering that uses our GPU pipeline engine.. and our prototype android app benchmark test always fails with black rendering frame detection assertion.

Practice:
Spend more than a month to debug a GPU pipeline system based on directed acyclic graph based rendering algorithm.

New abilities added :
Able to debug OpenGL ES code on Android using print statement placed in source code using binary search.

But why?
I was aware of the issue over a month and just ignored it thinking it's a driver bug in my android device.. but when the api was used by one of Android dev, he reported the same issue. In the same day at night 2:59AM ....

Satan came to me and told me that " ok listen man, here is what I am gonna do with you today, your new code will be going production in a week, and the renderer will give you just one black frame after random time, and after today 3AM, your code will not show GL Errors if you debug or trace. Buhahahaha ahhaha haahha..... Puffff"

And he was gone..
Thanks satan for not killing me.. I will not trust stable production code anymore enevn though every line is documented and peer reviewed.

Dear Target App Developers,

I think you left some debug code in your app...

I know you want to know if someone is using a VPN and that's cool with me, I get it.

But when I'm on a VPN your app constantly pops up a modal with my IP address every time I change a page in your app....

Might want to look things over a bit closer before you put it in production ;)

Pushed to production with a debug message left in. Whoops, debug message includes the private key. Ummmm...

If they followed my suggestion and went straight to debugging the server issues they would have been solved it from week 1 and everyone would have thought the migration had a minor performance hiccup. In fact, we have already done such at least twice before and nobody batted an eye.

Instead they self-labelled the migration a failure on first error, setting the stage for apologizing to the client, and put themselves on the spot for a whole staging / production signoff, replication / backup worfklow, almost a blue-green "seamless" deployment reminiscent of DigitalOcean.

Well they're not DigitalOcean, and anyone who has spent any time understanding users knows they will not participate in "new system" tests long enough to find or report issues.

So of course the migration stretched out to almost three months up until the whole reason for the migration - the rapidly escalating risk of the old provider disappearing - hit like a freight train and now they have to go through the problem of debugging the server like I told them to on week 1. Only this time they've set the client mindset against it, lost any chance of reverting, have had grave risk for data loss, and are under pressure to debug other people's code in real-time.

This is why I don't trust devs to do ops. A dev's first solution to any problem is to throw tech at it.

It is not on production anymore, but it was for long enough. Someone thought it would be a great idea to be able to debug a web app while signed in as a user reporting a problem. How to do it? It's easy. Just check on every request if magic HTTP parameter SIGN_IN_AS=id is present and if it is, sign in as this user. Of course, it worked also with admin account with hard-to-guess id=1.

> Moment you thought you did a good job, but ended up failing
> Times the bug wasn't actually your fault
> Times you took the blame for a junior or other dev
> Times someone took the blame for you
> Times you got away with something you shouldn't have.
> Most valuable data loss
> The bug you never fixed
> Most satisfying bug to fix
> Times where a "simple" task turned out to be not so simple
> Debug code left in production?
> Moments you wish you could undo
> Most satisfying optimization
> Have you ever been ranted about?

We support a system we inherited from another company, it’s an online document store for technical specifications of electronic devices used by loads of people.

This thing is the biggest pile of shite I’ve ever seen, it wasn’t written by developers but rather by civil engineers who could write vb...so needless to say it’s classic asp running on iis, but it’s not only written in vbscript oh god no, some of it is vb other parts is jscript (Microsoft’s janky old JavaScript implementation) and the rest is php.

When we first inherited it we spent the best part of 2 months fixing security vulnerabilities before we were willing to put it near the internet - to this day I remain convinced the only reason it was never hacked is that everything scanning it thought it was a honeypot.

We’ve told the client that this thing needs put out of its misery but they insist on keeping it going. Whenever anything goes wrong it falls to me and it ends up taking me days to work out what’s happening with it. So far the only way I’ve worked out how to debug it is to start doing “Response.AddHeader(‘debug’, ‘<thing>’) on the production site and looking at the header responses in the browser.
I feel dirty doing that but it works so I don’t really care at this point

FUCK I hate this thing!

When you find this in production code and git blames you for this. Luckily no one can every see this log output, because the next statement closes the frame.

The real life of me as a trainee developer:

New system works locally but fails to work in production and dev.

Proceeds at futile attempts to debug for hours to find out that my connection strings in the transforms were nested inside logging.

Please Please Please remove your shitty ass 'console.log' before commit i don't need your fucking "debug mode" in production

Oooh I have quite a few,
My favourite: accidently left a log. Debug("bollocks") in a try catch this made it through testing and does (still) occasionally go into production log files.

Worst: wrote an interceptor for jboss with the intent of checking cache for some lookup data. I picked the wrong one of two similarly named methods and instead queried the database, I effectively wrote a denial of service utility into our app

Debugception, when you are debuggin someones debug code in a production environment but nobody knows what he was debugging because he left the company.

You know what's hard...

Fixing a bug which occurred in production without having any logs because you log that useful info at debug level. 😧

Now take pen and paper, do calculation on your own and speculate what would have happened in production.

Been put on debug duty, shit fucking SUCKS ASS.

Demotivating as hell seeing other people implementing cool features while you're doing this stupid shit trying to reproduce bugs that appear in production. Fucking hell.

So, someone from support department asked me to check the app as it was displaying blank information on a page.

I started debugging on the api which i found is doing the proboem. I checked and it was working a moment before but not now. Switched in debug mode to see what went wrong and it works again. This happen multiple times.

Before doing anything else i asked our api developer to check if api is working. He said it should work now and problem has been fixed.

Later i found out that he was doing debugging/changing code on production server instead of his local machine or test server.

Why shouldn't I clone production DB locally in order to debug an issue / recreate a bug? What is the alternative?

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.

A CASE AGAINST BLUE PRISM
Let's review one of the worst weeks I had with Blue Prism

Monday: Yay! Solved one of the problems we've been carrying around for a week before.

One of the robots suddenly became slow. Like, REAL slow. A process that would take 3 minutes per record now takes 45, and that broke apart all the following schedule.

There were no updates on the application server, the production machine, the robot, it just became slow. And not always slow; a process manually run from console room would work, a process in debug room would work, it's just the scheduled part that caused problems.

It turned out, BP didn't seem to like that particular combination of schedulation + process + machine. Moving the process to a different machine seemingly fixed that. IDK why.

Tuesday: One of our processes waits for a code to appear in the page, and when that happens, it memorizes this code. However, now it is always returning blank. Worked for months, now it breaks every single time.
After half a day of debugging a bug which DIDN'T HAPPEN IN DEBUG MODE YET AGAIN, at 11pm I decided to just place a nonsensical timeout in page before reading and call it a day.

WEDNESDAY: a scheduled process didn't start. "No sessions created". Thanks Blue Prism, very cool.

THURSTAY: This time, schedulation did start, but the process is "waiting". As in: it's 9:30 am, the process has been stuck in the same step since 6:00 am. Turns out, it blocked during a navigate stage; you need to send a string to clipboard using the standard BP action for that, then paste and click "enter", but for some reason the standard BP object sent "ORRCO" instead of "ORRICO" to clipboard, which obviously returned no results and then... the process just didn't feel like doing things anymore. No errors, no logs, nothing: just sitting on its ass. Because fuck you that's why.

Friday: another process uses a very moderate amount of scripts to work. Nothing really fancy, just a couple of lines of code to place in page some IDs and selector to help BP do its thing, otherwise selecting these elements would be a nightmare.

But

Failed while invoking javascript method:Exception from HRESULT: 0x80020101-> at mshtml.HTMLWindow2Class.IHTMLWindow2_execScript(String code, String language)

The same script -it's not dynamically generated-worked yesterday, the day before and the day after. But sometimes it will not. Why? The answer, my friend, is blowin'' in the wind

So I am debugging a connector library for an api that users curl.

I am fighting my ass off with errors and a lack of debug, testing or thought for CI

Take a poke around this set of classes only to find.

Postman token in the opts. and a removal of ssl check. What you straight copied from postman.

Like seriously clean up your fucking code if you are gonna put something out as production ready to your team.

Console.log('fuck');

Sometime ago I had to debug a override to a Joomla component made by the trainne.

So the component work just fine on development but not in production. I made the transition myself so I knew the code was the same.

This is my first rant, quite long, hope you like it.

So I thought something went wrong in the database, I searched and didn't find a thing.

Then I decided to check the code... hold yourself now

The component had to support internalization ..so instead of using the proper function to see what the current language is, he made a if with the URL hard coded..

Like http:://dummy.com/en/

:)

I have this system that receives an image on Base64 or the URL (in that case the system makes an HTTP request and saves re response to disk). It works beautiful when I run the tests, it doesn't work at all in production (all the resulting files are corrupted). I don't know how to start to debug it so I'm going to bed and let my future self to resolve the issue.

Today the DBA-team (needed to keep our Oracle mess running) decided to change datatypes in tables that has been in production for 20 years. Without a headsup.

Great success!
Just took us 5 hours to debug good ol' visual basic code......
Why would you do such a thing!?

Sometimes, certain features don't work in my app if it's uploaded to the production track in Google Play.

It works in debug mode.
It works in release mode.
It works in the internal testing track.
...but it won't work in the production track? Y'know, the one that actually matters? Theoretically, there shouldn't really be a difference if the same exact APK was uploaded to the internal testing and production track, but... idk dude.

As a result, I implemented a secret way to test a feature in the production build (it's an app to remotely control OBS Studio): if the first connection you added is named "yayeet_" and you open the disclaimer, it tests the feature. Luckily, I got some of the stuff figured out, but I just thought the way I had to test it in production was dumb.

Sometimes I don't get "don't test on production".
And I'm definitely not a front-end guy, I only have debug and release in mobile development.
And I definitely often test on release, because it may be broken while debug build works fine.
You know what that means?
1. Test locally
2. Try to fix issues
3. Realize that this issues would ever appear ONLY locally
4. Move to staging and test
5. Fix issues
6. Realize that most of them are caused by workarounds for localhost
7. Move to production
8. Realize that everything is fucked up and you don't have any idea why, because `h5aqq2 was called on null"

It's been a good month where honestly I had nothing to rant about. Pretty much doing my own project setting up ELK.

But last few days I had to return to the reality called teammates....

It where it ok... I mentored one of them, then did the code review yesterday

And that's when the shit hit the fan.

I told them to do X but then they did Y instead thinking that they were smart.

In hindsight they seem to have no idea wtf they were doing, inexperienced and couldn't even use console.log and JSON.stringify to debug object states...

Which course now reminded what's wrong with this team, you got people jumping around stacks and projects so they're all mediocre on all of them. Rather than having specific people being good at one of them (aka more experienced than a noob).

And if course this morning, manager asked me to look into something on a program I haven't support in a while (there are a free people that are more experienced and know the current state better). And he said this is quick and urgent... And actually when he said that I'm like uh.... don't think so....

And last thing is we had to rerun a report in production so needed the shipper ten to do it. Asked them look yesterday, users were waiting.

Today... Still not done. And well I actually can run the report myself locally.. takes 5mins but in production they need to reload the data but that should take at most 20mins... Either way... Nothing was done.

Oh and I just remembered I raised a request to it SA group to have some not script installed... That not done either.

And this is why relying on others it at least these people is a bad idea..... Unless your are capable of firing them...

What is worse code that works in debug but not in production or code that doesn't work in debug but does work in production. Both without usable stack traces.

Currently batteling both....

Someone forgot to disable debug reports in production

I want to build a "report-error" function into the app to make errors in production easier to track and debug. But i am unsure, what it should do actually.
Create a GitHub Issue? Then every Hoster needs a specific GitHub account and repo for that.
Send an Email? Then connectivity-related issues are not sendable. Send an Email with the frontend? Brr. Any Tips?

I got my first client at upwork almost a week ago and the experience has been awful so far, not because of this client but because of the codebase, it's so bad, it is running DEBUG=True on production and if I turn it DEBUG=False things break for some fucking reason that makes no sense (I don't think that's true but the previous developer states it). The website is running on pythonanywhere which is weird, bootstrap is a nightmare, the database needs to be in sync all the time using a manage.py command that executes tasks received through a webhook from a Hubspot shit that has all the information. Just adding a simple edit/verify profile on that site is such a fucking nightmare. The whole project its full of holes and things that are just screaming to break, its like a fucking house of cards that falls to the ground the second I edit something and it looks like its my fault. I'm thinking of telling the client that I will no longer work on this project