My mentor/guider at my last internship.

He was great at guiding, only 1-2 years older than me, brought criticism in a constructive way (only had a very tiny thing once in half a year though) and although they were forced to use windows in a few production environments, when it came to handling very sensitive data and they asked me for an opinion before him and I answered that closed source software wasn't a good idea and they'd all go against me, this guy quit his nice-guy mode and went straight to dead-serious backing me up.
I remember a specific occurrence:
Programmers in room (under him technically): so linuxxx, why not just use windows servers for this data storage?
Me: because it's closed source, you know why I'd say that that's bad for handling sensitive data
Programmers: oh come on not that again...
Me: no but really look at it from my si.....
Programmers: no stop it. You're only an intern, don't act like you know a lot about thi....
Mentor: no you shut the fuck up. We. Are. Not. Using. Proprietary. Bullshit. For. Storing. Sensitive. Data.
Linuxxx seems to know a lot more about security and privacy than you guys so you fucking listen to what he has to say.
Windows is out of the fucking question here, am I clear?

Yeah that felt awesome.

Also that time when a mysql db in prod went bad and they didn't really know what to do. Didn't have much experience but knew how to run a repair.
He called me in and asked me to have a look.
Me: *fixed it in a few minutes* so how many visitors does this thing get, few hundred a day?
Him: few million.
Me: 😵 I'm only an intern! Why did you let me access this?!
Him: because you're the one with the most Linux knowledge here and I trust you to fix it or give a shout when you simply can't.

Lastly he asked me to help out with iptables rules. I wasn't of much help but it was fun to sit there debugging iptables shit with two seniors 😊

He always gave good feedback, knew my qualities and put them to good use and kept my motivation high.

Awesome guy!

Is this the code life
Another scrum meeting
Caught in the the Node life
No escape from reality
Open your eyes
Look up to the screens and see..
I'm just a dev boy
Doing some debugging
Because there's warnings here
Errors there
Segment faults
Everywhere
Anytime you distract
Takes another hour from me
From me

*piano starts

Mama. Just committed a bug
Merge the branch to production
Did it fast for milestones
Mama. The repo has just begun
But now they going to throw the stack away.
Mama. U u u uu
Didn't mean to code in LAMP
But it's the only stack i know how to setup
In Ubuntu. Without docker
I really don't get vagrant

*piano

It's too late
My team is done
Some dev is working in Nepal
A UX dev. Now what is that?
Goodbye everybody
I've got to go
Gotta leave this lame meeting
And face the truth
Oh nooooo. I i interns
(they have questions)
I want to debug
I don't want to stay till 3 in the morning

*epic guitar

I see a litlle dev over there
Let's code review, let's code review
Did he do the last commit?
Coding in the white board
Very very frightening me
That's bug(that's a bug)
That's a bug (that's a bug)
What the f*ck did you do that?
Magnificcooooooo

I was just coding and nobody liked it
He was coding and nobody liked it, spare his some time to do his debugging

Easy man. Here go. Will you let me code?
A meeting. No,we will not let you code. ( let me code)
A meeting. we will not let you code. ( let me code)
A meeting. we will not let you code. ( let me code)
We will not let you code
Never never let you go
Never let you code, oh
No no no no no no no
Oh mama mia, mama mia ( dude, you've gotta let me code)
Screw you guys, I'm gonna code and commit. Commit. Comiiiiitt!

*epic guitar

So you think you can review me and spit in my eye?
So you think you can dump me and erase my branch?
Oh baby, cant do this to me baby
I've just have to log out.
I've just have to log outta here

*epic guitar solo

Nothing really matters
The users will not care
Nothing really matters
To them

Any way this code blows

Debugging in production be like...

The time my Java EE technology stack disappointed me most was when I noticed some embarrassing OutOfMemoryError in the log of a server which was already in production. When I analyzed the garbage collector logs I got really scared seeing the heap usage was constantly increasing. After some days of debugging I discovered that the terrible memory leak was caused by a bug inside one of the Java EE core libraries (Jersey Client), while parsing a stupid XML response. The library was shipped with the application server, so it couldn't be replaced (unless installing a different server). I rewrote my code using the Restlet Client API and the memory leak disapperead. What a terrible week!

I extracted a tangled action to its own api, and wrote a test for it.

The test failed.

I added debugging, more debugging, all the debugging. It still fails. But I can at least see why it fails!

It turns out the api finds and updates the wrong user. It finds and updates the wrong user EVEN WHEN THERE ARE NO OTHER USERS.

WHAT THE SALAMI.

Also, the user lookup it uses is extremely roundabout and takes several seconds with ~2 million users. Normally I'd fix the lookup, but it has been in production for several years, and I'm terrified it will break something if I fix it.

Blargherhagrid.

A bug in production, so I was debugging and the boss says to me "how do you stay so calm?" and I answer "just like a surgeon does't have to freak out with blood, a developer doesn't have to freak out with bugs"

Freaking tech support.
Freaking sparkhire.

Their 'one-way interview' bs only supports flash. Flash. in production. in 2019. Flash died years ago, and its support ends next year. What the crap?

Anyway, I finally decided I should do the interview since they already have all of my information anyway. Thanks, "privacy-conscious" third party. Totally appreciate it.

I spent half an hour and couldn't get flash working on their site (but all other sites were fine), so I contacted their support. I gave them all the relevant specs (inc. ofc browser), the steps to reproduce, and all of my attempts at fixing the issue.

To their credit, I recieved a response within a few minutes. To their discredit: their response was: "What browser are you using?" This question was followed by my report (including, ofc, my browser and all the other overlooked details), immediately followed by a "debugging info" section appended by their support service that also included my browser, os, and other specs.

Learn to fucking read.

Their suggestion? Use google chrome. Barring that: record your 20-30 minute video by holding your phone in front of your face the entire time. I am so not kidding.

They also asked what page i was having difficulty on. You guessed it: the page url was also included within that "debugging info" section.

It wasn't a form letter, either. I'd understand if it was all automated, but it was a real person who was really typing up the emails, and really didn't bother reading a damned thing.

I did end up getting flash working, but their "tech support" (script-reader) was entirely useless.

Today's highlights include:

The offshore team has put code gems in production featuring the example code generated on project startup that you're supposed to delete or overwrite, an API endpoint that just returns the value 5, and various debugging console.logs. it's a delight reading their code.

My boss also forgot the meeting he called me in for so I've been sitting here waiting for 20 minutes when I could have gone home. I'm glad it's friday

I tried writing this rant before, but I was (and still am) in too good of a mood so it was lengthy, meandering, and over-specific. so I'll summarize(ish).

summary:
* miscommunication
* working weekends
* incompetence and/or screwy integrations

summary of the summary:
* I can't fix someone else's mess if you don't talk to me!

Summary^3: #TODO: learn telepathy

Shortened rant:
Bossman at work signed up a very lucrative client by promising them something he couldn't deliver because he misunderstood and miscommunicated scope -- anti-fraud, if you've been following my rants.

Their signup (all four...) are screwy and cause issues and nobody knows why. I didn't write the code, have barely even glanced through it, and it uses a third-party (Clover) that's rather screwy.

Bossman has been asking me to do various things concerning the merchant, but has never been around to provide specifics, so I'm left to guess. I've done my best, but due to the aforementioned screwiness, I really have no idea what's going on. I just sort of muddled my way through.

Bossman also asked me, super late on Friday night (after 8:30pm), to rename one of the merchants because there are two with the same name (with different Clover creds, etc.) and that's just confusing. I didn't see the message because late and tired, and he didn't follow up or text/call me until two days later (today, Sunday). I also thought these were strictly for diagnosing and were de-listed. I had no idea the merchant was live and people were actually purchasing things for it. Had I known this I would have freaked out and demanded specifics on Thursday/Friday because wtf? debugging in production? with broken merchants? selling things for real money? scary bad? hello?

Anyway, I didn't see his message until he texted me about it at like 5pm today while I was about 2 hours from my computer. He's understandably frustrated, and I totally don't blame him, but fuck, miscommunication is a serious problem in this company, and that's amazing because it's so freaking small.

But the short version is that I'm likely going to get blamed for all this, Clover screwiness included. Bossman and I set up a call for 10am tomorrow and I'm positive he's going to try pinning it on me. Totally not going to let him, but his social is lv16+ while mine's like. 2 or 3. 😕 I'll see how it goes.

Really though, I should read @rutee07's book and just roast these fuckers.

OMFG I don't even know where to start..
Probably should start with last week (as this is the first time I had to deal with this problem directly)..

Also please note that all packages, procedure/function names, tables etc have fictional names, so every similarity between this story and reality is just a coincidence!!

Here it goes..

Lat week we implemented a new feature for the customer on production, everything was working fine.. After a day or two, the customer notices the audit logs are not complete aka missing user_id or have the wrong user_id inserted.
Hm.. ok.. I check logs (disk + database).. WTF, parameters are being sent in as they should, meaning they are there, so no idea what is with the missing ids.

OK, logs look fine, but I notice user_id have some weird values (I already memorized most frequent users and their ids). So I go check what is happening in the code, as the procedures/functions are called ok.

Wow, boy was I surprised.. many many times..
In the code, we actually check for user in this apps db or in case of using SSO (which we were) in the main db schema..
The user gets returned & logged ok, but that is it. Used only for authentication. When sending stuff to the db to log, old user Id is used, meaning that ofc userid was missing or wrong.

Anyhow, I fix that crap, take care of some other audit logs, so that proper user id was sent in. Test locally, cool. Works. Update customer's test servers. Works. Cool..

I still notice something off.. even though I fixed the audit_dbtable_2, audit_dbtable_1 still doesn't show proper user ids.. This was last week. I left it as is, as I had more urgent tasks waiting for me..

Anyhow, now it came the time for this fuckup to be fixed. Ok, I think to myself I can do this with a bit more hacking, but it leaves the original database and all other apps as is, so they won't break.
I crate another pck for api alone copy the calls, add user_id as param and from that on, I call other standard functions like usual, just leave out the user_id I am now explicitly sending with every call.
Ok this might work.

I prepare package, add user_id param to the calls.. great, time to test this code and my knowledge..

I made changes for api to incude the current user id (+ log it in the disk logs + audit_dbtable_1), test it, and check db..
Disk logs fine, debugging fine (user_id has proper value) but audit_dbtable_1 still userid = 0.

WTF?! I go check the code, where I forgot to include user id.. noup, it's all there. OK, I go check the logging, maybe I fucked up some parameters on db level. Nope, user is there in the friggin description ON THE SAME FUCKING TABLE!!
Just not in the column user_id...

WTF..Ok, cig break to let me think..

I come back and check the original auditing procedure on the db.. It is usually used/called with null as the user id. OK, I have replaced those with actual user ids I sent in the procedures/functions. Recheck every call!! TWICE!! Great.. no fuckups. Let's test it again!
OFC nothing changes, value in the db is still 0. WTF?! HOW!?

So I open the auditing pck, to look the insides of that bloody procedure.. WHAT THE ACTUAL FUCK?!
Instead of logging the p_user_sth_sth that is sent to that procedure, it just inserts the variable declared in the main package..
WHAT THE ACTUAL FUCK?! Did the 'new guy' made changes to this because he couldn't figure out what is wrong?! Nope, not him. I asked the CEO if he knows anything.. Noup.. I checked all customers dbs (different customers).. ALL HAD THIS HARDOCED IN!!! FORM THE FREAKING YEAR 2016!!! O.o

Unfuckin believable.. How did this ever work?!

Looks like at the begining, someone tried to implement this, but gave up mid implementation.. Decided it is enough to log current user id into BLABLA variable on some pck..
Which might have been ok 10+ years ago, but not today, not when you use connection pooling.. FFS!!

So yeah, I found easter eggs from years ago.. Almost went crazy when trying to figure out where I fucked this up. It was such a plan, simple, straight-forward solution to auditing..

If only the original procedure was working as it should.. bloddy hell!!

1. high severity production incident was asked to look into at the end of the day.
2. needed fix in ui.
3. fixed and deployed in 1 hour.
4. issue remained. debugging began.
5. gave up at 1 AM and went to sleep.
6. woke up at 6 and after debugging for 2 hours, identified to be a back end issue.
7. worked with back end team for the fix, and 6 hours and 3 deployments later, it worked.
8. third party vendor reported they are still not receiving one parameter from us.
9. back end team realised they forgot to ask ui to send another parameter.
10. added the parameter in ui, redeployed ui.
11. build and deployment tool broke down. got it fixed. delay of 1.5 hours.
12. finally things are in place. total time 26 hours.

13. found half bottle of vodka, leftover from last weekend. *Priceless*

Spent 4 hours debugging a “button” styling, worked fine locally but not on production.!

After striking out “cache” issues, “browser” versions, “fonts”, “sass errors” the error was with a stupid chrome extension that appended a css class attribute to the “HTML” tag 😡

And the other developer thought that was a part of what was written in the code !!

Hate these kinda plugins that manipulate the DOM 😪

P S the plugin is "Grammarly".

I started my internship at the end of the year..
Fuck my ass!!! This code I have to work with is a huge pile of shit.
The code base I need to work with is around 40k LOC. It is a mixture of C++, C, Java, Python, Bash and I think I saw some lonely js files around.

A list of awesome parts:
- Paths are hard coded.
- Redundant code everywhere
- No documentation or inline comments available

Most of the comments in the code are just old code that is not used anymore. But the cherry on the turd is the class that should provide all kind of useful functions in my daily routine. About ninety percent of the functions have the same description or nothing. Sometimes a function name says "readSomethingFromSomewhere" but instead it writes something to a file. It is really confusing and I need to check everything twice instead of rely on what the function name promises.
I have also learned why copy paste isn't that good. The brief descriptions of every method in a files are always the same.

getName() - Description: Fork child process
getIp() - Description: Fork child process
getIpv6() - Description: Fork child process.

Surprise: None of these functions forks a child process. :D

Another awesome feature is the thing that they store up to five different versions of libraries. Everyone with slight modifications but no hint which one you need to use. Sometimes it is the newest, sometimes the oldest which is running in production. Another case of try and error.
Oh and my dev machine is a potato with a power supply and a fan. I started with NetBeans and every time I compiled the code it sounds like the machine wants to lift off and leave for a better place. (At this point I switched to Emacs and everything runs smoothly now)

At first I thought that I'm just not that good at coding and understanding a big project from scratch but some colleagues have the same problem. The whole system is very inflexible and it is all about "std::cout"-debugging to check if your changes do what you want them to do.

Currently I'm just trying to fix this mess to make the life for the next student or employee easier. The first month was just frustrating as hell. I need to ask so many questions and most of the time the answer was "I don't know, haven't touched this code in years". Needless to say that my progress isn't that awesome but at least I get a nice payment for 20 hours of work a week.

Fuck sake, so my bank has been migrating/rolling out new IT system and app/site have been broken for about a week (others noted evidence of devs debugging in production)

Assuming I don't lose my money as some mischievous assholes will inevitably exploit the fuck up, and rob the bank, I will be moving my funds to a different bank...

In mean time I'm trying to prepare for uni, and they're making a ton of semi-random changes in addition to rolling out a site with course details and info along those line, and good fucking god is it bad.

Is is slow as fuck? Check. Does it use never-seen-before naming for standard things? Check! Is the UI pulled from late 90's? YOOU BETCHA! Are the pages bloated with unnecessary content? Fuck yeah! Do I get SQL exceptions when I finally locate my course? Of course I do. Does clicking "back" take me back to the landing page instead of previous page, when I'm several steps deep? .....

I could keep going, but don't feel like ranting and feel more like punching someone in the throat.repeatedly.

People don't seem to know how to properly do print-debugging, so here's a simple guide:

1. A log of "aaaaaa" or "got here" isn't as helpful as you think when ALL OF THEM ARE THE FUCKING SAME. You put a descriptive label or copy verbatim the conditional statement. This saves time matching statements, allows one to watch multiple branches at once, and allows others to understand and help faster when dragged in to help.
2. When trying to see where code fucks up, before each line, paste said line into a proper print statement for your language. If there's, say, a function call or some shit, have it output something like "functionCall(varA=<varA contents>,varB=<varB contents);" Most normal lines should be like this too, but it's especially helpful for calls and comparisons.
If need be, add return values after if they're not shown in another print statement later.
This allows for a trail of execution AND the line that fucks up will be the last in the log, making finding it easier when dealing with hangs and such.
3. Putting something unique like "DEBUG: " or something in front of all statements ensures you can just search for them to ensure you're not rolling one out to production. It also separates debug output from normal output at a glance, making digging through logs faster.

Woops! I was debugging a particularly snarky issue in Production the other day. This morning I realized about 60 minutes into my new coding I hadn't changed my profile back. I was testing form submissions on a live customer's site.

Our production database has tables that exist only to be recompiled into a single table. Like, one record across 4 tables. Each record is only for one other record in the other 3 tables.

I LOST 2 HOURS DEBUGGING IT BECAUSE IT COULD NOT RECOMPILE THIS FUCKING TABLE. FUCK!

If they followed my suggestion and went straight to debugging the server issues they would have been solved it from week 1 and everyone would have thought the migration had a minor performance hiccup. In fact, we have already done such at least twice before and nobody batted an eye.

Instead they self-labelled the migration a failure on first error, setting the stage for apologizing to the client, and put themselves on the spot for a whole staging / production signoff, replication / backup worfklow, almost a blue-green "seamless" deployment reminiscent of DigitalOcean.

Well they're not DigitalOcean, and anyone who has spent any time understanding users knows they will not participate in "new system" tests long enough to find or report issues.

So of course the migration stretched out to almost three months up until the whole reason for the migration - the rapidly escalating risk of the old provider disappearing - hit like a freight train and now they have to go through the problem of debugging the server like I told them to on week 1. Only this time they've set the client mindset against it, lost any chance of reverting, have had grave risk for data loss, and are under pressure to debug other people's code in real-time.

This is why I don't trust devs to do ops. A dev's first solution to any problem is to throw tech at it.

Best debugging trick ever:

Wear your fucking glasses while coding so that you do not mistake COMMA(,) with a DOT (.).

So by
1. Doing that (which obviously aren't a huge number) and
2. Cleaning my screen (yes that).

I was able to wrap my head around the issue that almost wasted one day.
So what I intended to pass as string concatenation join operation value actually was being passed as an argument to the underlying function (that wasn't taking care of it and returning a timestamp from thin air).

Murphy's Law in production and practice.
Nice!

Depressing music continues......!

So, I've had a personal project going for a couple of years now. It's one of those "I think this could be the billion-dollar idea" things. But I suffer from the typical "it's not PERFECT, so let's start again!" mentality, and the "hmm, I'm not sure I like that technology choice, so let's start again!" mentality.

Or, at least, I DID until 3-4 months ago.

I made the decision that I was going to charge ahead with it even if I started having second thoughts along the way. But, at the same time, I made the decision that I was going to rely on as little external technology as possible. Simplicity was going to be the key guiding light and if I couldn't truly justify bringing a given technology into the mix, it'd stay out.

That means that when I built the front end, I would go with plain HTML/CSS/JS... you know, just like I did 20+ years ago... and when I built the back end, I'd minimize the libraries I used as much as possible (though I allowed myself a bit more flexibility on the back end because that seems to be where there's less issues generally). Similarly, any choice I made I wanted to have little to no additional tooling required.

So, given this is a webapp with a Node back-end, I had some decisions to make.

On the back end, I decided to go with Express. Previously, I had written all the server code myself from "first principles", so I effectively built my own version of Express in other words. And you know what? It worked fine! It wasn't particularly hard, the code wasn't especially bad, and it worked. So, I considered re-using that code from the previous iteration, but I ultimately decided that Express brings enough value - more specifically all the middleware available for it - to justify going with it. I also stuck with NeDB for my data storage needs since that was aces all along (though I did switch to nedb-promises instead of writing my own async/await wrapper around it as I had previously done).

What I DIDN'T do though is go with TypeScript. In previous versions, I had. And, hey, it worked fine. TS of course brings some value, but having to have a compile step in it goes against my "as little additional tooling as possible" mantra, and the value it brings I find to be dubious when there's just one developer. As it stands, my "tooling" amounts to a few very simple JS scripts run with NPM. It's very simple, and that was my big goal: simplicity.

On the front end, I of course had to choose a framework first. React is fine, Angular is horrid, Vue, Svelte, others are okay. But I didn't want to bother with any of that because I dislike the level of abstraction they bring. But I also didn't want to be building my own widget library. I've done that before and it takes a lot of time and effort to do it well. So, after looking at many different options, I settled on Webix. I'm a fan of that library because it has a JS-centric approach. There's no JSX-like intermediate format, no build step involved, it's just straight, simple JS, and it's powerful and looks pretty good. Perfect for my needs. For one specific capability I did allow myself to bring in AnimeJS and ThreeJS. That's it though, no other dependencies (well, at first, I was using Axios because it was comfortable, but I've since migrated to plain old fetch). And no Webpack, no bundling at all, in fact. I dynamically load resources, which effectively is code-splitting, and I have some NPM scripts to do minification for a production build, but otherwise the code that runs in the browser is what I actually wrote, unlike using a framework.

So, what's the point of this whole rant?

The point is that I've made more progress in these last few months than I did the previous several years, and the experience has been SO much better!

All the tools and dependencies we tend to use these days, by and large, I think get in the way. Oh, to be sure, they have their own benefits, I'm not denying that... but I'm not at all convinced those benefits outweighs the time lost configuring this tool or that, fixing breakages caused by dependency updates, dealing with obtuse errors spit out by code I didn't write, going from the code in the browser to the actual source code to get anywhere when debugging, parsing crappy documentation, and just generally having the project be so much more complex and difficult to reason about. It's cognitive overload.

I've been doing this professionaly for a LONG time, I've seen so many fads come and go. The one thing I think we've lost along the way is the idea that simplicity leads to the best outcomes, and simplicity doesn't automatically mean you write less code, doesn't mean you cede responsibility for various things to third parties. Those things aren't automatically bad, but they CAN be, and I think more than we realize. We get wrapped up in "what everyone else is doing", we don't stop to question the "best practices", we just blindly follow.

I'm done with that, and my project is better for it!

I am scratching my head since 2 days cause a rather large Dockerfile doesn't work as expected.

CMD Execution just leads to "File not found".

Thanks, that's as useless as one ply toilet paper...

Whoever wrote the Dockerfile (not me…) should get an oscar...

Even in diarrhea after eating the good one day old extra hot china takeout from dubious sources I couldn't produce such a dumpster fire of bullshit.

The worst: The author thought layering helps - except it doesn't really, as it's a giant file with roughly 14 layers If I count correctly.

I just found out the problem...

The author thought it would be great to add the source files of the node project that should be built as a volume to docker... Which would work I guess....

Except that the author is a clueless chimp who thought at the same time seemingly that folder organization means to just pour everything into one folder....

Yeah. That fucker just shoved everything into one folder.

Yeeeeeesssssssss.

It looks like this:
source
docker-compose.mounts.yml
docker-compose.services.yml
docker-compose.yml
Dockerfile-development
Dockerfile-production
Dockerfile
several bash scripts
several TS / JS / config files
...

If you read the above.... Yes.
He went so far to copy the large Dockerfile 3 times to add development and production specific overrides.

I can only repeat what I said many times before: If you don't like doing stuff, ask for fucking help you moron.

-.-

*gooozfraba*

Anyways...

He directly mounts this source directory as a volume.

And then executes a shell script from this directory...

And before that shit was copied in the large gooozfraba Dockerfile into the volume.

Yeeeaaah.

We copy stuff inside the container, then we just mount on start the whole folder and overwrite the copied stuff.

*rolls eyes* which is completely obvious in this pit latrine of YML fuckery called Dockerfile.

As soon as I moved the start script outside the folder and don't have it running inside the folder that is mounted via volume, everything works.

Yeah.... Maybe one should seperate deployment from source files, runtime related stuff from build stuff.

*rolls eyes*

I really hate Docker sometimes. This is stuff that breaks easily for reasons, but you cannot see it unless you really grind your teeth and start manually tracing and debugging what the frigging fuck the maniac called author produced.

- They had an error in production
- Almost one year without looking in this codebase
- Last backend dev in the building
- 1h debugging and reading the Stack Trace
- Had a feeling, and changed the place of a single asterisk (groovy's spread operator in the wrong place)
- Now everything is working, our PM is happy, and the client didn't even noticed
- Probably the shortest commit I've ever done

It was a good day :D

So we released to production today (Friday), not my decision.
All pages work fine expect for the one page which I added a new feature.
It worked fine in Chrome and Edge. But after release a customer who requested the feature said it doesn't work for him. Screenshot showed he was using IE.
Horror time.. it was evident that it has to be the changes to the JavaScript I did, but why does the whole page doesn't work.

So I started debugging. Nothing works on that page in IE11, it doesn't even load the fucking script file. Then I dared to change mode to IE10, it actually gave me an error in my script file. The bad IE has actually picked a mistake that other browsers didn't.

So, the mistake is fun part too.
I had the following jQuery (or Jake Weary) call
$.getJSON(
'/url',
{
argA: a, argB, b, argC:c
},
function (){
// did something
}
);

In second argument, I accidentally typed comma instead of colon. Chrome and Edge ran the script perfectly passing all the arguments.IE 11 failed to load script without giving any error and only IE 10 gave an error of expecting a colon.
I do not know which browser to blame.

PS I didn't try in Firefox, safari, etc.

Two and a half hours debugging an ancient, poorly written prototype being used in production to local an obscure bug causing double product sizes to appear.
Eventually, after navigating the 3 level deep compatibility layer of SQL views, I found out the client had attached the size option to the product twice with two different prices.

Debugception, when you are debuggin someones debug code in a production environment but nobody knows what he was debugging because he left the company.

Spent all morning debugging legacy code that I need to migrate.

Most of the time is just waiting for it to load --pieces of data-- entire tables from the database and then filter out the records it doesn't want using some app logic.

WHAT SORT OF MONKEY WRITES CODE LIKE THIS? HOW WAS THIS EVEN ALLOWED INTO PRODUCTION...

I have to open Notepad to write down my chain of thoughts, steps, and things to check once the next breakpoints are hit so that I don't forget them.

So in theory I'm being paid all morning to sit around and do nothing.

That sounds great but I'm falling asleep... Shoulda worked from home...

What was I saying again...yea...

DON'T HIRE MONKEYS!!! THEY WRITE SHIT CODE THAT WASTES EVERYONE'S TIME EVEN AFTER THEY LEAVE...

I'm going to lunch now... Hopefully Notepad has enough into for me to remember what I was doing...

I'm live debugging on the production machine on an 1x year old system in the middle of the night ....

Debugging a task, that's sending emails to too many customers.
Supervisor: "Never mind, just test in production, there is a dry run flag for the tasks."
Just in case I test locally...
Flags tried:
--dryrun="TRUE" => Error, failed to send mail.
--dryrun=TRUE => Error, failed to send mail.
--dryrun="true" => Not trying to send mail.

If it's THIS PICKY a little more documentation would be nice.
And by a little more I mean: more than the task base class in a giant php monstrosity without phpdocs expecting its code to be self-documenting.

think the downside to rust is safe is holy shit I haven't even seen passable code

everybody shits on JavaScript developers but I think their code quality is obviously better (cuz otherwise if you make it unreadable or a mess there's no debugging that)

the problem with rust is it's safe so you can write the dumbest shit and if the compiler says it works then it works so you just keep it there

over and over I just see this

90% of this codebase is literally useless moves and abstractions that you could've cut out
to be fair I'm only now learning how to reduce clutter in rust (but I do have literal brain damage)
but I'm not sure I've ran into a codebase not suffering from this

also on the other hand it inspires me to care as little as these guys. the bar is so low you can just ignore the bar. one of us, one of us, one of us. if it compiles it is production ready, ayyo

So, someone from support department asked me to check the app as it was displaying blank information on a page.

I started debugging on the api which i found is doing the proboem. I checked and it was working a moment before but not now. Switched in debug mode to see what went wrong and it works again. This happen multiple times.

Before doing anything else i asked our api developer to check if api is working. He said it should work now and problem has been fixed.

Later i found out that he was doing debugging/changing code on production server instead of his local machine or test server.

I actually learnt this last year but here I go in case someone else steps into this shit.

Being a remote work team, every other colleague of mine had some kind of OS X device but I was working this Ubuntu machine.

Turns out we were testing some Ruby time objects up to a nanosecond precision (I think that's the language defaults since no further specification was given) and all tests were green in everyone's machine except mine. I always had some kind of inconsistency between times.

After not few hours of debugging and beating any hard enough surface with our heads, we discovered this: Ruby's time precision is up to nanoseconds on Linux (but just us on OS X) indeed but when we stored that into PostgreSQL (its time precision is up to microseconds) and retrieved it back it had already got its precision cut down; hence, when compared with a non processed value there was a difference. THIS JUST DOES NOT HAPPEN IN OS X.

We ended up relying on microseconds. You know, the production application runs on Ubuntu too. Fuck this shit.

Hope it helps :)

P.s.: I'm talking about default configs, if anyone knows another workaround to this or why is this the case please share.

Production goes down because there's a memory leak due to scale.

When you say it in one sentence, it sounds too easy. Being developers we know how it all goes. It starts with an alert ping, then one server instance goes down, then the next. First you start debugging from your code, then the application servers, then the web servers and by that time, you're already on the tips of your toes. Then you realize that the application and application servers have been gradually losing memory over a period of time. If the application is one that don't get re-deployed ever so often, the complexity grows faster. No anomaly / change detection monitor can detect a gradual decrease of memory over a period of months.

When your code misbehaves in production and you need all the ducks in the world to fix the problem😂😓

(Marshmallow doing some serious debugging!)

Note to devs here. Please don't choose a framework for the hype at your work. Use it on your own time or company hackaton/learning time.

I'm looking at you angular.
Production ready doesn't mean sanity ready.
Now because some dev choose such technology for arbitrary reasons. (hype, latest acronym on CV). I spend more time debugging and understanding than I would if some simpler technology was chosen. Look at all the options then choose the simplest one that has and seems to have active maintenance. Zen of python is the best thing to happen in programming and I think everyone, even if you don't like Python should follow it. Save you and your colleagues brain time and ask for advice.
Also IMO react is probably third or second best option, higher if one requirement is to be react native. Angular is even lower because it's complexity is unforgivable when a dev has not enough front-end experience.

What's your workspace setup?

Curious because it took awhile and a lot of experimenting/thinking to get mine setup the way it is, but now I can't even think properly unless I have things setup that way after booting up in the morning.

Here goes:

Workspace 1: General stuff, personal email. social media, random research for non work related things, etc

Workspace 2: My main project local development, includes terminals, database, browser research for bugs, debugging software, error logs, etc.

Workspace 3: My main project, production workspace, consoles, browser, etc related to production server, you get the idea

Workspace 4: local dev on my side project

I found it crucial to setup workspace 2 and 3, it has helped me avoid countless stupid errors, like, for example, accidentally working on production terminal and wanting to rip my hair out wondering why the fuck _____ isn't working, then realizing, oh shit, i'm on production, not local. Huge brainspace bandwidth saver when I setup like this.

How about you?

So I am debugging a connector library for an api that users curl.

I am fighting my ass off with errors and a lack of debug, testing or thought for CI

Take a poke around this set of classes only to find.

Postman token in the opts. and a removal of ssl check. What you straight copied from postman.

Like seriously clean up your fucking code if you are gonna put something out as production ready to your team.

Console.log('fuck');

A CASE AGAINST BLUE PRISM
Let's review one of the worst weeks I had with Blue Prism

Monday: Yay! Solved one of the problems we've been carrying around for a week before.

One of the robots suddenly became slow. Like, REAL slow. A process that would take 3 minutes per record now takes 45, and that broke apart all the following schedule.

There were no updates on the application server, the production machine, the robot, it just became slow. And not always slow; a process manually run from console room would work, a process in debug room would work, it's just the scheduled part that caused problems.

It turned out, BP didn't seem to like that particular combination of schedulation + process + machine. Moving the process to a different machine seemingly fixed that. IDK why.

Tuesday: One of our processes waits for a code to appear in the page, and when that happens, it memorizes this code. However, now it is always returning blank. Worked for months, now it breaks every single time.
After half a day of debugging a bug which DIDN'T HAPPEN IN DEBUG MODE YET AGAIN, at 11pm I decided to just place a nonsensical timeout in page before reading and call it a day.

WEDNESDAY: a scheduled process didn't start. "No sessions created". Thanks Blue Prism, very cool.

THURSTAY: This time, schedulation did start, but the process is "waiting". As in: it's 9:30 am, the process has been stuck in the same step since 6:00 am. Turns out, it blocked during a navigate stage; you need to send a string to clipboard using the standard BP action for that, then paste and click "enter", but for some reason the standard BP object sent "ORRCO" instead of "ORRICO" to clipboard, which obviously returned no results and then... the process just didn't feel like doing things anymore. No errors, no logs, nothing: just sitting on its ass. Because fuck you that's why.

Friday: another process uses a very moderate amount of scripts to work. Nothing really fancy, just a couple of lines of code to place in page some IDs and selector to help BP do its thing, otherwise selecting these elements would be a nightmare.

But

Failed while invoking javascript method:Exception from HRESULT: 0x80020101-> at mshtml.HTMLWindow2Class.IHTMLWindow2_execScript(String code, String language)

The same script -it's not dynamically generated-worked yesterday, the day before and the day after. But sometimes it will not. Why? The answer, my friend, is blowin'' in the wind

Soooo I am an apprentice who just started his third year. Everybody in my team (3 ppl) left for better jobs.
I am now basically front and backend lead, teaching four new employees our restapi, web and javafx frontend.
At the same time I fix errors happening in production and develop new features.
I guess there are many great rants to come, so stay tuned :D
Going to write about things like tests that got disabled months ago after migrating to gradle, no documentation, finding out how to set up new development workstations with an outdated script missing important steps, management, print debugging in production and much more :)
Oh and it is not that bad, I learned more in the last month than in the two years before. (not saying my team was bad)

"Hey before we launch, can you reintroduce that bug you fixed on Friday? The other team needs it for debugging."

Why the fuck would you need debugging code in production and why the fuck do we want to readd something that was causing problems? Shaping up to be a great week already.

I'm an iOS developer and I cringe when I read job specs that require TDD or excessive unit testing. By excessive I mean demanding that unit tests need to written almost everywhere and using line coverage as a measure of success. I have many years of experience developing iOS apps in agencies and startups where I needed to be extremely time efficient while also keeping the code maintainable. And what I've learned is the importance of DRY, YAGNI and KISS over excessive unit testing. Sadly our industry has become obsessed with unit tests. I'm of the opinion that unit tests have their place, but integration and e2e tests have more value and should be prioritised, reserving unit tests for algorithmic code. Pushing for unit tests everywhere in my view is a ginormous waste of time that can't ever be repaid in quality, bug free code. Why? Because leads to making code testable through dependency injection and 'humble object' indirection layers, which increases the LoC and fragments code that would be easier to read over different classes. Add mocks, and together with the tests your LoC and complexity have tripled. 200% code size takes 200% the time to maintain. This time needs to be repaid - all this unit testing needs to save us 200% time in debugging or manual testing, which it doesn't unless you are an absolute rookie who writes the most terrible and buggy code imaginable, but if you're this terrible writing your production code, why should your tests be any better? It seems that especially big corporate shops love unit tests. Maybe they have enough money and resources to pay for all these hours wasted on unit tests. Maybe the developers can point their 10,000 unit tests when something goes wrong and say 'at least we tried'? Or maybe most developers don't know how to think and reason about their code before they type, and unit tests force them to do that?

Currently debugging a project that was written over 4 years ago...

At first all was well in the world, besides the ever present issue off our goddamn legacy framework. This framework was written 7 years ago on top of an existing open source one, because the existing one was 'lacking some features' & 'did not feel right'.

Now those might be perfectly fine reasons to write a layer on top of a framework, but please, for all future devs sanities, write fucking documentation and maintain it if you're going to use said framework in all major projects!!

Anyhow back to the situation at hand, I'm getting familiar with the project, sighing at the use of our stupid legacy framework, attempting to recreate the reported bugs...

Turns out I can't, well I get other bugs & errors, but not the reported ones. I go to the production server, where I suddenly do can reproduce them...

Already thinking, fuck my life, and scared for the results... I try a 'git status' on the production server....

And yep, there it is, lo and behold, fucking changes on production, that are not in git, fuck you previous dev who worked on this and your stupid lazy ass modifcations on production!

Bleh, already feeling royally pissed, there's only 1 thing I can do, push changes back to git in a seperate branch, and pray I can merge them back in master on my dev environment without to much issues...

Only I first have to get our sysadmi. to allow pushing from a production server back to our git server...

Sigh, going to put on my headphones, retreat to my me space and try to sort out this shitpile now...

Debugging is fine, totally part of the job!

Constantly fixing sh1t and new reports of another pile of sh1t coming every day like somebody is throwing them with shovels at us just to open the codebase that is written by the folks who aren't here anymore with some list of obscure libraries that is last maintained about 5 y ago is not ok.

It is not buggy codebase it is actually coddy bugbase!

I tried to be vocal several times to change technology to more suitable one, to make some improvements and to remove code smells(there is a ton of it, smells like organic garbage dumpster with rotten eggs) but "everything works" and there is no real "value for the customers" in that(fixing, refactoring etc.)!!!

Yea it works with sh1t ton of bugs reported every week. Nobody gives a shit, just contempt with their mediocre lives solving bug at the time while i feel like I'm wasting my time and talent on wrong people and fixing other's shit.

That is what happens when prototype becomes product and ships to production because numbers, money and sh1t!

this is why we who care about our career can't have nice things! I am not god damn pest control, I am f*ckin developer.

TL;DR As time goes by, I'm feel deeply in love with linux. An infatuation? :D

Before, I really dont mind how the file system works, permission setup, library installation, etc. as long I finished my project (before like 90% of the time I copy paste cmds). But now, after many hair pulling while debugging times, crying while rolling on the floor moments, and painful production deployments (wtf! it's working on my machine/dev server rants), it helps me clearly realized how amazing it is. I might be relatively new with the OS compare to others so maybe what I feel like now is like having a crush on someone in a bus :). But still, I just wanted to say thank you to all who are giving their time in developing/improving linux distros - you are heroes!

I'm hoping that I can contribute something soon :)

senti_mode off

I see the jqgrid people have full debugging mode on.

Someone forgot to disable debug reports in production

New question.
When debugging/troubleshooting, what does your desktop look like?

I have a total of 8 production environments to look after, each of which have their appropriate dev environments. Troubleshooting for me typically starts with VisualVM, 6-8 Putty sessions across the environments, at least one dbms session, WinSCP with at least 4 sessions, text editor with minimum of five open files and at least thirty tabs open in Chrome. Oh yeah, forgot outlook and Skype (typically with at least three team mates and usually a group chat).

All is well when I'm in the zone, but good forbid for someone to ask me to show them the article/bug report I just read that sent me down the rabbit hole.

MRW I deploy to production server and forget to add a server domain in "OAuth redirect domains" in Firebase.
Before that I was debugging for 6 hours without success.