How they teach http errors the easy way..

!rant

This was over a year ago now, but my first PR at my current job was +6,249/-1,545,334 loc. Here is how that happened... When I joined the company and saw the code I was supposed to work on I kind of freaked out. The project was set up in the most ass-backward way with some sort of bootstrap boilerplate sample app thing with its own build process inside a subfolder of the main angular project. The angular app used all the CSS, fonts, icons, etc. from the boilerplate app and referenced the assets directly. If you needed to make changes to the CSS, fonts, icons, etc you would need to cd into the boilerplate app directory, make the changes, run a Gulp build that compiled things there, then cd back to the main directory and run Grunt build (thats right, both grunt and gulp) that then built the angular app and referenced the compiled assets inside the boilerplate directory. One simple CSS change would take 2 minutes to test at minimum.

I told them I needed at least a week to overhaul the app before I felt like I could do any real work. Here were the horrors I found along the way.

- All compiled (unminified) assets (both CSS and JS) were committed to git, including vendor code such as jQuery and Bootstrap.
- All bower components were committed to git (ALL their source code, documentation, etc, not just the one dist/minified JS file we referenced).
- The Grunt build was set up by someone who had no idea what they were doing. Every SINGLE file or dependency that needed to be copied to the build folder was listed one by one in a HUGE config.json file instead of using pattern matching like `assets/images/*`.
- All the example code from the boilerplate and multiple jQuery spaghetti sample apps from the boilerplate were committed to git, as well as ALL the documentation too. There was literally a `git clone` of the boilerplate repo inside a folder in the app.
- There were two separate copies of Bootstrap 3 being compiled from source. One inside the boilerplate folder and one at the angular app level. They were both included on the page, so literally every single CSS rule was overridden by the second copy of bootstrap. Oh, and because bootstrap source was included and commited and built from source, the actual bootstrap source files had been edited by developers to change styles (instead of overriding them) so there was no replacing it with an OOTB minified version.
- It is an angular app but there were multiple jQuery libraries included and relied upon and used for actual in-app functionality behavior. And, beyond that, even though angular includes many native ways to do XHR requests (using $resource or $http), there were numerous places in the app where there were `XMLHttpRequest`s intermixed with angular code.
- There was no live reloading for local development, meaning if I wanted to make one CSS change I had to stop my server, run a build, start again (about 2 minutes total). They seemed to think this was fine.
- All this monstrosity was handled by a single massive Gruntfile that was over 2000loc. When all my hacking and slashing was done, I reduced this to ~140loc.
- There were developer's (I use that term loosely) *PERSONAL AWS ACCESS KEYS* hardcoded into the source code (remember, this is a web end app, so this was in every user's browser) in order to do file uploads. Of course when I checked in AWS, those keys had full admin access to absolutely everything in AWS.
- The entire unminified AWS Javascript SDK was included on the page and not used or referenced (~1.5mb)
- There was no error handling or reporting. An API error would just result in nothing happening on the front end, so the user would usually just click and click again, re-triggering the same error. There was also no error reporting software installed (NewRelic, Rollbar, etc) so we had no idea when our users encountered errors on the front end. The previous developers would literally guide users who were experiencing issues through opening their console in dev tools and have them screenshot the error and send it to them.
- I could go on and on...

This is why you hire a real front-end engineer to build your web app instead of the cheapest contractors you can find from Ukraine.

Spent most of the day debugging issues with a new release. Logging tool was saying we were getting HTTP 400’s and 500’s from the backend. Couldn’t figure it out.

Eventually found the backend sometimes sends down successful responses but with statusCode 500 for no reason what so ever. Got so annoyed ... but said the 400’s must be us so can’t blame them for everything.

Turns out backend also sometimes does the opposite. Sends down errors with HTTP 200’s. A junior app Dev was apparently so annoyed that backend wouldn’t fix it, that he wrote code to parse the response, if it contained an error, re-wrote the statusCode to 400 and then passed the response up to the next layer. He never documented it before he left.

Saving the best part for last. Backend says their code is fine, it must be one of the other layers (load balancers, proxies etc) managed by one of the other teams in the company ... we didn’t contact any of these teams, no no no, that would require effort. No we’ve just blamed them privately and that’s that.

#successfulRelease

I know a guy, about 50 years old. He is a self-taught programmer since he was young, and he has always used Visual Basic (never anything newer than VB6).

He once needed to interface with a web application I wrote, so I asked him to send me a POST HTTP request. He didn't know what I was talking about. No notion of REST, sockets, HTTP, nothing.

The he showed me his code. Actually, his codes. He had multiple copies of the project, one for each version, and he even kept multiple variations of the software in different separate folders. He probably doesn't know what "version control" even means.
You think this is messy. You didn't see the actual code (it's a huge application!).

Spaghetti all over the place. Meaningful variable names, what are they? Default names for the controls, like button1, button2, etc, with forms with more than 30 buttons and text fields. This was the most incomprensibile code I have ever seen.

You might think that this guy is just a hobbyist.
No.
He sells his applications. To companies. They are obviously full of errors, but they buy them.

Now, if you're still with me, two questions come into my mind:

- why?? I hate this, because it's impossible to prove to a non-technical person that this is *not* software development.
- how do I know that, to someone else, I am not like him? How can I be sure that I know and will know what needs to be known?

I hate Wordpress. I hate Wordpress. I hate Wordpress.

Wordpress can take a big shit on itself and crawl into a deep dark hole far away from all that is good.

Who even uses Wordpress? Bloggers? Come on, let’s be honest, they’re using more intuitive sites like weebly, wix, and square space. So WHAT is Wordpress for? I’ll tell you, it’s just to FUCKING TORTURE PEOPLE.

So, being the “techy guy” of the family, a relative contacts me asking for some help with their website because they need to install an SSL certificate but they don’t know how to. I tell them I’d gladly do it because, sure, they’re family and how long can it possibly take to install a certificate? I’ve done it before!

Well, I get to work and log into the sluggish Wordpress dashboard and try to use a plugin that would issue a LetsEncrypt certificate because they are free and just as good as any other SSL. But one plugin after the next I keep getting errors about how my hosting wouldn’t allow it.

So I contact GoDaddy (don’t get me fucking started) and ask them about the issue. The guy tells me it’s “policy” to only be able to use GoDaddy’s certificates. How much do they cost? Oh, how about $100 a year?! Fuck you.

I figured out the only way to escape this hell was to ask them to open an economy Linux hosting account with cPanel on GoDaddy (the site was formerly hosted on a “Managed Wordpress” account which is just bullshit for not wanting to give you any control over your own goddamn content). So now I have to deal with migrating the site.

GoDaddy representative tells me that it should only take 20 minutes for me to do this (I’ve already spent way too much time on this but whatever) so I go forward with the new account. I decide I should migrate the site by exporting a backup and manually placing everything on the new server. Doesn’t it end up taking an entire hour to back up a 200MB site because GoDaddy throttled the processing speed?!

So, it’s another hour later and I’ve installed all the databases and carried over all the files. At this point, I’m really at the end of my rope and can’t wait to install the certificate and be done with this fuckery.

I install the certificate and finally get ready to be on my way, but then I see it. A warning. A warning from my browser telling me the site is only partially secure. It turns out the certificate was properly installed but whoever initially made the site HARDCODED ALL THE LINKS to images, websites, and style sheets to be http instead of https.

I’m gonna explode.

I swear, I’m gonna fucking explode.

After a total of 5 hours of work, I finally get the site secure by using search and replace on every fucking file.

Wordpress can go suck a big one. Actually, Wordpress can go suck the largest fuckin one in existence and choke on it.

TL;DR I agree to install an SSL certificate but end up with much more work than I bargained.

Yesterday the web site started logging an exception “A task was canceled” when making a http call using the .Net HTTPClient class (site calling a REST service).

Emails back n’ forth ..blaming the database…blaming the network..then a senior web developer blamed the logging (the system I’m responsible for).

Under the hood, the logger is sending the exception data to another REST service (which sends emails, generates reports etc.) which I had to quickly re-direct the discussion because if we’re seeing the exception email, the logging didn’t cause the exception, it’s just reporting it. Felt a little sad having to explain it to other IT professionals, but everyone seemed to agree and focused on the server resources.

Last night I get a call about the exceptions occurring again in much larger numbers (from 100 to over 5,000 within a few minutes). I log in, add myself to the large skype group chat going on just to catch the same senior web developer say …
“Here is the APM data that shows logging is causing the http tasks to get canceled.”

FRACK!

Me: “No, that data just shows the logging http traffic of the exception. The exception is occurring before any logging is executed. The task is either being canceled due to a network time out or IIS is running out of threads. The web site is failing to execute the http call to the REST service.”

Several other devs, DBAs, and network admins agree.

The errors only lasted a couple of minutes (exactly 2 minutes, which seemed odd), so everyone agrees to dig into the data further in the morning.

This morning I login to my computer to discover the error(s) occurred again at 6:20AM and an email from the senior web developer saying we (my mgr, her mgr, network admins, DBAs, etc) need to discuss changes to the logging system to prevent this problem from negatively affecting the customer experience...blah blah blah.

FRACKing female dog!

Good news is we never had the meeting. When the senior web dev manager came in, he cancelled the meeting.

Turned out to be a hiccup in a domain controller causing the servers to lose their connection to each other for 2 minutes (1-minute timeout, 1 minute to fully re-sync). The exact two-minute burst of errors explained (and proven via wireshark).

People and their petty office politics piss me off.

It fucking staggers me how many backend/devops-y people don't understand what a client side "request timeout" is, versus a server side one.

What does it mean:

The client was fed up with the servers bullshit, and decided to piss off and not wait around for the server to take forever to respond, because life's too short.

How not to solve/debug this issue:

- "I've checked the API request in tool xyz, and it works fine for me"

Congratulations, you've figured out how to call an API once, in isolation to the rest of the application, and without any excessive load. And using a different client to me, with a different configuration. Lets get back to actually looking at the issue shall we?

- "I only see HTTP 200's in the logs"

Yep, you probably will in most circumstances, because its the client complaining about it taking too long, not the server. If the server was telepathetic and knew what the client was thinking/doing at all times, we wouldn't have half of the errors we do.

- "Ah ok, I understand ... so how do I solve this?"

Your asking me? I don't fucking know, I didn't build the server! Put better logging in place and figure out why sometimes it takes forever.

Jesus fucking christ

Use your status codes

413 Payload too large, oh f**k

Hi lil puppies what's your problem?

*proxy vomits*

Have you eaten something wrong....

*proxy happily eats requests and answers correctly*

Hm... Seems like you are...

*proxy vomits dozen of requests at once*

... Not okay.

Ok.... What did u you get fed you lil hellspawn.

TLS handshake error.

Thousands. Of. TLS. Handshake. Errors.

*checking autonomous system information*

Yeah... Requests come from same IP or AS. Someone is actively bombing TLS requests on the TLS terminator.

Wrong / outdated TLS requests.

Let's block the IP addresses....

*Pats HAProxy on the head*

*Gets more vomit as a thank you no sir*

I've now added a list of roughly 320 IP adresses in 4 h to an actively running HAProxy in INet as some Chinese fuckers seemingly find it funny to DDOS with TLS 1.0... or Invalid HTTP Requests... Or Upgrade Headers...

Seriously. I want a fucking weekend you bastards. Shove your communism up your arse if you wanna have some illegal fun. ;)

Inmates are trying to take over the asylum again.

Got a message from the web team manager deeply concerned because since switching to the new logging framework, the site is significantly slower.

She provided no proof or any data to what 'significantly slower' means.

#1 The 'new logging' has been in place and logging for 5 years. We only recently depreciated the ILogger interface ('new' ILogger interface only has 1 method instead of 5)
#2 The 'old logging' was modified 5 years ago, so even if you were using the 'old' interface, the underlying implementation is still the same.

She tried to push the 'it wasn't this slow before' argument, so I decided to do some fact based analysis.
Knowing they deployed their logging changes couple of weeks ago, I opened up AppDynamics, looked at the average call time to Splunk (along with a few other http calls they are doing)
- caching services - 5ms
- splunk - 30ms
- Order Service - 350ms
- Product Data Service -525ms

Then I look at the data they are logging, for the month of June, over 5 million messages. At 30ms each, that's almost 42 hours spent logging errors...yes errors. Null reference exceptions, Argument exceptions, easily fixable stuff.

So far for the month of July (using the 'new' logging), almost 2.5 million errors. Pretty close so far with June's numbers.

My only suggestion was to fix the bugs in their code so they don't log so many errors.

Her response.."Can we have one of our developers review your logging code? We believe we can find ways to optimize the http requests"

Oh good Lord. I'm not a drinking man..but ...I might start.

Right, I've been here before.
Our app requires an internet connection, and one of our clients wants to roll it out on a strictly managed network.
We told them which addresses our app communicates with and their network team opened them up for traffic. Should work, right?
Nope, doesn't work.

So I request them to use Fiddler to do some debugging of the network traffic, and lo and behold, it does work when Fiddler is active.
One important detail is that Fiddler uses it's own SSL certificate to debug HTTPS communications. I've had moments where expired certificates were the cause of things not working and running Fiddler "fixes" this because of their own certificate.

So I point this out in numerous mails to their network team, every time I get a response saying "nah, that can't be it".
I keep insisting "I have had this before, please check if any installed Root CA Certificates is expired"
At this point I'm certain they have updates turned off on these machines, and their certificates must not have been updated for a long time.

At one point they come back to me. "Hey, when Fiddler is off, WireShark shows the app communicating with ICMP calls, but when it's on it shows HTTP calls instead".

...YOU'RE THE SUPPOSED NETWORK EXPERTS?! You think data can be send via ICMP? Do you even know what ICMP is? Of course you'll see ICMP calls when the network is rejecting the packages instead of HTTP calls when everything's fine.
(ICMP is used to communicate errors)

I'm trying to keep my patience with these guys until they find exactly what's wrong because even I am somewhat grasping at straws right now. But things like this makes me doubt their expertise...

Boss: "So I'm taking the next week off. In the mean time, I added some stuff for you to do on Gitlab, we'd need you to pull this Docker image, run it, setup the minimal requirement and play with it until you understand what it does."

Me: "K boss, sounds fun!" (no irony here)

First day: Unable to login to the remote repository. Also, I was given a dude's name to contact if I had troubles, the dude didn't answer his email.

2nd day: The dude aswered! Also, I realized that I couldn't reach the repository because the ISP for whom I work blocks everything within specific ports, and the url I had to reach was ":5443". Yay. However, I still can't login to the repo nor pull the image, the connection gets closed.

3rd day (today): A colleague suggested that I removed myself off the ISP's network and use my 4G or something. And it worked! Finally!! Now all I need to do is to set that token they gave me, set a first user, a first password and... get a 400 HTTP response. Fuck. FUCK. FUUUUUUUUUUUUUUUUUUUCK!!!

These fuckers display a 401 error, while returning a 400 error in the console log!! And the errors says what? "Request failed with status code 401" YES THANK YOU, THIS IS SO HELPFUL! Like fuck yea, I know exactly how t fix this, except that I don't because y'all fuckers don't give any detail on what could be the problem!

4th day (tomorrow): I'm gonna barbecue these sons of a bitch

(bottom note: the dude that answered is actually really cool, I won't barbecue him)

I used to be a sysadmin and to some extent I still am. But I absolutely fucking hated the software I had to work with, despite server software having a focus on stability and rigid testing instead of new features *cough* bugs.

After ranting about the "do I really have to do everything myself?!" for long enough, I went ahead and did it. Problem is, the list of stuff to do is years upon years long. Off the top of my head, there's this Android application called DAVx5. It's a CalDAV / CardDAV client. Both of those are extensions to WebDAV which in turn is an extension of HTTP. Should be simple enough. Should be! I paid for that godforsaken piece of software, but don't you dare to delete a calendar entry. Don't you dare to update it in one place and expect it to push that change to another device. And despite "server errors" (the client is fucked, face it you piece of trash app!), just keep on trying, trying and trying some more. Error handling be damned! Notifications be damned! One week that piece of shit lasted for, on 2 Android phones. The Radicale server, that's still running. Both phones however are now out of sync and both of them are complaining about "400 I fucked up my request".

Now that is just a simple example. CalDAV and CardDAV are not complicated protocols. In fact you'd be surprised how easy most protocols are. SMTP email? That's 4 commands and spammers still fuck it up. HTTP GET? That's just 1 command. You may have to do it a few times over to request all the JavaScript shit, but still. None of this is hard. Why do people still keep fucking it up? Is reading a fucking RFC when you're implementing a goddamn protocol so damn hard? Correctness be damned, just like the memory? If you're one of those people, kill yourself.

So yeah. I started writing my own implementations out of pure spite. Because I hated the industry so fucking much. And surprisingly, my software does tend to be lightweight and usually reasonably stable. I wonder why! Maybe it's because I care. Maybe people should care more often about their trade, rather than those filthy 6 figures. There's a reason why you're being paid that much. Writing a steaming pile of dogshit shouldn't be one of them.

Am I the only developer in existence who's ever dealt with Git on Windows? What a colossal train wreck.

1. Authentication. Since there is no ssh key/git url support on Windows, you have to retype your git credentials Every Stinking Time you push. I thought Git Credential Manager was supposed to save your credentials? And this was impossible over SSH (see below). The previous developer had used an http git URL with his username and password baked in for authentication. I thought that was a horrific idea so I eventually figured out how to use a Bitbucket App password.

2. Permissions errors
In order to commit and push updates, I have to run Git for Windows as Administrator.

3. No SSH for easy git access
Here's where I confess that this is a Windows Server machine running as some form of production. Please don't slaughter me! I am not the server admin.
So, I convinced the server guy to find and install some sort of ssh service for Windows just for the off times we have to make a hot fix in production. (Don't ask, but more common than it should be.)
Sadly, this ssh access is totally useless as the git colors are all messed up, the line wrap length and window size are just weird (seems about 60 characters wide by 25 lines tall) and worse of all I can't commit/push in git via ssh because Permissions. Extremely aggravating.

4. Git on Windows hangs open and locks the index file
Finally, we manage to have Git for Windows hang quite frequently and lock the git index file, meaning that we can't do anything in git (commit, push, pull) without manually quitting these processes from task manager, then browsing to the directory and deleting the .git/index.lock file.

Putting this all together, here's the process for a pull on this production server:
Launch a VNC session to the server. Close multiple popups from different services. Ask Windows to please not "restart to install updates". Launch git for Windows. Run a git pull. If the commits to be pulled involve deleting files, the pull will fail with a permissions error. Realize you forgot to launch as Administrator. Depending on how many files were deleted in the last update, you may need to quit the application and force close the process rather than answer "n" for every "would you like to try again?" file. Relaunch Git as Administrator. Run Git pull. Finally everything works.

At this point, I'd be grateful for any tips, appreciate any sympathy, and understand any hatred. Windows Server is bad. Git on Windows is bad.

So there I am sitting in front of my laptop, and trying to npm i and I am getting all sorts of sha mismatch errors.

After lot of debug I conclude it is coming from the proxy as it refuses to download and supplies the error page.

It says it's because I'm using the old proxy so they give me the new URL which I set up and it works.

All good until my password expires. I use our bash script to change it. NPM is buggered again throwing the same errors.

Go to IT, tell them the saga begins.

After a countless hours of looking at the log files we notice that the npm registry is set to http instead of the standard https (thanks bash script). so our firewall blocks the download.

Sorted, finally.

Almost. NPM now works fine, but when I go and I play around with node and axios, I get my requests time out. My instinct says its the bloody proxy again.

So I hit up my trusted WIN Support guy and he confirms that the url is not blocked. So he starts monitoring whats going on and turns out, every time I run the node app, node casually ignores the system-wide proxy settings and tries to send the request as the PC rather then my username.

Since the pc's don't have rights on the proxy it is being refused...

Thank fuck for the corporate proxies, without them, I could just develop things not ever learning these quirks of node...

403 Error freaks me more than 404 Error.. Idk why?

SIM 800L
The fucking nail to my coffin. This thing is so unreliable. I fixed on issue get the next one. Then get an error trying to make a http request, with little information on the web. Eventually switch to FTP which is working for a while. Then suddenly nothing is working properly. Even the serial communication has errors. This process took over 6 months. Constant debugging and headscraching involved. After hundreds of hours I give up. I'm going to switch to a Raspberry Pi Zero with an UMTS Stick attached. This is going to cost way more battery time but my project needs to be finished by july and I'm tired of this shitty little module.

The world must truly hate me.

I refactored my code a lot in order to easily integrate different APIs endpoints but then I can't use the one I specifically did this for, because of TLS errors.
My browsers all agree that it's fine, but curl and the 2 http libraries I tested can't fetxh any data.

END ME!

Frontend:" im going to chage some bits because I have been thinking....and when we get errors Im confused because we got code and message"

Me:"yes but that's fine. The http status code should tell you what happened"

Frontend:"oh, you want me to check status code??"

WHYYYYY THIS GUYYYYYY

He even tells u like he is right tone of voice

Someone wrote some error handling middleware for the whole application.
Then someone wrote some default classes for HTTP errors; BadRequest, Unauthorized and so on.

If I didn't know any better, I'd think throwing one of these default errors would give the proper status code, instead of default everything to 500.

But alas, I do know better.

I am the responsible for the atlassian Suite at work, as I maintain the systems, set them up, and stuff.

One day, our crowd (the authentication and authorization application) just went crazy. At like lunch time it could not connect to the AD anymore. No reasons. Throwing XSRF errors (cross site scripting), because http would connect to https. "won't do it, fuck you" it told me. Out of the blue. Noone changed anything. And yea, seriously. Noone did.
It just refused to connect (as connecting to AD is connecting yourself with you own api. And refusing yourself talking to yourself). It runs behind a proxy. Therefore http/https. Well, this worked for years. But out of sudden not anymore.

Yea. Fuck you.

It was reported some hours later, at like 3pm, as people could not login to the applications using crowd as authentication and authorization server.

Tried to debug the system, where nothing was did, to make it work. At best time to fail.

First workaround: if you are logged into one of the other applications of atlassian, just refresh the site, so your SSO token gets a refresh and you are signed on again.

Then I searched more and more. And more.
But nothing worked, nothing helped.

So I addressed an emergency maintenance, take down the whole Suite, restart crowd, to apply some changes to it's settings, not knowing what happening then, because all connections of SSO will then be released. Sent out the mail like 30 minutes beforehands.

While waiting for the window, I just typed my credentials... And redid, and redid, so to type and being bored.
Three minutes before the window...

It just worked again.

Well. Wtf. Serioudl
Just came back.

No Intrusion, no changes at all. Just came back, as nothing has happened.

Kind of best part of this story... A headhunter messaged me on my way home to offer me a job as an Atlassian Suite SysAdmin for a company, at kinda the double of my salary.

At first I was thinking to go there, and when someone then asked me sth about Atlassian just start to laugh and then leave still laughing...

But then I very nicely respond that I dont want to cry at work. And wished him best luck.

I am doing some bad upgrades now on our Suite. Very painful.
And I looked into the start scripts. Some Look like the untalented intern tells another one to write scripts. Seriously wtf.

Today I followed the guide to Update a confluence and change database to Postgres. Didnt work, Postgres error.
Try it again, jquery won't load. Next try, tomcat not starting anymore. Did same thing. Every fucking time.

Yea. Maintenance window to get a nice new export soon. Will only take an hour.

To switch database in confluence, you need to set it up very fresh. And then Import your export.
Export takes an hour at our system.
Importing maybe the same time. Hope it will work (hint: Nope).

Oh, can be nice also. Just tell the Bitbucket to migrate databases, there is a fucking setting for it. Enter new database, ready, go, finished.

At least they don't raise costs very much every kinda year.
Oh sorry, yes, they do.

Im building an API with NancyFx (HTTP stuff) and mobgoDB (database). i had very strage errors when starting queres. and after 2 fucking days if debugging i found it!
accidentally I wrote somewhere in a custom type conversion "ObjectId" i stead of "OwnerId".

F M L
"""""""""
#MongoDB #NancyFx #CSharp/.Net

Why are most (cloud-based) websites failing? They just spew out 5xx HTTP errors all the time. I can't even register properly on a website anymore. What the funk, man.