Good Morning!, its time for practiseSafeHex's most incompetent co-worker!

Todays contestant is a very special one.

*sitcom audience: WHY?*

Glad you asked, you see if you were to look at his linkedin profile, you would see a job title unlike any you've seen before.

*sitcom audience oooooooohhhhhh*

were not talking software developer, engineer, tech lead, designer, CTO, CEO or anything like that, No No our new entrant "G" surpasses all of those with the title ..... "Software extraordinaire".

*sitcom audience laughs hysterically*

I KNOW!, wtf does that even mean! as a previous dev-ranter pointed out does this mean he IS quality code? I'd say he's more like a trash can ... where his code belongs

*ba dum tsssss*

Ok ok, lets get on with the show, heres some reasons why "G" is on the show:

One of G's tasks was to build an analytics gathering library for iOS, similar to google analytics where you track pages and events (we couldn't use google's). G was SO good at this job he implemented 2 features we didn't even ask for:

- If the library was unable to load its config file (for any reason) it would throw an uncatchable system integrity error, crashing the app.
- If anything was passed into any of the functions that wasn't expected (null, empty array etc.) it would crash the app as it was "more efficient" to not do any sanity checks inside the library.

This caused a lot of issues as some of the data needed to come from the clients server. The day we launched the app, within the first 3 hours we had over 40k crash logs and a VERY angry client.

Now, what makes this story important is not the bugs themselves, come on how many times have we all done something stupid? No the issue here was G defended all of this as the right thing to do!

.. and no he wasn't stoned or drunk!

G claimed if he couldn't get the right settings / params he wouldn't be able to track the event and then our CEO wouldn't have our usage data. To which I replied:

"So your solution was to not give the client an app instead? ... which also doesn't give the CEO his data".

He got very angry and asked me "what would you do then?". I offered a solution something like why not have a default tag for "error" or "unknown" where if theres an issue, we send up whatever we have, plus the file name and store it somewhere else. I was told I was being ridiculous as it wasn't built to track anything like that and that would never work ... his solution? ... pull the library out of the app and forget it.

... once again giving everyone no data.

G later moved onto another cross-platform style project. Backend team were particularly unhappy as they got no spec of what needed to be done. All they knew was it was a single endpoint dealing with very complex model. There was no Java classes, super classes, abstract classes or even interfaces, just this huge chunk of mocked data. So myself and the lead sat down with him, and asked where the interfaces for the backend where, or designs / architecture for them etc.

His response, to this day frightens me ... not makes me angry, not bewilders me ... scares the living shit out of me that people like this exist in the world and have successful careers.

G: "hhhmmm, I know how to build an interface, but i've never understood them ... Like lets say I have an interface, what now? how does that help me in any way? I can't physically use it, does it not just use up time building it for no reason?"

us: "... ... how are the backend team suppose to understand the model, its types, integrate it into the other systems?"

G: "Can I not just tell them and they can write it down?"

**
I'll just pause here for a moment, as you'll likely need to read that again out of sheer disbelief
**

I've never seen someone die inside the way the lead did. He started a syllable and his face just dropped, eyes glazed over and he instantly lost all the will to live. He replied:

" wel ............... it doesn't matter ... its not important ... I have to go, good luck with the project"

*killed the screen share and left the room*

now I know you are all dying in suspense to know what happened to that project, I can drop the shocking bombshell that it was in fact cancelled. Thankfully only ~350 man hours were spent on it

... yep, not a typo.

G's crowning achievement however will go down in history. VERY long story short, backend got deployed to the server and EVERYTHING broke. Lead investigated, found mistakes and config issues on every second line, load balancer wasn't even starting up. When asked had this been tested before it was deployed:

G: "Yeah I tested it on my machine, it worked fine"

lead: "... and on the server?"

G: "no, my machine will do the same thing"

lead: "do you have a load balancer and multiple VM's?"

G: "no, but Java is Java"

... and with that its time to end todays episode. Will G be our most incompetent? ... maybe.

Tune in later for more practiceSafeHex's most incompetent co-worker!!!

So, I needed a package installed on one of our Unix servers. The package manager--which is obsolete garbage--was failing with a message which can only be described as a variant of "Go fuck yourself". A quick Google search didn't help.

3 espressos and an eternity later, I have descended into a manic state. My hair has turned grey and I have started lactating. As a last-ditch effort, I try a new search query on Google, and the first link takes me to a forum with a thread discussing a similar issue. The last post in the thread has a solution which works for me. After fixing the issue, everything in the world feels right and I decide to thank the generous poster, who is like an angel to me at this point.

Guess what? The poster is none other than me. 8 months back, I had created a user account on the forum just to post the solution to a similar issue I had on another server.

I absolutely HATE "web developers" who call you in to fix their FooBar'd mess, yet can't stop themselves from dictating what you should and shouldn't do, especially when they have no idea what they're doing.

So I get called in to a job improving the performance of a Magento site (and let's just say I have no love for Magento for a number of reasons) because this "developer" enabled Redis and expected everything to be lightning fast. Maybe he thought "Redis" was the name of a magical sorcerer living in the server. A master conjurer capable of weaving mystical time-altering spells to inexplicably improve the performance. Who knows?

This guy claims he spent "months" trying to figure out why the website couldn't load faster than 7 seconds at best, and his employer is demanding a resolution so he stops losing conversions. I usually try to avoid Magento because of all the headaches that come with it, but I figured "sure, why not?" I mean, he built the website less than a year ago, so how bad can it really be? Well...let's see how fast you all can facepalm:

1.) The website was built brand new on Magento 1.9.2.4...what? I mean, if this were built a few years back, that would be a different story, but building a fresh Magento website in 2017 in 1.x? I asked him why he did that...his answer absolutely floored me: "because PHP 5.5 was the best choice at the time for speed and performance..." What?!

2.) The ONLY optimization done on the website was Redis cache being enabled. No merged CSS/JS, no use of a CDN, no image optimization, no gzip, no expires rules. Just Redis...

3.) Now to say the website was poorly coded was an understatement. This wasn't the worst coding I've seen, but it was far from acceptable. There was no organization whatsoever. Templates and skin assets are being called from across 12 different locations on the server, making tracking down and finding a snippet to fix downright annoying.

But not only that, the home page itself had 83 custom database queries to load the products on the page. He said this was so he could load products from several different categories and custom tables to show on the page. I asked him why he didn't just call a few join queries, and he had no idea what I was talking about.

4.) Almost every image on the website was a .PNG file, 2000x2000 px and lossless. The home page alone was 22MB just from images.

There were several other issues, but those 4 should be enough to paint a good picture. The client wanted this all done in a week for less than $500. We laughed. But we agreed on the price only because of a long relationship and because they have some referrals they got us in the door with. But we told them it would get done on our time, not theirs. So I copied the website to our server as a test bed and got to work.

After numerous hours of bug fixes, recoding queries, disabling Redis and opting for higher innodb cache (more on that later), image optimization, js/css/html combining, render-unblocking and minification, lazyloading images tweaking Magento to work with PHP7, installing OpCache and setting up basic htaccess optimizations, we smash the loading time down to 1.2 seconds total, and most of that time was for external JavaScript plugins deemed "necessary". Time to First Byte went from a staggering 2.2 seconds to about 45ms. Needless to say, we kicked its ass.

So I show their developer the changes and he's stunned. He says he'll tell the hosting provider create a new server set up to migrate the optimized site over and cut over to, because taking the live website down for maintenance for even an hour or two in the middle of the night is "unacceptable".

So trying to be cool about it, I tell him I'd be happy to configure the server to the exact specifications needed. He says "we can't do that". I look at him confused. "What do you mean we 'can't'?" He tells me that even though this is a dedicated server, the provider doesn't allow any access other than a jailed shell account and cPanel access. What?! This is a company averaging 3 million+ per year in revenue. Why don't they have an IT manager overseeing everything? Apparently for them, they're too cheap for that, so they went with a "managed dedicated server", "managed" apparently meaning "you only get to use it like a shared host".

So after countless phone calls arguing with the hosting provider, they agree to make our changes. Then the client's developer starts getting nasty out of nowhere. He says my optimizations are not acceptable because I'm not using Redis cache, and now the client is threatening to walk away without paying us.

So I guess the overall message from this rant is not so much about the situation, but the developer and countless others like him that are clueless, but try to speak from a position of authority.

If we as developers don't stop challenging each other in a measuring contest and learn to let go when we need help, we can get a lot more done and prevent losing clients. </rant>

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Ladies and gentlemen, prepare yourselves for a rant with a capital R, this is gonna be a long one.

Our story begins well over a year ago while I was still in university and things such as "professionalism" and "doing your job" are suggestions and not something you do to not get fired. We had multiple courses with large group projects that semester and the amount of reliable people I knew that weren't behind a year and in different courses was getting dangerously low. There were three of us who are friends (the other two henceforth known as Ms Reliable and the Enabler) and these projects were for five people minimum. The Enabler knew a couple of people who we could include, so we trusted her and we let them onto the multiple projects we had.

Oh boy, what a mistake that was. They were friends, a guy and a girl. The girl was a good dev, not someone I'd want to interact with out of work but she was fine, and a literal angel compared to the guy. Holy shit this guy. This guy, henceforth referred to as Mr DDTW, is a motherfucking embarrassment to devs everywhere. Lazy. Arrogant. Standards so low they're six feet under. Just to show you the sheer depth of this man's lack of fucks given, he would later reveal that he picked his thesis topic "because it's easy and I don't want to work too hard". I haven't even gotten into the meat of the rant yet and this dude is already raising my blood pressure.

I'll be focusing on one project in particular, a flying vehicle simulator, as this was the one that I was the most involved in and also the one where shit hit the fan hardest. It was a relatively simple-in-concept development project, but the workload was far too much for one person, meaning that we had to apply some rudimentary project management and coordination skills that we had learned to keep the project on track. I quickly became the de-facto PM as I had the best grasp on the project and was doing a lot of the heavy lifting.

The first incident happened while developing a navigation feature. Another teammate had done the basics, all he had to do was use the already-defined interfaces to check where the best place to land would be, taking into account if we had enough power to do so. Mr DDTW's code:
-Wasn't actually an algorithm, just 90 lines of if statements sandwiched between the other teammate's code.
-The if statements were so long that I had to horizontal scroll to see the end, approx 200 characters long per line.
-Could've probably been 20 normal-length lines MAX if he knew what a fucking for loop was.
-Checked about a third of the tiles that it should have because, once again, it's a series of concatenated if statements instead of an actual goddamn algorithm.
-IT DIDN'T FUCKING WORK!

My response was along the lines of "what the fuck is this?". This dipshit is in his final year and I've seen people write better code in their second semester. The rest of the team, his friend included, agreed that this was bad code and that it should be redone properly. The plan was for Mr DDTW to move his code into a new function and then fix it in another branch. Then we could merge it back when it was done. Well, he kept on saying it was done but:
-It still wasn't an algorithm.
-It was still 90 lines.
-They were still 200 characters wide.
-It still only checked a third of the tiles.
-IT STILL DIDN'T FUCKING WORK!

He also had one more task, an infinite loop detection system. He watched while Ms Reliable did the fucking work.

We hit our first of two deadlines successfully. We still didn't have a decent landing function but everything else was nice and polished, and we got graded incredibly well. The other projects had been going alright although the same issue of him not doing shit applied. Ms Reliable and I, seeing the shitstorm that would come if this dude didn't get his act together, lodged a complaint with the professor as a precautionary measure. Little did I know how much that advanced warning would save my ass later on.

Second sprint begins and I'm voted in as the actual PM this time. We have four main tasks, so we assign one person to each and me as a generalist who would take care of the minor tasks as well as help out whoever needed it. This ended up being a lot of reworking and re-abstracting, a lot of helping and, for reasons that nobody ever could have predicted, one of the main tasks.

These main tasks were new features that would need to be integrated, most of which had at least some mutual dependencies. Part of this project involved running our code, which would connect to the professor's test server and solve a server-side navigation problem. The more of these we solved, the better the grade, so understandably we needed an MVP to see if our shit worked on the basic problems and then fix whatever was causing the more advanced ones to fail. We decided to set an internal deadline for this MVP. Guess who didn't reach it?

Hitting the character limit, expect part 2 SOON

I hate people... I hate stupid people even more...

A person asked on slack about where download a Programming Language server called Railo. The official site is no longer up because the software was forked and acquired by a new company.

I suggested just to download that fork since it's more stable. They said no, they needed to mimic their production environment. Makes sense, so I left it alone since I couldn't help further.

Another person on slack asked which version of Railo they need. The OPs response was, "Oh whatever version you have."

My response was... "WTF... the latest version of Railo is 4.3 and the fork is 4.5... the only difference is the new name and a couple of security fixes. If you want to mimic production then you need the exact copy.. otherwise, the fork will be your best bet."

Nope.. I need Railo... any version. They say again.

I just can't understand what will lead an so called Software Company, that provides for my local government by the way, to use an cloud sever (AWS ec2 instance) like it were an bare metal machine.

They have it working, non-stop, for over 4 years or so. Just one instance. Running MySQL, PostgreSQL, Apache, PHP and an f* Tomcat server with no less than 10 HUGE apps deployed. I just can't believe this instance is still up.

By the way, they don't do backups, most of the data is on the ephemeral storage, they use just one private key for every dev, no CI, no testing. Deployment are nightmares using scp to upload the .war...

But still, they are running several several apps for things like registering citizen complaints that comes in by hot lines. The system is incredibly slow as they use just hibernate without query optimizations to lookup and search things (n+1 query problems).

They didn't even bother to get a proper domain. They use an IP address and expose the port for tomcat directly. No reverse proxy here! (No ssl too)

I've been out of this company for two years now, it was my first work as a developer, but they needed help for an app that I worked on during my time there. I was really surprised to see that everything still the same. Even the old private key that they emailed me (?!?!?!?!) back then still worked. All the passwords still the same too.

I have some good rants from the time I was there, and about the general level of the developers in my region. But I'll leave them for later!

Is it just me or this whole shit is crazy af?

I'm not sure if this entirely qualifies and I might have ranted about it a few years ago but fuck it.

My last internship. Company was awesome and my mentor/technical manager got along very well with me to the point that he often asked me to help out with Linux based stuff (he preferred Linux but was a C# guy and wasn't as familiar with it as me (Linux)).

We had to build an internal site thingy (don't remember what it was) and we delivered (me and some interns) and then the publishing moment came so I went to out project manager (a not-as-technical one) and asked if he could install a LetsEncrypt certificate on the site (he knew how and was one of the only ones who had direct access to the server).

He just stared at us and asked why the fuck we needed that since it was an internal thing anyways.
I kindly told that since it's free and can secure the connection, I preferred that and since its more secure, why the fuck not?
He wasn't convinced so it was off.

Next day I came in early and asked my mentor if he could do the SSL since he usually had access to that stuff. He stared at me with "what?" eyes and I explained what the PM said.

Then he immediately ssh'd in and got the damn cert with "we're going to go secure by default, of course!"

A minute later it was all set.

A situation i just dealt with on my tech help Discord server:

Beginning:
"my gpu isn't working and i need it to update my bios"

Ending:
"i have to have a cpu and ram to update my bios?"

this kid thought he only needed a mobo, GPU and PSU

he's 19

why

So yeah, we released our app for 26k people to use two days ago. Due to circumstances not under our control, that was developed way faster than it should've. Today we're two days in and everything is going horrible. 26 thousand people are having trouble using it (not every problem is our fault, but ofc they don't give a shit) and our support team is not big enough to handle every request. We're not able to find any more people to help us handle it, so some developers are being used for support. But that slows the bugfixes a lot. We're at risk of losing our jobs because we will not be able to make it work in due time.

In Italy, at the beginning of the year a particular type of invoice became necessary and to handle that everyone needs a particular kind of email called "certified", you know, because Italy, and today our certified email server went offline for the whole fucking day because the provider was an overload of feces. We were overwhelmed by the amount of people that needed help.

I don't even know what to say, if we don't fix it we're fucking fucked, like literally.

I really hope everyone is having a nice new year. For us, this is going terrible.

I had spent the last year working on a online store power by woocommerce with over 100k products from various suppliers. This online store utilized a custom API that would take the various formats that suppliers offer their inventory in and made them consistent. Now everything was going swimmingly initially, but then I began adding more and more products using a plug-in called WP all import. I reached around 100k products and the site would take up to an entire minute to load sometimes timing out. I got desperate so I installed several caching plugins, but to no avail this did not help me. The site was originally only supposed to take three to four months but ended up taking an entire year. Then, just yesterday I found out what went wrong and why this woocommerce website with all of these optimizations was still taking anywhere from 60 to 90 seconds to load, or just timing out entirely. I had initially thought that I needed a beefier server so I moved it to a high CPU digitalocean VM. While this did help a little bit, the site was still very slow and now I had very high CPU usage RAM usage and high disk IO. I was seriously stumped the Apache process was using a high amount of CPU and IO along with MYSQL as well. It wasn't until I started digging deeper into the database that I actually found out what the issue was. As I was loading the site I would run 'show process list' in the SQL terminal, I began to notice a very significant load time for one of the tables, so I went to go and check it out. What I did was I ran a select all query on that particular table just to see how full it was and SQL returned a error saying that I had exceeded the maximum packet size. So I was like okay what the fuck...
So I exited my SQL and re-entered it this time with a higher packet size. I ran a query that would count how many rows were in this particular table and the number came out to being in the millions. I was surprised, and what's worse is that this table belong to a plugin that I had attempted to use early in the development process to cache the site. The plugin was deactivated but apparently it had left PHP files within the wp content directory outside of the actual plugin directory, so it's still executing scripts even though the plugin itself was disabled. Basically every time I would change anything on the site, it would recache the whole thing, and it didn't delete any old records. So 100k+ products caching on saves with no garbage collection... You do the math, it's gonna be a heavy ass database. Not only that but it was serialized data, so when it did pull this metric shit ton of spaghetti from the database, PHP then had to deserialize it. Hence the high ass CPU load. I had caching enabled on the MySQL end of things so that ate the ram. I was really desperate to get this thing running.
Honest to God the main reason why this website took so long was because the load times made it miserable to work on. I just thought that the hardware that I had the site on was inadequate. I had initially started the development on a small Linux VM which apparently wasn't enough, which is why I moved it to digitalocean which also seemed to not be enough, so from there I moved to a dedicated server which still didn't seem to be enough. I was probably a few more 60-second wait times or timeouts from recommending a server cluster to my client who I know would not be willing to purchase it. The client who I promised this site to have completed in 3 months and has waited a year. Seriously, I would tell people the struggles that I would go through with this particular site and they would just tell me to just drop the site; just take the money, just take the loss. I refused to, this was really the only thing that was kicking my ass. I present myself as this high-and-mighty developer like I'm just really good at what I do but then I have this WordPress site that's just beating the shit out of me for a year. It was a very big learning experience and it was also very humbling as well, it made me realize that I really don't know as much as I think I might. It was evidence that there is still so much more to learn out there, I did learn a lot from that experience especially about optimizing websites the different types of methods to do that particular lonely on the server side and I'll be able to utilize this knowledge in the future.
I guess the moral of the story is, never really give up. Ultimately things might get so bad that you're running on hopes and dreams. Those experiences are generally the most humbling. Now I can finally present the site that I am basically a year late on to the client who will be so happy that I did not give up on the project entirely. I'll have experienced this feeling of pure euphoria, and help the small business significantly grow their revenue. Helping others is very fulfilling for me, even at my own expense.

Anyways, gonna stop ranting. Running out of characters. If you're still here... Ty for reading :')

I work on a warehouse dev team. One day this past year, I was trying to deploy a new build to a QA server. Earlier that day I had been looking at the logs on the production server and had left the ssh session open. I had been working for less than a year out of college at this point and shouldn't have had access to deploy to the production server.

Long story short I deployed my QA build to the production server and saw there were problems connection to our production database. Then my heart dropped in my chest as I realized I had just brought down our production server.

I managed to get the server back up by rolling back in about 5 minutes and no one ever knew except some people on my team.

I felt horrible for the longest time. Later in the year another guy that joined my team that has about 20 years of experience under his belt did the exact same thing, but needed help rolling it back. Needless to say, that made me feel a lot better. 😂

Definitely the worst moment of my year.

At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).

At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.

I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.

A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.

First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.

Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.

So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.

Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care.

Welp, this made my night and sorta ruined my night at the same time.

He decided to work on a new gaming community but has limited programming knowledge, but has enough to patch and repair minor issues. He's waiting for an old friend of his to come back to start helping him again, so this leads to me. He needed a custom backend made for his server, which required pulling data from an SQL/API and syncing with the server, and he was falling behind pace and asked for my help. He's a good friend that I've known for a while, and I knew it wouldn't take to long to create this, so I decided to help him. Which lead to an interesting find, and sorta made my night.

It wasn't really difficult, got it done within an hour, took some time to test and fix any bugs with his SQL database. But this is where it get's interesting, at least for me. He had roughly a few hundred people that did beta testing of the server, anyways, once the new backend was hooked in and working, I realized that the other developer he works with had created a 'custom' script to make sure there are no leaks of the database. Well, that 'custom' script actually begins wiping rows/tables (Depends on the sub-table, some get wiped row by row, some just get completely dropped), I just couldn't comprehend what had happened, as rows/tables just slowly started disappearing. It took me a while of checking, before checking his SQL query logs (At least the custom script did that properly and logged every query), to realize it just basically wiped the database.

Welp, after that, it began to restrict the API I was using, and due to this it identified the server as foreign access (Since it wasn't using the same key as his plugin, even though I had an API key created just so it could only access ranks and such, to prevent abuse) and begin responding not with denied, but with a lovely "Fuck you hacker!" This really made my night, I don't know why, but I was genuinely laughing pretty hard at this response.

God, I love his developer. Luckily, I had created a backup earlier, so I patched it and just worked around the plugin/API to get it working. (Hopefully, it's not a clusterfuck to read, writing this at 2 am with less than an hour of sleep, bedtime! Goodnight everyone.)

TL;DR: When picking vendors to outsource work to, vet them really well.

Backstory:
Got a large redesign project that involves rebuilding a website's main navigation (accessibility reasons).
Project is too big just for our dev team to handle with our workload so we got to bring a 3rd party vendor to help us. We do this often so no big deal.
But, this time the twist was Senior Management already had retained hours with a dev shop so they want us to use them for project. Okay...

It begins:
Have our scope / discovery meeting about the changes and our expected DevOps workflow.

Devs work Local and push changes to our Github, that kicks off the build and we test on Dev, then it goes to Staging for more testing & PM review. Once ready we can push to prod, or whenever needed. All is agreed, everyone was happy.

Emailed the vendors' project manager to ask for their devs Github accounts so we can add them to the project. Got no reply for 3 days.

4th day, I get back "Who sets up the Github accounts?"

fuck me. they've never used Github before but in our scope meeting 4 days ago you said Github was fine...??

Whatever, fuck it. I'll make the accounts and add them.

Added 4 devs to the repo and setup new branch. 40min later get an email that they can't setup dev environment now, the dev doesn't know how to setup our CMS locally, "not working for some reason."

So, they ask for permission to develop on our STAGING server.. "because it's already setup"... they want to actively dev on our staging where we get PM/Senior Management approvals?

We have dev, staging, production instances and you want to dev in staging, not dev?... nay nay good sir.

This is whom senior management wants us to use, already paid for via retainer no less. They are a major dev shop and they're useless...

😢😭

Cant wait for today's progress checkup meeting. 😐😐

/rant

Ex-colleague asked for help in regards to an old project we were working at my old job

Him:"We're experiencing an internal server error. What do we do?"
Me:"Restart tomcat, dude"
Him:"How?"

Then I explained how by finding tomcat in ps -ef in a Redhat server, because he's a Linux noob and needed a lesson in how services works. Proceeded to explain how to restart tomcat with an online guide available.

Him:"Couldn't find tomcat in any of the servers"
Me:"Are you sure? Send me screenshots"
Him: sent screenshots
Me:"it's there. Look carefully."
Him: finds it and proceeded to restart tomcat.
Him: "Can't restart. Some catalina.sh is stopping it."
Me:"Figure it out. You can do it".

Half a day passed...

Him:"I give up. If I restart the server, will tomcat also restart?"
Me:"Up to you man. It will work but it's bad practice."

He restarted the server vand now everything is honky dory. I feel sorry for him though.

My work product: Or why I learned to get twitchy around Java...

I maintain a Java based test system, that tests a raster image processor. The client is a Java swing project that contains CORBA bindings to the internal API of the raster image processor. It also has custom written UI elements and duplicated functionality that became available in later versions of Java, but because some of the third party tools we use don't work with later versions of Java for some reason, it's not possible to upgrade Java to gain things as simple as recursive directory deletion, yes the version of Java we have to use does not support something as simple as that and custom code had to be written to support it.

Because of the requirement to build the API bindings along with the client the whole application must be built with the raster image processor build chain, which is a heavily customised jam build system. So an ant task calls out to execute a jam task and jam does about 90% of the heavy lifting.

In addition to the Java code there's code for interpreting PostScript files, as these can be used to alter the behaviour of the raster image processor during testing.

As if that weren't enough, there's a beanshell interface to allow users to script the test system, but none of the users know Java well enough to feel confident writing interpreted Java scripts (and that's too close to JavaScript for my comfort). I once tried swapping this out for the Rhino JavaScript interpreter and got all the verbal support in the world but no developer time to design an API that'd work for all the departments.

The server isn't much better though. It's a tomcat based application that was written by someone who had never built a tomcat application before, or any web application for that matter and uses raw SQL strings instead of an orm, it doesn't use MVC in any way, and insane amount of functionality is dumped into the jsp files.

It too interacts with a raster image processor to create difference masks of the output, running PostScript as needed. It spawns off multiple threads and can spend days processing hundreds of gigabytes of image output (depending on the size of the tests).

We're stuck on Tomcat seven because we can't upgrade beyond Java 6, which brings a whole manner of security issues, but that eager little Java updated will break the tool chain if it gets its way.

Between these two components we have the Java RMI server (sometimes) working to help generate image data on the client side before all images are pulled across a UNC network path onto the server that processes test jobs (in PDF format), by reading into the xref table of said PDF, finding the embedded image data (for our server consumed test files are just flate encoded TIFF files wrapped around just enough PDF to make them valid) and uses a tool to create a difference mask of two images.

This tool is very error prone, it can't difference images of different sizes, colour spaces, orientations or pixel depths, but it's the best we have.

The tool is installed in both the client and server if the client can generate images it'll query from the server which ones it needs to and if it can't the server will use the tool itself.

Our shells have custom profiles for linking to a whole manner of third party tools and libraries, including a link to visual studio 2005 (more indirectly related build dependencies), the whole profile has to ensure that absolutely no operating system pollution gets into the shell, most of our apps are installed in our home directories and we have to ensure our paths are correct for every single application we add.

And... Fucking and!

Most of the tools are stored as source bundles in a version control system... Not got or mercurial, not perforce or svn, not even CVS... They use a custom built version control system that is built on top of RCS, it keeps a central database of locked files (using soft and hard locks along with write protecting the files in the file system) to ensure users can't get merge conflicts by preventing other users from writing to the files at all.

Branching is heavy weight and can take the best part of a day to create a new branch and populate the history.

Gathering the tools alone to build the Dev environment to build my project takes the best part of a week.

What should be a joy come hardware refresh year becomes a curse ("Well fuck, now I loose a week spending it setting up the Dev environment on ANOTHER machine").

Needless to say, I enjoy NOT working with Java. A lot of this isn't Javas fault, but there's a lot of things that Java (specifically the Java 6 version we're stuck on) does not make easy.

This is why I prefer to build my web apps in python or node, hell, I'd even take Lua... Just... Compiling web pages into executable Java classes, why? I mean I understand the implementation of how this happens, but why did my predecessor have to choose this? Why?

AWS Contractor

I've been putting a web application together that I'm looking to have published on AWS. Not having too much experience with AWS, I am looking to hire a contractor. I've had a number of quotes from different AWS admin's ranging from $40 an hour to $200 an hour, from 1-days worth of work to 2-months worth of work!

I'm not really sure what to make of it or to whom to trust. I believe they’re using my ignorance to overcharge me. I've listed my requirements below, could you guys use your professional experiences to let me know what you think is reasonable charge and where best I could find someone to help me.

My application is a US shopping website where people can set up an online shop and upload their products and maintain an inventory of the items.

This is what I’m looking for setup and configuration with the following two areas:

1) AWS SYSTEMS…

* AIM - Set up my server admin users.

* EC2 - Web Hosting.

* RDS - Fast DB.

* SES - To send emails.

* S3 Buckets - Uploaded image hosting.

Route 53 - I don’t know but someone said I should have this.

* Elastic Load Balancing - For, well, load balancing.

2) SCRIPTS…

* A script that would back up the database once a day and save it to a private S3 Bucket.

* A script that will run once a day that calls an internal API, and POST a query to it.

* A script that runs once every 90 days, to refresh the SSL using ZeroSSL.com

Is there anything that I've missed such as security systems, firewalls, auto scaling and CDNs?

The quotes that I've received arranged from $320 to $64,000. I know I am being abused because of my ignorance. I would never overcharge someone because the customer doesn't know the efforts of the work. I hope someone here can help to understand the efforts needed and can tell me the true cost.

Thank you

I need some help speccing a server.
We have 5 video editors/ 3D animators, who’s currently working on their own machine and keeping a version on their machine.

I want a server that allows 5 people to work on it simultaneously. Preferably a few SSDs for the current project, then 50 TB++ of long storage. What raid should I use, how much ram, cpu, what system should I run, etc etc

#Suphle Rant 11: Laravel board launch

The launch took almost 2 weeks more than originally slated, because I sought to install it manually, just as an outsider would. Installation steps had been documented, automated tests for the installation tests were passing. When time came to actually execute the binary from the terminal, we went from one obstacle to the other. First, were the relatively minor Composer/Roadrunner issues, eventually resolved by the helpful RR maintainers who sat with me through a Discord server for about 2 hours until their command ran the way I needed it to.

Next was the Psalm scare: One of my value propositions was the guarantee of eliminating all type related bugs in Suphle apps. I intended to use Psalm for that. Wrote tests as usual. Turns out the library behaves differently under conditions differing from raw CLI usage. I resurrected threads I'd opened since December that were left unattended, and with some help from the maintainer, we eventually got it to do what I need it to do.

I was all the more frightened by the fact that Transphporm had caused me to renege on one of my earlier promises. I can only miss so many targets. After this, the docs had to be updated with all the changes effected to accurately integrate those two. Project installation and initialization commands were ran rigorously to ensure all progresses smoothly.

Tagged one final release and suddenly became impatient to launch on our local Laravel group chat where I've been a member for the last 4+ years, where we've had a rollercoaster of emotions. In that time, I've refined my launch speech to suit that audience -- obviously, countless times. Not just a tame "It's my pleasure to announce what I've been working on", but near 40 messages going into details about the inner workings, why it was built, how it compares. An expose that dove deeper than I would anywhere else.

I scheduled a time for them to tune in and got some encouraging anticipation. Ended up deflated after posting the whole thing. Only about 5 persons interacted. 1 (who I've chatted with outside the board) was quite enthusiastic. Feverishly checked the docs but commented it was overwhelming and he'd need more time. Already starred the repository.

For some context, there are give or take 250 members on that board. Not all are active but activity there easily reaches a crescendo when the topic discussed is about inanities like what 3rd party services to use for SMS, how to receive salaries from abroad, or job openings. I was optimistic when the acquaintance mentioned above published a payment library and met a riotuous welcome as one of their own. Maybe, they are simply not fond of me and the speech should have been passed off to someone else.

I checked Packagist installs -- not more 10. For 3 years, I'd been hyped up for that night; but for some reason, the audience I considered myself closest to flopped, woefully. Thankfully, this isn't the main launch. I'm still holding out hope for that. If it fails, I would have sunk an immeasurable amount of effort and time, that nobody will compensate me for. That is the one place I go to see those more advanced than me in PHP. I constantly learn there and find stimulating conversations there.

Now, I can no longer predict reception from other presentations. All I can do now is hope

I was trying to set up my own "cloud" for iot experiments. I planned to use Intel Edison with mqtt broker (using mosca) and a node js app for providing API for mobiles and browser. And also to do other book keeping.

I spent the half day trying figure how to expose these servers to internet.

I configured ddns in noip.com and ddns settings configured to it in my router.
Port forwarded to the local server services I needed.

And then tested. Worked perfectly on any device in my router connected network. Tested on mobile network. Bam! It fucking doesn't work.

Then connected another router.
Double port forwarded. Again worked perfectly on router network. And failed on mobile.
Tested if ddns is right. Did nslookup it was fine as fuck.
Then disabled port forwarding. Did dmz. Nope. Nadda. No luck.

Then scratched my head so hard that I lost more already losing hairs.

Then remembered about router hardware firewall. Disabled it.
Tested
And there it didn't work.
My dreams shattered like a fucking deer hit by car on highway.
Didn't work.
Then I see the IP pointing to my router in nslookup. Its 172.20.xx.xx. Its a fucking private IP.

My Asshole ISP is running another private network behind firewall. Which I fucking can't port forward
Now I think how much of a noob and idiot I am. Fuck this shit. Fuck all of these shit.
I am going for SaaS option for mqtt broker.
(Or help me?)
Once again.
Asshole ISP.
Fuck your firewall.
(PS: I had test the next day. FML)

# This isn't THAT bad, but since I never had any collab before this one, this is the worst so far

I'm in a web development school where we need to do a yearly project. At the beginning, we started with the idea of doing an online wallet that would handle crypto-currencied (#blockchains), and other currencies too.
On the paper that sounds good, but the dude decided to create a NodeJS server api, and let's be honest, this was a gas factory. I couldn't help him because he was too fast in his ideas, and the third member was a bit more useful because he was the one creating the mobile app, so all he needed was an url that the dude couln't manage to create.

After a few weeks he started over the project, then over again a few weeks later, before coming to us and saying it was too difficult. We said "yeah, I mean you're own your own since the beginning, no wonders!" "Uh do you guys care if we change the whole project to do something else? Like a CV library"

Went a moment where he tried to over sell some incredible (read "overly common") features that already existed 10 years ago on some famous websites (ie. Monster), and he then eventually told me that this idea came from his new job, and that they needed this library. So we would have to work for his company for free. Nice.
The third guy and me came with a new idea (image recognition with IA and stuff), and we saw the dude maybe 5 times the whole week while we're supposed to work together

PHP dev help/advice needed!
We have problems with mysql. Still stuck with mariaDB, I'm using indexes (correct ones) and we have problems with scaling. we have a few tables with over 100mil rows, 1 of them is being read every morning with a subselect that counts unique rows, and fails every time because of timeout/lock, the temp table size was increased and helped for a little while but as time goes on the table grows and the problem reappears. I'm reading from a slave server that was purposely created for read only, yet we still have problems. We're using managed dedicated servers for out hosting and they aren't willing to optimise the database configs for our needs. What are the easiest options for scaling at this point? Going fully dedicated server and perconaDB? NOsql? Sharding the server? Anyone got any good blogposts or something to read about this? your own experience?

I needed to migrate one DB to another with one sql suite but instead I fucked up and suddenly disconnected both DBs, without being able to reconnect them again
I waisted a whole day for debugging, but found nothing
And guess which magic fixed all issues? On and Off a service of an app
On and Off!!!
The fun thing is that restarting the server didn't help, but the only service helped

Help needed.
Anyone who has worked with OpenBazaar REST APIs. I cannot seem to get any response from the endpoints even though I think I've set the server correct. Any sort of help would be highly appreciated.

P.S. (Rant) the docs and the "helpful slack community" are total shit.

Follow up sorta...

So I got pulled into a support issue on a day off. Some system was facing timeouts on our servers so had to investigate.

Over the weekend as part of the release, I released the ELK stack I built and today I used that to help.

Pretty much immediately pinpointed which machine was hanging though still had to investigate and confirm so split between KQL and checking the server logs.

One thing I've always griped about is how no one created schema docs for it mongo collections so can't easily figure out what they do or your to get the document needed.

Well guess it's my turn.... Because only I know the schema :)

Okay, I hope a few people can help me with this; what are the benefits/reasons to use MS technologies? I'm talking about .NET, ASP, Windows Server, Powershell...

I've never understood it. I love Nodejs because you don't have any packages unless you ask for them. Alpine Linux is amazing! It runs on 8MB of RAM from fresh and doesn't need much more space to install.

You want .NET core? 140MB download. You're configuring database connection strings? Feel free to type in whatever you like, it'll parse and replace with some magic variables that have come from some other random file.

I was using Powershell recently, needed to set an env variable. Bash is happy with "export name=value". You want to do that in Powershell? I just googled it and found an entire 40-minute read discussing how to set env vars. Why?! It should be one command, and I don't know who thought that "Get-ChildItem" was _obviously_ referring to env variables.

It seems to me that everywhere MS has got their hands on development-wise, it inherits the typical sales bullshit. No no, you can't call them "websockets", they have to be branded "SignalR" and add tons of overhead. You can't say "disable notifications" it has to be "focus assist". I'm really surprised something as simple as a keyboard hasn't become a "varied user input device" or something of the like.

Am I alone in thinking this?