This is going to be a long rant, coz this is the only way to vent out my frustration against our tech head.

Yesterday, while our fucking twat tech head was playing around in company aws account, he terminated the production server. By mistake, apparently. Coz he doesn't know shit about server management. But that egoist ass won't admit and fucked the production server.

And then ran away. We developers sprang into action. Updated dns to point to staging server, setup virtual hosts, env files, point to prod database, force flush dns cache. All systems were up and running in 30 mins. And since it was staging server, it had lot of untested features and codes, and we spent rest of the day fixing the bugs.

And that tech head, who ran away hiding his tail between his legs, after he fucked the server, came back after systems were up. And started cracking jokes, that "so many features got released in 1 day" . "We cut server cost by shutting down 1 server."

We were struggling and working in full throttle to make the services running again. And that fuckity fucker was cracking jokes.

And I don't even know what excuse he gave to ceo for the downtime. I am pretty sure he would have made up some crappy excuse to hide his fucking mistake. That ass never admits his mistake. I am thinking to go to ceo today and tell the real story and get that faggot head fired or at least a strict warning.

Current work project is microservices architecture out of 4 - 8 components.

It is fully Infrastructure as a Code automatized. I just change somewhere code, git pushing

And it automatically invokes Gitlab CI, terraform, ansible, kubernetes helm charts.
Auto checking itself with unit and integration tests in autoredeployed staging env. Then it saves tested results to docker registry and asks for one button verificating click to be rereleased to prod.

I just go for drink or eat food. While all the stuff is happening.

And I am proud that all the infrastructure, backend and frontend I made on my own.

I don't need to remember how to Deploy it. It is all automatized

There is a company providing a very speciffic service. And it has a core application for that svc, supported by a core app team.
That company also has other services, which are derivations of the core one. So every svc depends on core.

Now that we're clear on that... I was working in a team of one of the subservices. We very strongly depended on core. In fact, our svc was useless if integration w/ core broke down.

The core team had an annoying habbit. They refused to version their webservices and they LOVED to push api updates w/o any warnings. Our prod, test, other envs used to fail bcz of core api changes quite often. Mgmt, IT head was aware of the problem and customers' complaints as well.

So as a result, once core api changes we're all in a panic mode: all prior priorities are lowered and revival of prod is to be our main focus. Core api is not docummented, the changes are not clear, so we have to reverse engineer the shit out of it. We manage to patch our prod up w/ hotfixes, but now we have tech debt. While working on the debt, core api changed again, in test env. Mgmt pushes debt back and reallocates us to hotfix test. Hotfix is 80% done when another core api breaks. Now mgmt asks us to drop wtv we're working on and fix that new break. By the time we're to deploy the hotfix, another api breaks in another env. The mgmt..... You get the picture :)

2 years go by, nothing has changed so far.

I'm having an existential crisis with this client.

We are spending millions of $s every year to make sure the product's performance is perfect. We are testing various scenarios, fine-tuning PLABs: the environment, application, middleware, infra,... And then we provide our recommendations to the client: "To handle load of XX parallel users focusing on YY, yy and Zy APIs, use <THIS> configuration".

And what the client does?
- take our recommendations and measure the wind speed outside
- if speed is <20m/s and milk hasn't gone bad yet, add 2x more instances of API X
- otherwise add 3xX, 1xY and give more CPUs to Z
- split the setup in half and deploy in 2 completely separate load-balanced prod environments.
- <do other "tweaking">

- bomb our team with questions "why do we have slow RTs?", "why did the env crash?", "why do we have all those errors?", "why has this been overlooked in PLABs?!?"

If you're improvising despite our recommendations, wtf are we doing here???

One day I will crack. Hopefully, not sometime soon.

Today in horror stories: setting up the local dev environment for a project I'm taking over creates changes and notifications in prod.

Oh and my local dev env is somehow receiving data from actual patients. I don't know how.

You know how my 'about' here says I create features from bugs?!? Well if you didn't before, now you do.. :P

Anyhow, today the most bizzare thing happened.. Customer played 'reverse uno card' on me..

Meaning?!

They reported a feature missing on uat env, when in fact I fixed a bug they have on prod..

Not sure how I should feel about this, but it sure made my day! (: I just hope they will not open a bug report for this missing bug..

I hate having to deal with our IT service desk. Every time it takes enormous energy to get to the right people and make them understand that no, you are not an idiot, but you actually have a technical issue.

Sure thing they do have a few competent nice folks there too I've gotten to know over time and they indeed have to deal with a ton of dumb non-tech savvy idiots on a daily basis. However, if my job title mentions "software" and "engineer" they should at least assume I'm an idiot in tech. Or something. Every single time I need to open a ticket, even for the simplest "add x to env y", I need to quadruple check that the subject line is moron-friendly because otherwise they would take every chance to respond "nah we can't do that", "that's not us", or "sry that's not allowed". And then I would need to respond, "yes you do:) your slightly more competent colleague just did this for us 2 weeks ago".

Now you might imagine this is on even another level when the problem is complex.

One of our internal apps has been failing because one of the internal APIs managed by a service desk team responds a 500 status code randomly but only when called with a specific internal account managed by another service desk team.
(when I say "managed by", that doesn't mean they maintain it, it just mean they are the only ones who would have access to change something)

Yesterday I spent over a fucking hour writing a super precise essay detailing the issue, proving a million times it's not on our end and that they need to fix it. Now here is an insight to what beautiful "IT service" our service desk provides:

1) ticket gets assigned to a "Connectivity Engineer" lady

2) few hours later she responds and asks me to give her the app and environment IDs and grant her access to those
(naturally everything in my email was ignored including these two IDs)

3) since the app needs to be in prod for the issue, I make a copy isolating the failing part and grant her access to the original "for reference" and the copy to play with

4) few hours later I get an email from the env that some guy called P made changes to the actual app, no changes to the copy
(maybe they immediately fixed the app even though I asked them to only touch the copy)
I also check the env and the live app had been shared with another 2 people giving them editing rights:)

5) another few hours pass and the lady responds that she had been chatting with P (no mention of who tf that guy is) and that P has a suggestion that might work and I should test it, "please see screen shot" for details:

These motherfuckers sent me a fucking screenshot of the env config file where "P has edited a few parameters" that might help. The screenshot had a 16 line part of the config json with a bunch of IDs and Base64 params which HE EDITED LOCALLY.

Again, because I needed a few iterations to realise what I've just witnessed:
These idiots modified some things in the main app (not the copy) for hours. Then came to the conclusion that the config needs some IDs and params updated. They downloaded the config json. Edited it locally. Did not fucking upload it back to the main or test app. Did not test it live. Did not CC in or direct the guy with changes to me. Did not send me the modified config file. Did not even paste the new IDs into the email. But TOOK A FUCKING SCREENSHOT OF THE MODIFIED FILE AND SENT THAT SHIT TO ME. And then had the audacity to ask me to test it when they had access to it and that's literally their fucking job.

I had to compare the fucking screenshot to the live config file and manually type in the changes.

And no, it still doesn't work. And Now I have to get back to them showing it still fails the same way but I just can't deal with these people. Fuck. Was hoping by the time I write it all down it'd be better, and it does feel a bit better, but I still need to get this app fixed. And I can only do it through these... monkeys. I just can't. Talking to these people drains my life energy... I'm just sad.

Started a new job as junior developer. One of my first task was to sent a simple notification on an event in out product. Write the code, test that it works, push to devops.

Code compiles, tests pass, it’s deployed to internal test env. Check that my notification works in the test env. No problem.

It’s deployed to the customers test environment. It works and customer accepts it for prod.

We release to prod and of course it fails. Seems to be a simple string.Format that fails for god knows why. After 3h of debugging on prod without success we decide to roll it back.

Today we decided to try it on a backup of the prod db since one of the strings was taken from the db. Still working. No matter what data I input when trying it locally it still wont reproduce the issue we saw on prod.

Fuck this

Does someone know a site where i can get professional level help/guides/tutorials with system architecture questions? Like best practices for implementing common features? (Something like stackoverflow but where u actually get an answer instead of insults)

Googling for tutorials gives very basic/demo level results that might not be great for scale/security in prod env

I just joined a new company.
Their CI pipeline is to give root access to staging and prod servers to every developer in the company and the manually git pull each repository (8-10 repos per server) and manually set nginx and port configurations. And if this wasn't enough, all of the 30 sites they have are basically the same site and they make the changes manually for each tenant (no env file). I'm amazed at how hard some people are willing to work.

2 AIX workloads got messed up during patching. Apparently they had a local database and they were in PROD env.

"Not a biggie - there's always a netbackup to restore the whole workload" - I said and reached out to netbackup folks.

Turns out even though netbackup was configured properly it had never even initiated successfully bcz backup NIC was not whitelisted in network layer. Never ever in 3 years.

Service guys were piiiiiiiiiiiiised big time

Yesterday whole 12 hours we were working on deployment about a feature X that has deadline yesterday itself.
Everything damn perfectly running on Test env but not on Prod.
We made Prod into Dev/Test/Fucking garabage env. Haha.
I was laughing to myself at same time crying hard in my deep heart.

Business guys chasing PM
PM chasing us
And from morning till night we were in same room. Had lunch, and dinner only went out for toilet and to refil water bottles.

And found that feature Y is not working at same time that is related to our feature X. Fucking we have been wasted hours on it.

One of my devs got so fucked up emotionally that he messed up the code (not his fault) he didnt had his lunch and dinner. Had to console him later that its not his fault. Poor guy not sure whether he slept or not; will find out in few hours.

Anyways reported a bug.
But that bug assigned to us for fixing.
Are you fucking kidding me.
Anyways no choice. Had to do it.
Hope today everything goes good or horribly bad. FYI no deployments on Friday damn we are in stalememt till Monday.

Fuck that bug
Or
May be fuck our stupiditiy while makiing mistakes.

did on my last project:
1 .Using QA env as dev env
2. Deploy in production not completely tested stuff (90% tested)
3. Run with errors in prod
4. Manual fix in prod
5. Git versioning

lol......a random deployment made me discover an expired certificate in a lower env.....the same will expir in prod in a few days.....I would be fucking screwed if that happened.....best part it only a redeploy/restart would have triggered the whole damn thing

To ensure everything goes well on prod, the chief QA set an env called 'stable' after prod deployment to stimulate true prod situation. Then we found that some apis on 'stable' crashed more often then those on prod.

Any one running Symfony on a Docker container in production? I currently try to migrate our dev env to a docker compose setup (from a "monolith" vagrant vm). I'm atually not stuck at a Symfony specific thing, but on a, I guess Docker specific one(?), The issue is, I need to read and write with two users to one folder (in my case the /application/var/cache folder). Since I mount my whole code into the docker container (to use an IDE on the local files), I've got a volume (not mounted to the outside world) for that folder. (As far, as good). Now this folder is owned by root and root is also the user I get when I enter the container. When I then run a cli script, that writes to this folder, every thing works (as it's run by root) and the resulting entries in the cache dir are owned by root. Trouble starts when the php fpm process tries to write stuff in there too (as it's run by www-data).
If I add `USER www-data` (or create a new user foobar and add `USER foobar`) the container exits with status 0

So I guess the question is, is anyone running an Symfony app on Docker in Prod, if so how do you solve this? Or another question would be what is the best practice to do this? Sure on dev I could just `chmod 777` the whole folder or run the php-fpm process as root, but if that thing ever goes to prod, I wouldn't sleep very well...

I work in a small team. As the senior dev I tens to focus on important tasks that shape the core of the product but some times I can’t divide my self when there are multiple tasks at hand, so I pass some tasks to the an other mid level dev.

So the task was to create an automation in order to CD (continuously deliver) an order from WHMCS of the (git versioned) product to customers UAT, PROD envs.

To get a background this is an old guy with “constricted” experience in PHP/jQuery/Joomla/Wordpress.

So when we were breaking up the tasks he told me he would like to implement this so i gave him the task as i was busy with core features.

I was like what could go wrong? I know he doesn’t know much about CI/CD but he can read right? He will google right? He will search for CI/CD solutions that do this out of the box right? He will design on paper or what ever and do small POCs right? He will design the flow first before starting the implementation right? RIGHT?

So fast forward to today I had a call with him this morning about some DB staff. And he wanted to show me his progress…

His solution is:
(parentheses is my brain)
1. Customer completes WHMCS order (perfect)
2. Web Hook 🪝 action (YES)
3. cpanel gets source and “automatic!” Init, all using pure PHP code ignoring the usage of the current framework (ok… something is missing)
4. cpanel web hooks(?) WHMCS to send email to customer with the envs initial setup page(?)
5. Customer opens link and adds setup info (ok fuck, fuck, fuck)

(Ok stay cool composed, lets ask some questions maybe he thought it all in a cool way I can’t get my mind around)

Me: So how are you gonna get the correct version from the repo to the env and init the correct schema?

Dev: I haven’t thought about it yet.

Me: Are we gonna save each version to a file system then your code is going to fetch them?

Dev: I haven’t really thought about it we will see. But look on customer init user setup I implemented a password strength validation and it also checks if the password is the same.

So after this Pokémon encounter I politely closed teams. Stood up drank some (a lot) coffee ☕️. Put out the washed laundry while reflecting on life’s good things, while listening to classical music 🎼 .

Then I sat on my office chair drank some more coffee, put some linking park starting with in that order:
“Numb” then “What I’ve Done” and ended with “In the end, it does really fucking matter”

I was hoping that I never have to build stupid websites again, but here I am...

The thing I hate the most, is the hassle to have an easy to update dev, staging and prod environment. Fuck wordpress, fuck drupal, fuck joomla.

git pull && composer install && npm install should be all that's needed to get the latest code in an environment.

composer require *** to install plugins. No stupid web interface where users install plugins in production env.

I don't want to create database dumps just because these fuckers think that you should store configurations in the database.

Is there any clean CMS primarily for professional programmers? Or are they all just made for retarded subhumans?

Half a day wasted. FUCK!

I use grafana loki and mimir/prometheus for telemetry. A few days ago I queried loki to see if logging is still working. Yesterday I changed the datasource to mimir, changed the query parameters to get metrics from another env, ran the query, and... Querier [mimir] crashed.

Wtf.
Error says it got too much data to chew on.

So I spend 4 hours playing with the querier and grpc limits, balancing between limit errors and OOMKills [2G ram].

I got suspicious about oomk. Why would it...

Then I tried to shrink the timeframe to 15min. Still oomk. Down to 5min -- now it worked. But the number of different metrics returned was over 1k

then I look once again at the query. And ofc it is ´{env="prod"}´

turns out, forgetting that you're querying metrics with a logs' query is an expensive and frustrating mistake. Esp. at 3am.

idk why it even returned me anything...

One of those debugging days where minutes feel like hours, and hours like days.

I had the bad luck of being asked to dive into a legacy project which was unmaintained for months, but of course it's still on prod. And very suddenly the urgent need arrises to change stuff.

Yet: the docker stack won't work. It builds fine but the stack crashes.

Long story short: some internal api URI were renamed and at some point one internal api started to always require an access token. Which we set for the stage, prod env yet somebody forgot to mention that to the devs of legacy-project.

That ain't too bad.

WHAT IS FUCKING BAD IS THAT YOUR SHITTY APP SWALLOWS THE ERROR MESSAGE!

I mean it's bad enough I have to `var_dump && die` your app since you never bothered to setup a xdebug that I could use out of the box, yet egregious fact that your app would catch a valid exception but transforms it into an "internal warning" is borderline insane!

It's ok to throw exceptions. It's ok to let your service die. That's how other will know what and where to fix it. (You may want to restrict the data visible to the outside, but that's a whole different conversation.)

Whats the fucking purpose of our companys dev test and prod env. Dev always only has a single instance. Sometimes clustered services run as cluster on test. Producing headaches because the clustering behaviour couldnt be seen on a single instance and Prod lacks all the nice deployment tools off dev/test. Fuck thinking you could dev then test and prod without any major reconfiguration and headaches. And all because the Storage costs is RETARDEDLY expensive because the backup EVERYTHING with ridiculess overkill. That results in headaches when requesting new servers. Took an old Workstation from the shelves and made it my vm slave so at least i could reliably deploy to test.. Fuck this process

What is the best approach for Continuous Integration / Continuous Delivery/Deployment? I'm using SVN as code repository and I need to identify the files that goes to Test/QA env. and the ones that goes do Prod env. (by Commit message or something else) via sFTP.
Any help would be appreciated. Already tested Jenkins, GoCD and Jetbrains Teamcity.

In my initial days as a web developer, i was assigned a task, to implement a cart share functionality in an e commerce company.
I made the functionality and tested on my system.
Result: working good.

Pushed it to beta testing environment.
Resilt: working good.

Pushed to pre production environment.
Result: working good.

Pushed to live site.
Result: 😀 Error in live site..
So a call comes to me from my team lead..
Asks what was the issue...

Me: i dont know either.
....

After 3-4 hrs:
I found the reason.
My system, beta test env, pre prod env are all having latest php version (5.6 i guess)
But the live server had old version of php.

Me: laughed like anything.
I didn't know that these things would matter in such a great level.

Moral of the story:
Be one with the force (server in this case)