I did a simple bar graph to show the status of a db migration. Each bar is a computer with a db file. As I was bored waiting, I added a King Kong hanging from the longest bar.

Running a huge migration on our production db.
Takes 45+15 minutes (migration+tests) on my lappy. So. I'm going to be bored and slightly worried for the next hour.

Hopefully the production servers will be faster.

I HATE TESTING DB MIGRATIONS! SHIT TAKES BLOODY FOREVER!

This one takes 20 freaking minutes each attempt, and I need to run it. yet again.

$@%&!

I'm going on vacation next week, and all I need to do before then is finish up my three tickets. Two of them are done save a code review comment that amounts to combining two migrations -- 30 seconds of work. The other amounts to some research, then including some new images and passing it off to QA.

I finish the migrations, and run the fast migration script -- should take 10 minutes. I come back half an hour later, and it's sitting there, frozen. Whatever; I'll kill it and start it again. Failure: database doesn't exist. whatever, `mysql` `create database misery;` rerun. Frozen. FINE. I'll do the proper, longer script. Recreate the db, run the script.... STILL GODDAMN FREEZING.

WHATEVER.

Research time.
I switch branches, follow the code, and look for any reference to the images, asset directory, anything. There are none. I analyze the data we're sending to the third party (Apple); no references there either, yet they appear on-device. I scour the code for references for hours; none except for one ref in google-specific code. I grep every file in the entire codebase for any reference (another half hour) and find only that one ref. I give up. It works, somehow, and the how doesn't matter. I can just replace the images and all should be well. If it isn't, it will be super obvious during QA.

So... I'll just bug product for the new images, add them, and push. No need to run specs if all that's changed is some assets. I ask the lead product goon, and .... Slack shits the bed. The outage lasts for two hours and change.

Meanwhile, I'm still trying to run db migrations. shit keeps hanging.

Slack eventually comes back, and ... Mr. Product is long gone. fine, it's late, and I can't blame him for leaving for the night. I'll just do it tomorrow.

I make a drink. and another.
hard horchata is amazing. Sheelin white chocolate is amazing. Rum and Kahlua and milk is kind of amazing too. I'm on an alcoholic milk kick; sue me.

I randomly decide to switch branches and start the migration script again, because why not? I'm not doing anything else anyway. and while I'm at it, I randomly Slack again.

Hey, Product dude messaged me. He's totally confused as to what i want, and says "All I created was {exact thing i fucking asked for}". sfjaskfj. He asks for the current images so he can "noodle" on it and ofc realize that they're the same fucking things, and that all he needs to provide is the new "hero" banner. Just like I asked him for. whatever. I comply and send him the archive. he's offline for the night, and won't have the images "compiled" until tomorrow anyway. Back to drinking.

But before then, what about that migration I started? I check on it. it's fucking frozen. Because of course it fucking is.

I HAD FIFTEEN MINUTES OF FUCKING WORK TODAY, AND I WOULD BE DONE FOR NEARLY THREE FUCKING WEEKS.

UGH!

OMG people please stop being so fucking lazy... help me help you... RN I have multiple support people asking me to fix a bug that they can't even describe (and honestly I doubt it exists) and a fellow developer who refuses to give me a DB or migration to test his patch but wants me to merge it urgently. FOH and die, y'all.

...He hired a shit dev who did the same work in 3 times less than what I asked for.

He's now back crying to fix his Fuck up.

You ask how I know he is shit. He SSH-ed into the server. Worked directly off the production files. Worst of all, he installed phpmyadmin, changed the db structure without even writing a fucking migration !!!

How the hell am I supposed to know what he changed!! It's gonna be a long night 😥

First company I worked for, overall it was a good experience, but at one point they promoted a consultant to project manager, and their planning skills were about as good as their people skills, which is to say, appalling.

We had a project update for a huge client, that required, for political BS reasons, that most of the team spend several weeks on-site, 300km away from home.
Go-live was approaching, and the plan was: migration starts Friday night, shortly after midnight (so actually, Saturday) once the client’s IT confirms DB is backed up. Expected duration: 5 hours.

- So, you expect me to work from midnight to 5am on Saturday? And when do we start working on Friday?

- 9am, of course.

- 9am!? So you actually planned a 20 hour work day? (Note: legal maximum here is 10 hours in a day, 40 a week)

- And we have to be there on Saturday 1pm to recheck everything is running smooth.

wtaf were they thinking?

Contex: Working on a c++ frankenstein code (mixture of legacy and new stuff whith things depending on the client using it)

User Story: Migration from oracle to SQLite for half of the DB data

Summoner: One client wants to keep using legacy for now, therefore we need an strategy chooser templated singleton...

Satan 666 = Singletons + Static methods + Different compilation units

Result: 3/4 of the files of the full backend being modified for the migration.

Conclusion: When will be loaded on production company will probably lose many clients due to unspected bugs everywhere.

Insert potato here

One of the largest companies on the continent. Uses Oracle on AWS RDS with the beefiest resources available. It comes to the point where lowering the number of CPUs boosts the DB performance up (concurrency). Point is - Oracle is sweating hard during our tests. You can almost feel the smell of those hot ICs on AWS servers.

And then someone at higher levels, while sitting on a pooper, has a great idea: "I know! Let's migrate to Aurora! They say it's so much faster than anything there is!"

*migration starts*

Tests after migration: the database on the largest instance possible shits itself at 10% of the previous load: the CPU% is maxed out (sy:60%,us:40%), IO is far, far from hitting the limits.

Is it really possible Aurora will cope with the load better than Oracle? Frankly, I haven't seen any database perform better than Oracle yet. Not sure if it's worth to invest time in this adventure..

One of our projects migrated their file-repository to another one during a major release.

Instead of giving this task to an experienced programmer, they gave it to the head of the respective dev department due to the usual release panic.

Soo.... He wrote the migration tool. It was executed during the release. Everything seemed fine so far.

A few days later. Someone from the above project came to my team due to some "strange behaviour on the production database".
They reported that they couldn't download some of the user's documents due to unknown reasons.

After quickly analyzing the current state of the new file-repository, we concluded that the affected documents did not exist in the new repository.

Then we took a look at the so called migration tool...
Well.. After nearly 30 min. we knew the root cause for that.
They only migrated the first 4 levels of the folder structure. Due to the assumption that "we don't use deeper nesting". (Facepalm)

As the head of their department wrote it, no one seems to questioned it either. Nor did they made a code review and ended up with a tool with hard coded urls to the production db, no version control, no build tool, no ci, nothing. Breaking nearly every possible company standard.

However.. That's not it. When analyzing their migration tool we noticed another even more dangerous thing.
They mixed up the id generation of the migrated documents resulting in a random assignment between customers and documents. Which is quite bad as this contains sensitive information. E.g. passports

They offered us quite a nice amount of money to fix this until EOB. We declinded as it was simply not possible in that time, but agreed to support them with the new tool.

After some time I heard that they migrated production again. And they fucked it up again. They never talked to us after we offered them support...

The third and final migration was written by us. Not only migrated it correctly. It was also way faster. By factor 20.

In the end we haven't gained anything from this rushed project as the penalties were piling up due to this fucked up migration.
After all this time I'm not sure who is to blame. In my opinion, partly all of them.
Head of department who can't and shouldn't code.
Seniors who didn't review the code and didn't ask for help.
Release mgmt who put way too much pressure on the devs.

Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!

April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!

July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....

August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...

September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'

somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..

I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.

Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.

Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.

Can I?

I'm currently between jobs and have a few rants about my previous job (naturally). In retrospect, it's somewhat therapeutic to range about the sheer brainfuckery that has taken place. Enjoy!

First, let me set the scene: legacy B2B web app made with LEMP stack and sencha ext.js 3 + 4 (don't ask) and a lot of madness. Let's call that app "Alpha".

Alpha is a self made CMS build for typical ERP stuff. Yes, a self made CMS: entities are containers, containers have types and fields and values. Like so many legacy PHP apps, it does not have a dedicated FE: the HTML is rendered on the server and then spewed out to the browser.

Easy right? Coding like it's 1999! But there was a twist: Because everything is basically a container, the HTML-templates are saved in the DB. Along with the nessary JS and the CSS. And the translation variables. Why? Because fuck you! That's why. Who needs a git history anyways.

For some reason, Alpha was kinda slow.

There was also an editor, that allowed you to modify templates (web, mail, pdf) on the fly in prod. Because templates contain repeating data (header/footer), one template could contain additional templates. Much confusion. You could change templates via migration (slow, boring) or just ctrl-c/ctrl-v that sucker (fast, much excitement).

Did I mention Alpha was slow?

On with the rant: e-mails! How do they work? Noone knows. How to send mails asynchronous in PHP? Witchcraft is the only possible answer to that riddle. Here is your enterprise™ solution:
1. create mail
2. insert mail into DB
3. WAIT UP TO 59 SECONDS FOR A FUCKING CRON TO SEND MAIL

Why? "Because that way, we can resend mails in case the network is down :)"

Same procedure for the SOAP-API (db-queue + cron). You read that right: all requests to various other systems are processed once a minute.

Alpha slow.

Alpha was only one of several systems. Imagine a bunch of monolithic php apps, interconnected via SOAP, REST and GraphQL like a godamn intergalactic orgy. Image having to debug that cluster fuck.

Let's say there is a bad request. These things happen. No biggie. Remember the db-queue? Let's try to send the bad request a second time! And a third time! Still no luck? How odd. Let's create a specific file in a specific directory: a LOCK-file. Now, "the db-queue is on hold and no request gets processed :)"

Golly gee thanks Alpha.

Anyhow, did you know that MySQL has a join limit of 61 tables?

Mine: not listening to early warning sings that shit won't work. As a result we implemented the wrong technology and had to roll back after a month in production. Lost some customers data, though we had backups, so it was just a delta of a day or so. Still, not pretty, had a rough night writing scripts at 3am to check data validity after.

I will throw in another one from a colleague of mine. He was running some database migrations and ended up pointing the DB migration tool to production by accident. Oops. That one required a restore as well, if I remember correctly.

get to the office at 5:45AM to do a migration because the client doesnt want to automate it.. but pays extra for the inconvenience...

Good to see people are learning and don't use hard coded credentials and store them in the DB.
Now they just have to quit putting them into migration files

I so hate working with legacy db code! I wish I could migrate all to entity framework but the migration is a pain in the ass and would take hours which the client wouldn't like.

Come on, WordPress! Why are you such an asshole? I just want to migrate and move on with my life.
Is it because I started with Joomla? Is it because I cheated you with Flask?
Can you please, please be nice to the same db you made? Am I asking to much for?

So many choices for backend I'm fucking confused. Yesterday, I tried Django and i found it little similar to RoR(in generating things, db migration things).
I'm currently working in NodeJS.
I even don't know should I rant or cry.
And of course frontend is another thing same like this.....
And I'm not much experienced to differentiate them and know which is better and where it will fit.
Can anyone tell me in simple way which framework fits where?

I thought today was a good day to look at how I will deal with database migrations for this node.js/sql-server application. I read up docs for a few migration frameworks but the ones I found seem to make things too complex.

I am tempted to just roll my own by storing a db version in a table, numbering .sql scripts in a folder and running all the higher numbered scripts when the application starts.

Anyone know is there any gold standard for this sort of thing or anything to watch out for?

Guessing my rant free streak is over. Trying to connect to a mongo atlas cluster. Just migrated from mlab as mongo Inc is discontinuing the heroku add on.
Migration went well. I can connect to atlas cluster via mongo shell.
Reactive mongo claims it supports dns seed list. I add mongodb+srv connection string. Doesn't work.
I go back to atlas and allow all ips access (migrating staging dB first to make sure all is well so I can whitelist all ips) - > send a request-> mongo error. No primary node is available.
Disconnect from my network, connect to another network, same thing. I push the connection string to my server, test using an ssl connection to make a request, still no primary node available. I am about to lose my mind.

just 'Hello world' me trying to make a restful api.
*Got Ktor, loved the koltin, hated the deploy, quit.
*Got Django, loved the python, hated the sql migration, quit.
*Got Node, loved everything, hated mongo, can't quit now...
*Got Firebase DB now, I feel the hate monster...ghostly voices, saying, Work my slave, build it... dont stop, 'cause we're right behind you...

....and we're waiting for you

Why does Microsoft make entity migration default and every tutorial show them? I have other ways of migrating my db schema changed. Newbies come to me with this cluster already in motion and I have to help straighten it out. Then the newbie says "why do all the tutorials show it". I say because the people making the tutorials are trying to quickly create content for a tutorial.

Dont be that guy, ok. Just clean up your shit and don't let shit go through you.
I will git blame you. I will judge you.
#leaveitbetterthanyoufoundit

DB migrations give the chills.

This one went well. It worked as expected but that "uneasy" feeling lingers...

me: FE in work, but doing fullstack on my passion projects and somewhat confident on small VPSs - heck, I have a beard, I can do server stuff :) - migrating a WP site that just wont work, copied everything, didn't work, used a migration tool, didn't work, always getting "Connection refused"... must be something with the SSL certificates.. 3 fckn days passed by and nothing when I stumbled upon a forum post with similar issue where the guy stated: I tried all the obvious like copying files, db, certificates, enabled ssl on apache... then it hit me, this is a new installation, I didn't enabled SSL in apache sudo a2enmode ssl restarted apache and BOOM everything is working
part of me was like how stupid you have to be - but the other part is like I guess I learn something every day, this is how you migrate a WP site with the domain #IloveIT

Wordpress folks , please help me out here.

Sometime ago, I was making an android app for this small NGO which had a functional website recently created in wordpress. My goal was to make their users go mobile, as I liked watching their uploaded videos, but the website interface was just horrible. The different parts of the website were :
- A blog feed, managed via wordpress, publically accessible via json based (horrible) rest apis
- A video playlist/player managed via wordpress but the videos were put seperately in aws s3, accessible to registered users, and
- a login/signup page, managed via a wordpress plugin.

I was able to parse their blog feed into my app.
I tried to create users/login/signup via the rest apis for the app, but couldn't because wordpress doesn't expose its database. so i went with firebase based login/signup/forgot pwd system.
I was also able to make api calls for aws s3. It actually didn't required any authentication, the bucket was private but was made inaccessible based on a client side isLoggedIn boolean provided by the wordpress plugin. So i also imitiated the pattern and hide the video access in my app behind the login check provided by firebase.

Now they are not satisfied by the solution because while registration, all the data goes into my auth db and registration details db, present in the firebase. and when a user registers via wordpress, the details goes into their wordpress plugin. We need a way to have accounts of users on both the platforms and there doesn't seem to be a way from either the firebase or wordpress to perform such migration.

So do you know any plugin or ways by which we could do it? note neither they nor i know enough php do come up with a custom php based solution. i can try tho, but then again, the plugin they used is known as "Profile builder" and afaik, it doesn't expose any data in any manner