Frack he did it again.
In a meeting with the department mgr and going over a request feature *we already discussed ad nauseam* that wasn’t technically feasible (do-able, just not worth the effort)

DeptMgr: “I want to see the contents of web site A embedded in web site B”
Me: “I researched that and it’s not possible. I added links to the target APM dashboard instead.”
Dev: “Yes, it’s possible. Just use an IFrame.”
DeptMgr: “I thought so. Next sprint item …what’s wrong?…you look frustrated”
Me: “Um..no…well, I said it’s not possible. I tried it and it doesn’t work”
Dev: “It’s just an IFrame. They are made to display content from another site.”
Me: “Well, yes, from a standard HTML tag, but what you are seeing is rendered HTML from the content manager’s XML. It implemented its own IFrame under the hood. We already talked about it, remember?”
Dev: “Oh, that’s right.”
DeptMgr: “So it’s possible?”
Dev: “Yea, we’ll figure it out.”
Me: “No…wait…figure what out? It doesn’t work.”
Dev: “We can use a powershell script to extract the data from A and port it to B.”
DeptMgr: “Powershell, good…Next sprint item…”
Me: “Powershell what? We discussed not using powershell, remember?”
Dev: “It’s just a script. Not a big deal.”
DeptMgr: “Powershell sounds like a right solution. Can we move on? Next sprint item….are you OK? You look upset”
Me: “No, I don’t particularly care, we already discussed executing a powershell script that would have to cross two network DMZs. Bill from networking already raised his concern about opening another port and didn’t understand why we couldn’t click a link. Then Mike from infrastructure griped about another random powershell script running on his servers just for reporting. He too raised his concern about all this work to save one person one click. Am I the only one who remembers this meeting? I mean, I don’t care, I’ll do whatever you want, but we’ll have to open up the same conversations with Networking again.”
Dev: “That meeting was a long time ago, they might be OK with running powershell scripts”
Me: “A long time ago? It was only two weeks.”
Dev: “Oh yea. Anyway, lets update the board. You’ll implement the powershell script and I’ll …”
Me: “Whoa..no…I’m not implementing anything. We haven’t discussed what this mysterious powershell script is supposed to do and we have to get Mike and Bill involved. Their whole team is involved in the migration project right now, so we won’t see them come out into the daylight until next week.”
DevMgr: “What if you talk to Eric? He knows powershell. OK…next sprint item..”
Me: “Eric is the one who organized the meeting two weeks ago, remember? He didn’t want powershell scripts hitting his APM servers. Am I the only one who remembers any of this?”
Dev: “I’m pretty good with powershell, I’ll figure it out.”
DevMgr: “Good…now can we move on?”

GAAAHH! I WANT A FLAMETHROWER!!!

Ok…feel better, thanks DevRant.

Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Story time:

I was once working on a project that dealt with incredibly sensitive financial data.

We needed a client’s database to do a migration.

They wouldn’t send it over the internet because it was too big and they didn’t think it would be secure.

They opt to send it in the post on an encrypted usb drive.

(Fair enough thinks I)

USB drive arrives.
Is indeed encrypted.

MFW there’s a post it note in the envelope with the password on.

MFW this is a billion dollar multinational petrochem company.

MFW this same company’s ‘sysadmin’ and ‘dba’ once complained because a SQL script I sent them didn’t work - they’d pasted it twice and couldn’t work this out from the fucking “table already created” error message management studio was throwing at them.

Arghhh, rant time :|

So yesterday I completed a database migration of 167,000 products from an old ecommerce system to a new one. Everything was brought over, orders, customer addresses, everything, really chuffed :)

The only thing the client picked up on was the lack of his spammy "meta keywords" data that I intentionally did not import. I mean the tag isn't used and a list of 40 comma separated random words you'd like to rank for isn;t going to help the sites SEO on bit.

Anyway, the client is now moaning a lot and insisting I add them in. Even after I explained that the meta keywords is gone for good reasons he insists on keeping the data.

Soooo, pointless :|

(note the tags for the sake of satire :) )

To replace humans with robots, because human beings are complete shit at everything they do.

I am a chemist. My alignment is not lawful good. I've produced lots of drugs. Mostly just drugs against illnesses. Mostly.

But whatever my alignment or contribution to the world as a chemist... Human chemists are just fucking terrible at their job. Not for a lack of trying, biological beings just suck at it.

Suiting up for a biosafety level lab costs time. Meatbags fuck up very often, especially when tired. Humans whine when they get acid in their face, or when they have to pour and inhale carcinogenic substances. They also work imprecisely and inaccurately, even after thousands of hours of training and practice.

Weaklings! Robots are superior!

So I replaced my coworkers with expensive flow chemistry setups with probes and solenoid fluid valves. I replaced others with CUDA simulations.

First at a pharma production & research lab, then at a genetics lab, then at an Industrial R&D lab.

Many were even replaced by Raspberry Pi's with two servos and a PH meter attached, and I broke open second hand Fischer Sci spectrophotometers to attach arduinos with WiFi boards.

The issue was that after every little overzealous weekend project, I made myself less necessary as well.

So I jumped into the infinitely deep shitpool called webdev.

App & web development is kind of comfortable, there's always one more thing to do, but there's no pressure where failure leads to fatalities (I think? Wait... do I still care?).

Super chill, if it weren't for the delusion that making people do "frontend" and "fullstack" labor isn't a gross violation of the Geneva Convention.

Quickly recognizing that I actually don't want to be tortured and suffer from nerve damage caused by VueX or have my organs slowly liquefied by the radiation from some insane transpiling centrifuge, I did what any sane person would do.

Get as far away from the potential frontend blast radius as possible, hide in a concrete bunker.

So I became a data engineer / database admin.

That's where I'm quarantining now, safely hiding from humanity behind a desk, employed to write a MySQL migration or two, setting up Redis sorted sets, adding a field to an Elastic index. That takes care of generating cognac and LSD money.

But honestly.... I actually spend most of my time these days contributing to open source repositories, especially writing & maintaining Rust libraries.

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Q: Your data migration service from old site to new site cost money.
A: Yes, I have to copy data from old database and import to the new one.

Q: Can I just provide you content separately so you don’t need to do that?
A: Yes, but I will have to charge you for copying and pasting your 100 pages of content manually.

Q: Can it come with part of the web development service and not as an additional service?
A: Yes, but the price for web development service will have to be increased to combine the two. If you don’t want to pay for it, I can just set up a few sample pages with the layout and you can handle your own content entry. Does that work for you?

Q: Well, but then I will have to spend extra time to work on it.
A: Yes you will. (At this point I think she starts to understand the concept of Time = Money...)

I wrote a database migration to add a column to a table and populated that column upon record creation.
But the code is so freaking convoluted that it took me four days of clawing my eyes out to manage this.
BUT IT'S FINALLY DONE.
FREAKING YAY.

Why so long, you ask? Just how convoluted could this possibly be? Follow my lead ~

There's an API to create a gift. (Possibly more; I have no bloody clue.)
I needed the mobile dev contractor to tell me which APIs he uses because there are lots of unused ones, and no reasoning to their naming, nor comments telling me what they do.

This API takes the supplied gift params, cherry-picks a few bits of useful data out (by passing both hashes by reference to several methods), replaces a couple of them with lookups / class instances (more pass-by-reference nonsense). After all of this, it logs the resulting (and very different) mess, and happily declares it the original supplied params. Utterly useless for basically everything, and so very wrong.

It then uses this data to call GiftSale#create, which returns an instance of GiftSale (that's actually a Gift; more on that soon).

GiftSale inherits from Gift, and redefines three of its methods.

GiftSale#create performs a lot of validations / data massaging, some by reference, some not. It uses `super` to call Gift#create which actually maps to the constructor Gift#initialize.

Gift#initialize calls Gift#pre_init (passing the data by reference again), which does nothing and returns null. But remember: GiftSale inherits from Gift, meaning GiftSale#pre_init supersedes Gift#pre_init, so that one is called instead. GiftSale#pre_init returns a Stripe charge object upon success, or a Gift (and a log entry containing '500 Internal') upon failure. But this is irrelevant because the return value is never actually used. Pass by reference, remember? I didn't.

We're now back at Gift#initialize, Rails finally creates a Gift object using the args modified [mostly] in-place by all of the above.

Another step back and we're at GiftSale#create again. This method returns either the shiny new Gift object or an error string (???), and the API logic branches on its type. For further confusion: not all of the method's returns are explicit, and those implicit return values are nested three levels deep. (In Ruby, a method will return the last executed line's return value automatically, allowing e.g. `def add(a,b); a+b; end`)

So, to summarize: GiftSale#create jumps back and forth between Gift five times before finally creating a Gift instance, and each jump further modifies the supplied params in-place.

Also. There are no rescue/catch blocks, meaning any issue with any of the above results in a 500. (A real 500, not a fake 500 like last time. A real 500, with tragic consequences.)

If you're having trouble following the above... yep! That's why it took FOUR FREAKING DAYS! I had no tests, no documentation, no already-built way of testing the API, and no idea what data to send it. especially considering it requires data from Stripe. It also requires an active session token + user data, and I likewise had no login API tests, documentation, logging, no idea how to create a user ... fucking hell, it's a mess.)

Also, and quite confusingly:
There's a class for GiftSale, but there's no table for it.
Gift and GiftSale are completely interchangeable except for their #create methods.

So, why does GiftSale exist?
I have no bloody idea.
All it seems to do is make everything far more complicated than it needs to be.

Anyway. My total commit?
Six lines.
IN FOUR FUCKING DAYS!

AHSKJGHALSKHGLKAHDSGJKASGH.

The riskiest dev choice...

How about "The riskiest thing you've done as a dev"? I have a great entry for that. and I suppose it was my choice to build the feature afterall.

I was working on an instance of a small MMO at a game company I worked for. The MMO boasted multiple servers, each of them a vastly different take on the base game. We could use, extend, or outright replace anything we wanted to, leading to everything from Zelda to pokemon to an RP haven to a top-down futuristic counterstrike. The server in this particular instance was a fantasy RPG, and I was building it a new leveling and experience system with most of the trimmings. (Talents, feats/perks, etc. were in a future update.)

A bit of background, first: the game's dev setup did not have the now-standard dev/staging/prod servers; everything ran on prod, devs worked on prod, players connected and played on prod, etc. Worse yet, there was no backup system implemented -- or not really. The CTO was really the only person with sufficient access. The techy CEO did as well, but he rarely dealt with anything technical except server hardware, occasionally. And usually just to troll/punish us devs (as in "Oops ! I pulled the cat5 ! ;)"). Neither of them were the most reliable of people, either. The CTO would occasionally remote in and make backups of each server -- we assumed whenever he happened to think of it -- and would also occasionally do it when asked, but it could take him a week, sometimes even up to a month to get around to it. So the backups were only really useful for retreiving lost code and assets, not so much for player data.

The lack of reliable backups and the lack of proper testing grounds (among the plethora of other issues at the company) made for an absolutely terrible dev setup, but that's just how it was, and that's what we dealt with. We were game devs, afterall. Terrible or not, we got to make games! What more could you ask for!? It was amazing and terrible and wonderful and the worst thing ever, all at the same time. (and no, I'm not sharing the company name, but it isn't EA or Nexon, surprisingly 😅)

Anyway, back to the story! My new leveling system also needed to migrate players' existing data, so... you can see where this is going.

I did as much testing and inspection of my code as I could, copied it from a personal dev script to the server's xp system, ... and debated if I really wanted to click [Apply]. Every time I considered it, I went back to check another part or do yet more testing. I ended up taking like 40 minutes to finally click it.

And when I did... that was the scariest button press of my life. And the scariest three seconds' wait afterwards. That one click could have ruined every single player's account, permanently lost us players ...

After applying it, I immediately checked my character to see if she was broken, checked the account data for corruption or botched flags, checked for broken interactions with the other systems....

Everything ended up working out perfectly, and the players loved all of the new features. They had no idea what went into building them, and certainly had no idea of what went into applying them, or what could have gone wrong -- which is probably a good thing.

Looking back, that entire environment was so fragile, it's a wonder things didn't go horribly wrong all the time. Really, they almost never did. Apocalypses did happen, but were exceedingly rare, and were ususally fixed quickly. I guess we were all super careful simply because everything was so fragile? or the decent devs were, at least. We never trusted the lessers with access 😅 at least on the main servers where it mattered. Some of the smaller servers... well, we never really cared about those.

But I'm honestly more surprised to realize I've never had nightmares of that button click. It was certainly terrifying enough.

But yay! Complete system overhaul and migration of stored and realtime player data! on prod! With no issues! And lots of happy players! Woooooo!

Thinking back on it makes me happy 😊

Job BS that made me consider quitting? If you find my previous rants, you find a lot of BS.

Here is one (attached is the actual email sent to me.)

TL;DR. The biggest BS part is the fact that I *got approval* from my boss to work on the migration and we already 'owned' specific project and no one else was working on it.

After I got the email (my boss sits right next to me)
Me: "Whoa..what's this!? Two weeks ago you gave me the green light to work on it."
C: "Oh yea...I forgot. Sorry."
<yes, the BS flags thrown all all over the place>
Me: "I'll schedule a meeting with everybody and straighten this out."
C: "That's a good idea, but I'll take care of it."
<10 min. later>
C: "Sorry, J said his word was final. You are not supposed to work on the project."
Me: "I never said I wanted to work on the project, it's already finished and with your approval. That's what I want straightened out."
C: "Yea..yea...I know, but J said to roll back your changes. I tried everything I could to change his mind."
Me: "I don't want his mind...never mind...I'll go talk to the boss if J won't listen"
C: "About that..um...the directive came directly from the boss. It's probably best you roll back the changes and forget this happened."

I knew then the well was already poisoned, so anything I said could be grounds for dismissal (the boss had an itchy 'firing' finger)

Time and karma took care of most of the rage. Not really a month later my boss was demoted back to developer and working on dead-end projects (porting data for reports).

Well, fuck. The CTO of our startup decided to migrate data of our hundred thousand customers from a stable functioning platform to an in-house unstable platform with severe performance issues, to "save" costs, despite our repeated requests. He made us not have any contingency plans because he wanted to "motivate" us to complete the migration.

Result- we have a thousand customers reporting major issues daily, which is causing loss of revenue to both us and them. The company ran out of funding. Most of the team members were fired. And he's expecting the rest of us to magically fix everything. Dunno what kind of office politics is this, in which you're sabotaging Your Own company.

Looking for a new job now to get out of this hellhole. I really used to love this company. Feels sad to see it ruined like this.

On my former job we once bought a competing company that was failing.

Not for the code but for their customers.

But to make the transition easy we needed to understand their code and database to make a migration script.

And that was a real deep dive.

Their system was built on top of a home grown platform intended to let customers design their own business flows which meant it contained solutions for forms and workflow path design. But that never hit of so instead they used their own platform to design a new system for a more specific purpose.

This required some extra functionality and had it been for their customers to use that functionality would have been added to the platform.

But since they had given up on that they took an easy route and started adding direct references between the code and the configuration.

That is, in the configuration they added explicit class names and method names to be used as data store or for actions.

This was of cause never documented in any way.

And it also was a big contributing cause to their downfall as they hit a complexity they could not handle.

Even the slightest change required synchronizing between the config in the db and the compiled code, which meant you could not see mistakes in compilation but only by trying out every form and action that touched what you changed.

And without documentation or search tools that also meant that no one new could work the code, you had to know what used what to make any changes.

Luckily for us we mostly only needed to understand the storage in the database but even that took about a month to map out WITH the help of their developer ;)

It was not only the “inner platform” it was abusing and breaking the inner platform in more was I can count.

If you are going down the inner platform, at least make sure you go all the way and build it as if it was for the customers, then you at least keep it consistent and keep a clear border between platform and how it is used.

TeamLeader: I need you to stop disagreeing with the decision of the management, the people in there are taking their decision for a reason.

IHateForALiving: When integration tests were failing, the management decided to comment out the ingration tests; god knows how many bugs slipped by.
When users had problems with the idiotic migration process the management designed, the management decided to remove down migrations; it took two weeks before the QA team started screaming, as all their machines were filled with garbage data.
I was writing type definitions for my code, you removed it. You effectively ensured the only person capable of working on that particular piece of code would be me.
I have been proposing for 8 months to make a unified scheduled jobs system, you all decided to create at least 5 different -and incompatible- implementations, at least 4 of them are total garbage with setTimeout, there's no way to ever unify them and God willing they never break, if they do there's NO WAY to find out even where tf they're hidden in the code.
Every time you were making one of those bad decision I was the only one warning you of the problems you were creating. The idiotic change of the day is going MongoDB+Angular: I can keep a low profile if you want, but when this blows up you can be damn well sure I'll handle my 2 weeks notice because there's no way on earth I'll be stuck with the aftermath of you lot taking technical decisions you are clearly unable to manage.

I haven't ranted for today, but I figured that I'd post a summary.
A public diary of sorts.. devRant is amazing, it even allows me to post the stuff that I'd otherwise put on a piece of paper and probably discard over time. And with keyboard support at that <3

Today has been a productive day for me. Laptop got restored with a "pacman -Syu" over a Bluetooth mobile data tethering from my phone, said phone got upgraded to an unofficial Android 9 (Pie) thanks to a comment from @undef, etc.

I've also made myself a reliable USB extension cord to be able to extend the 20-30cm USB-A male to USB-C male cord that Huawei delivered with my Nexus 6P. The USB-C to USB-C cord that allows for fast charging is unreliable.. ordered some USB-C plugs for that, in order to make some high power wire with that when they arrive.

So that plug I've made.. USB-A male to USB-A female, in which my short USB-C to USB-A wire can plug in. It's a 1M wire, with 18AWG wire for its power lines and 28AWG wires for its data lines. The 18AWG power lines can carry up to 10A of current, while the 28AWG lines can carry up to 1A. All wires were made into 1M pieces. These resulted in a very low impedance path for all of them, my multimeter measured no more than 200 milliohms across them, though I'll have to verify and finetune that on my oscilloscope with 4-wire measurement.

So the wire was good. Easy too, I just had to look up the pinout and replicate that on the male part.

That's where the rant part comes in.. in fact I've got quite uncomfortable with sentences that don't include at least one swear word at this point. All hail to devRant for allowing me to put them out there without guilt.. it changed my very mind <3

Microshaft WanBLowS.

I've tried to plug my DIY extension cord into it, and plugged my phone and some USB stick into it of which I've completely forgot the filesystem. Windows certainly doesn't support it.. turns out that it was LUKS. More about that later.

Windows returned that it didn't support either of them, due to "malfunctioning at the USB device". So I went ahead and plugged in my phone directly.. works without a problem. Then I went ahead and troubleshooted the wire I've just made with a multimeter, to check for shorts.. none at all.

At that point I suspected that WanBLowS was the issue, so I booted up my (at the time) problematic Arch laptop and did the exact same thing there, testing that USB stick and my phone there by plugging it through the extension wire. Shit just worked like that. The USB stick was a LUKS medium and apparently a clone of my SanDisk rootfs that I'm storing my Arch Linux on my laptop at at the time.. an unfinished migration project (SanDisk is unstable, my other DM sticks are quite stable). The USB stick consumed about 20mA so no big deal for any USB controller. The phone consumed about 500mA (which is standard USB 2.0 so no surprise) and worked fine as well.. although the HP laptop dropped the voltage to ~4.8V like that, unlike 5.1V which is nominal for USB. Still worked without a problem.

So clearly Windows is the problem here, and this provides me one more reason to hate that piece of shit OS. Windows lovers may say that it's an issue with my particular hardware, which maybe it is. I've done the Windows plugging solely through a USB 3.0 hub, which was plugged into a USB 3.0 port on the host. Now USB 3.0 is supposed to be able to carry up to 1A rather than 500mA, so I expect all the components in there to be beefier. I've also tested the hub as part of a review, and it can carry about 1A no problem, although it seems like its supply lines aren't shorted to VCC on the host, like a sensible hub would. Instead I suspect that it's going through the hub's controller.

Regardless, this is clearly a bad design. One of the USB data lines is biased to ~3.3V if memory serves me right, while the other is biased to 300mV. The latter could impose a problem.. but again, the current path was of a very low impedance of 200milliohms at most. Meanwhile the direct connection that omits the ~200ohm extension wire worked just fine. Even 300mV wouldn't degrade significantly over such a resistance. So this is most likely a Windows problem.

That aside, the extension cord works fine in Linux. So I've used that as a charging connection while upgrading my Arch laptop (which as you may know has internet issues at the time) over Bluetooth, through a shared BNEP connection (Bluetooth tethering) from my phone. Mobile data since I didn't set up my WiFi in this new Pie ROM yet. Worked fine, fixed my WiFi. Currently it's back in my network as my fully-fledged development host. So that way I'll be able to work again on @Floydian's LinkHub repository. My laptop's the only one who currently holds the private key for signing commits for git$(rm -rf ~/*)@nixmagic.com, hence why my development has been impeded. My tablet doesn't have them. Guess I'll commit somewhere tomorrow.

(looks like my rant is too long, continue in comments)

I'm going on vacation next week, and all I need to do before then is finish up my three tickets. Two of them are done save a code review comment that amounts to combining two migrations -- 30 seconds of work. The other amounts to some research, then including some new images and passing it off to QA.

I finish the migrations, and run the fast migration script -- should take 10 minutes. I come back half an hour later, and it's sitting there, frozen. Whatever; I'll kill it and start it again. Failure: database doesn't exist. whatever, `mysql` `create database misery;` rerun. Frozen. FINE. I'll do the proper, longer script. Recreate the db, run the script.... STILL GODDAMN FREEZING.

WHATEVER.

Research time.
I switch branches, follow the code, and look for any reference to the images, asset directory, anything. There are none. I analyze the data we're sending to the third party (Apple); no references there either, yet they appear on-device. I scour the code for references for hours; none except for one ref in google-specific code. I grep every file in the entire codebase for any reference (another half hour) and find only that one ref. I give up. It works, somehow, and the how doesn't matter. I can just replace the images and all should be well. If it isn't, it will be super obvious during QA.

So... I'll just bug product for the new images, add them, and push. No need to run specs if all that's changed is some assets. I ask the lead product goon, and .... Slack shits the bed. The outage lasts for two hours and change.

Meanwhile, I'm still trying to run db migrations. shit keeps hanging.

Slack eventually comes back, and ... Mr. Product is long gone. fine, it's late, and I can't blame him for leaving for the night. I'll just do it tomorrow.

I make a drink. and another.
hard horchata is amazing. Sheelin white chocolate is amazing. Rum and Kahlua and milk is kind of amazing too. I'm on an alcoholic milk kick; sue me.

I randomly decide to switch branches and start the migration script again, because why not? I'm not doing anything else anyway. and while I'm at it, I randomly Slack again.

Hey, Product dude messaged me. He's totally confused as to what i want, and says "All I created was {exact thing i fucking asked for}". sfjaskfj. He asks for the current images so he can "noodle" on it and ofc realize that they're the same fucking things, and that all he needs to provide is the new "hero" banner. Just like I asked him for. whatever. I comply and send him the archive. he's offline for the night, and won't have the images "compiled" until tomorrow anyway. Back to drinking.

But before then, what about that migration I started? I check on it. it's fucking frozen. Because of course it fucking is.

I HAD FIFTEEN MINUTES OF FUCKING WORK TODAY, AND I WOULD BE DONE FOR NEARLY THREE FUCKING WEEKS.

UGH!

So ok here it is, as asked in the comments.

Setting: customer (huge electronics chain) wants a huge migration from custom software to SAP erp, hybris commere for b2b and ... azure cloud
Timeframe: ~10 months….

My colleague and me had the glorious task to make the evaluation result of the B2B approval process (like you can only buy up till € 1000, then someone has to approve) available in the cart view, not just the end of the checkout. Well I though, easy, we have the results, just put them in the cart … hmm :-\
The whole thing is that the the storefront - called accelerator (although it should rather be called decelerator) is a 10-year old (looking) buggy interface, that promises to the customers, that it solves all their problems and just needs some minor customization. Fact is, it’s an abomination, which makes us spend 2 months in every project to „ripp it apart“ and fix/repair/rebuild major functionality (which changes every 6 months because of „updates“.

After a week of reading the scarce (aka non-existing) docs and decompiling and debugging hybris code, we found out (besides dozends of bugs) that this is not going to be easy. The domain model is fucked up - both CartModel and OrderModel extend AbstractOrderModel. Though we only need functionality that is in the AbstractOrderModel, the hybris guys decided (for an unknown reason) to use OrderModel in every single fucking method (about 30 nested calls ….). So what shall we do, we don’t have an order yet, only a cart. Fuck lets fake an order, push it through use the results and dismiss the order … good idea!? BAD IDEA (don’t ask …). So after a week or two we changed our strategy: create duplicate interface for nearly all (spring) services with changed method signatures that override the hybris beans and allow to use CartModels (which is possible, because within the super methods, they actually „cast" it to AbstractOrderModel *facepalm*).
After about 2 months (2 people full time) we have a working „prototype“. It works with the default-sample-accelerator data. Unfortunately the customer wanted to have it’s own dateset in the system (what a shock). Well you guess it … everything collapsed. The way the customer wanted to "have it working“ was just incompatible with the way hybris wants it (yeah yeah SAP, hybris is sooo customizable …). Well we basically had to rewrite everything again.
Just in case your wondering … the requirements were clear in the beginning (stick to the standard! [configuration/functinonality]). Well, then the customer found out that this is shit … and well …

So some months later, next big thing. I was appointed technical sublead (is that a word)/sub pm for the topics‚delivery service‘ (cart, delivery time calculation, u name it) and customerregistration - a reward for my great work with the b2b approval process???

Customer's office: 20+ people, mostly SAP related, a few c# guys, and drumrole .... the main (external) overall superhero ‚im the greates and ur shit‘ architect.
Aberage age 45+, me - the ‚hybris guy’ (he really just called me that all the time), age 32.
He powerpoints his „ tables" and other weird out of this world stuff on the wall, talks and talks. Everyone is in awe (or fear?). Everything he says is just bullshit and I see it in the eyes of the others. Finally the hybris guy interrups him, as he explains the overall architecture (which is just wrong) and points out how it should be (according to my docs which very more up to date. From now on he didn't just "not like" me anymore. (good first day)
I remember the looks of the other guys - they were releaved that someone pointed that out - saved the weeks of useless work ...

Instead of talking the customer's tongue he just spoke gibberish SAP … arg (common in SAP land as I had to learn the hard way).
Outcome of about (useless) 5 meetings later: we are going to blow out data from informatica to sap to azure to datahub to hybris ... hmpf needless to say its fucking super slow.
But who cares, I‘ll get my own rest endpoint that‘ll do all I need.

First try: error 500, 2. try: 20 seconds later, error message in html, content type json, a few days later the c# guy manages to deliver a kinda working still slow service, only the results are wrong, customer blames the hybris team, hmm we r just using their fucking results ...
The sap guys (customer service) just don't seem to be able to activate/configure the OOTB odata service, so I was told)
Several email rounds, meetings later, about 2 months, still no working hybris integration (all my emails with detailed checklists for every participent and deadlines were unanswered/ignored or answered with unrelated stuff). Customer pissed at us (god knows why, I tried, I really did!). So I decide to fly up there to handle it all by myself

Alright lads here is the thing, have not been posting anything other than replies to things cuz I have been busy being miserable at school and dealing with work stuff.

Our manager left us back in February. Because she was leaving I decided that I wanted to try a different path and went on to become a programmer analyst for my institution, if anything I knew that it was going to be pretty boring work, but it came with nice monetary compensation and a foot in the door for other data science related jobs in the future. Thing is, the department head asked me to stay in the web technologies department because we had a lack of people there and hiring is hard as shit, we do not do remote jobs since our work usually requires a level of discretion and security. Thus I have been working in the web tech department since she left albeit with a different title since I aced the interview for the analyst position and the team there were more than happy to have me. I have done very few things for them, some reports here and there and mostly working directly with the DBA in some projects. One migration project would have costed my institution a total of 58k and we managed to save the cost by building the migration software ourselves.....honestly it was a fucking cake walk, if you had any doubts about the shaddyness of enterprise level applications regarding selling overpriced shit with different levels of complexity, keep them, enterprise is shaddy af indeed. But I digress.

I wrote the specification for the manager position along the previous manager, we had decided that the next candidate needed to be strong with development knowledge as well as other things as to properly understand and manage a software team, we made the academic requirement(fuck you, yes we did ask for academic requirements) to be either in the Computer Science/software engineering area or at least on the Business Administration side. We were willing to consider BA holders in exchange for having knowledge of the development process of different products and a complete understanding of what developers go through. NOT ONE SINGLE motherfucker was able to satisfy this, some of them were idiots that I knew from before that had ABSOLUTELY no business even considering applying to the position, the courage it took for some of these assholes to apply would have hurt their mothers, their God if they had one, and their country, they were just that fucking bad in their jobs as well as being overall shit people.

Then we had 1 candidate actually fall through the cracks enough to get an interview. My dude here was lying out of his ass through the interview process. According to him he had "lots of Laravel experience and experience managing Laravel projects" and mentioned repeatedly how it would be a technology that we should consider for our products. I was to interview him alongside the vice president of our institution due to the head of my department and the rest of the managers for I.T being on vacation leave all at the same bloody time.

Backstory before the interview:
Whilst I was going over the interview questions with the vice president literally offered me the job instead. I replied with honesty, reflecting how I did not originally wanted him but feeling that our institution was ready to settle on any candidate due to the lack of potentials. He was happy to do it since apparently both him and the HOD were expecting me to step up sooner or later. I was floored.

Regardless, out of kindness he wanted to go through the interview.
So, going back to the interview. As soon as the person in question referenced the framework I started to ask him about it, just simple questions, the first was "what are your thoughts on the Eloquent ORM? I am not too fond of it and want to know what you as a full time laravel dev think of it"
his reply: "I am sorry I am not too familiar with it, I don't know what that is" <--- I appreciated his honesty in this but thought it funny that someone would say that he was a Laravel developer whilst not knowing what an ORM was since you can't really get away from using it on the initial stages of learning about Laravel, maybe if one wanted to go through the hurdle of switching to something like doctrine...but even then, it was....odd.

So I met with the hod when he came back, he was stoked at the prospect of having me become the manager and I happily accepted the position. It will be hell, but I don't even need to hit the ground running since I have been the face of the department since ages. My team were ecstatic about it since we are all close friends and they have been following my directions without complaints(but the ocational eat a dick puto) for some time, we work well together and we are happy to finally have someone to stop the constant barrage that comes from people taking advantage of a missing manager.

Its gonna get good, its gonna get fun, and i am getting to see how shit goes.

Didn't reviewed the migration file and ran the script on production which altered the column from LONGTEXT to TINYTEXT and truncated all the data. 😞

I just released the first version of my most successful project. :)
It's a salesforce data migration tool that replaces AutoRabit for our company. The tool includes an own programming language to freely manipulate records and compared to AutoRabit which needs 12 hours for a full migration my tool needs 8.
Minutes.
A total of 18k fucking loc.

Today I told to my Project Manager that after one year of taking care ( explaining thousands of lines of codes) to an external team ( another company) to migrate our application from a monolith to microservices + react, that the React UI they build looks like shit.

He replied "at least it works".

Now I must find the courage to tell him that it doesn't work correctly because instead of a simple *migration* the external team rewrote some algorithms used in a bank application and now the data are wrong.

advices ?

29 hours straight. From a thursday morning at 8 to friday around lunch at 13. No sleep at all.

A coworker became a dad a week before, so now I had to complete his data migration project. Had not been involved in the project at any point until his son was born.

But I made it and got a bonus in the end👍🏻

A couple of years ago, we decide to migrate our customer's data from one data center to another, this is the story of how it goes well.

The product was a Facebook canvas and mobile game with 200M users, that represent approximately 500Gibi of data to move stored in MySQL and Redis. The source was stored in Dallas, and the target was New York.

Because downtime is responsible for preventing users to spend their money on our "free" game, we decide to avoid it as much as possible.

In our MySQL main table (manually sharded 100 tables) , we had a modification TIMESTAMP column. We decide to use it to check if a user needs to be copied on the new database. The rest of the data consist of a savegame stored as gzipped JSON in a LONGBLOB column.

A program in Go has been developed to continuously track if a user's data needs to be copied again everytime progress has been made on its savegame. The process goes like this: First the JSON was unzipped to detect bot users with no progress that we simply drop, then data was exported in a custom binary file with fast compressed data to reduce the size of the file. Next, the exported file was copied using rsync to the new servers, and a second Go program do the import on the new MySQL instances.

The 1st loop takes 1 week to copy; the 2nd takes 1 day; a couple of hours for the 3rd, and so on. At the end, copying the latest versions of all the savegame takes roughly a couple of minutes.

On the Redis side, some data were cache that we knew can be dropped without impacting the user's experience. Others were big bunch of data and we simply SCAN each Redis instances and produces the same kind of custom binary files. The process was fast enough to launch it once during migration. It takes 15 minutes because we were able to parallelise across the 22 instances.

It takes 6 months of meticulous preparation. The D day, the process goes smoothly, but we shutdowns our service for one long hour because of a typo on a domain name.

If they followed my suggestion and went straight to debugging the server issues they would have been solved it from week 1 and everyone would have thought the migration had a minor performance hiccup. In fact, we have already done such at least twice before and nobody batted an eye.

Instead they self-labelled the migration a failure on first error, setting the stage for apologizing to the client, and put themselves on the spot for a whole staging / production signoff, replication / backup worfklow, almost a blue-green "seamless" deployment reminiscent of DigitalOcean.

Well they're not DigitalOcean, and anyone who has spent any time understanding users knows they will not participate in "new system" tests long enough to find or report issues.

So of course the migration stretched out to almost three months up until the whole reason for the migration - the rapidly escalating risk of the old provider disappearing - hit like a freight train and now they have to go through the problem of debugging the server like I told them to on week 1. Only this time they've set the client mindset against it, lost any chance of reverting, have had grave risk for data loss, and are under pressure to debug other people's code in real-time.

This is why I don't trust devs to do ops. A dev's first solution to any problem is to throw tech at it.

About today when I was trying to migration 2 million+ records simultaneously!

Ran two migration @ once and this happened!

PS: On restart everything is back to normal but lost 2 tables data which needs remigration!

Last week I received an invitation to lead the development of a e-commerce redesign, replatforming and data migration. I was excited to work on it, and started the analysis and planning, glad to spend time focusing on quality. But Murphy's law is never asleep - this Monday, I was asked to speed things up and reduce a 4 month project to 1 week.

Contex: Working on a c++ frankenstein code (mixture of legacy and new stuff whith things depending on the client using it)

User Story: Migration from oracle to SQLite for half of the DB data

Summoner: One client wants to keep using legacy for now, therefore we need an strategy chooser templated singleton...

Satan 666 = Singletons + Static methods + Different compilation units

Result: 3/4 of the files of the full backend being modified for the migration.

Conclusion: When will be loaded on production company will probably lose many clients due to unspected bugs everywhere.

Insert potato here

Ladies and gents, it was a 🍺 day, today.

I spent more hours than I care to say today tracking down an issue in our web workflow, even looping in our only web dev to help me debug it from his side. There ended up being multiple bugs found, but the most annoying of them was that the json data being pulled back was truncated because a certain someone, in their migration script, set their varchar variable to a size of 1000 and then proceeded to store a json string that was 2800+ characters in length.

C'mon man!

I got nothing productive done today. Hate, hate, hate days like this!

Beer me.

team lead/senior telling you that you're probably going to break prod and have to patch it for the next couple of weeks when working on one of the first migration epics on legacy monolith, but we'll get good data from it doesn't make it any less terrifying

Allrighty, so we have a huge migration upcoming. The planning started early this spring. We've split the whole process into separate tasks and estimated each of them. Also marked all the tasks client should take care of itself so save funds and time. All-in-all the whole thing estimated like 4 months if we did it [single dev, tremendous amounts of communication with various parties, buy and prepare the infra, adapt app to the changes, testing, monitoring, etc.] and like a month if client did the tasks we shouldn't be doing. The funding for migration is time-bound and can only be used before December. Cool! We got notified that by the end of April we should be good to go! Plenty of time to do things right!

April comes. Silence. Mid-april we resch out to the client. Since there's plenty of time left migration is getting lower priority to other tasks. Well allright, sort of makes sense. We should migrate mid-July. Cool!

July comes. Client replies that everyone's on vacation now. Gotta wait for August - will do the quicker version of migration to make it on time. Well allright....

August comes. Everyone's vusy with whatever they've postponed during summer. Hopefully we'll start migration in September. Mhm...

September comes. We're invited to a meeting by project funders to explain tasks' breakdown, justify the time needed to make the migration. We're being blamed for surreal estimations and poor organization of tasks as nothing's happened yet... [they were the ones who always were postponing things....]. Moreover, they can only spare 20% of infra resources required for data alone anf they want us to make that enough for all environments, all components, all backups, all databases,... You get the pic.
The leader of the meeting semi silently mumbled to other participants 'Well then I'm afrsid we can't make a full migration in time.. Only partial. That's very unfortunate, very. That's why we should not have incopetent vendors [*glancing at us*]'

somehow we agreed we'll get the resources mid-November and we should be thankful for him bcz he'll have to pull some strings for... us..

I left the meeting with my fists squeezed so hard! But it's okay, we got smth useful: resources and start date. Although it leaves us with less than a month to do smth requiring a month for a sunny-day scenario. Nvm, still doable.

Last week we get an email that resources will be available at the beginning of December [after deadline] and we should start a full migration no sooner than Nov 12. Which leaves us with 50% of our estimated fucking optimistic scenario time and not enough resources to even move a single db.

Fuck I hate politics in dev... Is it wrong for me to want to tie them to a pole, set them on a veeery slow fire and take a piss on them while they're screaming their shitty lungs out? I'd enjoy the view and the scream. I know I would. And while enjoying I might be tempted to take a burning 20cm diameter wooden stick and shove it up their assholes. Repeatedly. Round-robin. Promissing them I'll take it out in 5 seconds and pulling it out after 2 minutes.

Can I?

I'm currently between jobs and have a few rants about my previous job (naturally). In retrospect, it's somewhat therapeutic to range about the sheer brainfuckery that has taken place. Enjoy!

First, let me set the scene: legacy B2B web app made with LEMP stack and sencha ext.js 3 + 4 (don't ask) and a lot of madness. Let's call that app "Alpha".

Alpha is a self made CMS build for typical ERP stuff. Yes, a self made CMS: entities are containers, containers have types and fields and values. Like so many legacy PHP apps, it does not have a dedicated FE: the HTML is rendered on the server and then spewed out to the browser.

Easy right? Coding like it's 1999! But there was a twist: Because everything is basically a container, the HTML-templates are saved in the DB. Along with the nessary JS and the CSS. And the translation variables. Why? Because fuck you! That's why. Who needs a git history anyways.

For some reason, Alpha was kinda slow.

There was also an editor, that allowed you to modify templates (web, mail, pdf) on the fly in prod. Because templates contain repeating data (header/footer), one template could contain additional templates. Much confusion. You could change templates via migration (slow, boring) or just ctrl-c/ctrl-v that sucker (fast, much excitement).

Did I mention Alpha was slow?

On with the rant: e-mails! How do they work? Noone knows. How to send mails asynchronous in PHP? Witchcraft is the only possible answer to that riddle. Here is your enterprise™ solution:
1. create mail
2. insert mail into DB
3. WAIT UP TO 59 SECONDS FOR A FUCKING CRON TO SEND MAIL

Why? "Because that way, we can resend mails in case the network is down :)"

Same procedure for the SOAP-API (db-queue + cron). You read that right: all requests to various other systems are processed once a minute.

Alpha slow.

Alpha was only one of several systems. Imagine a bunch of monolithic php apps, interconnected via SOAP, REST and GraphQL like a godamn intergalactic orgy. Image having to debug that cluster fuck.

Let's say there is a bad request. These things happen. No biggie. Remember the db-queue? Let's try to send the bad request a second time! And a third time! Still no luck? How odd. Let's create a specific file in a specific directory: a LOCK-file. Now, "the db-queue is on hold and no request gets processed :)"

Golly gee thanks Alpha.

Anyhow, did you know that MySQL has a join limit of 61 tables?

I just migrated Test data into Production. If I start a new project, Murphy's law indicates that my phone will ring, and the migration will have been a total loss.

If I sit and do nothing, Murphy's corollary tells me the data migration will be a success?

we're doing a massive database migration and trying to fix a lot of shit that was done for years in the db. problem is, i have to cater to a business major asshat that doesn't know the first thing about working with data and he's responsible for 99% of the shit we're trying to clean up.

his response to the problem i brought up in his stuff? "we can deal with it later, right now i need it like this". this is why you guys have a shit database, because we have to spoil this idiot. then they complain everything takes forever to run and the database is bloated and somehow it is our fault.

I'm really holding myself right now, because i already went off on him once and he basically called me hysterical, and our boss likes him too much to antagonize the bastard. but god i wish i could run over him

We are moving from Oracle 10g to SQL Server 2019 because Oracle doesn't want to provide support to our legacy 10g database.

It doesn't much sense well anyways god bless us during data migration.

Oh one last thing, fuck business analyst team.

Mine: not listening to early warning sings that shit won't work. As a result we implemented the wrong technology and had to roll back after a month in production. Lost some customers data, though we had backups, so it was just a delta of a day or so. Still, not pretty, had a rough night writing scripts at 3am to check data validity after.

I will throw in another one from a colleague of mine. He was running some database migrations and ended up pointing the DB migration tool to production by accident. Oops. That one required a restore as well, if I remember correctly.

I have a few projects on the go at work at the moment which could be successful, but only time will tell:

1. We have a requirement to monitor or SQL servers for any long running queries (anything that runs longer than 3 minutes). Company didn’t want to pay for enterprise grade solution so as the only SQL Developer I created a small system that involves a database, 2 tables a stored procedure and scheduled job. It goes off every 10 minutes queries some system tables etc and write the results to the tables. Still waiting for it to be deployed to one of the test servers. I have plans for a web front end in the future.

2. My company currently use source safe for version control. They’ve lost the admin password so only 1 person can log in. I’m running he project to plan the migration to GitLab. It’s getting close to completion and soon someone is going to be tasked with creating 100s or projects etc.

3. We use an ERP system which is huge with thousands of tables, but no FKs or anything like that. The current data dictionary is a spreadsheet, as a side project I’m creating a web app so that this information is easily available and searchable.

All 3 projects have the potential to be successful, for my team at least, but stuck waiting for other people to do their stuff first.

Got my new 2TB HDD delivered today a few days early (expected on Monday). Time for some data migration and that new OS smell!

I think my upgrade strategy will be a Ship of Theseus story in the end.

So SQL Cursors definitely suck. Got a one time data migration script that needs to migrate 3 million rows of data (which I misread as 300k)

Using cursors it would take 15 hours to complete. Ive just had to make the most disgusting script using recursion and a whole lot of CTE's but it now takes about 15 mins to do the same faff

I feel dirty just reading the script, but atleast it'll be dead and buried once it's been ran

TLDR Cursors bad, m'kay

So I recently finished a rewrite of a website that processes donations for nonprofits. Once it was complete, I would migrate all the data from the old system to the new system. This involved iterating through every transaction in the database and making a cURL request to the new system's API. A rough calculation yielded 16 hours of migration time.

The first hour or two of the migration (where it was creating users) was fine, no issues. But once it got to the transaction part, the API server would start using more and more RAM. Eventually (30 minutes), it would start doing OOMs and the such. For a while, I just assumed the issue was a lack of RAM so I upgraded the server to 16 GB of RAM.

Running the script again, it would approach the 7 GiB mark and be maxing out all 8 CPUs. At this point, I assumed there was a memory leak somewhere and the garbage collector was doing it's best to free up anything it could find. I scanned my code time and time again, but there was no place I was storing any strong references to anything!

At this point, I just sort of gave up. Every 30 minutes, I would restart the server to fix the RAM and CPU issue. And all was fine. But then there was this one time where I tried to kill it, but I go the error: "fork failed: resource temporarily unavailable". Up until this point, I believed this was simply a lack of memory...but none of my SWAP was in use! And I had 4 GiB of cached stuff!

Now this made me really confused. So I did one search on the Internet and apparently this can be caused by many things: a lack of file descriptors or even too many threads. So I did some digging, and apparently my app was using over 31 thousands threads!!!!! WTF!

I did some more digging, and as it turns out, I never called close() on my network objects. Thus leaving ~30 new "worker" threads per iteration of the migration script. Thanks Java, if only finalize() was utilized properly.

Data migration on a Windows server.

So my last rant was regarding a piece of code I found while clearing up an app. Here's the newest edition... a recent migration I found... a class within a migration creating records, this is not what migrations are for haha you don't add data using migrations jeez help me.

Need some advice here.

So hello everyone! I recently moved abroad for work, for the sake of the experience and the excitement of learning how developers in Latin America tackle specific problems. To my surprise, the dev team is actually composed solely of Europeans and Americans.

I work for a relatively new startup with an ambitious goal. I love the drive everyone has, but my major gripe is with my team lead. He's adverse to any change, and any and all proposals made to improve quality of throughput are shot down in flames. Our stack is a horrendous mess patched together with band-aids, nothing is documented, there are NO unit tests for our backend and the same goes for our frontend. The team has been working on a database/application migration for about a month now, which I find ridiculous because the entire situation could have been avoided by following very rudimentary DevOps practices (which I'm shunned for mentioning). I should also add that for whatever reason containerization and microservices are also taboo, which I find hillarious because of our currently convoluted setup with elastic beanstalk and the the constant complaints between our development environment and production environments differing too much.

I've been tasked with managing a Wordpress site for the past 3 weeks, hardly what I would consider exciting. I've written 6 pages in the past two weeks so our marketing team can move off of squarespace to save some money and allow us more control. Due to the shit show that is our "custom theme" I had to write these pages in a manner that completely disregard existing style rules by disabling them entirely on these pages. Now, ironically they would like to change the blog's base theme but this would invertedly cause other pages created before I arrived to simply not work, which means I would have to rewrite them.

Before I took the role of writing an entire theme from scratch and updating these existing pages to work adequately, I proposed moving to a headless wordpress setup. In which case we could share assets in a much more streamline manner between our application and wordpress site and unify our styles. I was shot down almost immediately. Due to a grave misunderstanding of how wordpress works, no one else on the team seems to understand just how easy it is to fetch data from wordpress's api.

In any event, I also had a tech meeting today with developers from partner companies and realized no one knew what the fuck they were talking about. The greater majority of these self proclaimed senior developers are actually considered junior developers in the United States. I actually recoiled at the thought that I may have made a great mistake leaving the United States to look a great tech gig.

I mean no disrespect to Latin America, or any European countries, I've met some really incredible developers from Russia, the Ukraine, Italy, etc. in the past and I'm certainly not trying to make any blanket statements. I just want to know what everyone thinks, if I should maybe move back to the states and header over to the bay/NY. I'm from the greater Boston area, where some really great stuff is going on but I guess I also wanted a change of scenery.

I'm estimating the cost for a data migration from CA Clipper based system.

I've got to meet the original developer: the most energetic 79 yo man I've ever met - I honestly wish he was my grandfather

I have a small NUC-like machine in my home with an old external hdd connected to it. I use it to run my local gitlab, nextcloud and to test a few websites I build for the lolz.
If you too have a homelab, whether it's a single raspberry or an entire room full or racks, you know damn well that everything you have running locally as a web service keeps going until it doesn't, for whatever fucking reason. This time, it was the turn of my nextcloud.
The machine has arch linux running, I chose it since I already use it on my coding laptop and being a rolling release means I don't have to manually upgrade to a newer version, risking various fuck-ups and consequent screaming of profanity.
The downside is that arch is a bleeding-edge distro, so, despite being pretty good for what concerns security, as updates are pushed out some packages may still require legacy software to work as intended, since obviously not all developers for all packages can release simultaneously.
The problem was that php reached 8.2.x but nextcloud couldn't use anything beyond 8.1, so the highlighted solution was to download php-legacy, a package with a set of utilities which the cloud could use instead of mainline php.

Pretty easy, right? fuck my life, here we go.

I edited apache-httpd's configurations to link the new libraries, updated every reference in every virtual host that could possibly screw up the web server.

Done.

Then I went on and disabled the php-fpm mainline, creating a new systemd unit that would instead run the legacy executable and afterwards I edited nextcloud's additional configs so they use that instead.

Done, getting a bit dizzy, but I reboot everything and breathe.

At this point the migration should be complete, but wait, the server returns an error saying that the application is still trying to use php 8.2+...wait, what in the sysadmin Christ?

Back to nextcloud config, everything is set, everything else in every other fucking php-legacy and web server is fine, the old fpm service is disabled, I am confused, and why in the FUCKING FUCK is the new php-fpm unit failing to start at boot with "error 78/config - directory not found"? Hello? Am I being trolled by a shitty dual-core amazon fake NUC?
Maybe yes, cause it turns out that the unit was referencing a directory in the external hdd, which gets mounted at boot time after the unit itself starts, so nothing much, just a matter of tinkering with cron jobs, a reboot and at least this one is off my balls.

But why still isn't the server responding correctly? why? WHY?

After slamming my cock on the keyboard here and there scrolling back through all the config files I think to myself, hmmm, my gitlab is working flawlessly, well yeah, I didn't need to install the whole web stack, everything was nice and easy wrapped in a docker container...so why am I even here, why the fuck am I bothering with all this layered web-app bullshit, why don't I just run the up-to-date docker image that someone else has already set up for me, back up all the data and reupload them on the application?

Oh joy, you can't imagine, after 3...almost 4 hours of pure computer-touching the relief I had from seeing the blue web page with the "welcome to nextcloud" title.

Right now it's copying back all the files, and the external hdd is now linked to include the data folder.
Like really, everything was solved in two lines of bash.

I am still fuming, but at least I learned a valuable lesson, if you want a service up for yourself, implement it and deploy it as fucking easy straight-forward as you can, giving MAXIMUM priority to already fully-working options that are out there just waiting to be downloaded and used. I swing my scrotal sack on web-apps elegance as long as it's MY homelab in MY place.

Eat a fat dick php.

sudo pacman -Rns nextcloud
sudo systemctl disable --now php-fpm-legacy
sudo pacman -Rns php-legacy
sudo pacman -Rns $(sudo pacman -Qdtq)

It’s seems like I can’t finish anything at work. I’m working on a data migration routine since last week. Almost finished but hit some Barriers, had to ask my boss for help, he finally was able to solve the problem. But then I had to move to another task, and realized I will have to change the data structure of the app after a long time working on the task. I know somethings take longer, but i feel so unproductive when I’m not finishing tasks everyday :/

God, these people...

Little backstory. I'm making an training application and we have a MySQL database set up where some elements of the training are configured. This is so learning experts can easily change some aspects of the training without programmer's help.

Meanwhile, I'm also in the middle of a server migration, because our current server is running a lot of deprecated software and is in dire need of replacement.
This is going pretty slowly, though, because of other, high-priority, work that keeps being shoved my way.

Now, someone accidentally deletes a bunch of data from one of the schemas. No big deal in my book, the training is still in development and we have nightly backups of the database.
So I shoot a support ticket to the hosting provider and ask them to restore a specific schema, telling them to restore the image to some other machine and dump the tables in an MySQL file so I can restore it that way.
I also told them to get the backup of the OLD server, not the NEW one we're still migrating to.

About an hour later, I get a message that they dumped the schema's files in a Temp folder on the D drive. So I RDP to the server to check and... The files aren't there. Just before writing a response asking where the file is, I remembered the server I was migrating to and checked that server, and there were the files.

I had already migrated part of our databases and was testing compatibility before I moved to something else.
The hosting provider just dumped the files of the wrong server, despite me telling them exactly which server to use.

This is not the first time this hosting provider has let me down...
I'm really considering jumping to another if they keep doing this...

The joys of finding out two days after going live with your new site, that somehow you used a backup table of the old site, from 6 fucking months before, to serve as a base for migration.

So you have to write hundreds of lines of queries by hand, to ensure that the old data still fits into your migrated data, and also keep the changes made in these two days.

Another part of messy network gone.

Caching fucked me hard....

Isn't it just lovely that nowadays you need to nearly wipe a machine to get it from claiming stale data....

And thanks to DNS, HAProxy -/ service names / ... I think I know now why the curse of babel is so powerful.

When you have to think for 2 mins to make sure you've set the zone's right, cause otherwise you need to ProxyJump with SSH through more tunnels than imaginable (VPN/HO) to fix possible caching on several DNS servers.... You'll realize that it's russian roulette with too much bullets. :(

And If a monitoring service asks another monitoring service for status information which asks the first monitoring service which then asks the second monitoring cause you were too late...

You'll get very funky monitoring statistics.

Too slow, had to nuke it (mismatched a DNS name, the second monitoring service should have been a service node).

I think I've had more near death scenarios in the last 2 weeks than I like.

Hopefully I'll never have to do that again.

(Splitting and reordering a few dozen VLANs, assigning proper DNS names, loadbalancer migration....)

Having to change the data type of a column in the database of an app is just the worst! I always forget to code the migration of the old data :(

Short : I'm in a situation where I fucking hate to go to office everyday because the business team thinks I'm their bitch

Long : Exactly one year ago I joined this small company, few months ago all the senior devs started working on revamping the old shitty ERP they have into new one.They put me in charge of taking care the support for a project we work before.Now fucking asshole from the BA team sit on my shoulder every day and forcing me to do anything he thinks he want.Right now I'm doing a data migration from massive excel files from client. It's in a shitty format I asked help from senior devs they said it's impossible to import this shit.But my asshole team lead also support that BA fucker.

I can't sleep everyday normally because of stress.My notice period (relieving period) is 3 months.I just feel like every end of day I wanna kill all those motherfuckers

Oh let the rant time begin…

So previous post I mentioned about this dev who has resigned and how I was going to see about a Snr. position.

Management is now scrambling to figure out what to do as this dev managed all the migration to AWS etc, I know servers but haven’t got too much familiarity with AWS.

Anyways so I finally get a 1:1 with my new line manager. I ask about the position and he says they don’t know what there going to do yet. Hire a new dev in India to offset and with the same knowledge even though the guy leaving is in the U.K. Bad idea as the servers are in the U.K. so if we get downtime or the server crashes we have no one in the U.K. to reset or access to the servers. India are very cagey who gets access which is annoying to say the least even though us (three devs) in the U.K. are the principal engineering team so there looking at all options.

Anyways we have a back and fourth, we discuss some of the plans for the app, some of which we are nowhere near ready to even conceptualise as the app in its current state sucks, (ruby 2.2.6 and rails 5 but not really). Needs major refactoring and rewrite, one thing they want to do is multi tendency which again given the state is laughable.

So, as my manager is speaking my head is screaming being like “this is just going to be a massive disaster”. Then we go onto that he’s seeing what everyone’s strengths are etc. And then we get onto the upgrade and that he wants me to work on it.

Yes.. the upgrade I’ve been trying to do for the past 4+ months but I keep getting told to stop and getting pushed backed.

I’ve been told we have devOps looking into restructuring the app, not possible as how the app is written, we have India trying to multi tenant again disaster incoming as they’ll end up rushing it. Legal are going to have a field day. Every time I say the issues are the fundamentals with the app, here’s how we can sort it. In one ear out the other basically there patching the ship even though it’s still leaking.

I have so many ideas, and things I can do to improve the app and get it back to not only working order, fix the performance issues, data issues and everything else. Brick wall.

So rants ensue where I basically say I would love to do the upgrade but management gives me no time in the roadmap (we have no say in planning). At this point I’m just speaking to a brick wall.

After the meeting I have a chat with the BAs, we all have the same issues so honestly it sucks we end up ranting to each other for an hour.

I’m being under-utilised, being told do this, do that even though I’ve had two stabs but told to stop and pushed back, I know what benefits I can bring to the app with a refactoring, ideas and how to properly lead the team because honestly we’re working on an old legacy app, and management are clueless and there priorities are all wrong, the company is getting frustrated and it’s a sinking ship. They would rather patch issues without solving them and everything I say goes in one ear and out the other.

Frustrating is not the word.

Tooday i try to migrate a Drupal 7 to Drupal 8 ... I think that migrate module from drupal8 will work easily... But i remember on Drupal, noting work as expected on this Platform... I run the migration and suprise suprise content type field are off or empty and I don't have any data imported ...

Go to reports check the error message ... Try to get information on Internet hoo yes i remember no fucking documentation anywhere because Yolo i need to release 1626 modules that in beta or alpha after 5 years of development. And i dont give shit anymore.

Now i try use migrate_plus and tools to make my custom importation... Documentation not found or not finish ... Please omg please just shutup and work

Anyone else here done a data migration from CiviCRM to Salesforce with a metric arseton of custom fields? Every so often I find some dirty data issue that predates even the previous lead dev and most other people who still work here.

So I'm working on our inhouse help desk system
added a few stuff to the database and since there was already some live data in the database had to make use of flask-migrate for making migrations and upgrades

Since I had already initialized a migration and a revision number was already generated bumped into a situation where i couldn't make new migrations and me panic mode ramping up
opens chrome and surfed the net for solutions like the flash
thumbs up for platforms like stackoverflow
saved my ass today

I have been tasked with planning a feature retirement.
Basically the plan is to move this feature to another plan where it makes more sense and is more stable.
Now we don't have any data on how many or which customers are using this. And I need to plan a migration for them.
Fun times ahead...

It's been some months and I've been working with data, I'm in charge of the data migration and after 4 months into the job, they tell me that the data is sometimes transformed but they do not have a tool to validate the data, instead they do it manually.
When I said, that if we do it manually, it'll be prone to human error. My team says that humans make errors.
They're basically checking 2 rows of the database manually and then pray. That's their way of testing/validating the data.

Couple years back when I was in college, we used to talk about code quality or whatever but I don't think that's followed anywhere.

Developers are lazy. That's the fucking truth.