Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "never deploy on a friday"
-
Currently on an internship, PHP mostly, little bit of Python and the usual web stuff, and I just had the BEST FUCKING DAY EVER.
Wake up and find out I'm out of coffee, oh boy here we go.
Bus leaves 10 minutes late, great gonna miss my train.
Trains just don't wanna ride today, back in a bus I go, what's normally a 10 minute train travel is now a 90 minute bus ride.
Arrive at internship, coffee machine is broke, non problem, I'll just lose it slowly.
NOW HERE COMES THE FUCKING GOOD PART!!
Alright, so I'm working on a CMS that can be used just about on any device you want, mobile or desktop, it's huge, billion's of rows of scientific data. Very specific requirements and low error margins. Now, yesterday I was really enjoying myself here until today, Project manager walks in, comes to my desk and hands me a Samsung Gear S3, an Apple watch and some cheap knockoff. He tells me that before the Friday deploy, THE ENTIRE CMS SHOULD WORK ON THOSE WATCHES!
I mean, don't get me wrong, I like a challenge but it's just not right, I mean, I'm still not sure what the right way to handle tables on phones is, but smart watches, just no. Besides that, I've never worked with any Apple devices, let alone WatchOs, nor have I worked with Android Wear.
Also, Project Manager is a total dickhead, he's the kinda guy that prefers a light theme, doesn't clean up his code, writes 0 documentation for an API, 1 space = tab, pure horror.
So after almost flipping my desk, I just called my school coach to announce I'm leaving this internship. After a brief explanation he decides to come over, and guess what, according to the Project Manager I wasn't supposed to do that, I was supposed to test if it would be possible.
FUCKING ASSFUCKFACE9 -
Valve deployed a CS:GO update this Friday, guess what. It had quite a few bugs.
Never deploy on a Friday, please.6 -
deploying app 101: never deploy anything on a friday unless you want your saturday to be another monday
-
Long rant ahead.. 5k characters pretty much completely used. So feel free to have another cup of coffee and have a seat 🙂
So.. a while back this flash drive was stolen from me, right. Well it turns out that other than me, the other guy in that incident also got to the police 😃
Now, let me explain the smiley face. At the time of the incident I was completely at fault. I had no real reason to throw a punch at this guy and my only "excuse" would be that I was drunk as fuck - I've never drank so much as I did that day. Needless to say, not a very good excuse and I don't treat it as such.
But that guy and whoever else it was that he was with, that was the guy (or at least part of the group that did) that stole that flash drive from me.
Context: https://devrant.com/rants/2049733 and https://devrant.com/rants/2088970
So that's great! I thought that I'd lost this flash drive and most importantly the data on it forever. But just this Friday evening as I was meeting with my friend to buy some illicit electronics (high voltage, low frequency arc generators if you catch my drift), a policeman came along and told me about that other guy filing a report as well, with apparently much of the blame now lying on his side due to him having punched me right into the hospital.
So I told the cop, well most of the blame is on me really, I shouldn't have started that fight to begin with, and for that matter not have drunk that much, yada yada yada.. anyway he walked away (good grief, as I was having that friend on visit to purchase those electronics at that exact time!) and he said that this case could just be classified then. Maybe just come along next week to the police office to file a proper explanation but maybe even that won't be needed.
So yeah, great. But for me there's more in it of course - that other guy knows more about that flash drive and the data on it that I care about. So I figured, let's go to the police office and arrange an appointment with this guy. And I got thinking about the technicalities for if I see that drive back and want to recover its data.
So I've got 2 phones, 1 rooted but reliant on the other one that's unrooted for a data connection to my home (because Android Q, and no bootable TWRP available for it yet). And theoretically a laptop that I can put Arch on it no problem but its display backlight is cooked. So if I want to bring that one I'd have to rely on a display from them. Good luck getting that done. No option. And then there's a flash drive that I can bake up with a portable Arch install that I can sideload from one of their machines but on that.. even more so - good luck getting that done. So my phones are my only option.
Just to be clear, the technical challenge is to read that flash drive and get as much data off of it as possible. The drive is 32GB large and has about 16GB used. So I'll need at least that much on whatever I decide to store a copy on, assuming unchanged contents (unlikely). My Nexus 6P with a VPN profile to connect to my home network has 32GB of storage. So theoretically I could use dd and pipe it to gzip to compress the zeroes. That'd give me a resulting file that's close to the actual usage on the flash drive in size. But just in case.. my OnePlus 6T has 256GB of storage but it's got no root access.. so I don't have block access to an attached flash drive from it. Worst case I'd have to open a WiFi hotspot to it and get an sshd going for the Nexus to connect to.
And there we have it! A large storage device, no root access, that nonetheless can make use of something else that doesn't have the storage but satisfies the other requirements.
And then we have things like parted to read out the partition table (and if unchanged, cryptsetup to read out LUKS). Now, I don't know if Termux has these and frankly I don't care. What I need for that is a chroot. But I can't just install Arch x86_64 on a flash drive and plug it into my phone. Linux Deploy to the rescue! 😁
It can make chrooted installations of common distributions on arm64, and it comes extremely close to actual Linux. With some Linux magic I could make that able to read the block device from Android and do all the required sorcery with it. Just a USB-C to 3x USB-A hub required (which I have), with the target flash drive and one to store my chroot on, connected to my Nexus. And fixed!
Let's see if I can get that flash drive back!
P.S.: if you're into electronics and worried about getting stuff like this stolen, customize it. I happen to know one particular property of that flash drive that I can use for verification, although it wasn't explicitly customized. But for instance in that flash drive there was a decorative LED. Those are current limited by a resistor. Factory default can be say 200 ohm - replace it with one with a higher value. That way you can without any doubt verify it to be yours. Along with other extra security additions, this is one of the things I'll be adding to my "keychain v2".11 -
In horror movies. Characters do things that they them-self know it could kill them. Like getting in basement.
As a developer we all know to never deploy code on Friday. Here we are taking the wrong turn.2 -
So last week I really fucked up
I had this new implementation that was supposedly to be integrating smoothly into the rest of the service. It depended on a serialized model made by a data scientist. I test it in local, in QA environment: no problem.
So, Friday, 4pm, I decide to deploy to production. I check once from the app: the service throw an error. Panic attack, my chief is at my desk, we triy to understand what went wrong. I make calls with cUrls: no problem. Everything seems fine. I recheck from the app again: no problem.
We dedice to let it in prod, as the feature work. I go get some beers with the guys, to celebrate the deploy.
Fast-forward the next morning, 11am, my phone ring: it's a colleague of my chief. "Please check Slack, a client is trying to use the feature, it's broken"
FUUUUUUUUUUUUCK!!!
Panic attack again. I go to the computer, check the errors: two types of errors. One I can fix, the other from a missing package on the machine that the data guy used.
Needless to say, I had a fairly good weekend.
Lessons learned:
- make sure Dev, QA and Prod are exactly the same (use Ansible or Container)
- never deploy on a Friday afternoon if you don't have a quick way to revert1 -
Here: this is BDD, a Linter, and Git. CI your shit to staging before you CI to production. Never deploy on a Friday.
-
I was thinking about the problems one of our clients faced with the launch of their project the other day, because things were rushed, stuff was omitted and in the end they could not meet the launch date, and I started making a list of hard lessons I learned over the years that would have helped them avoid this situation.
Feel free to add yours in the comments.
- Never deploy on Friday
- Never make infrastructure changes right before a launch
- Always have backups. Always!
- Version control is never optional
- A missed deadline is better than a failed launch
- If everything is urgent, nothing is important
- Fast and cheap, cheap and quality, quality and fast. Only one pair at a time can be achieved
- Never rush the start or the end of a project
- Stability is always better that speed
- Make technical decisions based on the needs of the project two years from now
- Code like you will be the only maintainor of the project two years from now. You probably will...
- Always test before you deploy
- You can never have too many backups (see above)
- Code without documentation is a tool without instructions
- Free or famous does not necessarily mean useful or good
- If you need multiple sentences to explain a method, you should probably refactor
- If your logic is checked beforehand, writing the code becomes way easier
- Never assume you understand a request the first time around. Always follow up and confirm
There are many more that should be on this list, but this is what came to mind now.2 -
Every first of a month is our most intense day because we have the most data throughput then due to lazy people we get data from. And boy will it be even more intense when something broke because we were forced to deploy a new feature the day before...