Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "do-release-upgrade"
-
Things have been a little too quiet on my side here, so its time for an exciting new series:
practiseSafeHex's new life as a manager.
Episode 1: Dealing with the new backend team
It's great to be back folks. Since our last series where we delved into the mind numbing idiocy of former colleagues, a lot has changed. I've moved to a new company and taken a step up as a Dev manager / Tech lead. Now I know what you are all thinking, sounds more dull and boring right? Well it wouldn't be a practiseSafeHex series if we weren't ...
<audience-shouting>
DEALING! ... WITH! ... IDIOTS!
</audience-shouting>
Bingo! so lets jump right in and kick us off with a good one.
So for the past few months i've been on an on-boarding / fact finding / figuring out this shit-storm, mission to understand more about what it is i'm suppose to do and how to do it. Last week, as part of this, I had the esteemed pleasure of meeting face to face with the remote backend team i've been working with. Lets rattle off a few facts to catch us all up:
- 8 hour time difference to me
- No documentation other than a non-maintained swagger doc
- Swagger is reporting errors and several of the input models are just `Type: String`
- The one model that seems accurate, has every property listed as optional, including what must be the primary key
- Properties go missing and get removed at the drop of a hat and we are never told.
- First email I sent them took 27 days to reply, my response to that hasn't been answered so far 31 days later (new record! way to go team, I knew we could do it!!!)
- I deal directly with 2 of them, the manager and the tech lead. Based on how things have gone so far, i've nick named them:
1) Ass
2) Hole
So lets look at some example of their work:
- I was trying to test the new backend, I saw no data in QA. They said it wouldn't show up until mid day their time, which is middle of the night for us. I said we need data in our timezone and I was told: a) "You don't understand how big this system is" (which is their new catch phrase) b) "Your timezone is not my concern"
- The whole org started testing 2 days later. The next day a member from each team was on a call and I was asked to give an update of how the testing was going on the mobile side. I said I was completely blocked because I can't get test data. Backend were asked to respond. They acknowledged they were aware, but that mobile don't understand how big the system is, and that the mobile team need to come up with ideas for the backend team, as to how mobile can test it. I said we can't do anything without test data, they said ... can you guess what? ... correct "you don't understand how big the system is"
- We eventually got something going and I noticed that only 1 of the 5 API changes due on their side was done. Opened tickets. 2 days later asked them for progress and was told that "new findings" always go to the bottom of the backlog, and they are busy with other things. I said these were suppose to be done days ago. They said you can't give us 2 days notice and expect everything done. I said the original ticket was opened a month a go *sends link* ......... *long silence* ...... "ok, but you don't understand how big the system is, this is a lot of work"
- We were on a call. Product was asking the backend manager (aka "Ass") a question about a slight upgrade to the new feature. While trying to talk, the tech lead (aka "Hole") kept cutting everyone off by saying loudly "but thats not in scope". The question was "is this possible in the future" and "how long would it take", coming from management and product development. Hole just kept saying "its not in scope", until he was told to be quiet by several people.
- An API was sending down JSON with a string containing a message for the user with 2 bits of data inside it. We asked for one of those pieces to also come down as a property as the string can change and we needed it client side. We got that. A few days later we found an edge case and asked for the second piece of data to be a property too. Now keep in mind, they clearly already have access to them in order to make the string. We were told "If you keep requesting changes like this, you are going to delay the release of the backend by up to 2 weeks"
Yes folks, there you have it, the most minuscule JSON modifications, can delay your release by up to 2 weeks ........ maybe I should just tell product, that they don't understand how big the app is, and claim we can't build it on our side? Seems to work for them
Thats all the time we have for today,
Tune in for more, where we'll be looking into such topics as:
- If god himself was an iOS developer ... not
- Why automate when you can spend all day doing it by hand
- Its more time-efficient to just give everything a story point of 5
- Why waste time replying to emails ... when you can do nothing instead
See you all next week,
practiseSafeHex14 -
Yesterday (or the day before that depending on your timezone and day-night schedule - this Friday) my OnePlus 6T arrived. After only 2 days of time between placing the order and actually getting the phone, quite impressive!
The DHL guy asked me upon receipt - is it the OnePlus 6T? - Yes it is!! - "An amazing device it is!", he said. And honestly.. he couldn't be more right.
I might be a bit biased on this because after all I did just spend €630 on this phone. But it feels so snappy, high quality, the 8GB of RAM is just.. it blows my mind. But I'm sure that the other reviews did this sort of jazz already.
The things that set this phone apart for me though were the following.
When I get a new phone or tablet, usually the first thing I do is rooting it. This one was no different, about an hour after receipt it was successfully rooted and loaded with Magisk. Currently I'm still in the phase of "getting to know the phone", wherein fuckups are usual. This time again being no different - I removed some apps and apparently did something to it that the search engines - both Google and DuckDuckGo - didn't quite like, as both of them would crash upon application launch. Me in full panic mode of course, desperately trying to find the stock ROM (which doesn't seem to be present in its usual form) or a new set of GApps (which didn't resolve the issue). OnePlus does seem to offer its OTA updates in zip archives though. So I downloaded its latest update (same as what was on the device) and applied it.
That's when the nerdgasm happened.
The "update" was simply a matter of going into the settings, tapping this and that and applying the update. No recovery, no unrooting, no nothing. The update just went like that despite the phone being rooted and just having had TWRP flashed to it. I always wanted this sort of thing, which even the Nexus couldn't offer - having the cake and eating it too. Being able to root the device and muck around with it while still being able to update the device timely without too many hurdles. This fucking thing does it!!!
That is to say, after my initial nerdgasm I did find that it bulldozed over my su binary (effectively unrooting the thing), custom emoji I've set (iOS 12 because fuck Google's most recent emoji set) and some other things. But those are easy to install back, much more so than it would've been to download a whole Android release and dirty flash it, as it was on the Nexus.
Other than that, battery life, dash charging (edit: on that topic, it does remain cool like a cucumber despite getting 15-20W of power jammed into it, quite impressive!), snappiness, the usual jazz.. eh, as I said earlier that's the usual reviewer stuff. But this feature of being able to upgrade the phone while it's modified, that's something which seems to be severely underrated by those.
Oh and during kernel builds, I couldn't quite get the source to work - probably due to my lack of experience with builds of Android kernels - but I did find that this phone actually exposes its kernel config through /proc/config.gz as it should. None of my MediaTek devices do this, so that's something that I found really appealing. Always nice to see when a manufacturer exposes this information to give you a stock sort of config that you can be rest assured will work configuration-wise. And it allows you to see what the stock kernel is actually built with, which again is really nice. I quite like this! It really encourages further development.11 -
alias break-ubuntu="do-release-upgrade"
# Only works if new distro is available, otherwise you have to find another way to break ubuntu -
Anyone reading these emails we are sending?
I work at a small place. A few users are using an application at our place that I develop and maintain. We all work remotely.
I announce by email to these few users a new version release of said application because of low level changes in the database, send the timeline for the upgrade, I include the new executable, with an easy illustrated 2 minutes *howto* to update painlessly.
Yet, past the date of the upgrade, 100% of the application users emailed me because they were not able to use the software anymore.
----------------
Or I have this issue where we identified a vulnerability in our systems - and I send out an email asking (as soon as possible) for which client version users are using to access the database, so that I patch everything swiftly right. Else everything may crash. Like a clean summary, 2 lines. Easy. A 30 second thing.
A week pass, no answer, I send again.
Then a second week pass, one user answers, saying:
> well I am busy, I will have time to check this out in February.
----------------
Then I am asking myself:
* Why sending email at all in the first place?
* Who wrote these 'best practices textbooks about warning users on schedule/expected downtime?'
*How about I just patch and release first and then expect the emails from the users *after* because 'something is broken', right? Whatever I do, they don't read it.
Oh and before anyone suggest that I should talk to my boss about this behavior from the users, my boss is included in the aforementioned 'users'.
Catch-22 much ? Haha thanks for reading
/rant7 -
!rant && story
tl;dr I lost my path, learned to a lot about linux and found true love.
So because of the recent news about wpa2, I thought about learning to do some things network penetration with kali. My roommate and I took an old 8gb usb and turned it into a bootable usb with persistent storage. Maybe not the best choice, but atleast we know how to do that now.
Anyway, we started with a kali.iso from 2015, because we thought it would be faster than downloading it with a 150kpbs connection. Learned a lot from that mistake while waiting apt-get update/upgrade.
Next day I got access to some faster connection, downloaded a new release build and put the 2015 version out it's misery. Finally some signs of progress. But that was not enough. We wanted more. We (well atleast I) wanted to try i3, because one of my friends showed me to /r/unixporn (btw, pornhub is deprecated now). So after researching what i3 is, what a wm is AND what a dm is, we replaced gdm3 with lightdm and set i3 as standard wm. With the user guide on an other screen we started playing with i3. Apparently heaven is written with two characters only. Now I want to free myself from windows and have linux (Maybe arch) as my main system, but for now we continue to use thus kali usb to learn about how to set uo a nice desktop environment. Wait, why did we choose to install kali? 😂
I feel kinda sorry for that, but I want to experiment on there before until I feel confident. (Please hit me up with tips about i3)
Still gotta use Windows as a subsystem for gaming. 😥3 -
story of a release
v2.1.0 major changes went live : new features, bug fixes, optimisations. also included releases for 2 associated libraries
release process tasks:
- do code
- update test cases
- test sample app
- test on another sample app
- get code reviewed and approved by senior ( who takes his own sweet time to review and never approves on first try)
- get code reviewed again
- merge to develop after 20 mins( coz CICD pipeline won't finish and allow merging before that)
- merge to master after 20 mins( coz CICD again)
- realise that you forgot to update dates in markdown files as you thought the release will be on 10th sept and release is happennig on 12th sept coz of sweet senior's code fucking/reviewing time
- again raise a branch to develop
- again get it a review approval by sr (who hopefully gives a merge approval in less time now)
- again get it merged to develop after waiting for 20 mins
- again get it merged to master after waiting for 20 mins
- create a clean build aar file
- publish to sonatype staging
- publish to sonatype release
- wait for 30 mins to show while having your brain fucked with tension
- create a release doc with all the changes
- update the documentation on a wyswig based crappy docs website
- send a message to slack channels
- done
===========
why am i telling you this? coz i just found a bug in a code that i shipped in that release which still got in after all the above shitty processes. its a change of a 3 lines of code, but i will need to do all the steps again. even though i am going through the same shitty steps for another library version upgrade that depends on this library 😭😭
AND I AM THE ONE WHO CAUGHT IT. it went unnoticed because both of those shitty samples did not tested this case. now i can keep mum about it and release another buggy build that depends on it and let the chaos do its work, or i can get the blame and ship a rectification asap. i won't get any reward or good impression for the 2nd, and a time bomb like situation will get created if i go with 1st :/
FML :/6 -
In the ever-growing saga of the upgrade, here is another one.
In the daily scrum meeting, I chat about the upgrade, standard stuff.
The other dev pipes up - "Oh we had a meeting about that this morning and were going with a different approach"
Me - "wait, we're doing what now? You do know I've spent a month so far just on this upgrade?*
*silence*
Anyways I continue working on the upgrade, few meetings while I try to find out what's going on.
Spoken to BA, my line manager and the other dev didn't get much basically saying yeah this is how we're handing it now.
Well it turns out after writing a big long message to the other dev, he decided *yesterday* in a manager meeting (he's kind of a manager but not really) to propose a new approach and they all just leapt at the chance even though it's going to take way longer (2 years estimate) to patch up the system version by version until we get to the latest release.
So at some point today he sends me a message to stop what I'm doing and go and help with a product release and that we *are* doing this new approach and that he made the decision yesterday. I'm sorry but since when did he become my manager micromanaging me haha
So as the only one doing the upgrade, I only got told of this change in passing, the other dev said that he decided yesterday and didn't bother to tell me as he had other stuff to work on and neither did my line manager.
Seriously what the hell.
So hopefully the things I've worked on and done might get used in a year or two haha6 -
Is it me or software subscriptions make developers lazy?
There is a great photo editing software: Capture One. Every year they release a new major version, so users need to buy an upgrade. In the past developers packed a bunch of big changes into major update, also they released 3 minor updates yearly, and every minor update brought some cool features. But then they added subscription model which was cheaper then perpetual model. And at the same time major updates became not that cool. Developers started to add enterprise features needed by museums, features involving other camera brands users, changes targeted at newbies and so on. For perpetual model users most of these changes are not worth 80-255 EUR yearly (depends on license type and offs) but is ok for subscription model users because they continue using the software and even small updates and enhancements are fine for them.
Not every major update is that weak but many of them are not worth upgrade. And developers are not motivated to do more cool stuff because subscription model users will continue paying for their subscriptions.1 -
To the reactjs-centered fucks who develop the popular web component viewing software called storybook: have you ever heard about semver?
89 alpha/beta/rc releases for a minor update 6.3 -> 6.4 with "100's of fixes and enhancements" "in preparation of the HUGE 7.0 release". Gee I wonder will it have 1000's of bugfixes? How bug-ridden is this software?
Every minor upgrade since 5.x is backwards-incompatible and requires a day of frustration finding out in how many more fucking NPM packages you split your codebase just because it's cool. I know move fast and break things, but some of us have other things to do than resolving node_modules incompatibilities you know. "No just hit 'npx sb upgrade' you say". I did, I really did! And the browser showed a blank screen of death with tons of cryptic React errors, it really did! Thank God you abstracted away all your dependencies in that sb command, now you can't even read the docs about what could have gone wrong with a specific sub-package. You have @storybook/html but the docs redirect to React pages, so good luck if you use something else
This is so sad... like.. the IDEA of storybook is great. But why did faith put the capacity to develop such a tool into the hands of people who think the world centers around React and JSX.. HTML should have been the default, and then you build on top of that for your fav framework, not the other way around -
In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.
In today's episode - SystemD vs CGroups!
Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.
One can read about these interactions in the manual page "systemd.resource-control"
All is cool so far. So what happened to me today?
Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.
The test server was never properly stress-tested. This would prove to be an issue.
When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.
2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.
Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.
This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().
So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.
Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.
This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)
So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.
It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.
What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...
Oh, and apache + http/2 = huge TID sink. -
Currently having very funny project lead, who gives on the spot estimates for 9 years old very pathetic quality code having Android app in security domain. Memory leaks, bad practices, typos, CVEs etc. you name it we have it in our source of the app.
Since 5-6 sprints of our project, almost 50% of user stories were incomplete due to under estimations.
Basically everyone in management were almost sleeping since last 7-8 years about code quality & now suddenly when new Dev & QA team is here they wanted us to fix everything ASAP.
Most humourous thing is product owner is aware about importance of unit test cases, but don't want to allocate user stories for that at the time of sprint planning as code is almost freezed according to him for current release.
Actually, since last release he had done the same thing for each sprint, around 18 months were passed still he hadn't spared single day for unit testing.
Recently app crash issue was found in version upgrade scenario as QAs were much tired by testing hundreds of basic trivial test cases manually & server side testing too, so they can't do actual needful testing & which is tougher to automate for Dev.
Recently when team's old Macbook Pros got expired higher management has allocated Intel Mac minis by saying that few people of organization are misusing Macbooks. So for just few people everyone has to suffer now as there is no flexibility in frequent changing between WFH & WFO. 1 out of those Mac minis faced overheating & in repair since 6 months.
Out of 4 Devs & 3 QAs, all 3 QAs & 2 Devs had left gradually.
I think it's time to say goodbye 😔3 -
Anti climactic story time (as in there's no promotion in this story):
Sometime ago there were some organizational changes happening in my company that put me in a very tricky place. Theoretically, I was put on a level that was supposed to be an upgrade from my previous level. Practically, it didn't come with any benefits and it was actually a downgrade because anyone who joined the company in the six months before these changes was in the same level as me (who'd been in for roughly 2 years).
It felt really insulting because I was about to be actually promoted. My manager and his manager tried to gaslight me into believing that I'm not at all affected in any way, before giving in and agreeing that a mistake was made. I was promised that next year it'll be corrected and I'll be promoted two levels. Even the HR assured me of that. I knew it was too good to be true but I was too demotivated to find another job.
Fast forward one year. My bosses are all praises for the work I put in. But, no two level promotion. Reason? They tried but couldn't get the management to agree. The boss apologized to me and asked me if I wanted him to try again. What an insolent arse!
Fast forward one more, extremely glum year.
This time I am part of a different team so the team lead is different but the manager is same. The team lead really went all out with showing appreciation for me. He talked for almost an hour(!) about how I exceeded his expectations and went on to claim that his app's release would have been impossible if it weren't for me, the new team member. It was really humbling and satisfying. But what did I get? A limp handshake from the manager with fucking loose change.
Silver lining. At least the manager did away with the 'well wisher, on your side' pretense this time. No mentions of failed promises, just regular empty promises for the future.
Fast forward 3 months.
Still here. Recovering. I am mulling over a much better offer than what my current boss can give me. Thinking about how long it takes before I'm in the dumpster again. I have stopped giving any fucks about anything here. I try to do the minimum required unless it benefits me in some way.
The end.4 -
Trying to fix an urgent issue with our Xamarin iOS app and a known bug in Xamarin "IOException: Sharing violation on path /Assets.xcassets/AppIcon.appiconset/Icon-1024.png" is blocking me.
Luckily I still have my old laptop from my last upgrade on standby, boot it up and it's not using the affected version of Xamarin. 😃
Instead this one has the also know "/ios/release/mono/mini/mini-arm64.c:5439, condition `native_offset % 4 == 0' not met" blocking issue when debugging. 🤦♂️
I just want to do some work. ☹3