devRant Android users - if you are a user who can consistently reproduce a devRant app crash and wouldn't mind helping me please let me know. You can comment here or email me at david@devrant.io

I'm trying to get rid of a couple of remaining crashes that seem to be happening in the Android app but we've never really been able to reproduce any of them. Thanks!

So...
I'm penetrationtesting a network and the servers on said network
The network administrator and IT security officer knows this, because they hired me..

TL;DR a scan caused the network to crash.

Today I received a very angry email going "Stop scanning NOW!" from one of the IT departments.

Apparently I crashed their login server and thus their entire network...
It happened d the first time I scanned the network from the outside and they had spend an entire day figuring out how and repairing the service they thought was the problem, but then it crashed again, when I scanned from within the network.

Now they want to send me a list of IP's that I'm not allowed to scan and want to know exactly what and when I'm scanning...

How crap can they be at their job, if they weren't able to spot a scan... The only reason they found out it was me was because the NA had whitelistet my IP, so that I could scan in peace...

Spent 2 hours last night (leaving an hour late) with the IT guy hunting down a problem that affected at least 12 other teams. It didn't crash the app, but did prevent MANY scripts from working, and thus nothing could be committed.
I found the culprit, made a solution, and posted in the email chain my solution. (it required a code review and a client-side update)
Someone responded asking for another dev to confirm my report. That dev did and them dumbed it down for those who can't understand programming talk. Then EVERY EMAIL after that thanked that dev for "fixing the root problem" and "solving their scripts".
And just now, the PO for the bug was replaced to that dev's team. (previously was my team's PO)

Anyone reading these emails we are sending?

I work at a small place. A few users are using an application at our place that I develop and maintain. We all work remotely.

I announce by email to these few users a new version release of said application because of low level changes in the database, send the timeline for the upgrade, I include the new executable, with an easy illustrated 2 minutes *howto* to update painlessly.

Yet, past the date of the upgrade, 100% of the application users emailed me because they were not able to use the software anymore.

----------------

Or I have this issue where we identified a vulnerability in our systems - and I send out an email asking (as soon as possible) for which client version users are using to access the database, so that I patch everything swiftly right. Else everything may crash. Like a clean summary, 2 lines. Easy. A 30 second thing.

A week pass, no answer, I send again.

Then a second week pass, one user answers, saying:

> well I am busy, I will have time to check this out in February.

----------------

Then I am asking myself:
* Why sending email at all in the first place?

* Who wrote these 'best practices textbooks about warning users on schedule/expected downtime?'

*How about I just patch and release first and then expect the emails from the users *after* because 'something is broken', right? Whatever I do, they don't read it.

Oh and before anyone suggest that I should talk to my boss about this behavior from the users, my boss is included in the aforementioned 'users'.

Catch-22 much ? Haha thanks for reading

/rant

1) API Server crash due to Uncaught Error and none of us realised it until 30 mins later (no email alerts, no mechanism in place to notify devs): expensive mistake

2) api server and database server on one frickin vps, with no caching mechanisms or memoisation on frontend: Worst nightmare of my life

Me: "Can you get the user to send us a recent crash dump?"
(Support person forwards my email)
User: "The most recent ones are from November"
Support person: "It seems they haven't had any this month"
Me: (?????!!) "Which November?"

The eventual crash dump was datestamped with today's date.

This shit makes me fucking rage! Ok, so here it goes. I use Multiswipe (multiswipe.com) and I'm actually very happy with it. It works as advertised and doesn't crash. Then I bought a new laptop. I wiped the old one and installed multiswipe on the new one. After launching I'm greeted with the registration (I bought it) where it claims I have already registered the software on a different computer. So I reach out to the owner and ask him to help out and am told

"MultiSwipe can be installed and updated as many times as you want as long as it is in the same machine, but if you change laptops then you will need to purchase a new license, this is because MultiSwipe implements a serious of optimizations depending on your touchpad brand and model and as such it's physically linked to it."

This sounds like horseshit to me. If I download a fresh copy, wouldn't it optimize depending on the new system?

But ok. I'll purchase another one, only to be told that my e-mail address is already in use. After reaching out I am told that I HAVE TO CREATE A NEW EMAIL ADDRESS.

Jon, if you read this, I seriously love your software but what the fuck is this fuckery. Like, seriously, how god damn hard can it be to allow customers to purchase multiple licences?

I'm so angry that I'm considering pouring time into cracking it and sending him the new version with the text "nvm, got it to work."

I'm open to suggestions....

TLDR;

When governments started printing money to cure new pandemic and crash current market with great inflation I took all my savings, got a loan and bought biggest property I could afford. Every major news station was talking about end of world, but this was not I was scared of. I was scared of the helicopter money that would wipe my 5 years old savings.

When I was about to sign loan papers to buy my first apartment I got an email that my contract will end in 3 months. I said ok, the contractor company will find me something else.

I asked and they assured me they will do it. After my contract end just before summer holidays there was silence from contracting company and then after 5 years of me earning them piles of money, after finished project and congratulations from customer they offered me most shitty job they had where people resign after a week. I said I don’t want to land in another shit hole bring it back to life for another 2-5 years and kill myself when they offer me same shit afterwards so I resigned.

It was so fucked up that even the boss from the client I was contracting asked me if I lost my job cause I finished all that they wanted. I said it’s not your fault man. I will be ok, but I wasn’t.

I had apartment I couldn’t move in cause I needed to renovate. Loan I needed to pay. Rented apartment, accountant and business that was loosing money cause I was without contract, the world was locked down and everyone was depressed.

I said ok, I still have some savings left so I I started looking for something new but market was dead. Everyone was gone for holidays after winter lockdown. I was burning money and trying to figure out what to do.

After 2 months of nothing, when I started thinking about finding some temporary job to not loose everything I worked for, things moved. I started attending hiring meetings and solving tests everyday, also from big four gang but I didn’t passed trough hr due to how they say I’m to independent and I need to look for consulting business or do something on my own.

People asked why I don’t do something on my own and I politely answered that I want to work there.

I was about to run out of money when I got a call that company is looking for me cause I was doing similar things they want to do. During interviews it was pleasant small talk about what id did over those years and what they want to do, 2 days later I joined small team. I barely managed to survive a month for a first paycheck.

Since then we created new product for a company. Now the person who hired me is leaving and I think I should also leave the ship and find other things to do.

A software had been developed over a decade ago. With critical design problems, it grew slower and buggier over time.
As a simple change in any area could create new bugs in other parts, gradually the developers team decided not to change the software any more, instead for fixing bugs or adding features, every time a new software should be developed which monitors the main software, and tries to change its output from outside! For example, look into the outputs and inputs, and whenever there's this number in the output considering this sequence of inputs, change the output to this instead.
As all the patchwork is done from outside, auxiliary software are very huge. They have to have parts to save and monitor inputs and outputs and algorithms to communicate with the main software and its clients.
As this architecture becomes more and more complex, company negotiates with users to convince them to change their habits a bit. Like instead of receiving an email with latest notifications, download a csv every day from a url which gives them their notifications! Because it is then easier for developers to build.
As the project grows, company hires more and more developers to work on this gigantic project. Suddenly, some day, there comes a young talented developer who realizes if the company develops the software from scratch, it could become 100 times smaller as there will be no patchwork, no monitoring of the outputs and inputs and no reverse engineering to figure out why the system behaves like this to change its behavior and finally, no arrangement with users to download weird csv files as there will be a fresh new code base using latest design patterns and a modern UI.
Managers but, are unaware of technical jargon and have no time to listen to a curious kid! They look into the list of payrolls and say, replacing something we spent millions of man hours to build, is IMPOSSIBLE! Get back to your work or find another job!
Most people decide to remain silence and therefore the madness continues with no resistance. That's why when you buy a ticket from a public transport system you see long delays and various unexpected behavior. That's why when you are waiting to receive an SMS from your bank you might end up requesting a letter by post instead!
Yet there are some rebel developers who stand and fight! They finally get expelled from the famous powerful system down to the streets. They are free to open their startups and develop their dream system. They do. But government (as the only client most of the time), would look into the budget spending and says: How can we replace an annually billion dollar project without a toy built by a bunch of kids? And the madness continues.... Boeings crash, space programs stagnate and banks take forever to process risks and react. This is our world.

Gah... Outlook just froze because it can't connect to the mail server.... While I was drafting a long email....

Hope it doesn't crash, hope it doesn't crash.... Please please please...

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

That moment you setup 17 domains on sparkpost as a email delivery system

make your account secure with 2 factor authentication like a good infoSec enthusiast

Go on with your life

Having a Phone crash but nothing to worry because you made them backupz

Restore backupz

once again go on with your happy life.

Having to setup a different bounce action on sparkpost

logging in to sparkpost to make the adjustments

opening google authenticator

realising the backup you restored was before you added the sparkpost entry

mailing sparkpost asking to deactivate 2factor authentication

Having them tell me that they have no access to Google authenticator so they can't help me and all they can do for me is delete my account if i answer their 7569357 questions that i entered a year ago ..

--
You have access to your database yes ? You can delete my account but you can't adjust a fcking Boolean column from true to false? #@?#&!

Why even offer a feature where you have apparently no control over. Stuff like this happens all the time and almost no one saves that fcking authenticator secret.

Make people use authenticators to keep the hackers out, forces them out instead.

Bullshittery continues. This time around, absolutely innocent, clamav is root cause. For once not incompetent idiot, but piece of software. IDK if that makes me happy or upset.

So our email server that I configured and took care of died. RIP. Damn, better put it back together ASAP. So Im under pressure, while still pissed at everything that I ranted before (actually my last 2 rants were throttled, and in total all of that happened past 60 minutes but devrant rate limiting) I start auditing logs. You imagine, we kindda need it NOW, and it's second time last month clamav is pulling stunts and MTA refuses (properly) to work without antivirus. So pressurized, I look at logs, what the fuck went wrong.

clamav deamonize() failed - cannot allocate memory

Hmm. Intresting, but sounds like bullshit. I know server is quite micro becouse they wanted to save on costs as much as possible, but it has well over half a gig free ram just before it crashes (like 800MB) with that message. Is it allocating almost gig in one call or what? Looked carefully at trusty htop while it was starting, and indeed, suddenly it just dies with quite a bit of ram free, almost as much as it weights already. And I remember booting it up when I was configuring it, and it had fair bit of headroom.

Google, help me friend... Okay, great, so apparently at some point clamav loads virus DB into ram (dafuq?), and than forks, which causes spike of 2x the ram usage, and than immidietely frees it up.

Great, that sounds like great design decision... At least I know, I can just slap on SWAP file, restart it and call it a day.

It worked, swap file is almost empty (used 15megs, 900 megs free ram, whatever).

That leaves me wandering, who figured out to load DB to ram? That means pretty much that clamav will eat a little bit more ram each vir db update, and that milisecond "double ram" spike will confuse innocent people who just wanted to run clamav and it worked last *long period of time* and now crashes without warning without any changes to configuration.

Maybe there is logical explanation, I want to know it.

Settle an argument for our development team. We have infrastructure to report crashes when they occur, via a simple online form submission. The form provides basic fields for a description and an email address, and also posts some basic telemetry (rebuilding what it can of call stack, variables etc). Currently the email address is optional (you can submit the form and leave it blank). The form is not mandatory (the user can hit 'send' to submit online, 'save' to save the crash report to transfer in other ways, or 'cancel' to just ignore it entirely (irregardless of their choice, depending on where the crash has occurred, they may be able to continue using the application or it might exit).

A suggestion has been put forward to make the email address mandatory. Surprisingly, this has kicked off an incredibly polarising debate, so I thought I would put it to devRant to see what is the consensus here.

I'm trying not to bias the discussion by stating with the considerations at play, but would encourage you to think about them before chiming in!