Good Morning!, its time for practiseSafeHex's most incompetent co-worker!

Todays contestant is a very special one.

*sitcom audience: WHY?*

Glad you asked, you see if you were to look at his linkedin profile, you would see a job title unlike any you've seen before.

*sitcom audience oooooooohhhhhh*

were not talking software developer, engineer, tech lead, designer, CTO, CEO or anything like that, No No our new entrant "G" surpasses all of those with the title ..... "Software extraordinaire".

*sitcom audience laughs hysterically*

I KNOW!, wtf does that even mean! as a previous dev-ranter pointed out does this mean he IS quality code? I'd say he's more like a trash can ... where his code belongs

*ba dum tsssss*

Ok ok, lets get on with the show, heres some reasons why "G" is on the show:

One of G's tasks was to build an analytics gathering library for iOS, similar to google analytics where you track pages and events (we couldn't use google's). G was SO good at this job he implemented 2 features we didn't even ask for:

- If the library was unable to load its config file (for any reason) it would throw an uncatchable system integrity error, crashing the app.
- If anything was passed into any of the functions that wasn't expected (null, empty array etc.) it would crash the app as it was "more efficient" to not do any sanity checks inside the library.

This caused a lot of issues as some of the data needed to come from the clients server. The day we launched the app, within the first 3 hours we had over 40k crash logs and a VERY angry client.

Now, what makes this story important is not the bugs themselves, come on how many times have we all done something stupid? No the issue here was G defended all of this as the right thing to do!

.. and no he wasn't stoned or drunk!

G claimed if he couldn't get the right settings / params he wouldn't be able to track the event and then our CEO wouldn't have our usage data. To which I replied:

"So your solution was to not give the client an app instead? ... which also doesn't give the CEO his data".

He got very angry and asked me "what would you do then?". I offered a solution something like why not have a default tag for "error" or "unknown" where if theres an issue, we send up whatever we have, plus the file name and store it somewhere else. I was told I was being ridiculous as it wasn't built to track anything like that and that would never work ... his solution? ... pull the library out of the app and forget it.

... once again giving everyone no data.

G later moved onto another cross-platform style project. Backend team were particularly unhappy as they got no spec of what needed to be done. All they knew was it was a single endpoint dealing with very complex model. There was no Java classes, super classes, abstract classes or even interfaces, just this huge chunk of mocked data. So myself and the lead sat down with him, and asked where the interfaces for the backend where, or designs / architecture for them etc.

His response, to this day frightens me ... not makes me angry, not bewilders me ... scares the living shit out of me that people like this exist in the world and have successful careers.

G: "hhhmmm, I know how to build an interface, but i've never understood them ... Like lets say I have an interface, what now? how does that help me in any way? I can't physically use it, does it not just use up time building it for no reason?"

us: "... ... how are the backend team suppose to understand the model, its types, integrate it into the other systems?"

G: "Can I not just tell them and they can write it down?"

**
I'll just pause here for a moment, as you'll likely need to read that again out of sheer disbelief
**

I've never seen someone die inside the way the lead did. He started a syllable and his face just dropped, eyes glazed over and he instantly lost all the will to live. He replied:

" wel ............... it doesn't matter ... its not important ... I have to go, good luck with the project"

*killed the screen share and left the room*

now I know you are all dying in suspense to know what happened to that project, I can drop the shocking bombshell that it was in fact cancelled. Thankfully only ~350 man hours were spent on it

... yep, not a typo.

G's crowning achievement however will go down in history. VERY long story short, backend got deployed to the server and EVERYTHING broke. Lead investigated, found mistakes and config issues on every second line, load balancer wasn't even starting up. When asked had this been tested before it was deployed:

G: "Yeah I tested it on my machine, it worked fine"

lead: "... and on the server?"

G: "no, my machine will do the same thing"

lead: "do you have a load balancer and multiple VM's?"

G: "no, but Java is Java"

... and with that its time to end todays episode. Will G be our most incompetent? ... maybe.

Tune in later for more practiceSafeHex's most incompetent co-worker!!!

Hacking/attack experiences...
I'm, for obvious reasons, only going to talk about the attacks I went through and the *legal* ones I did 😅 😜

Let's first get some things clear/funny facts:
I've been doing offensive security since I was 14-15. Defensive since the age of 16-17. I'm getting close to 23 now, for the record.

First system ever hacked (metasploit exploit): Windows XP.
(To be clear, at home through a pentesting environment, all legal)
Easiest system ever hacked: Windows XP yet again.

Time it took me to crack/hack into today's OS's (remote + local exploits, don't remember which ones I used by the way):
Windows: XP - five seconds (damn, those metasploit exploits are powerful)
Windows Vista: Few minutes.
Windows 7: Few minutes.
Windows 10: Few minutes.
OSX (in general): 1 Hour (finding a good exploit took some time, got to root level easily aftewards. No, I do not remember how/what exactly, it's years and years ago)
Linux (Ubuntu): A month approx. Ended up using a Java applet through Firefox when that was still a thing. Literally had to click it manually xD
Linux: (RHEL based systems): Still not exploited, SELinux is powerful, motherfucker.

Keep in mind that I had a great pentesting setup back then 😊. I don't have nor do that anymore since I love defensive security more nowadays and simply don't have the time anymore.

Dealing with attacks and getting hacked.

Keep in mind that I manage around 20 servers (including vps's and dedi's) so I get the usual amount of ssh brute force attacks (thanks for keeping me safe, CSF!) which is about 40-50K every hour. Those ip's automatically get blocked after three failed attempts within 5 minutes. No root login allowed + rsa key login with freaking strong passwords/passphrases.

linu.xxx/much-security.nl - All kinds of attacks, application attacks, brute force, DDoS sometimes but that is also mostly mitigated at provider level, to name a few. So, except for my own tests and a few ddos's on both those domains, nothing really threatening. (as in, nothing seems to have fucked anything up yet)

How did I discover that two of my servers were hacked through brute forcers while no brute force protection was in place yet? installed a barebones ubuntu server onto both. They only come with system-default applications. Tried installing Nginx next day, port 80 was already in use. I always run 'pidof apache2' to make sure it isn't running and thought I'd run that for fun while I knew I didn't install it and it didn't come with the distro. It was actually running. Checked the auth logs and saw succesful root logins - fuck me - reinstalled the servers and installed Fail2Ban. It bans any ip address which had three failed ssh logins within 5 minutes:
Enabled Fail2Ban -> checked iptables (iptables -L) literally two seconds later: 100+ banned ip addresses - holy fuck, no wonder I got hacked!

One other kind/type of attack I get regularly but if it doesn't get much worse, I'll deal with that :)

Dealing with different kinds of attacks:

Web app attacks: extensively testing everything for security vulns before releasing it into the open.
Network attacks: Nginx rate limiting/CSF rate limiting against SYN DDoS attacks for example.
System attacks: Anti brute force software (Fail2Ban or CSF), anti rootkit software, AppArmor or (which I prefer) SELinux which actually catches quite some web app attacks as well and REGULARLY UPDATING THE SERVERS/SOFTWARE.

So yah, hereby :P

My first job: The Mystery of The Powered-Down Server

I paid my way through college by working every-other-semester in the Cooperative-Education Program my school provided. My first job was with a small company (now defunct) which made some of the very first optical-storage robotic storage systems. I honestly forgot what I was "officially" hired for at first, but I quickly moved up into the kernel device-driver team and was quite happy there.

It was primarily a Solaris shop, with a smattering of IBM AIX RS/6000. It was one of these ill-fated RS/6000 machines which (by no fault of its own) plays a major role in this story.

One day, I came to work to find my team-leader in quite a tizzy -- cursing and ranting about our VAR selling us bad equipment; about how IBM just doesn't make good hardware like they did in the good old days; about how back when _he_ was in charge of buying equipment this wouldn't happen, and on and on and on.

Our primary AIX dev server was powered off when he arrived. He booted it up, checked logs and was running self-diagnostics, but absolutely nothing so far indicated why the machine had shut down. We blew a couple of hours trying to figure out what happened, to no avail. Eventually, with other deadlines looming, we just chalked it up be something we'll look into more later.

Several days went by, with the usual day-to-day comings and goings; no surprises.

Then, next week, it happened again.

My team-leader was LIVID. The same server was hard-down again when he came in; no explanation. He opened a ticket with IBM and put in a call to our VAR rep, demanding answers -- how could they sell us bad equipment -- why isn't there any indication of what's failing -- someone must come out here and fix this NOW, and on and on and on.

(As a quick aside, in case it's not clearly coming through between-the-lines, our team leader was always a little bit "over to top" for me. He was the kind of person who "got things done," and as long as you stayed on his good side, you could just watch the fireworks most days - but it became pretty exhausting sometimes).

Back our story -
An IBM CE comes out and does a full on-site hardware diagnostic -- tears the whole server down, runs through everything one part a time. Absolutely. Nothing. Wrong.

I recall, at some point of all this, making the comment "It's almost like someone just pulls the plug on it -- like the power just, poof, goes away."

My team-leader demands the CE replace the power supply, even though it appeared to be operating normally. He does, at our cost, of course.

Another weeks goes by and all is forgotten in the swamp of work we have to do.

Until one day, the next week... Yes, you guessed it... It happens again. The server is down. Heads are exploding (will at least one head we all know by now). With all the screaming going on, the entire office staff should have comped some Advil.

My team-leader demands the facilities team do a full diagnostic on the UPS system and assure we aren't getting drop-outs on the power system. They do the diagnostic. They also review the logs for the power/load distribution to the entire lab and office spaces. Nothing is amiss.

This would also be a good time draw the picture of where this server is -- this particular server is not in the actual server room, it's out in the office area. That's on purpose, since it is connected to a demo robotics cabinet we use for testing and POC work. And customer demos. This will date me, but these were the days when robotic storage was new and VERY exciting to watch...

So, this is basically a couple of big boxes out on the office floor, with power cables running into a special power-drop near the middle of the room. That information might seem superfluous now, but will come into play shortly in our story.

So, we still have no answer to what's causing the server problems, but we all have work to do, so we keep plugging away, hoping for the best.

The team leader is insisting the VAR swap in a new server.

One night, we (the device-driver team) are working late, burning the midnight oil, right there in the office, and we bear witness to something I will never forget.

The cleaning staff came in.

Anxious for a brief distraction from our marathon of debugging, we stopped to watch them set up and start cleaning the office for a bit.

Then, friends, I Am Not Making This Up(tm)... I watched one of the cleaning staff walk right over to that beautiful RS/6000 dev server, dwarfed in shadow beside that huge robotic disc enclosure... and yank the server power cable right out of the dedicated power drop. And plug in their vacuum cleaner. And vacuum the floor.

We each looked at one-another, slowly, in bewilderment... and then went home, after a brief discussion on the way out the door.

You see, our team-leader wasn't with us that night; so before we left, we all agreed to come in late the next day. Very late indeed.

Sister = bee ( who isn't a stranger to Ubuntu)
Me = Cee

Bee: can I use your laptop?
Cee : why ? Use yours ,it's works fine.
Bee : no I want to use yours and I need to work with windows.
Cee: 🤯
Bee : my work can only be done using windows.
Cee : fine do whatever ( doesn't want to argue )
* Le bee opens MS word, and starts her work *
Cee : 😤😤Seriously?
Bee : I don't like libre
Cee : 😑😑😑^∞
* Few moments later *
Bee : my work is done ,you can have your laptop,btw it's updating.
Cee : 😑😑😑😑😑
* 2000 years later *
*Opens Ubuntu *
*Getting a weird bug*
*Tried to fix *
*Can't open OS files * 👏👏👏🎆
* Windows not shutdown properly *
* Opens windows *
* Not able to login via pin *
* Password ? not accepted *
* Changes outlook password *
* Please chose a password you haven't chosen before *
* Logs in *
* types old pin to change pin *
*You've entered wrong pin too many times *
*System hanging a lot *
* Removes pin *
* Gets huge mcAfee restart system popups , every 10 sec *
* Just shutdown , feels irritated for the rest of the day*

* Regrets dual booting, shd have wiped the windows partition 😫😫*

*Wonders,what the hell did my sister even do to my laptop ?*

Pranks again today. Mother of God the level of those pranks is becoming high as fuck.

Define high?
Having to debug shit at system (cron, firewalling, users, sometimes even digging through logs/dmesg) level because weird shit happens all day long.

This is upping my Linux skills a lot though! I love it 😍

Bossman freaks out over every little thing (ironically unless it's important).

Bossman also just set our papertrail 500 filter to forward him a copy of every "critical alert."

Fuck me.

He has zero technical knowledge and zero reading comprehension. He literally forwarded one and said "This has one lots of info in it. explain it to me." (It was a log usage notification. in plain English. It had maybe three numbers in it.)

There's lots of useless "500"s in the list we don't care about. API Guy used the finer as a debugging alert system, and peppered his logs with "500 internal hey look at this." In fact, none of the 500s were even interesting; all but one was spam.

All day I've been tending to freakouts and accusations of me not doing anything. Ugh.

So, some time ago, I was working for a complete puckered anus of a cosmetics company on their ecommerce product. Won't name names, but they're shitty and known for MLM. If you're clever, go you ;)

Anyways, over the course of years they brought in a competent firm to implement their service layer. I'd even worked with them in the past and it was designed to handle a frankly ridiculous-scale load. After they got the 1.0 released, the manager was replaced with some absolutely talentless, chauvinist cuntrag from a phone company that is well known for having 99% indian devs and not being able to heard now. He of course brought in his number two, worked on making life miserable and running everyone on the team off; inside of a year the entire team was ex-said-phone-company.

Watching the decay of this product was a sheer joy. They cratered the database numerous times during peak-load periods, caused $20M in redis-cluster cost overrun, ended up submitting hundreds of erroneous and duplicate orders, and mailed almost $40K worth of product to a random guy in outer mongolia who is , we can only hope, now enjoying his new life as an instagram influencer. They even terminally broke the automatic metadata, and hired THIRTY PEOPLE to sit there and do nothing but edit swagger. And it was still both wrong and unusable.

Over the course of two years, I ended up rewriting large portions of their infra surrounding the centralized service cancer to do things like, "implement security," as well as cut memory usage and runtimes down by quite literally 100x in the worst cases.

It was during this time I discovered a rather critical flaw. This is the story of what, how and how can you fucking even be that stupid. The issue relates to users and their reports and their ability to order.

I first found this issue looking at some erroneous data for a low value order and went, "There's no fucking way, they're fucking stupid, but this is borderline criminal." It was easy to miss, but someone in a top down reporting chain had submitted an order for someone else in a different org. Shouldn't be possible, but here was that order staring me in the face.

So I set to work seeing if we'd pwned ourselves as an org. I spend a few hours poring over logs from the log service and dynatrace trying to recreate what happened. I first tested to see if I could get a user, not something that was usually done because auth identity was pervasive. I discover the users are INCREMENTAL int values they used for ids in the database when requesting from the API, so naturally I have a full list of users and their title and relative position, as well as reports and descendants in about 10 minutes.

I try the happy path of setting values for random, known payment methods and org structures similar to the impossible order, and submitting as a normal user, no dice. Several more tries and I'm confident this isn't the vector.

Exhausting that option, I look at the protocol for a type of order in the system that allowed higher level people to impersonate people below them and use their own payment info for descendant report orders. I see that all of the data for this transaction is stored in a cookie. Few tests later, I discover the UI has no forgery checks, hashing, etc, and just fucking trusts whatever is present in that cookie.

An hour of tweaking later, I'm impersonating a director as a bottom rung employee. Score. So I fill a cart with a bunch of test items and proceed to checkout. There, in all its glory are the director's payment options. I select one and am presented with:

"please reenter card number to validate."

Bupkiss. Dead end.

OR SO YOU WOULD THINK.

One unimportant detail I noticed during my log investigations that the shit slinging GUI monkeys who butchered the system didn't was, on a failed attempt to submit payment in the DB, the logs were filled with messages like:

"Failed to submit order for [userid] with credit card id [id], number [FULL CREDIT CARD NUMBER]"

One submit click later and the user's credit card number drops into lnav like a gatcha prize. I dutifully rerun the checkout and got an email send notification in the logs for successful transfer to fulfillment. Order placed. Some continued experimentation later and the truth is evident:

With an authenticated user or any privilege, you could place any order, as anyone, using anyon's payment methods and have it sent anywhere.

So naturally, I pack the crucifixion-worthy body of evidence up and walk it into the IT director's office. I show him the defect, and he turns sheet fucking white. He knows there's no recovering from it, and there's no way his shitstick service team can handle fixing it. Somewhere in his tiny little grinchly manager's heart he knew they'd caused it, and he was to blame for being a shit captain to the SS Failboat. He replies quietly, "You will never speak of this to anyone, fix this discretely." Straight up hitler's bunker meme rage.

Some days I feel like I work in a different universe.

Last night our alerting system sent out a dept. wide email regarding a high number of errors coming from the web site.

Email shows the number of errors and a summary of the error messages.

Ex. 60 errors
59 Object reference not set to an instance of an object
1 The remote server returned an unexpected response: (413) Request Entity Too Large

Web team responds to the email..

"Order processing team's service is returning a 413 error. I'll fill out a corrective action ticket in the morning to address that error in their service. "

Those tickets are taken pretty seriously by upper mgmt, so I thought someone on the order processing team would point out the 1 error vs. 59 (coming from the web team's code).

Two hours go by, nobody responds, so I decide to jump into something that was none of my business.

"Am I missing something? Can everyone see the 59 null reference exceptions? The 413 exception only occurred once. It was the null reference exceptions that triggered the alert. Looking back at the logs, the site has been bleeding null reference exceptions for hours. Not enough for an alert, but there appears to be a bug that needs to be looked into."

After a dept. managers meeting this morning:

MyBoss: "Whoa..you kicked the hornets nest with your response last night."
Me: "Good. What happened?"
<Dan dept VP, Jake web dept mgr>
MyBoss: "Dan asked Jake if they were going to fix the null reference exceptions and Jake got pissed. Said the null reference errors were caused by the 413 error."
Me: "How does he know that? They don't log any stack traces. I don't think those two systems don't even talk to one another."
<boss laughs>
MyBoss:"That's what Dan asked!..oh..then Jake started in on the alert thresholds were too low, and we need to look into fixing your alerting code."
Me: "What!? Good Lord, tell me you chimed in."
MyBoss: "Didn't have to. Dan starting laughing and said there better be a ticket submitted on their service within the next hour. Then Jake walked out of the meeting. Oh boy, he was pissed."
Me: "I don't understand how they operate over there. It's a different universe.
MyBoss: "Since the alert was for their system, nobody looked at the details. I know I didn't. If you didn't respond pointing out the real problem, they would have passed the buck to the other team and wasted hours chasing a non-existent problem. Now they have to take resources away from their main project and answer to the VP for the delay. I'm sure they are prefixing your name right now with 'that asshole'"
Me: "Not the first, won't be the last."

I swear to god, I'm going to track down the dipshit who just made my day hilariously painful.

So here I am, finishing up this project that's been going on for what feels like an eternity, when I get an email "why doesn't order X show up in this other system?".

I mean, it's a common thing they can take 15 minutes to push across, so the usual quick glance and what do you know, it's just sitting there as if it's waiting to be pushed through, than an hour later... it's still there, so I start digging, maybe a data issue, nope looks all good, customer details, payment details, products...

just another order, jump on the logs and all looks fi......... wait.... why does this postcode have 3 digits and not 4 , Australia has 4 digit postal codes fyi, looks at order again, 3 digits, look at log, 3....hold on why's it only 3 digits, checks code, handled as string... ok..... where the fuck would it drop a digit.... frontend requires 4 digits, validation requires 4 digits... how the fuck did you get 3 digits in... I can't see anything anywhere that logically makes sense for this🤔

Drops address into google and it's a postcode starting with 0.

Jumps on DB and the fucker is an int in the postcode table. For all you playing at home 0123 <> 123

I don't know if I should feel bad, or impressed, it's been 7 years since this table was created, and 7 years before someone managed to live in one of these parts of the country with a leading 0.

QA didn't spot this years ago,
No one tested this exact scenario,
The damn thing isn't even documented as a required delivery area, but here we are!

Kudos good sir, you broke it! 🤜 🤛
You sir may get your order now!

Developer vs Tester

(Spoiler alert: developer wins)

My last developent was quite big and is now in our system testing department. So last week i got every 20 minutes a call from the tester, that something did not work as expected. For about 90% of the time i looked at the testing setup or the logs and told him, that the data is wrong or he used the tool wrong. After a couple of days i got mad because of his frequent interruptions. So I decided to make a list. Every time he came to me with an "error" i checked it and made a line for "User Error" or "Programming Error". He did not liked that much, because the User Error collum startet to grow fast:

User Errors: ||||| |||
Programming Errors: |||

Now he checks his testing data and the logs 3 times before he calls me and he hardly finds any "errors" anymore.

So I looked at our dashboard and noticed a banner mentioning scheduled maintenance set for 7:00 AM. And I thought to myself, "I never released an update, and even if I had, the maintenance would be performed 15 minutes after the build finished, not at 7:00 AM." So I emailed my coworkers, asking if they had put up the banner, no, no. I started pulling my hair out trying to figure out what caused this banner to be created. Was there some old job that was just now running? I combed through the server logs, thousands of entries later, and I found the banner was installed by some user with the IP 172.18.0.1...which was the local machine. I went through all the users on the system, running atq to see if anyone had jobs scheduled. And there was one job scheduled, under the root user. At that moment, I legit thought to myself, "have we been hacked? How is that possible?" It's wasn't! Then I looked under /var/spool/atjobs to see what the job actually was. And then I saw it. My weekly updater cron job had installed updates and had scheduled a maintenance window to reboot the system. And I smiled, realizing that my code was now sentient.

Worst hack/attack I had to deal with?

Worst, or funniest. A partnership with a Canadian company got turned upside down and our company decided to 'part ways' by simply not returning his phone calls/emails, etc. A big 'jerk move' IMO, but all I was responsible for was a web portal into our system (submitting orders, inventory, etc).

After the separation, I removed the login permissions, but the ex-partner system was set up to 'ping' our site for various updates and we were logging the failed login attempts, maybe 5 a day or so. Our network admin got tired of seeing that error in his logs and reached out to the VP (responsible for the 'break up') and requested he tell the partner their system is still trying to login and stop it. Couple of days later, we were getting random 300, 500, 1000 failed login attempts (causing automated emails to notify that there was a problem). The partner knew that we were likely getting alerted, and kept up the barage. When alerts get high enough, they are sent to the IT-VP, which gets a whole bunch of people involved.
VP-Marketing: "Why are you allowing them into our system?! Cut them off, NOW!"
Me: "I'm not letting them in, I'm stopping them, hence the login error."
VP-Marketing: "That jackass said he will keep trying to get into our system unless we pay him $10,000. Just turn those machines off!"
VP-IT : "We can't. They serve our other international partners."
<slams hand on table>
VP-Marketing: "I don't fucking believe this! How the fuck did you let this happen!?"
VP-IT: "Yes, you shouldn't have allowed the partner into our system to begin with. What are you going to do to fix this situation?"
Me: "Um, we've been testing for months already went live some time ago. I didn't know you defaulted on the contract until last week. 'Jake' is likely running a script. He'll get bored of doing that and in a couple of weeks, he'll stop. I say lets ignore him. This really a network problem, not a coding problem."
IT-MGR: "Now..now...lets not make excuses and point fingers. It's time to fix your code."
IT-VP: "I agree. We're not going to let anyone blackmail us. Make it happen."

So I figure out the partner's IP address, and hard-code the value in my service so it doesn't log the login failure (if IP = '10.50.etc and so on' major hack job). That worked for a couple of days, then (I suspect) the ISP re-assigned a new IP and the errors started up again.

After a few angry emails from the 'powers-that-be', our network admin stops by my desk.
D: "Dude, I'm sorry, I've been so busy. I just heard and I wished they had told me what was going on. I'm going to block his entire domain and send a request to the ISP to shut him down. This was my problem to fix, you should have never been involved."

After 'D' worked his mojo, the errors stopped.

Month later, 'D' gave me an update. He was still logging the traffic from the partner's system (the ISP wanted extensive logs to prove the customer was abusing their service) and like magic one day, it all stopped. ~2 weeks after the 'break up'.

My University distributes all worksheets over an online system. To access the files one has to download them each time first. So to get rid of all this annoying clicking in the browser, I just programmed a service, which logs onto the website ,crawls trough every folder, searches for new files and downloads them if they do not exist on my computer. Kind of proud as this is pretty much the first really useful program I developed lol

Have been trying to setup Netdata as a monitoring system for a while now and finally got it working!

Instead of the built-in webhooks I just did a curl to a url containing a php page/file which error logs the status and description (just for testing).

It took me way too long to get it to work but BAM.
Immediately made a new cpu load rule (one minute high load):
The satisfaction of getting an error message in the php logs containing my custom rule as warning and a minute later as critical 😍

Netdata ❤

TLDR - you shouldn't expect common sense from idiots who have access to databases.

I joined a startup recently. I know startups are not known for their stable architecture, but this was next level stuff.
There is one prod mongodb server.
The db has 300 collections.
200 of those 300 collections are backups/test collections.
25 collections are used to store LOGS!! They decided to store millions of logs in a nosql db because setting up a mysql server requires effort, why do that when you've already set up mongodb. Lol 😂
Each field is indexed separately in the log.
1 collection is of 2 tb and has more than 1 billion records.
Out of the 1 billion records, 1 million records are required, the rest are obsolete. Each field has an index. Apparently the asshole DBA never knew there's something called capped collection or partial indexes.

Trying to get approval to clean up the db since 3 months, but fucking bureaucracy. Extremely high server costs plus every week the db goes down since some idiot runs a query on this mammoth collection. There's one single set of credentials for everything. Everyone from applications to interns use the same creds.
And the asshole DBA left, making me in charge of handling this shit now. I am trying to fix this but am stuck to get approval from business management. Devs like these make me feel sad that they have zero respect for their work and inability to listen to people trying to improve the system.

Going to leave this place really soon. No point in working somewhere where you are expected to show up for 8 hours, irrespective of whether you even switch on your laptop.
Wish me luck folks.

Started part time job at a company, had to log my time on timesheets. Said fuck this and now the whole company logs their hours on a custom web based time logging system which I built.

Watch out for these fucking bug bounty idiots.

Some time back I got an email from one shortly after making a website live. Didn't find anything major and just ran a simple tool that can suggest security improvements simply loading the landing page for the site.

Might be useful for some people but not so much for me.

It's the same kind of security tool you can search for, run it and it mostly just checks things like HTTP headers. A harmless surface test. Was nice, polite and didn't demand anything but linked to their profile where you can give them some rep on a system that gamifies security bug hunting.

It's rendering services without being asked like when someone washes your windscreen while stopped at traffic but no demands and no real harm done. Spammed.

I had another one recently though that was a total disgrace.

"I'm a web security Analyst. My Job is to do penetration testing in websites to make them secure."
"While testing your site I found some critical vulnerabilities (bugs) in your site which need to be mitigated."
"If you have a bug bounty program, kindly let me know where I should report those issues."
"Waiting for response."

It immediately stands out that this person is asking for pay before disclosing vulnerabilities but this ends up being stupid on so many other levels.

The second thing that stands out is that he says he's doing a penetration test. This is illegal in most major countries. Even attempting to penetrate a system without consent is illegal.

In many cases if it's trivial or safe no harm no foul but in this case I take a look at what he's sending and he's really trying to hack the site. Sending all kinds of junk data and sending things to try to inject that if they did get through could cause damage or provide sensitive data such as trying SQL injects to get user data.

It doesn't matter the intent it's breaking criminal law and when there's the potential for damages that's serious.

It cannot be understated how unprofessional this is. Irrespective of intent, being a self proclaimed "whitehat" or "ethical hacker" if they test this on a site and some of the commands they sent my way had worked then that would have been a data breach.

These weren't commands to see if something was possible, they were commands to extract data. If some random person from Pakistan extracts sensitive data then that's a breach that has to be reported and disclosed to users with the potential for fines and other consequences.

The sad thing is looking at the logs he's doing it all manually. Copying and pasting extremely specific snippets into all the input boxes of hacked with nothing to do with the stack in use. He can't get that many hits that way.

After working for this company for only a couple years, I was tasked with designing and implementing the entire system for credit card encryption and storage and token management. I got it done, got it working, spent all day Sunday updating our system and updating the encryption on our existing data, then released it.

It wasn't long into Monday before we started getting calls from our clients not being able to void or credit payments once they had processed. Looking through the logs, I found the problem was tokens were getting crossed between companies, resulting in the wrong companies getting the wrong tokens. I was terrified. Fortunately I had including safe guards tying each token to a specific company, so they were not able to process the wrong cards. We fixed it that night.

And this, ladies and gentlemen, is why you need properly tested backups!

TL;DR: user blocked on old gitlab instance cascade deleted all projects the user was set as owner.

So, at my customer, collegue "j" reviews gitlab users and groups, notices an user who left the organisation
"j" : ill block this user
> "j" blocks user
> minutes pass away, working, minding our own business
> a wild team devops leader "k" appears
k: where are all the git projects?
> waitwut?.jpg
> k: yeah all git projects where user was owner of, are deleted
> j.feeling.despair() ; me.feeling.despair();
> checks logs on server, notices it cascade deletes all projects to that user
> lmgt log line
> is a bugreport reported 3(!) years ago
> gitlab hasnt been updated since 3 years
> gitlab system owner is not present, backup contact doesnt know shit about it
> i investigate further, no daily backup cron tasks, no backup has been made whatsoever.
> only 'backups' are on file system level, trying to restore those
> gitlab requires restore of postgres db
> backup does not contain postgres since the backup product does not support that (wtf???)
> fubar.scene
> filesystem restore finished...
> backup product did not back up all files from git tree, like none of refs were stored since the product cannot handle such filenames .. Git repo's completely broken

Fuck my life

When I was at university in my last semester of my bachelor's, I was doing a game programming paper and our last assignment was to group up and make a game. So I go with one of the guys I know and this other dude since his previous game was really neat. Then two randoms joined that from my first impressions of their games wasn't much at all (one guy made four buttons click and called it a game in Java when we had to make games in c++ and the other guy used an example game and semi modded it.

Anyways we get to brain storming, totally waste too much time getting organised because the guy that volunteered (4 buttons guy) was slow to getting things sorted. Eventually we get to making the game and 4 buttons guy hasn't learnt how to use git, I then end up spending 3 hours over Skype explaining to him how to do this. He eventually learns how to do things and then volunteers to do the AI for the game, after about a week (this assignment is only 5 weeks long) he hasn't shown any progress, we eventually get to our 3rd week milestone no progress from him and the modder, with only three classes left we ask them both to get stuff done before a set deadline (modder wanted to do monsters and help 4 buttons with AI) both agreed and deadline rolls up and no work is shown at all, modest shows up extremely late and shows little work.

4 buttons guy leaves us a Skype message the day of our 2nd to last class,, saying he dropped the paper...

Modder did do some work but he failed to read all the documentation I left him (the game was a 2d multiplayer crafting game, I worked so hard to make a 2d map system with a world camera) he failed to read everything and his monsters used local coordinates and were stuck on screen!

With about a week left and not too many group meetings left we meet up to try and get stuff done, modder does nothing to help, the multiplayer is working my friend has done the crafting and weapon system and the map stuff is working out well. We're missing AI and combat, with our last few hours left we push to get as much stuff done, I somehow get stuck doing monster art, AI is done by the other two and I try to getting some of the combat and building done.

In the end we completely commented all of modders work because well it made us look bad lol. He later went to complain to my free claiming I did it and was a douchebag for doing so. We had to submit our developer logs and the three of us wrote about how shitty it was to deal with these two.

We tried out best not to isolate ourselves from them and definitely tried to help but we were swamped with our other assignments and what we had to work on.

In the end leaving and not helping right when the deadline is close was what I call the most shittiest thing team mates can do, I think sticking together even if we were to fail was at least a lot better.

My Sunday Morning until afternoon. FML. So I was experiencing nightly reboots of my home server for three days now. Always at 3:12am strange thing. Sunday morning (10am ca) I thought I'd investigate because the reboots affected my backups as well. All the logs and the security mails said was that some processes received signal 11. Strange. Checked the periodics tasks and executed every task manually. Nothing special. Strange. Checked smart status for all disks. Two disks where having CRC errors. Not many but a couple. Oh well. Changing sata cables again 🙄. But those CRC errors cannot be the reason for the reboots at precisely the same time each night. I noticed that all my zpools got scrubbed except my root-pool which hasn't been scrubbed since the error first occured. Well, let's do it by hand: zpool scrub zroot....Freeze. dafuq. Walked over to the server and resetted. Waited 10 minutes. System not up yet. Fuuu...that was when I first guessed that Sunday won't be that sunny after all. Connected monitor. Reset. Black screen?!?! Disconnected all disks aso. Reset. Black screen. Oh c'moooon! CMOS reset. Black screen. Sigh. CMOS reset with a 5 minute battery removal. And new sata cable just in cable. Yes, boots again. Mood lightened... Now the system segfaults when importing zroot. Good damnit. Pulled out the FreeBSD bootstick. zpool import -R /tmp zroot...segfault. reboot. Read-only zroot import. Manually triggering checksum test with the zdb command. "Invalid blckptr type". Deep breath now. Destroyed pool, recreated it. Zfs send/recv from backup. Some more config. Reboot. Boots yeah ... Doesn't find files??? Reboot. Other error? Undefined symbols???? Now I need another coffee. Maybe I did something wrong during recovery? Not very likely but let's do it again...recover-recover. different but same horrible errors. What in the name...? Pulled out a really old disk. Put it in, boots fine. So it must be the disks. Walked around the house and searched for some new disks for a new 2 disk zfs root mirror to replace the obviously broken disks. Found some new ones even. Recovery boot, minimal FreeBSD Install for bootloader aso. Deleted and recreated zroot, zfs send/recv from backup. Set bootfs attribute, reboot........
It works again. Fuckit, now it is 6pm, I still haven't showered. Put both disks through extensive tests and checked every single block. These disks aren't faulty. But for some reason they froze my system in a way so that I had to reset my BIOS and they had really low level data errors....? I Wonder if those disks have a firmware problem? So that was most of my Sunday. Nice, isn't it? But hey: calm sea won't make a good sailor, right?

In the before time (late 90s) I worked for a company that worked for a company that worked for a company that provided software engineering services for NRC regulatory compliance. Fallout radius simulation, security access and checks, operational reporting, that sort of thing. Given that, I spent a lot of time around/at/in nuclear reactors.

One day, we're working on this system that uses RFID (before it was cool) and various physical sensors to do a few things, one of which is to determine if people exist at the intersection of hazardous particles, gasses, etc.

This also happens to be a system which, at that moment, is reporting hazardous conditions and people at the top of the outer containment shell. We know this is probably a red herring or faulty sensor because no one is present in the system vs the access logs and cameras, but we have to check anyways. A few building engineers climb the ladders up there and find that nothing is really visibly wrong and we have an all clear. They did not however know how to check the sensor.

Enter me, the only person from our firm on site that day. So in the next few minutes I am also in a monkey suit (bc protocol), climbing a 150 foot ladder that leads to another 150 foot ladder, all 110lbs of me + a 30lb diag "laptop" slung over my shoulder by a strap. At the top, I walk about a quarter of the way out, open the casing on the sensor module and find that someone had hooked up the line feed, but not the activity connection wire so it was sending a false signal. I open the diag laptop, plug it into the unit, write a simple firmware extension to intermediate the condition, flash, reload. I verify the error has cleared and an appropriate message was sent to the diagnostic system over the radio, run through an error test cycle, radio again, close it up. Once I returned to the ground, sweating my ass off, I also send a not at all passive aggressive email letting the boss know that the next shift will need to push the update to the other 600 air-gapped, unidirectional sensors around the facility.

Setting Newyear's resolution to 4k..

failed: could not find X window system.
Please check the configuration logs.

So, at the start of November last year I completed a big system for a client. It took me months to complete.

Most frustrating was the sheer amount of pressure the client applied to get it completed. Emails every day, phone calls where the client was "checking" on my progress etc etc. All the annoying stuff.

Only plus side was the fact they paid in full a few after is was completed.

I've just released the system is still on a test AWS account and I haven't heard from the client for well over a month now.

I've just logged into the system and took a look at the logs. The client logged in once the day it was completed and hasn't done anything since.

I mean what was the point of all the pressure if they were just going to let the thing gather dust?

I'm pretty annoyed to be honest as I experienced a few fairly borderline stressful months due to that project.

Ah well, the image below was me after looking at the system logs :)

The application has a system for sending reports and errors.
Client: "I have a problem with the app, I can not log in (android phone)"
I check the logs: "sent from iPhone"

(the person has no other account and registered telephone)

24th, Christmas: BIND slaves decide to suddenly stop accepting zone transfers from the master. Half a day of raging and I still couldn't figure out why. dig axfr works fine, but the slaves refuse a zone update according to tcpdump logs.

25th, 2nd day: A server decides to go down and take half my network with it. Turns out that a Python script managed to crash the goddamn kernel.

Thank you very much technology for making the Christmas days just a little bit better ❤️

At least I didn't have anything to do during either days, because of the COVID-19 pandemic. And to be fair, I did manage to make a Telegram bot with fancy webhooks and whatnot in 5MB of memory and 18MB of storage. Maybe I should just write the whole thing and make another sacred temple where shitty code gets beaten the fuck out of the system. Terry must've been onto something...

Java dev here. I rewrote an app and replaced a system call to ssh with a modern jaxrs post for uploading a file and (new) some additional data.
I even used a stream.

1 hour in production, first client doesn't get his file. Log says OutOfMemoryError: heap.

Me: wtf? I already use streams.
Looking at the Jersey library. Docs say nothing. An issue from 2013 says: oh if you silly don't use the Apache httpclient addon, we disable chunking and buffer the whole body, because our tests fail with the jdk included http client otherwise.

Me: meh.

No warning in the logs. Thank you soooooo much! Who could have known?

This was some time ago. A Legendary bug appeared. It worked in the dev environment, but not in the test and production environment.

It had been a week since I was working on the issue. I couldn't pinpoint the problem. We CANNOT change the code that was already there, so we needed to override the code that was written. As I was going at it, something happened.

---
Manager: "Hey, it's working now. What did you do?"
Me: *Very confused because I know I was nowhere close to finding the real source of the problem* Oh, it is? Let me check.
Also me: *Goes and check on the test and prod environment and indeed, it's already working*
Also me to the power of three: *Contemplates on life, the meaning of it, of why I am here, who's going to throw out the trash later, asking myself whether my buddies and I will be drinking tonight, only to realize that I am still on the phone with my manager*
Me again: "Oh wow, it's working."
Manager: "Great job. What were the changes in the code?"
Me: "All I did was put console logs and pushed the changes to test and prod if they were producing the same log results."
Manager: "So there were no changes whatsoever, is that what you mean?"
Me: "Yep. I've no idea why it just suddenly worked."
Manager: "Well, as long as it's working! Just remove those logs and deploy them again to the test and prod environment and add 'Test and prod fix' to the commit comment."
Me: "But what if the problem comes up again? I mean technically we haven't resolved the issue. The only change I made were like 20 lines of console logs! "
Manager: "It's working, isn't it? If it becomes a problem, we'll work it out later."
---

I did as I was told, and Lo and Behold, the problem never occurred again.

Was the system playing a joke on me? The system probably felt sorry for me and thought, "Look at this poor fucker, having such a hard time on a problem he can't even comprehend. That idiotic programmer had so many sleepless nights and yet still couldn't find the solution. Guess I gotta do my job and fix it for him. I'm the only one doing the work around here. Pathetic Homo sapiens!"

Don't get me wrong, I'm glad that it's over but..

What the fuck happened?

Want to make someone's life a misery? Here's how.

Don't base your tech stack on any prior knowledge or what's relevant to the problem.

Instead design it around all the latest trends and badges you want to put on your resume because they're frequent key words on job postings.

Once your data goes in, you'll never get it out again. At best you'll be teased with little crumbs of data but never the whole.

I know, here's a genius idea, instead of putting data into a normal data base then using a cache, lets put it all into the cache and by the way it's a volatile cache.

Here's an idea. For something as simple as a single log lets make it use a queue that goes into a queue that goes into another queue that goes into another queue all of which are black boxes. No rhyme of reason, queues are all the rage.

Have you tried: Lets use a new fangled tangle, trust me it's safe, INSERT BIG NAME HERE uses it.

Finally it all gets flushed down into this subterranean cunt of a sewerage system and good luck getting it all out again. It's like hell except it's all shitty instead of all fiery.

All I want is to export one table, a simple log table with a few GB to CSV or heck whatever generic format it supports, that's it.

So I run the export table to file command and off it goes only less than a minute later for timeout commands to start piling up until it aborts. WTF. So then I set the most obvious timeout setting in the client, no change, then another timeout setting on the client, no change, then i try to put it in the client configuration file, no change, then I set the timeout on the export query, no change, then finally I bump the timeouts in the server config, no change, then I find someone has downloaded it from both tucows and apt, but they're using the tucows version so its real config is in /dev/database.xml (don't even ask). I increase that from seconds to a minute, it's still timing out after a minute.

In the end I have to make my own and this involves working out how to parse non-standard binary formatted data structures. It's the umpteenth time I have had to do this.

These aren't some no name solutions and it really terrifies me. All this is doing is taking some access logs, store them in one place then index by timestamp. These things are all meant to be blazing fast but grep is often faster. How the hell is such a trivial thing turned into a series of one nightmare after another? Things that should take a few minutes take days of screwing around. I don't have access logs any more because I can't access them anymore.

The terror of this isn't that it's so awful, it's that all the little kiddies doing all this jazz for the first time and using all these shit wipe buzzword driven approaches have no fucking clue it's not meant to be this difficult. I'm replacing entire tens of thousands to million line enterprise systems with a few hundred lines of code that's faster, more reliable and better in virtually every measurable way time and time again.

This is constant. It's not one offender, it's not one project, it's not one company, it's not one developer, it's the industry standard. It's all over open source software and all over dev shops. Everything is exponentially becoming more bloated and difficult than it needs to be. I'm seeing people pull up a hundred cloud instances for things that'll be happy at home with a few minutes to a week's optimisation efforts. Queries that are N*N and only take a few minutes to turn to LOG(N) but instead people renting out a fucking off huge ass SQL cluster instead that not only costs gobs of money but takes a ton of time maintaining and configuring which isn't going to be done right either.

I think most people are bullshitting when they say they have impostor syndrome but when the trend in technology is to make every fucking little trivial thing a thousand times more complex than it has to be I can see how they'd feel that way. There's so bloody much you need to do that you don't need to do these days that you either can't get anything done right or the smallest thing takes an age.

I have no idea why some people put up with some of these appliances. If you bought a dish washer that made washing dishes even harder than it was before you'd return it to the store.

Every time I see the terms enterprise, fast, big data, scalable, cloud or anything of the like I bang my head on the table. One of these days I'm going to lose my fucking tits.

Fulfillment company: the order you put in our system is wrong, the hours we needed to correct this will be charged to you (Red: or rather my employer)

Me: *Checks logs and our server - finds out the order was made in their own webform/webapp*

Me: hey how come I can put in values in your webform that should not pass sanity checks, thats weird (in this case it was a product w/ a quantity of 0)

Fulfillment company: we don't do sanity checks or validation, we just find out when shit crashes and burns, nothing weird about that

Me: WTF

So apple wants you to open your system logs with Photoshop, truly? So the fuck up and crashes looks more "beautiful"?

Saturday 9.00 AM. I was sleeping, my colleague (on holiday) sent me a text: "We got a problem on our system, probably we ran out of space". I checked the log and found out that several cron jobs failed due to not enough space on the disk. I started deleting some unnecessary logs (we're paranoid) and ended up to squeeze the vm like a lemon to save some space. Sent an email to the sysadmin, "We got to add more space ASAP, users are getting 500 errror for almost everything". Silence. I thought to myself: "Until monday we're safe..". I did a df (96%) and sent a screen to the sysadmin, just to be sure that we understood each other. Finally monday comes, nobody worries about the issue. At noon I literally takled the guy of IT dept. "Yeah, we read your email. I think the sysadmin didn't take you seriously". "Why? Which part of 'we're running out of space' isn't serious?!!!". "He just told me that we have unlimited space on that vm". Unlimited space...sure.... "Right.....the disk is at 96%, buuuuut if he said so No news to worry. Don't call me if everything burns. Have a good day!!!"

So, I work in a game development studio, right?
We're trying to launch the title on as many platforms as reasonable, because as a social VR app we're kinda rowing upstream.
So far, Steam and Oculus have been fairly reasonable, if oddly broken and inconsistent.
Enter store 3.
Basically no in-game transaction support (our asking prompted them to *start* developing it. No, it's not very complete). No patch-update system (You want an update? Gotta download the whole fsckin' thing!). No beta-testing functionality for most of their stuff ("Just write the code like the example, it will work, trust us!"). No tools besides the buggy SDK (Wanna upload that new build? Say hello to this page in your web browser!).
So, in other words: Fun.

We've been trying to get actively launched for two months now. Keep in mind that the build has been up on Steam and Oculus for over a year and half a year (respectively), so the actual binary functionality is, presumably fine.

The best feedback we get back tends to be "Well, when we click the Launch button it crashes, so fail."
Meanwhile we're going back and forth, dealing with other-side-of-the-world timezone lag, trying to figure out what is so different from their machines as ours. Eventually we get them to start sending logs (and no, Windows Event logs are not sufficient for GAMES, where did you even get that idea????) except the logs indicate that the program is getting killed so terribly that the engine's built-in crash handler can't even kick in to generate memory dumps or even know it died.

All this boils down to today, where I get a screenshot of their latest attempt.

I just can't even right now.

Today's GDPR-Bullshittery.

So we are using an open source remote update system for updating our embedded devices.
And today we learned that, that system logs ip-adress'. And low and behold mr.GDPR says that is a no no.

So either we completely drops it, finds a new update system and implements it..
Sift through all the source code of the update system "fix" it and recompile it.
Or we setup a Man in the middle attack on ourselves. To mask the ip-adress'.

GDPR encouraging hacking ourselves I fucking love it!

At the institute I did my PhD everyone had to take some role apart from research to keep the infrastructure running. My part was admin for the Linux workstations and supporting the admin of the calculation cluster we had (about 11 machines with 8 cores each... hot shit at the time).

At some point the university had some euros of budget left that had to be spent so the institute decided to buy a shiny new NAS system for the cluster.

I wasn't really involved with the stuff, I was just the replacement admin so everything was handled by the main admin.

A few months on and the cluster starts behaving ... weird. Huge CPU loads, lots of network traffic. No one really knows what's going on. At some point I discover a process on one of the compute nodes that apparently receives commands from an IRC server in the UK... OK code red, we've been hacked.

First thing we needed to find out was how they had broken in, so we looked at the logs of the compute nodes. There was nothing obvious, but the fact that each compute node had its own public IP address and was reachable from all over the world certainly didn't help.
A few hours of poking around not really knowing what I'm looking for, I resort to a TCPDUMP to find whether there is any actor on the network that I might have overlooked. And indeed I found an IP adress that I couldn't match with any of the machines.

Long story short: It was the new NAS box. Our main admin didn't care about the new box, because it was set up by an external company. The guy from the external company didn't care, because he thought he was working on a compute cluster that is sealed off behind some uber-restrictive firewall.

So our shiny new NAS system, filled to the brink with confidential research data, (and also as it turns out a lot of login credentials) was sitting there with its quaint little default config and a DHCP-assigned public IP adress, waiting for the next best rookie hacker to try U:admin/P:admin to take it over.

Looking back this could have gotten a lot worse and we were extremely lucky that these guys either didn't know what they had there or didn't care.

All your disk are belong to us!

TLDR; I was editing the wrong file, let's go to bed.

We have this huge system that receives data from an API endpoint, does a whole bunch of stuff, going through three other servers, and then via some calculation based on the data received from the UI, and data received from the endpoint, it finally sends the calculated fields to the UI via websocket.

Poor me sitting for over 4 hours debugging and changing values in the logic file trying to understand why one of the fields ends up being null.

Of course every change needs a reboot to all the 4 servers involved, and a hard refresh of the UI.

I even tried to search for the word null in that file, but to no avail.

After scattering hundreds of console logs, and pulling my hair out, I found out that I am editing the wrong file.

I guess it's time for some sleep.

Client: There was an order issue... please look into it.

Me: Okay, whats the order reference id.

Client: I accidently deleted the order. Please check your system logs. It was a 20 pound order.

😠

Me (4 hours later): It was a 18 pound order with free shipping... and it failed cause they never finished the 3D secure.

😑

So I just got asked for a quote for developing an app for a client's friend. He wanted an app that requires me to build let's just say a combination of what you see on uber with the live tracking of your uber driver, seeing all cars around your location and determining the closest one (It wasn't necessarily cars) plus profiles and another app for another set of users (I can easily make this one and determine the logged in user and in turn tailor the features for that user but they wanted two). An admin portal also was included and I had to do various integrations with Google maps. In app purchases was also necessary. Logs as the app has to keep track of all activities basically. A wallet feature was also to be implemented, scheduling, rating and complains section was also something requested and finally a mini accounting system was also to be developed. I was going to do this singlehandly as a freelancer. Obviously this is a lot of work. I also gave them a timeline of about 3 months for development. Which meant I was going to be putting all my time into developing this. Front end and backend for the app and front-end and backend for the server and database architecture. I charged them $10,000 not only for the work but also because they were going to be making money off of the app. They go "wow and why does it cost so much"...Judging from their reactions I don't think they will move further with this with me because of costs...😂 I can't even begin to wonder why they think that isn't a fair price. I have learnt from previous work before that you always state a cost for which you are absolutely sure you would want to work for else you would start doing the work and once you see how little you are being paid for so much work you end up hating the work and completing it ends up being a difficult task.

So, we (I'm the backend guy and work with a UI dev) are building this product portfolio management tool for our client and they have a set of 250 users. The team has two point of contacts for the 250 users who maintain the master data, help users with data quality, tool guidance, reporting and other stuff. So one day one of these two support users come to me and say : Hey I'm not able to add new transactions coz a customer is missing.

We have the provision to create / maintain customers.

I check the production DB, application code, try creating the customer and then the transaction, everything works perfectly fine.

I ask the user for a screen sharing session, the user starts reproducing the error like this :

We have a 3 system landscape - Dev / Test and Prod

U : Logs into the test system url, creates the customer.

U : Points out the toast saying customer creation is successful.

U : opens a new tab, opens the production system, tries creating the transaction, searches for the customer and says " see !! cant find the customer here ! the master data management apps never work !! "

FML?.

Manager X: (logs a support ticket) "Agent is unable to access system using the password provided."

Me: "You're going to have to narrow it down a little, we have over 1000 active agents."

I hate the support side of my job...

Pentesting for undisclosed company. Let's call them X as to not get us into trouble.
We are students and are doing our first pentest at an actual company instead of assignments at school. So we're very anxious. But today was a good day.
We found some servers with open ports so we checked a few of them out. I had a set of them with a bunch of open ports like ftp and... 8080. Time to check this out.
"please install flash player"... Security risk 1 found!

System seemed to be some monitoring system. Trying to log in using admin admin... Fucking works. Group loses it cause the company was being all high and mighty about being secure af. Other shit is pretty tight though.

Able to see logs, change password, add new superuser, do some searches for USERS_LOGGEDIN_TODAY! I shit you not, the system even had SUGGESTIONS for usernames to search for. One of which had something to do with sftp and auth keys. Unfortunatly every search gave a SQL syntax error. Used sniffing tools to maybe intercept message so we could do some queries of our own but nothing. Query is probably not issued from the local machine.

Tried to decompile the flash file but no luck. Only for some weird lines and a few function names I presume. But decompressing it and opening it in a text editor allowed me to see and search text. No GET or POST found. No SQL queries or name checks or anything we could think of.

That's all I could do for today. So we'll have to think of stuff for next week. We've already planned xss so maybe we can do that on this server as well.

We also found some older network printers with open telnet. Servers with a specific SQL variant with a potential exploit to execute terminal commands and some ftp and smb servers we need to check out next week.

Hella excited about this!

If you guys have any suggestions let us know. We are utter noobs when it comes to this.

TLDR; WINE+me=system binaries gone. (HOWTHEFUCKDIDIDOTHAT) Kernel panic. Core program files gone. I'll never have it fixed right. Will backup, then install fedora tomorrow.

I really like games and I'm sure there are many of you who can relate. Imagine my perpetual pain, being on the job hunt, no money, and only my Linux laptop for games. (It's only Linux because of a stupid accident and a missing windows installation disk, partly explained in a previous rant). My stack of games my dad and I have played over the years, going back to populous and before, looked light enough for my laptop to run them smoothly. I wanted to see if I could get one to work. My eyes settled on simcity 4 and Sid Meier's railroad tycoon, 13 and 10 years old, respectively. Simcity didn't work as many times as I tried following online instructions. Disk 1 went fine. Disk 2 showed up as Disk 1. Didn't think much of it, so long as the computer could read the contents. I downloaded playonlinux as that could apparently do the complex stuff for me. Didn't work. I gave up with it after an hour and a half.

Next was railroads. Put the disk in aaaand it says SimCity disk 1 is in the tray. Fuck right off, thank you very much. Eject, put back, reject, eject, fiddle in wineconfig, eject, more of this, and voilà it read as railroads :) Ran autoplay.exe with wine, followed instructions, installed it, and it worked! Chose single player, then the map and setting, pressed play, and all the models of the buildings and track were floating in the air over a green plane, the UI is weird and the map doesn't represent anything but trains. All the fkin land is gone, laying track is gonna be a ballache.

I quit it and decided bedtime.
Ctrl+alt+t
sudo shutdown -h now
shutdown not found.
sudo reboot
reboot not found
Que?
Nope, I don't like this.
Force choked my laptop by the power button. Turned it on again.
Lines of text appear.
Saw a phrase I've only ever seen on Mr Robot.
Kernel panic.
Nooooo thanks, not today, this is fiction.
I turned it off and on. Same thing. I read the logs and some init files couldn't be found. I got the memory stick I used to install mint in the first place and booted from that. I checked the difference between my stick's bin and sbin and the laptop's, and it was indeed missing binaries. Fuck knows what else has happened, I only wanted to play games but now I don't know what is or isn't in my computer. How can I trust what's on it now?
I go downstairs and tell my dad. He says something about rpm, but this is Linux so it won't work. I learn that binaries can be copied over, so maybe I can fix it.
Go upstairs again, decide not to fix it. Fedora is light, has a good rep for security, and is even more difficult to get games on, which is my vice. There are more reasons, but the overriding one is that I'm spooked by the fact that something I did went into and removed system binaries, maybe even altered others, so I want something I'm less likely to do that with. Also my fellow cs students used to hate on it but my dad uses and recommended it so I want to try it.

Also, seriously, fuck wine/PlayOnLinux/my inability to follow instructions(?)/whatever demons haunt me. Take your pick, at least one if not more is to blame and I can't tell which, but it's prooooobably the third one.

It's going to be 16 hours before I touch my laptop again, comments before I backup then install fedora are welcome, especially if they persuade me to do differently.

P.S thanks for reading this mind dump of a post, I'm writing while it's fresh but I'm tired AF.

My biggest dev epiphany was also my dumbest one. We were working on a payment system for a roadside rescue company where an employee would register payments "in the field".

The challenge was automating input with typeahead and autocompletes in order to lessen the workload as manual input had to be an absolute minimum; this will be used by truck drivers/mechanics as they are trying to hurry to the next customer who has been waiting for 3 hours longer than we said we'd take.

We managed to make the invoice path first (customer has not paid, employee logs personalia needed for billing), but when it came to "paid on site" we almost upended the entire system trying to find a way to fetch user personalia outside of the invoice path.

Neither of us realized it during the days we were banging our heads against it. Realizing we don't need to make an invoice for a job that has been paid for was equal parts relief and utter embarrassment.

Probably my greatest lesson in how important it is to pull my head out of the code once in a while, and to ask myself what I'm trying to do and why.

The conversations that come across my DevOps desk on a monthly basis.... These have come into my care via Slack, Email, Jira Tickets, PagerDuty alerts, text messages, GitHub PR Reviews, and phone calls. I spend most of my day just trying to log the work I'm being asked to do.

From Random People:
* Employee <A> and Contractor <B> are starting today. Please provision all 19 of their required accounts.
* Oh, they actually started yesterday, please hurry on this request.

From Engineers:
* The database is failing. Why?
* The read-only replica isn't accepting writes. Can you fix this?
* We have this new project we're starting and we need you to set up continuous integration, deployment, write our unit tests, define an integration test strategy, tell us how to mock every call to everything. We'll need several thousand dollars in AWS resources that we've barely defined. Can you define what AWS resources we need?
* We didn't like your definition of AWS resources, so we came up with our own. We're also going to need you to rearchitect the networking to support our single typescript API.
* The VPN is down and nobody can do any work because you locked us all out of connecting directly over SSH from home. Please unblock my home IP.
* Oh, looks like my VPN password expired. How do I reset my VPN password?
* My GitHub account doesn't have access to this repo. Please make my PR for me.
* Can you tell me how to run this app's test suite?
* CI system failed a build. Why?
* App doesn't send logs to the logging platform. Please tell me why.
* How do I add logging statements to my app?
* Why would I need a logging library, can't you just understand why my app doesn't need to waste my time with logs?

From Various 3rd party vendors:
* <X> application changed their license terms. How much do you really want to pay us now?

From Management:
* <X> left the company, and he was working on these tasks that seem closely related to your work. Here are the 3 GitHub Repos you now own.
* Why is our AWS bill so high? I need you to lower our bill by tomorrow. Preferably by 10k-20k monthly. Thanks.
* Please send this month's plan for DevOps work.
* Please don't do anything on your plan.
* Here's your actual new plan for the month.
* Please also do these 10 interruptions-which-became-epic-projects

From AWS:
* Dear AWS Admin, 17 instances need to be rebooted. Please do so by tomorrow.
* Dear AWS Admin, 3 user accounts saw suspicious activity. Please confirm these were actually you.
* Dear AWS Admin, you need to relaunch every one of your instances into a new VPC within the next year.
* Dear AWS Admin, Your app was suspiciously accessing XYZ, which is a violation of our terms of service. You have 24 hours to address this before we delete your AWS account.

Finally, From Management:
* Please provide management with updates, nobody knows what you do.

From me:
Please pay me more. Please give me a team to assist so I'm not a team of one. Also, my wife is asking me to look for a new job, and she's not wrong. Just saying.

Ok so riddle me this. The service for an application were required to run to send clients insurance through (as per government regulations) was working fine all day working super fast. Rare but awesome. I get a call one hour prior to the office closing (I don't work weekdays) and I am told that all of a sudden insurance isn't sending.

My mind goes right to this fu**ing process. Sure enough it's stopped on the server. Well shit ok. I click start..... Nothing. I kill it from task manager.... Nothing. "SERVICE CAN'T START"

I'm like ok that's fine let's check event logs.... Nothing. No problem let's just run it not in a service container and see if there's an error. NOPE IT DOESNT LET ME.

Okok so that's cool let's just try reinstalling the app. NOPE CAN'T DO THAT WITHOUT RESTARTING THE WHOLE FUCKING SERVER WHICH BRINGS THE ENTIRE OFFICES MANAGEMENT SYSTEM OFFLINE BECAUSE THIS FUCKING APP NEEDS TO BE ON THE SAME GODDAMN SERVER.

In last episode of "How SystemD screwed me over", we talked about Systemd's PrivateTMP and how it stopped me from generating SSL certificates.

In today's episode - SystemD vs CGroups!

Mister Pottering and his team apparently felt that CGroups are underused (As they can be quite difficult to set up), and so decided to integrate them into SystemD by default. As well as to provide a friendlier interface to control their values.

One can read about these interactions in the manual page "systemd.resource-control"

All is cool so far. So what happened to me today?

Imagine you did a major system release upgrade of a production server, previously tested on a standalone server. This upgrade doesn't only upgrade the distribution however, it also includes the switch from SysVInit to SystemD. Still, everything went smooth before, nothing to worry now then, right? Wrong.

The test server was never properly stress-tested. This would prove to be an issue.

When the upgrade finishes, it is 4 AM. I am happy to go to bed at last. At 6 AM, however, I am woken up again as the server's webservices are unavailable, and the machine is under 100% CPU load. Weird, I check htop and see that Apache now eats up all 32 virtual cores. So I restart it, casting it off to some weird bug or something as the load returns to normal.

2 hours later, however, the same situation occurs. This time, I scour all the logs I can, and find something weird - Many mentions that Apache couldn't create a worker thread? That's weird.

Several hours of research and tinkering later, I found out the following:
1 - By default, all processes of a system that runs SystemD are part of several CGroups. One of these CGroups is the PID CGroup, meant to stop a runaway process from exhausting all PIDs/TIDs of a system.

This limit is, by default, set to a certain amount of the total available PIDs. If a process exhausts this limit, it can no longer perform operations like fork().

So now, I know the how and why, but how should I solve this? The sanest option would be to get a rough estimate of just how many threads the Apache webserver might need. This option, though, is harder, than apparent. I cannot just take the MaxRequestsWorkers number... The instance has roughly double the amount of threads already. The cause being, as I found out, the HTTP/2 module, which spawns additional threads that do not count towards this limit. So I have no idea what limit to set.

Or I could... Disable the limit for just the webserver via the TasksAccounting switch. I thought this would work. And it did seem to... Until I ran out of TIDs again - Although systemctl status apache2.service no longer reported the number of tasks or a task limit of the process, the PID CGroup stayed set to the previous limit. Later I found out that I can only really disable the Task Accounting for all the units of a given slice and its parents.

This, though, systemctl somewhat didn't make apparent (And I skimmed the manual, that part was my fault)

So... The only remaining option I had was to... Just set the limit to infinite. And that worked, at last.

It took me several hours to debug this issue. And I once again feel like uninstalling systemd again, in favor of sysvinit.

What did I learn? RTFM, carefully, everything is important, it is not enough to read *half* the paragraph of a given configuration option...

Oh, and apache + http/2 = huge TID sink.

FML, somebody here or somewhere wrote that al customers lie..

Just been a witnes to that.. Over skype (mind I reminded them to write to jira on several occasions so others can help if I am out of office) feature xy is not working.. I log in to server, I see no logs of person a doing anything with our system, let alone use the xy feature... Well duh, of course it doesn't work, it's not a freakin mind reader.. :/

Next time no help, no log checking, no nothing until they provide ss of what exactly they were doing.. :/ Fuuuuu....

Let me start this off by stating I'm a Java dev, and a noob with C++.

Thought it'd be cool to learn some OpenCL, since I want to do some maths stuff and why not learn something new.

So I sat down, installed Nvidia proprietary drivers, broke my x-org server, purged, reinstalled, rebooted and after a while I got stuff sorted out.

Then on to my IDE. I use CLion and it uses Cmake. C++ noob knows shit about Cmake, so struggle for two hours trying to figure out wtf is going on with the OpenCL libs and why they're only partially detected. Fml.

Finally, everything is configured and I'm set. I start working on a Hello World program using OpenCL. Finish it in 20 mins, all good. No output. Do some googling, check my program a million times. Nothing wrong here. Check the kernel, everything as in the tutorial.

I start checking error codes after a while reported by OpenCL (which I had no clue was a thing) and I get some code saying the program was not created properly (to run the kernel). No fucking clue what's up with that. Google around, find another tutorial, rewrite my code in case I'm using outdated code or something. Nothing.

Fast forward an hour, I find out that OpenCL has logs! So I grab some code from the website I found it on, and voila, I finally get some info on what's going on.

Get a load of this bs.

In the kernel file, so that OpenCL knows that it's a function to run, you have to put __kernel. But in all the places I read, it said to put it as _kernel.

Add the underscore, compile, run and everything is perfect.

Then I tried just putting 'kernel'. Also compiles and runs fine.

Two hours hours and my program was fixed by adding an underscore. IF ONLY C++ GAVE AN INDICATION OF WHAT BLEW UP INSTEAD OF SITTING BACK AND BEING LIKE "oh wow man feels bad, work some magic and try again" THEN THIS WOULD NOT HAVE TAKEN SO LONG.

Then again, it was OpenCL that was being shitty with its styling enforcement or whatever the hell the underscore business is. But screw it. C++ eats shit too for this. Sure, maybe Java babies you by giving you the exact error and position that the error took place at. But at least that way you don't waste hours of your life chasing invisible bugs 😠😠

I'm going to eat some food... Too much energy was consumed fighting the system... Then I'll get back to OpenCL because 😇 but that doesn't make it less bs.

My day couldn't start in a worse day.

We are having a demo this week and I worked yesterday after hours to get the product ready. Tested everything and we were all set for the demo content.

Today we installed the new version with my fixes and nothing worked. Today's version should be the golden version to prep the demo! Obviously everyone starts looking at me as to why nothing works both worried and eager to help. But I got so stressed I just wanted to dig a hole.
Luckily after going through the logs a colleague of mine pieced together something he heard about another colleague on another location we have submitted a fix (without telling our location) that f**ked up the whole system.

Luckily we reverted it since the system was better without it and got it stable again but after all of that I had to go rest because until we found the answer I was starting to think I couldn't get one thing right. I think it was the most stressful moment in 5+ years on this job

As of two days ago, I no longer use systemd on my Arch system, I switched to openRC.

Basically it all started right around 9 months ago, installed Arch on a new laptop, and whenever I would reboot (which was never very often, mainly kernel upgrades), about 7 out of 10 times it would crash when booting up. My solution for a while was "just don't reboot then".

I spent a while trying to figure out exactly what was causing the boots to fail. I tried disabling systemd units, just trying to narrow it down. I even got the logs from each failed boot, comparing it to a successful boot to find any differences just to have some idea of what the issue was.

One day I figured, it's possible that it could be an issue with systemd itself. So on my day off of work, I figured I'd try using a different init system, just to see if it would work 10 out of 10 times. Decided to try openRC, and sure enough, IT FUCKING WORKS!

Now, I don't hate systemd, I've always been on the fence about it. I feel like it just tries to do too much. I will say, it is fairly convenient to have a lot of things running off of one component, making them all compatible, BUT there's also the factor that one issue could potentially fuck shit up.

Hell, I'll say that it is easier to use systemd than openRC. Enabling unit files is easy as shit in systemd. But I personally like a challenge, and to learn new things, that's part of why I use Arch.

Anyways, I'm done with my rambling for today.

Some time ago I shared a story about negotiating a raise. After that I talked with boss a bit longer and he gave me a new assignment which is not really dev-related. His logic was that I know Java so I should be able to do this since system I'm going to work with is written in Java. Yeah, right.
I have to configure document-flow system, eDok, for our client. I have absolutely no idea about all this document processing and such, but oh well. It's his money.
To do so, my boss bought an serwer with Ubuntu and our client has installed it. I finished a beta version of my last project and today had to start working on this eDok shit. I tried to log in, but nothing was working. From the logs it looks like HDD has failed.
Well, at least it has happened now and not after I've configured everything 😅

So.. I'm migrating a physical server to a virtual (Hyper-V) one.

The physical server is running Windows Server 2008 R2 with IIS6 and Windows SQL Server 2012.

I've set up a VM with Windows Server 2016, IIS10 and Windows SQL Server 2017.

I'm testing with just moving 1 db at a time (we have about 20, 1 per client running this software and a few others) and I've already imported all of the IIS sites.

So the database import and IIS import went smoothly and was surprisingly without hassle but now I'm trying to run the website that I imported the database for and it is throwing 503 Errors at me.
I've been trying to find out the cause but for some reason IIS isn't making any logs.
It's not any 64/32bit system problems (they're both x64) and I can't seem to find anything wrong with the imported config.

Anyone got any ideas?

So a page has been sending errors for long, but we weren't able to find any way to debug it, no error code, and I don't have the authorization to see the logs so I had to wait for a co-worker to be back from holidays.

Now that they're here, I could have a chance to find what was the error.
And be really annoyed about it.

The error was provoked because the security system found a tautology in the data I sent.
(I send datas to build the page, and one parameter is called "Page". Since it was a page of management, I've sent "Gestion", which is management in French. So I sent "Page=Gestion", the security saw "ge=Ge" in it, poof, tautology, you shall not pass.)

That is so ridiculous.

How fucking sucking difficult is it too setup a static ip in AWS on a loadbalancer??? I spend the whole day figuring out how to use the nat gateway or other means and it still doesn't work. Debugging is almost impossible because they give you zero logs.

And all of this because a client wants to work with a whitelist for their shitty system on location.

I think I finally, really, comprehend why secret societies have historically been created... I mean the potentially logical ones. This train of thought is logically terrifying.

I want a logic check.

I've been jokingly mentioning some of my totally true, practically useless in most scenarios, skills/specific fields of knowledge/ability under a moniker of 'extremely useful, assuming apocalyptic event' for years. Things like advanced knowledge of Coefficients of glass expansion, Fortran, various things that have caused friends to refer to me as MacGyver after the reboot came out.

In recent years, I've personally encountered several varieties of the ones defined by helplessness, self-victimisation, some version of a real disability... that theyve expounded into a personified personal nemesis-- to flashily battle yet never overcome, etc... the vast majority perplexing me as to why that's a valid form of life to them... it's not that they never consider some other way; the ball is just quickly dropped and never picked back up.

College?(not that I'm a big fan) they wish they could but so expensive... aide? The form was hard/confusing/past-due...

Lookup/learn something more indepth than a tiktok? *some self-deprecating bs*

Yet it's "I always wanted to do/be/learn X"

Shows like 'How It's Made' fascinate, but don't inspire enough for a 5min google query.

In the dev world its a clear, inverted pyramid-- one of the first posts I saw when I rejoined here was ostream's rant on Apple sucking because after they stop support/updates you "can't" load a different OS... ofc you can. But several comments down... no mention of that... i think it was @LensFlare who was the only one in ~15 respondents to point out the core logical fallacy.

Basic shit is totally forgotten... try asking some random adults what plastic is made from... or pay attention to how many people declare they have a gluten "allergy".

I get people frequently telling me that things im pointing out as differences don't matter because "it's just semantics"... semantics is literally the epitome of "significance", with roots in 'meaning' and 'truth'

Back to the main issue... We are in a world where DIY is typically something you pay more to do as a catered experience than actually learning anything, people destroy their own arguments hopes of validity unwittingly often by stating the arguement, get 'offended' or 'triggered' by factual statements, propagate misinformation and bastardise words until MW needs money enough to print a new version, likely adding the misuse as an actual definition and basic knowledge and the thought to actually learn is vetoed by the existence of google translate, the wisdom of tiktok and the pure brillance of troubleshooting every random linux issue you have from not knowing basic CLI and thinking linux makes you cool, with chmod 777 because so many other dumbasses on forums keep propagating misinformation. Ask them what 777 means, most have no clue... as they didnt consider googling that one before putting it in a terminal several times.
The number of humans that actually know the basic shit that the infrastructure of the world is built on keeps decreasing... and we aren't even keeping a running tally.

The structure of the internet has the right idea... dns- 13 active master root servers, with multiple redundancies if they start dropping... hell ICANN is like a secret society but publicly known/obfuscated... the modern internet hasnt had a global meltdown... aside from the lack of censorship and global availability changing the social definition of a valid use of braincells to essentially propagating spam as if it's factual and educational.

So many 'devs' so few understanding what a driver is, much less how to write one... irl network techs that don't know what dhcp is or that their equiptment has logs... professionals in deducated fields like Autism research/coping... no clue why it was called "autism", obesity and malnutrition simultaneously existing in the same humans... it's like we need to prepare a subterranean life-supporting vault and stock it like Noah's ark... just including the basic knowledge of things that used to be common/obvious. I've literally had 2 different, early 20s, female, certified medical assistants taking my medical history legitimately ask if not having a uterus made it harder to get pregnant...i wish i was joking.

Any ideas better than a subterranean human vault system? It's not like we can simply store detailed explanations, guides, media... unless we find a way to make them into obfuscated tiktok videos apparently on nonsense or makeup tutorials.

Deadline was 2-3 days for product launch and doing distributed transactions was not an opinion as it requires heavy modifications.

I was doing money transfer app between one transactional system and one not transactional system so the way I did it was :
1. transfer money from one system to my app that was using Akka STM ( software transactional memory)
2. try to transfer money to second system
3. transfer money back on failure

There was no database, no state only transactional log as installing database would require to much time and paper work.

Sometimes transfer back failed so we need to look back at logs and search for money, it was quite easy cause there was error and there were not so many failed transactions like this.
About one or two in a month and everyone accepted that.

I started to write some sort of reconciliation thread but then was assigned to other work and it worked like this for couple of years transferring couple millions worth of transactions.

Fuck you mesa

I installed arch again due to some other mess up (another rant 😂) and this time I was facing problem with sleep.

The system froze every time I tried to wake it up from suspend state.

This is how I wasted past 2 days:
- Wifi device sleep issue, disabling wifi
- Probably sddm issue, let me try lightdm
- Last time I installed gdm, might try that as well
- Had to remove all the bloatware then I suspected plasma
- Checked system logs and found amdgpu error
- Tried disabling graphics using nomodeset, system failed to boot
- glx IP block hung!, it's the graphics driver
- Checked another arch forum, a guy was having a same problem after upgrade
- Downgraded mesa to 18.3.4-1

Worked like a charm though hibernate is still not working

Alright boys.. calling in my networking friends for help..

Recently switched my ISP and got a fibre optic installed (100Mbps).

Thr ISP provided a new TP-Link router which supports 5GHz as well as 2.4GHz.

Some of my devices support 5GHz and connect to that network which works flawlessly.

However, my phone does not support 5GHz and hence, have to connect on 2.4GHz.

Somehow, the main router as well as the access point, are not functioning well for 2.4GHz. Whenever the connection is established, it would work fine for a minute or two before the networks starts disconnecting.

Restart the device Wi-Fi and it works for few moments and the cycle repeats.

I am not sure of what is causing this issue.

For the records, the access point is an old D-Link router. Why I mention this? Because funnily whenever the access point cable is plugged into the main router and I login to the router, the system logs me into the access point router (D-Link instead of TP-Link).

Can someone please help me resolve this issue?

Fun fact: The D-Link was a giveaway by one of my dR friends @Bigus-Dickus

Newbie Linux User - Story about not working GUI

I am a proud Opensuse user for about a year, still struggling with some basic stuff, terminal, etc.
The story begins when a few days ago I try to login to the system. To my trusty Gnome. I get stuck on login loop;
successful login - > black screen for a second - > back to login screen.

Zero feedback, not a single error message

Stress level increases taking in count that I am at a climax at my university with tons of projects on my computer.

I assemble the Team A:
Me, Google, Stackoverflow, and for desperate times Russian Stackoverflow

Over 4 hours, found out that my user is affected by this, tried restoring default Gnome configuration, went through bunch of logs only to find out that every user gets the same errors, still only my not working. Even KDE denied to cooperate with the same result.

So what went wrong you may be thinking.
One line in file replaced by miniconda, that changed the PATH.

Linux is the best detective game that I've ever played.
Is it something that I should get used to?

I am new to c and cpp.
I used to exploit my college's competitive programming platform cus it had a bad architecture and almost no auth checks.
For every ajax request, they weren't sending auth tokens or any form of identification and ran all the programs without any logs and on the main thread and as root.. wtf, right?
But recently they've changed something to the site and I cannot run bash commands using system() call.
Is there any other way to execute bash commands using c and cpp.
I already configured a miner in their server but then they re-deployed it cos someone forked bomb the shit out of it.
I'm a noob in c and cpp btw!

It goes back in college days were,I started developing on Visual Basic for a college project as it was the only option.
As the scope was limited to a standalone application,we we're not allowed to use network.
Building up on the that,the project was to be done in a group of two with SRS and other stuff needed to done.
With my partner having no knowledge about the code,I took my ideas and Incorporated it into my project such as system logs,session tracking,data records,barcode reader,export data in various formats and so on.
The project got large eventually and professor's were curious to see the development of my project.
The project got showcased as the best project by professors and that overall gained my popularity in college and got me a job offer which I rejected in the end

It's finally annoyed me enough, I gotta ask...

Am I the only one getting annoyed at procedurals (the shows about crimes) and their accuracy irl, at least when it comes to timelines and alibis that are deemed as 100% uneditable fact... based on metadata and system log files???

Or is this another case of me assuming other humans must realise things i see as simple, should be common, sense?

If it's the latter... ANY system can be modded, timestamps have sooooo many ways of being altered from multiple angles, a few lines of code in an apk file on a dev mode device can make your gps show as anywhere in the world...

even a basic early 2000s runescape bot creator should know the basic methodology of this... and even OS-innate output of event logs can be set up to effortlessly mod themselves to selectively delete or rewrite upon the system's command to show them.

For anyone thinking im FoS/this isnt a simple concept of reality... or simply never considered the reality of how much less work it'd be to just commit whatever crime that high intellect people plan meticulously for years, often still getting caught... including via their own, or easily hacked timestamps/lack of alibi...

Wanna blow your mind?
...
If you remove all hdds/ssds and even all RAM, any external devices of any kind, cut off all networking, then put brand new, all 0s, storage and RAM in... then only connect directly to a totally direct connection to WAN (aka the internet via ISP... and your ISP has no malware etc)...

Without browsing anything, and a totally fresh, safe, OS newly installed to 0'd new ROM... no one actively finding/targeting you...

Can you already be infected with spyware/viruses/malware/etc?

YUP!

how? Someone who knows what dev u use and concise coding and drivers in binary... they rewrite your bios to turn on basic components, no fans/lights/etc... bios has the clock, u can be asleep, see/hear nothing while bios boots up totally dark, runs a cimmand to pull all the malware to u... youd be oblivious

So for Christmas my friend got me some USB's from a pretty reputable company. When I copied some folder (~1.5 gb) to it (it was exFAT format) it errored out around 31%, then my OS just unplugged it (I'm not using Windows, Linux person) then errored out. So I replugged it, tried again, and again same thing happened. So 3rd time, my OS just doesn't recognize it... I checked "lsblk" (a linux command to list all drives) and it doesn't appear. So I checked the logs of my system (not OS but system itself) and it says that it's a memory issue (so I know nothing about this cause I never saw something like this before, but I think the USB is formatless as in like it has no accepted format.) So I was extremely confused. I put it on GParted, which is a tool dedicated to formatting drives. Not as an app but I booted a USB with it, AND EVEN THAT DOESN'T RECOGNIZE IT. My dad suggested booting on windows and trying it. So I went on the windows installer again off a USB, opened command prompt, then notepad, then the file dialog (since explorer doesn't exist) and sure enough, even that doesn't recognize it. So my USB is absolutely cooked. All from 1 folder. Wow. Any ideas what to do with it to fix it, or should I just abandon it? Also merry christmas! :D

Anyone tried converting speech waveforms to some type of image and then using those as training data for a stable diffusion model?

Hypothetically it should generate "ultrarealistic" waveforms for phonemes, for any given style of voice. The training labels are naturally the words or phonemes themselves, in text format (well, embedding vectors fwiw)

After that it's a matter of testing text-to-image, which should generate the relevant phonemes as images of waveforms (or your given visual representation, however you choose to pack it)

I would have tried this myself but I only have 3gb vram.

Even rudimentary voice generation that produces recognizable words from text input, would be interesting to see implemented and maybe a first for SD.

In other news:
Implementing SQL for an identity explorer. Basically the system generates sets of values for given known identities, and stores the formulas as strings, along with the values.
For any given value test set we can then cross reference to look up equivalent identities. And then we can test if these same identities hold for other test sets of actual variable values. If not, the identity string cam be removed, or gophered elsewhere in the database for further exploration and experimentation.

I'm hoping by doing this, I can somewhat automate the process of finding identities, instead of relying on logs and using the OS built-in text search for test value (which I can then look up in the files that show up, and cross reference the logged equations that produced those values), which I use to find new identities.

I was even considering processing the logs of equations and identities as some form of training data perhaps for a ML system that generates plausible new identities but that's a little outside my reach I think.

Finally, now that I know the new modular function converts semiprimes into numbers with larger factor trees, I'm thinking of writing a visual browser that maps the connections from factor tree to factor tree, making them expandable and collapsible, andallowong adjusting the formula and regenerating trees on the fly.

What the hell is the point of this small projects team spending 2-3 months on developing extensive logging system for an internal application for inside and outside customers to use if your application isn’t going to log any of the fucking errors. Sure you write the failure status to the database, but it just says failure with an even more vague explanation than microsoft’s errors. “An error occurred”. No shit, that’s why I’m looking in the logs and database to debug the application to get these files on their merry way so our company can stay in compliance with the state, feds, and not pay out the wazzoo in fines. All our other applications state where the error occured such as “failed to connect to the email server”, why can’t this one.

Cleaning my system. Learned something new (do a vacuum on your journalctl logs).

Any way: biggest applications on my system in the screenshot.

Can I have an applause for google-cloud-cli for being bigger than open office?

That google-cloud-cli thing is not some advanced supertool that you installed for convenience, it's actually to get API access to certain google services and is really required. Install this thing, and magically - no idea how (there is no ENV key introduced, so it executes an app to resolve credentials or smth?) - your python applications are authenticated some how after logging in using a web interface.
- it's big
- not normal api key usage
- requires a web browser
- user of API key needs to know credentials of the owner of the key, so forget installing by someone else.

Google is crazy. Why would you build smth so different than any other API supplier. Why reinvent the wheel in a bad way. How can so much of it be client side.

Only positive thing is - that it works immediately after being surprised that it was really such big app that allows you API access.

It's impossible that such big system is safer than just a simple API key.

Also, if it provides interface and is requirement for other google libs it's just a freaking SDK.

I need some advice about setting up monitoring. For background I have 4 gameservers and it is already hard enough to to develop them, but I am also responsible for support and monitoring of bugs/logs and other shit.

I can't disconnect from my work anymore.

I want to setup some kind of monitoring system, that would check constantly on my gameserver mysql databases and show me charts of what's happening over time. In that case I could have a quick glance at some dashboard and wouldn't need to waste my time logging into game and checking everything.

I would also need to setup some notifications in case shit goes down.

Any advice what could i use for that?

New Project

M: Hey, check these two processes. Both took different paths for the same input. Here are the logs. Both are the same though.
Me: Ok... do we have a debugger?
M: No this product doesn't have a debugger
Me: Any unit tests i should know of?
M: We don't do unit testing. Everything is done in Integration Testing.
Me: Ok. So how can i check the db for this?
M: You can't, the access is restricted. You'll have to raise a ticket to other team with the sql output you need.
Me: Ok. So I hope you have the schema at least.
M: Yes we have the schema. But there was some issue last week so the values might not be there in the correct column. They may or may not be present where they are supposed to be.

Wtf am i supposed to do... fucking play football on ticketing system with the other team 😐

Follow up sorta...

So I got pulled into a support issue on a day off. Some system was facing timeouts on our servers so had to investigate.

Over the weekend as part of the release, I released the ELK stack I built and today I used that to help.

Pretty much immediately pinpointed which machine was hanging though still had to investigate and confirm so split between KQL and checking the server logs.

One thing I've always griped about is how no one created schema docs for it mongo collections so can't easily figure out what they do or your to get the document needed.

Well guess it's my turn.... Because only I know the schema :)