SkillsLinux administration, Bash, basics of PHP, jS and other languages
Joined devRant on 5/13/2016
Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Can I just say, I owe my soul, nerves and eternal gratitude to the folk over at Percona? They publish articles that have, on more than one occasion, saved my hide when a DB node wasn't working as it should, and I had to find out how to fix that.
Seriously, amazing, love those guys!2
I completely *detest* that the MongoDB *shell* is just a fucking jS interpreter with extra API calls sprinkled on top and whoever came up with that idea should have all their commits reverted immediately, working with that thing is a punishment!
I don't even know a way to parse and chew through the json it spits out in my own json viewers, as it's "Extended", and none of my editors understand that!
Ugh, haven't been this frustrated with a tool for a while...5
In my case, the only way to stay productive is to task switch often. Suffering of adhd, I get bored of researching / developing a specific solution rather quickly, and then have a huge issue staying focused.
That's also why I can't imagine being a programmer. Being a sysadmin, however, is great! Dealing with many different tickets a day.
Despite already having a few years of professional experience dealing with Linux servers, I still, to this day, confuse, which environment file gets sourced and when...
There's /etc/profile, /etc/bashrc, ~/.bash_profile, ~/.profile, ~/.bashrc
I think it's... Bashrc for interactive shells, profile for login shells.
But then I have examples like "ssh user@server 'echo $var'" that... Don't source any of the files!
You can enable user environment files for SSH that get sourced whenever a user logs on through SSH (~/.ssh/environment / environment specified for a key in ~/.ssh/authorized_keys)
Is there some sort of master environment file that gets sourced *every* time, no matter what kind of shell starts?1
What is it with web devs that can't write effective PHP applications that don't need a 1 GB of Memory Limit?
Where are the days that 32MBs of memory was fine per request? Ugh...2
*Frustrated user noises* Whyyyy, Grafana, why don't you implement any actual query forgery checks?!
So long as a user has access to the Grafana frontend, they can happily forge the requests going off to the backend, and modify them to return *whatever* data they want from the datasource.
No matter that they're a read-only user. That only stops them from modifying the dashboard definitions on the frontend, but doesn't enforce any sort of immutability on the BE...
If anyone had any tips on how to further secure it, I'm curious...5
I often wonder if our clients seriously think we have an all-knowing crystal ball of wisdom when they send tickets like "Cannot send emails - please check" while they have like 10 servers and email delivery is a complex matter on its own already.
If I didn't care what our clients think, I'd reply with equally informative email of "Maybe, who knows"...1
I swear, there will come a day when I stop confusing Grafana and Kibana. The two things sound too similar for their own good.4
Anyone has any idea how to debug occasional (Severa times a day) where one of our servers decides to mount a second copy of the same NFS? It triggers our monitoring system thinking there was a change to the mounts of the system, and I was able to verify through the mounts command, that it indeed had the same NFS mount mounted twice, with exactly the same parameters.
Is there a debug interface of some sort to see what initiated that mount? Or any tool to help me track it down? I've been stuck with this mysterious issue for a while now (As it's not really a priority, it doesn't break anything, but it bugs me and I wanna know)3
Boy, I sure do love trying to figure out why our master and slave MariaDBs differ in their execution plans, even after running analyze tables on the whole DB.
Or rather, I really hope the two boxes didn't somehow magically desynced, cuz that would then beg a question of why, and how to prevent that from happening again.
I hate how databases are so necessary nowadays, but are probably the most complex and black box software I deal with. There's just so much to consider...1
WHAT A PIECE OF CRAP
It's so precious that when it detects an existing mdraid signature, it just *won't* let me create a physical volume over it!
No matter that I run pvcreate with double-force switch.
It doesn't matter that the system doesn't even have a single MD device defined (Which can be easily checked in /proc/mdstat OR by checking the /dev subsystem)
I *hate* commands that are trying to be more clever than the admin sitting behind the keyboard.
Sure, leave this as the default behavior (It could save a lot of people's data I bet), but BLOODY HELL GIVE ME A SWITCH TO OVERRIDE THE CHECK YOU DUMBASS.
I swear... I feel like I'll get a frickin' brain hemorrhage from this "clever tool" -_-"5
Spent an hour figuring out why my dd command did not actually rewrite the specific portion of disk, only to find out that the skip argument applies only to input file.
If one wishes to skip onto a specific address of the output file, seek is the argument they... seek.
Ugh, little things in life...
Did I ever mention how much I hate reading through perl scripts?
Seriously, I can read through BASH hell anyday, Python's fine, PHP... But out of all of the scripting languages, Perl is just something that makes me want to scratch my eyes out.
It doesn't help it used to be the sysadmin's language of choice in the past.
Perl just hurts my brain.5
Maaaan, we all knew it was coming, we were warned, again and again, yet still, when Lets Encrypt's old root CA expired today, we found out a tool we were using to get new certs (Not cerbot, custom wrapper around acme-tiny) included the old root in the chain.
So... A few hours ago, some of our servers started having connection issues.
Great final 3 hours of today. Better luck next time I guess? Still, despite the little hickup, Lets Encrypt still remains as one of the biggest revolutions in the adoption of SSL, they're the good guys.5
Any Elasticsearch gurus here? I have a box with too many young gen GCs (one per 2 or 3 seconds), and irregular, very long old gen GCs (One per several hours, taking around a minute and freeing about 2/3's of the old gen space) -- I was thinking changing the new gen ratio from 2/3 to something like 3/4 or 4/5.
However, after reading an elastic article about settings to never touch... I'm no longer so sure...
Only other option I was considering is going from CMS to G1GC to cut back on the old gen GC time... A minute long downtime for Elastic is rather problematic.
Any thoughts? The box is rather old - running Elastic 5.6 with 20 GBs of heap, 207 shards and 306k docs.2
There are 2 kinds of websites:
1 - The bad kind where not accepting their cookies boots you off the site (And so are in breach of GDPR... IIRC)
2 - Sites that continue working, albeit in a degraded / suboptimal state, even when you refuse their cookies.
I wish more sites were of the second variety. I'm even the only person among my friends who actually bothers going through the consent forms and disallowing everything marketing-related.
OneTrust is good. It at least remembers my preferences.6
Lemme just say... Wow. Wireguard... It's so incredibly simple and elegant. I cannot believe how easy and how little reading it needed to set it up.
And unlike OpenVPN, the Android client is even able to override the system's DNS servers, meaning I can finally start blocking nosy apps from contacting their big brothers in the cloud via DNS blackholing!
Wow. Wireguard... 10/10. Simple, fast to set up, elegant.4
So, despite being pretty experienced with Linux server management, today, I failed, even after hours spent tinkering, to get Bumblebee working on an older laptop of mine (Intel i3 + Geforce 960m).
What's funnier is that before I wiped that laptop with a clean install, it was working, albeit it on an out of date kernel / driver combo.
Though curiously, despite using the newest release of Xubuntu, the Bumblebee PPA repo wasn't signed (Missing InRelease file), and further lacked one of the Package index files (For i386 i believe)
I'm about to sell the laptop tomorrow. Anyone has any hints or things I could have missed? I still have a day to work on it, and if I don't manage, I'll just put on a clean win install...4
After reading through an article by EFF about Google's new FLoC technology, I finally woke up and realized that Google isn't any different from the other huge organizations, and that it isn't here to fight for greater user privacy and rights... As such, I made a switch, going from Google to DuckDuckGo.
Sure, I still use YT, but other than that, I don't have a gmail address that'd be of importance, don't use their search engine, and now...
Got any tips on how to minimize tracking potential of an android-based rooted smartphone? (Stock rom, Xiaomi Mi 10T Pro). I know using a custom rom is an option, however, Lineage isn't officially supported for this model of phone, and I still value functionality over anything else, so... Unofficial daily drive OS is sub-optimal.
Alternative hardware solutions like PinePhone, although nice in theory, are way too behind the commercial brands in terms of technology and features, so that's a no go.5
Okay, yes, modsecurity WAF is amazing and all, but... When one tries to implement its rules atop an existing app that wasn't developed in accordance to the rules... That hurts.
How tf am I supposed to parse and present a 6.5GB / 22M line audit log to the client?! Just parsing that monstrosity once takes *minutes*, let alone doing any sort of sorting / analysis!
I feel sick. This is exactly why I am a sysadmin and not a programmer, I don't like writing analysis stuff, or programs more complex than a few hundred lines of bash... :|6
It makes me want to cry in frustration that I... actually love SystemD, as an *init* system. But with all the crap it brings along with that core part, it just makes it so much harder for me to really enjoy! Why can't it be modular? Why can't it be broken down into independently-installable packages, with the init system as a core? Is there some sort of internal API issue? Or does mister Pottering just does not want that to happen? The Linux world has always stayed by the idea of "1 package = 1 task", and it made the system management so much easier!
But now... When I switch to SystemD from SysVInit, I get... What SysVInit did + so much more I didn't ask for... I just... Don't understand it.3
Not fair! How come Apache gets its modsecurity module pre-compiled and available in public repositories, but Nginx hides it behind a paywall of their "Plus" package/version?
Ugh... At least they provide the module's source, so one can compile and deploy it by himself...6
I think I finally reached a point where I Have to completely reinstall my RPi.
Running Raspbian, I was under the impression their kernel releases worked the same way a pure Debian release worked - That the kernel was somewhere in the system repository.
Turns out it was, but in a different pool. And also turns out the new kernel and initramfs won't fit into my /boot as, for some reason, it is under 50 MBs in size. I dunno why, but I don't have any unallocated space left to grow the partition...
I have no idea why the boot is so small (Probably because, when I was setting the system up, I wasn't really that good with Linux yet, and just went with defaults).
What do you guys think - Is it better to run the native Raspbian system (Formerly RaspberryPi OS), or go with a pureblood Debian for Arm? (Yes, I already checked, my HW revision [3B+] is already compatible])1
>Client complains about a 30 minute downtime around midnight
>Client also pays only for a single VM on a HV that they don't even own themselves
>Replies with an offer of how to make the setup more resilient, going from 1 VM to 2 LBs/FE loadbalanced through BGP, and distributing traffic through HaProxy onto 2 BE machines that in turn talk to a Postgres Cluster with RepMgr for dynamic failover.
>No reply so far
So, none other than the father of our beloved Linux kernel - Linus Torvalds, just totally put an antivax guy down in the public kernel mailing list.
I think I love Linus even more now. He may not be a people person, but he sure does know how to totally rip people into shreds lol.
Okay, so, I have a functional snort agent instance, and it's spewing out alerts in it's "brilliant" unified2 log format.
I'm able to dump the log contents using the "u2spewfoo" utility (wtf even is that name lol... Unified2... something foo) but... It gives me... data. With no actual hint as to *what* rule made it log this. What is it that it found?
All I see are IDs and numbers and timings and stuff... How do I get this
sensor id: 0 event id: 5540 event second: 1621329398 event microsecond: 388969
sig id: 366 gen id: 1 revision: 7 classification: 29
priority: 3 ip source: *src-ip* ip destination: *my-ip*
src port: 8 dest port: 0 protocol: 1 impact_flag: 0 blocked: 0
mpls label: 0 vland id: 0 policy id: 0
into information like "SYN flood from src-ip to destination-ip"
So... I got a simple task of choosing the best fitting NIDS/MIDS, as well as deploying it, configuring to fit a specific use case and monitor its outputs for one client at work today...
I'm a little... Anxious. At a first glance, setting up like... Snort... Doesn't seem all that difficult, but I have no idea where this takes me and if what I come up with will ultimately be useful or not... Until now I did simple service configuration changes like apache, nginx, php... And a bit of database management with things like mariadb, mysql, postgresql, mongo or elastic... I feel so... Out of my usual waters.
Do you guys thing a person without a title in network security (or... Any title for that matter) can even manage this?...1
I seriously love rsync. Whoever made that utility is my hero. Not only that its CLI client is amazing and full of features, but rsync in daemon mode makes secure file synchronization a breeze! <38
Can someone, anyone, explain to me, how can Microsoft get away with *charging extra* for additional concurrent RDP sessions on a self-hosted instance of Windows Server?
And not only that, but apparently also charges extra once the box gets over a certain amount of system users, too.
As a Linux admin that's used to working in teams over SSH, it just completely baffles me.
It would be terrible if such a practice was in free software... But a system, that one already *pays* for to run?
Or did I understand something wrong from a colleague that claims that this is the reason why I can't get an account on one of our Windows Servers?6
It is currently 5:30 AM, I've been trying to upgrade a server by one single major OS release for the last 3.5 hours. All major apps were no issue, except one.
That thing runs on Ruby.
Ruby, as it turns out, doesn't really like the way Debian handles package management.
And now, I have Redmine that, even if I completely uninstall / reinstall, won't start
I went down several rabbit holes, trying to operatively find what the issue is. But I never got to the key issue.
Fuck ruby. Fuck Redmine. Fuck nothing-saying error messages. Fuck bundle. Fuck gem. Fuck it all.
I'm redirecting the Redmine domain onto the server backup I've made. Upgrading that thing is a nightmare.
Maybe now I can finally go to bed...5