24
netikras
269d

I previously worked as a Linux/unix sysadmin. There was one app team owning like 4 servers accessible in a very speciffic way.
* logon to main jumpbox
* ssh to elevated-privileges jumpbox
* logon to regional jumpbox using custom-made ssh alternative [call it fkup]
* try to fkup to the app server to confirm that fkup daemon is dead
* logon to server's mgmt node [aix frame]
* ssh to server directly to find confirm sshd is dead too
* access server's console
* place root pswd request in passwords vault, chase 2 mangers via phone for approvals [to login to the vault, find my request and aprove it]
* use root pw to login to server's console, bounce sshd and fkupd
* logout from the console
* fkup into the server to get shell.

That's not the worst part... Aix'es are stable enough to run for years w/o needing any maintenance, do all this complexity could be bearable.

However, the app team used to log a change request asking to copy a new pdf file into that server every week and drop it to app directory, chown it to app user. Why can't they do that themselves you ask? Bcuz they 'only need this pdf to get there, that's all, and we're not wasting our time to raise access requests and chase for approvals just for a pdf...'

oh, and all these steps must be repeated each time a sysadmin tties to implement the change request as all the movements and decisions must be logged and justified.

Each server access takes roughly half an hour. 4 servers -> 2hrs.

So yeah.. Surely getting your accesses sorted out once is so much more time consuming and less efficient than logging a change request for sysadmins every week and wasting 2 frickin hours of my time to just copy a simple pdf for you.. Not to mention that threr's only a small team of sysadmins maintaining tens of thousands of servers and every minute we have we spend working. Lunch time takes 10-15 minutes or so.. Almost no time for coffee or restroom. And these guys are saying sparing a few hours to get their own accesses is 'a waste of their time'...

That was the time I discovered skrillex.

Comments
  • 1
    I don't want to meet the person with a brain capable of imagining such thing. The process in itself is just insane...
    I understand why you wanted to get out of there. Hope you'll happier now!

    (On a side note, I laughed when reading "tties/tities".)
  • 3
    @Jilano oh.. Posted this with a phone so there are some typos. Glad to see my mistakes made someone happy :D
  • 1
    How does one get to the point of writing a custom SSH alternative? (There is a point in using mosh but not via SSH jump hosts.)
  • 2
    @zvyn there are plenty commercial alternatives already. Mostly they are used because they are able to provide much better auditing when compared to ssh. For instance fkup (title is made up not to reveal actual tchnologies used there) can record EVERYTHING there is on a terminal and later on security auditos can replay every single session there was made to that server. Another alternative can do that + implements custom authorization policy integrated with internal infrasctucture tools + it does log non-printable keystrokes (like ENTER, TAB, wtv would not appear on the terminal) + it can categorize those logs + authorized personnel can review those logs in plain-text + you name it.

    Huge enterprise companies do this thing and it would be unimaginably difficult to work there if they didn't. HOWEVER since these tools are ALTERNATIVES they are far not as stable and efficient as ssh. Not to mention lack of some essential ssh features.
  • 1
    @zvyn ssh alternatives do come in handy when limited personnel has to overlook unlimited infrastructure. I cannot immagine any team of sysadmins provisioning each ssh access request or attempting to figure out where the f*** did those SAN luns disappear when we are talking about infra with tens of thousands of servers.

    A darn zoo where you can find anything from RHEL 7.3 to RHv4 or SCO-unix or hp-ux <9 or tru64 or zlinux or hell knows what else.

    So one must have a way to reliably check what, when, how, by whom and whay was done on the server.
  • 0
    @netikras right, that (especially the auditing part) seems reasonable! Where I worked the sysadmins used Puppet, Ansible, etc. for managing "unlimited infrastructure" (I'm sure something like that is part of your set up as well). Also the "zoo" could be categorised into a a few OS versions per client. In that scenario updates would never involve logging on to the server and a request like "upload this PDF on that server" where answered with "build a Debian package, upload it to the repo and I'll add it to the appropriate puppet file".
Your Job Suck?
Get a Better Job
Add Comment