Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "systems on-calls"
-
Hey everyone,
We have a few pieces of news we're very excited to share with everyone today. Apologies for the long post, but there's a lot to cover!
First, as some of you might have already seen, we just launched the "subscribed" tab in the devRant app on iOS and Android. This feature shows you a feed of the most recent rant posts, likes, and comments from all of the people you subscribe to. This activity feed is updated in real-time (although you have to manually refresh it right now), so you can quickly see the latest activity. Additionally, the feed also shows recommended users (based on your tastes) that you might want to subscribe to. We think both of these aspects of the feed will greatly improve the devRant content discovery experience.
This new feature leads directly into this next announcement. Tim (@trogus) and I just launched a public SaaS API service that powers the features above (and can power many more use-cases across recommendations and activity feeds, with more to come). The service is called Pipeless (https://pipeless.io) and it is currently live (beta), and we encourage everyone to check it out. All feedback is greatly appreciated. It is called Pipeless because it removes the need to create complicated pipelines to power features/algorithms, by instead utilizing the flexibility of graph databases.
Pipeless was born out of the years of experience Tim and I have had working on devRant and from the desire we've seen from the community to have more insight into our technology. One of my favorite (and earliest) devRant memories is from around when we launched, and we instantly had many questions from the community about what tech stack we were using. That interest is what encouraged us to create the "about" page in the app that gives an overview of what technologies we use for devRant.
Since launch, the biggest technology powering devRant has always been our graph database. It's been fun discussing that technology with many of you. Now, we're excited to bring this technology to everyone in the form of a very simple REST API that you can use to quickly build projects that include real-time recommendations and activity feeds. Tim and I are really looking forward to hopefully seeing members of the community make really cool and unique things with the API.
Pipeless has a free plan where you get 75,000 API calls/month and 75,000 items stored. We think this is a solid amount of calls/storage to test out and even build cool projects/features with the API. Additionally, as a thanks for continued support, for devRant++ subscribers who were subscribed before this announcement was posted, we will give some bonus calls/data storage. If you'd like that special bonus, you can just let me know in the comments (as long as your devRant email is the same as Pipeless account email) or feel free to email me (david@hexicallabs.com).
Lastly, and also related, we think Pipeless is going to help us fulfill one of the biggest pieces of feedback we’ve heard from the community. Now, it is going to be our goal to open source the various components of devRant. Although there’s been a few reasons stated in the past for why we haven’t done that, one of the biggest reasons was always the highly proprietary and complicated nature of our backend storage systems. But now, with Pipeless, it will allow us to start moving data there, and then everyone has access to the same system/technology that is powering the devRant backend. The first step for this transition was building the new “subscribed” feed completely on top of Pipeless. We will be following up with more details about this open sourcing effort soon, and we’re very excited for it and we think the community will be too.
Anyway, thank you for reading this and we are really looking forward to everyone’s feedback and seeing what members of the community create with the service. If you’re looking for a very simple way to get started, we have a full sample dataset (1 click to import!) with a tutorial that Tim put together (https://docs.pipeless.io/docs/...) and a full dev portal/documentation (https://docs.pipeless.io).
Let us know if you have any questions and thanks everyone!
- David & Tim (@dfox & @trogus)53 -
C: application not working
Me: k. What changed?
C: we didn't make changes
Me: k... *gets a tech team (W) on the phone*
W: Hey, what's broken?
Me: C's application. How do things look?
W: running healthy. I'll check logs.
Me: thanks. *gets tech team (S) on the line*
S: hey, everything clear on our end, will check logs.
Me: thanks *gets tech team (U)*
U: hey! They asked us to deploy their new version today during normal deployment time. Is it acting up?
Me: C, what did you change?
C: nothing major, just how we connect to W and S...
W&S: are you shitting me???
Me: U, will you please roll it back?
C: no! Must stay on this version, you need to fix your side!!
Me: nope. *calls U boss (UG)*
UG: U, you have my permission to roll back, they need to fix. C, if your boss doesn't like it, have them call me.
*rollback fixes problem*
IF I FUCKING ASK YOU WHAT THE FUCK YOU CHANGED, YOU BETTER TELL ME THE TRUTH, OR I WILL STRIP YOUR CODE OFF OUR FUCKING SYSTEMS AND SHOVE IT DOWN YOUR THROAT. MY JOB IS TO HELP YOU AND YOU NEED TO BACK TO FUCK UP AND NOT GET IN THE WAY OF MY JOB OR YOU WON'T HAVE ONE ANYMORE.11 -
So this was a couple years ago now. Aside from doing software development, I also do nearly all the other IT related stuff for the company, as well as specialize in the installation and implementation of electrical data acquisition systems - primarily amperage and voltage meters. I also wrote the software that communicates with this equipment and monitors the incoming and outgoing voltage and current and alerts various people if there's a problem.
Anyway, all of this equipment is installed into a trailer that goes onto a semi-truck as it's a portable power distribution system.
One time, the computer in one of these systems (we'll call it system 5) had gotten fried and needed replaced. It was a very busy week for me, so I had pulled the fried computer out without immediately replacing it with a working system. A few days later, system 5 leaves to go work on one of our biggest shows of the year - the Academy Awards. We make well over a million dollars from just this one show.
Come the morning of show day, the CEO of the company is in system 5 (it was on a Sunday, my day off) and went to set up the data acquisition software to get the system ready to go, and finds there is no computer. I promptly get a phone call with lots of swearing and threats to my job. Let me tell you, I was sweating bullets.
After the phone call, I decided I needed to try and save my job. The CEO hadn't told me to do anything, but I went to work, grabbed an old Windows XP laptop that was gathering dust and installed my software on it. I then had to build the configuration file that is specific to system 5 from memory. Each meter speaks the ModBus over TCP/IP protocol, and thus each meter as a different bus id. Fortunately, I'm pretty anal about this and tend to follow a specific method of id numbering.
Once I got the configuration file done and tested the software to see if it would even run properly on Windows XP (it did!), I called the CEO back and told him I had a laptop ready to go for system 5. I drove out to Hollywood and the CFO (who was there with the CEO) had to walk about a mile out of the security zone to meet me and pick up the laptop.
I told her I put a fresh install of the data acquisition software on the laptop and it's already configured for system 5 - it *should* just work once you plug it in.
I didn't get any phone calls after dropping off the laptop, so I called the CFO once I got home and asked her if everything was working okay. She told me it worked flawlessly - it was Plug 'n Play so to speak. She even said she was impressed, she thought she'd have to call me to iron out one or two configuration issues to get it talking to the meters.
All in all, crisis averted! At work on Monday, my supervisor told me that my name was Mud that day (by the CEO), but I still work here!
Here's a picture of the inside of system 8 (similar to system 5 - same hardware)15 -
@netikras since when does proprietary mean bad?
Lemme tell you 3 stories.
CISCO AnyConnect:
- come in to the office
- use internal resources (company newsletter, jira, etc.)
- connect to client's VPN using Cisco AnyConnect
- lose access to my company resources, because AnyConnect overwrites routing table (rather normal for VPN clients)
- issue a route command updating routing table so you could reach confluence page in the intranet
- route command executes successfully, `route -n` shows nothing has changed
- google this whole WTF case
- Cisco AnyConnect constantly overwrites OS routing table to ENFORCE you to use VPN settings and nothing else.
Sooo basically if you want to check your company's email, you have to disconnect from client's VPN, check email and reconnect again. Neat!
Can be easily resolved by using opensource VPN client -- openconnect
CISCO AnyConnect:
- get a server in your company
- connect it to client's VPN and keep the VPN running for data sync. VPN has to be UP at all times
- network glitch [uh-oh]
- VPN is no longer working, AnyConnect still believes everything is peachy. No reconnect attempts.
- service is unable to sync data w/ client's systems. Data gets outdated and eventually corrupted
OpenConnect (OSS alternative to AnyConnect) detects all network glitches, reports them to the log and attempts reconnect immediatelly. Subsequent reconnect attempts getting triggered with longer delays to not to spam network.
SYMANTEC VIP (alleged 2FA?):
- client's portal requires Sym VIP otp code to log in
- open up a browser in your laptop
- navigate to the portal
- enter your credentials
- click on a Sym VIP icon in the systray
- write down the shown otp number
- log in
umm... in what fucking way is that a secure 2FA? Everything is IN the same fucking device, a single click away.
Can be easily solved by opensource alternatives to Sym VIP app: they make HTTP calls to Symantec to register a new token and return you the whole totp url. You can convert that url to a qr code and scan it w/ your phone (e.g. Google's Authenticator). Now you have a true 2FA.
Proprietary is not always bad. There are good propr sw too. But the ones that are core to your BAU and are doing shit -- well these ARE bad. and w/o an oppurtunity to workaround/fix it yourself.13 -
I really hate fucking Wordpress!
I hate it's stupid API, with it's stupid hooks and actions and all those stupid functions and no fucking logic to any of it!
I hate it's stupid plugin system, with all that fucking overhead that brings no real value and adds all that complexity for nothing!
I hate stupid fucking multiple calls for the same fucking assets, loading them over and over again because every stupid plugin calls them again and again!
I hate motherfucking SHORTTAGS, or whatever the fuck they are called!
I hate that every stupid fucking plugin and shortcode and fucking every little fucking piece of HTML comes from a different fucking place, with different fucking structure and different fucking classes and stupid fucking loading seaquences that make no fucking sense!
And I hate fucking page builders !!!!!
Fuck!!!!
I should be fucking coding on this fucking peace of shit, but I just cannot fucking take it any more!!!
IT NEEDS TO FUCKING DIE!
It should be relegated to the darkest corners of the internet and all the servers that have it's fucking code anyware on their systems should be disconnected and buried in the deepest pits of hell, just to be sure it never, EVER, surfaces again!!!
AAARRRRGGGHHHHHHH !!!!!!!!!5 -
THE FUCK!!!!
Some stupid assholes on my company have adquired the idiotic habit to call me to supervise some videoconferences using Skype, Hangouts or other fucking systems that i never heard about. I hate it because is a big fat waste of time.You don't need a engineer to handle your stupid password mismatch, your "why this is so slow? can you fix the internet". Or why the sound is so crappy or why don't you have a better microphone. This are your fucking computers, and your fucking calls. what the next? sending your emails? Wipe your asses? is not my fucking role.
Next time I will put your notebooks inside your assholes. Your crap will be better transmitted for sure. -
Was just recalling one of the worst calls I ever got in IT...
Many years ago we had a single rack for all of our servers, network and storage (pre virtualization too!).
We had a new security system installed in the building and the facilities manager let the guy into the server room to run all the sensor cables in because that is where they wanted their panel... the guy was too lazy to get up on the roof and in the attic repeatedly so after he checked it out he went around every where and drilled a hole straight up where he wanted the sensor wire to go... well the server room was not under an attic space... when he found he had drilled through to the out side... HE FILLED IT WITH EXPANDING FOAM.... the membrane on the roof was damaged... that night it rained... I got a call at 4 am that systems were acting funky and I went in... when I opened the door it was literally raining through the corners of the drop ceiling onto the rack... An excellent DR plan saved our asses but the situation cost the vendor's insurance company $30k in dead equipment and another $10k in emergency labor. Good thing for him we had so little equipment in that room back in.
Moral of the story... always have a good DR plan... you never know when it will rain in the server room.... :)3 -
Buckle up, it's a long one.
Let me tell you why "Tree Shaking" is stupidity incarnate and why Rich Harris needs to stop talking about things he doesn't understand.
For reference, this is a direct response to the 2015 article here: https://medium.com/@Rich_Harris/...
"Tree shaking", as Rich puts it, is NOT dead code removal apparently, but instead only picking the parts that are actually used.
However, Rich has never heard of a C compiler, apparently. In C (or any systems language with basic optimizations), public (visible) members exposed to library consumers must have that code available to them, obviously. However, all of the other cruft that you don't actually use is removed - hence, dead code removal.
How does the compiler do that? Well, it does what Rich calls "tree shaking" by evaluating all of the pieces of code that are used by any codepaths used by any of the exported symbols, not just the "main module" (which doesn't exist in systems libraries).
It's the SAME FUCKING THING, he's just not researched enough to fully fucking understand that. But sure, tell me how the javascript community apparently invented something ELSE that you REALLY just repackaged and made more bloated/downright wrong (React Hooks, webpack, WebAssembly, etc.)
Speaking of Javascript, "tree shaking" is impossible to do with any degree of confidence, unlike statically typed/well defined languages. This is because you can create artificial references to values at runtime using string functions - which means, with the right input, almost anything can be run depending on the input.
How do you figure out what can and can't be? You can't! Since there is a runtime-based codepath and decision tree, you run into properties of Turing's halting problem, which cannot be solved completely.
With stricter languages such as C (which is where "dead code removal" is used quite aggressively), you can make very strong assertions at compile time about the usage of code. This is simply how C is still thousands of times faster than Javascript.
So no, Rich Harris, dead code removal is not "silly". Your entire premise about "live code inclusion" is technical jargon and buzzwordy drivel. Empty words at best.
This sort of shit is annoying and only feeds into this cycle of the web community not being Special enough and having to reinvent every single fucking facet of operating systems in your shitty bloated spyware-like browser and brand it with flashy Matrix-esque imagery and prose.
Fuck all of it.20 -
So here I am sitting on my dusty laptop gaming laptop (because supposedly it would offer me better performance in compiling code and working with CUDA according to the people above me) at a research institute where I just started working at. I am told that there are some issues with the code and that it fails to build on Windows with MSVC that ships with Visual Studio 2017 and later.
I poor some hot tea from my insulated bottle I brought from home and start reading.
I look in this header file and what do I see - a custom uint24_t struct. Interesting...
I keep sifting through the code base. I find some functions that check and change Endianess. Ok, but the software is developed, built on and runs only on Win7 and later desktop systems. Never mind...
Further I find a custom "allocator" that is used throughout the whole code base. It has three inline static class member functions: allocate, copy and deallocate plus some private constructors. And these just wrap around the standard new and free calls. Some flavours of this class actually only deallocate (with a comment above them: "This allocator does not allocate. HANDLE WITH CARE!!!", which is btw the only "code documentation" I have managed to find).
But wait! What is this? A custom thread and mutex. Oh, and string, and vector.
Further down the rabbit hole I find a custom math library with a matrix class that does not support multiplication between a matrix and a vector. Perhaps not a use case I guess...
I continue and come across some UI-related calls. Interesting, I wonder what they are using as a framework. Oh, my...We have an extensive GUI custom framework written from scratch (drawing buttons and all).
All of this is to load an OBJ file and render it on the screen on a standard Windows PC in some way.
Very nice... ;_;1 -
The more I'm on here the more I remember all the shit I have had to deal with in the past.
Anyway, lets rant! I just moved cities after college to be closer to my family, I didnt have any work lined up at that stage but started job hunting the moment I was settled in, I did some freelance for smaller companies to stay afloat.
Eventually I got a job at this agency startup where "SEO" was there main focus, still very inexperienced they put me on frontend and data capturing but will teach me how to code using their systems in due time. At this stage I was getting paid minimum wage, but I was doing minimum work and it wasnt that bad.
A new investor bought 49% of the company and immediately moved into the office space to focus more on marketing (He was one of those scaly marketing guys that will sell you babies if he could get his hands on enough to make a profit).
This is where everything starts going to shit. He hires a bunch of "SEO Gurus", fills up the small office with people like sardines squished together. Development was still our main money maker at this stage, so there where 3 new more senior developers at this stage and I started learning a lot really fast.
Here are some of the issues we had to deal with:
1. Incentives - Great more money, haha! No, No, you where 5 minutes late so you only get half of the promised amount.
2. For every minute you are late we will deduct it from you paycheck (Did I mention I was getting paid minimum wage).
3. If you take a smoke break we will dock it from your pay.
4. Free gym membership to the gym downstairs, but you can only go once a week during your lunch.
5. No pay raises if you cant prove your worth on paper.
He on purposely made up shitty rules and regulations to keep us down and make as much profit as he could.
Here are some shitty stuff he has done:
1. We arent getting a 13th check this year because the company didnt make a big profit - while standing next to his brand new BMW.
2. Made changes over FTP on clients work because we where too slow to get to it, than blames me for it because its broken the next day and wants to give me a written warning for not resolving the issue Immediately. They went as far as wanting to fire me for this, gave me 1 day notice for meeting and that I can bring a lawyer to represent me (1 day notice is illegal, you need 5 days where I am from), so I brought a lawyer since my mom was a lawyer. They freaked the fuck out and started harassing me about this a week later.
3. Would have meetings all the time about how much money the company is making, but wont be raising our pay since no one has proven they are worth it yet.
4. Would full on yell at employees infront of the entire office if they accidentally made an mistake on a clients project.
One one occasion I took a week off for holiday, my coworker contacted me to ask a question and I answered that I will handle it when I am back the following week. Withing 2 hours my other boss phones me in a rage, "he is coming to fetch the company laptop from my house in 5 minutes, he will let me know when he arrives. Gives me no time to talk at all and hangs up - I have figured out what has happened by now so when he showed up he has this long speech about abandonment, and trust and loyalty to the company. So I pass him my laptop once he shut up and said: "You do know I am on holiday leave which you approved, right?", he goes even more silent and passes me back my laptop without saying anything, and drives off.
While the above was happening Douche manager back at the office has a rage as well and calls the whole office (25 people) to a meeting talking about how I abandoned the company and how disgraceful that is.
Those are the shitty experiences I can remember, there where many more like this. All of the above eventually led to me going into a deep depression and having panic attacks weekly, from being overworked or scared to step out of line. Its also the reason I almost stopped coding forever at that stage. I worked there for 2.5 years with the abuse.
I left 2 weeks after the last shit show, I am ok now and have my anxiety and depression well under control if not almost gone completely.
Ran into Douche Manager a few months ago after 9 years, the company got bought out and the first person they fired was him. LOL! He now has his own agency and is looking for Developers (They are hard to find he says), little does he know I spread his name far and wide to all and every Dev I knew and didnt know to avoid working for him at all costs. Seems like word of mouth still works in this digital age.
Thanks for reading this far!5 -
- A girl asks on FB how to deal with a problem in her Windows computer: the system is asking her to introduce the serial key.
- I comment her the possibility of using Linux in case her use cases are simple enough (web, music, videos).
- First reactions are even enthusiastic, some people who had good experiences join the thread to express their delight with Linux.
- Then a guy arrives to tell us how irresponsible we are, telling a poor girl who does not even know how to introduce the serial key... to use Linux (a super complex system!)
- So I tell the guy that Windows is not simple at all, and that most of the times, people just rely o knowing someone else with higher expertise than them, who always end up paying the price of solving the problems caused by Windows, so the users don't really feel how painful is Windows compared to other systems.
- The girl, who was enthusiastic at first, and seems to be not very bright, to say the least, completely misunderstands my answer. She interprets that I'm insulting the poor guys that act as IT service for free, and calls me a "know-all/smartass" (those words are not even close to their Spanish counterpart on pushing down people who know stuff, we are experts on that there, we didn't loose an empire in the 17th century by respecting the wise ones).
This is, in part, why I stopped helping those dumbasses 18 years ago. I forbid myself to learn anything new about Windows (at user level) so I couldn't help these ungrateful and ignorant people who don't make any effort to learn anything by themselves.19 -
I'm starting to feel super frustrated with my job.
Sometimes I feel like people who work for large tech companies must have it easy. My company is trying to do this digital transformation thing. Modern development practices Scrum, agile, CI/CD etc. So I was put on a team to work on a project with this new methodology. The idea was we would build the front end and interface with the core systems via service calls. Of course it didn't work out that simple and we had to add our own server side stuff but whatever. It's really hard without a point of reference for any of this stuff. We don't have established coding standards, the data we are working with is a mess, incompetent vendors, the infrastructure team supporting the environments can be such arrogant fucks when we need their help to get shit done. The team also doesn't have any members who really know the core systems well. I am the only developer on the team who is an employee of the company the rest are contractors who are in and out. Last week it was literally just me. This is my first job out of school btw I've been here a year now. I guess I just feel frustrated that I have to figure out so much on my own I don't really have many senior devs at the company I can look to. And on the team I've sorta ended up in an unofficial leadership position. Feels like a lot on my shoulders. I feel like if i could have worked for a bigger company I could learn to do a lot of things better. I feel like there's too much on me for the amount of experience I have or am I wrong ?5 -
Did anyone ever thought of just shutting down their systems, taking their bagpack and just straight away go on a break ?
No office phone calls!
No worries!
Just some quality you-tine.11 -
At my institution there is a sys admin that belongs to an entirely different department. They have their own systems on their own network, separated from ours. I do not care, nor do I mind at all, but this is the second time I've had to put their admin in his place.
The first instance was when we had a security firm gauge our systems for vulnerabilities etc. The one that they have was fine, but required some additional configurations on their Tomcat servers. The "sys-admin" contacted I.T (my department) in order to request assistance, the net manager was the one he contacted, and he told the dude that he is not familiar with the Tomcat environment that they have, but that I, the dev manager, would possibly give him some pointers. The net manager is my friend, and he knows how much of a dickhead I am, so he was careful in what he told him. So the dude calls me:
"Hey, I need some items fixed on my Tomcat servers, they told me you have to do it"
Me: "Who? those are your servers"
Him: "The net manager said that you would do it"
Me: "I am certain he didn't tell you that bud, no one here will take care of your servers, they are yours, I am not doing any configurations on your stuff, that is your job"
Him: "Can't you just do them?"
Me: "No, bye"
The little bitch escalated it to my department director, who told him exactly the same thing, the director did ask if I would be willing to assist, I told him no since even though his configurations were minimal, I was not going to put myself in the position to which that fucker's ineptitude would cause him to point fingers at me, director backed me up and told the fucker to deal with his own shit.
This year it came to my attention that not only do they have their owns servers, but their own SSO system. This moron contacted me, tagging VPS and such in the email to tell me that I had to configure his SSO because "they told me you had to do it". The same shit happened, but this time I put him on blast during a meeting and told him that as "sys admin" for his stuff it was his responsibility to deal with the SSO that they have, and to contact the vendor to ask for the specifications. In front of EVERYONE he asked me if I could do it for him, I fucking looooooooled and told him that he just admitted to not being able to do his job (for which he is paid handsomely) in front of the entire room of VPS. One VP asked me why I was not willing to help him, and I told the VP that it would be the equivalent of me taking his vehicle for services, it is not my vehicle, thus not my responsibility. The VP agreed and told the fucker to get on with it and do what I said: contact his vendor channels to figure it out himself since it was indeed his position.
Yet again he said that he didn't know about SSO configs and that he was "told that I would do it", everyone asked who the fuck told him that and he said that the vendor, they asked again how it was and he showed the message from the vendor telling him: "Have your SSO admin perform the following <bla bla bla bla>" they asked him who was the manager for the SSO that they had. He said that it was him. Then they asked him what logic made him believe that it should be me, he stated again "they told me it was him".
I could hear everyone's brains shortcircuiting as no one could believe someone would be this fucking dense.
I don't think he will continue to have his job for much longer. I understand not knowing something, and I would have been happy to give pointers since I do administer systems of that level, but I can't with the whole made up "they said he would do it"
Bitch who said that? just say that you want me to do it because you can't, I mean, I am still not fucking doing it, but damn. Fucking morons man.5 -
Got to love when the on-call takes their sweet-fucking time with the back-up. All this because the lazy douche did not what to verify the back-up was complete.
-
The networking group at my day job, hooooooolly crap I have some unprintable words. But keeping it professional:
* Days to turn around simple firewall whitelisting requests
* Expecting other teams to know the network layout despite not sharing that information anywhere and going out of their way to not share it
* Adding bureaucracy in the form of separate Word doc forms despite having a ticketing system - for no justifiable reason
* Breaking production systems multiple times per month
* Calling in with problems that are clearly network related, being told it’s our systems, and then the problems magically go away even though they swear they didn’t touch anything
* Outright verifiable lies or vague non-answers when they’re not talking to someone at the director level or a vendor from an outside company on conference calls
* Worse packet loss and throughput on our LAN than my home ISP
Doing anything with these clowns is my single biggest source of stress right now. I can’t wait until we get a full SDN stack set up and then we won’t have to deal with them for day-to-day needs any longer.
My boss swears it’s better that we’re not managing the network directly, but I’m pretty sure my friend’s dog could be loosed into the data center to chew on fiber, and eventually the pairs would be connected in such a way as to improve performance.1 -
When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.
Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.
When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.
Speaking of which, there's more than six microservices per developer!
Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?
When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:
1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.
Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).
Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.
When confronted/advised about these issues the response from the CTO/CEO
varies:
(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".
Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.
There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".
It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?2 -
Some humans are calm and thoughtful, some annoyingly complicated, while others with behaviours too difficult to comprehend.
I got a call from the office (former from 6 months ago) and it's from the G.M herself.
** Phone rings **
Hmm see who's calling...
Me: * Picks up phone and set it on loud speaker, so my partner can also listen *
Me: Hello Ma
G.M: Hey (calls me by my full name)
Me: It's really nice to he...
G.M: Why would you move the YETI server hosted on AWS to Azure! We have been faced with lots of challenges ever since and that has cost the company a lot.
Me: Pardon me Ma, but that...
G.M: That is a very bad and unacceptable behaviour from you and I can have the company sue you for this.
Me: Excuse me Ma, but...
G.M: I have spoken with the director of C.M.D quaters (A sister company) and explained the situation on ground about what you did before leaving without having any prior permission. What nonsense!
** At this point my partner let's call her "CC"... was more confused than me**
CC : **Panicking** Who's that? What did you do? I thought you said you no longer work at that firm, what's going on?
Now I'm confused cus I don't even know who to reply.
Me: **Signals CC to calm the fuck down**
G.M: ** Still talking and spitting out millions of threats to the guy who left the company with evil deeds in mind...**
CC who literally hates suspense and also a half cool and half crackhead kind of person... Tries stealing the phone from me so she could pour out whatever is on her mind to the caller because of how disgusted she felt, mostly for reasons I quite understood but nevertheless i kept the phone far from her reach while we both enjoyed the suiting voice of *a threat giver*
Honestly at this point my closest guess was "Joe, who must have fucked up big time" because Joe is the company's SysAdmin and has a lot of fucked up records (One time Joe tried to convert all system OSes to Linux even with our hydra servers with pre-installed windows running smoothly, his action caused a noticeable server down-time all for the reason of Joe being a Linux freak). He and only he has the power to transfer/switch/off/on servers at will. I really don't know what Joe must have done but sure thing is there is a fuck up somewhere.
Talking about me, I was only a developer enthroned only within his desk and secondly I no longer worked there. Who fucking calls a retired soldier about a lost battle after six freaking months later! Just fucking sink with your ship captain!
But how can I explain all of this to G.M without implicating Joe and also not look like snitch, I thought to myself.
While I was pondering within myself and the call which has long been disconnected, CC broke the silence.
CC: Giddy, Can you honestly explain me why your old company is calling talking about lawyers and suing you? Have you been lying to me about your work?
Me: *Explained the situation to CC*
CC: But why was she that saucy and acting a bitch? You should have spare me a minute with her.
Me: She wouldn't let me speak but we good CC. We good.
The woman that just called is the G.M. of the firm I had formerly worked with and she's also the wife to the M.D of the same firm which was my former direct Boss whom I respect a lot. Having a disjunct with the wife can also affect the relationship with the husband, which I don't want to lose. So we cool!
Maybe I should text her or maybe not... But before then
** Another call comes in **
It's her again.
GM: Hello Giddy (Sounding calm)
Me: (WTF. She called me by my first name and also sounds cool... More confused than a stray dog) ...Yeah Hello
GM: I just called to let you know that my accusation was wrong because I was misinformed. Joe Nosa was in charge on Systems but why didn't you correct me on that during our last conversation?
Me: ... 😲
CC: (Drags the phone) Hello and Good morning whosoever...
G.M: Sorry who am I speaking with?
CC: (Introduced herself) I overheard your last conversation with Giddy, and I demand you appogise to him both in written and in verbal because not only did you accused him falsely, you also almost bridge the trust between us which may have cost the relationship.
Me: ...
** Long awkward silence **
G.M: Hey Giddy, I'm sorry. Just angry about what went down recently.
Me: All good ma'am
CC: ** Hangs up **1 -
So I figure since I straight up don't care about the Ada community anymore, and my programming focus is languages and language tooling, I'd rant a bit about some stupid things the language did. Necessary disclaimer though, I still really like the language, I just take issue with defense of things that are straight up bad. Just admit at the time it was good, but in hindsight it wasn't. That's okay.
For the many of you unfamiliar, Ada is a high security / mission critical focused language designed in the 80's. So you'd expect it to be pretty damn resilient.
Inheritance is implemented through "tagged records" rather than contained in classes, but dispatching basically works as you'd expect. Only problem is, there's no sealing of these types. So you, always, have to design everything with the assumption that someone can inherit from your type and manipulate it. There's also limited accessibility modifiers and it's not granular, so if you inherit from the type you have access to _everything_ as if they were all protected/friend.
Switch/case statements are only checked that all valid values are handled. Read that carefully. All _valid_ values are handled. You don't need a "default" (what Ada calls "when others" ). Unchecked conversions, view overlays, deserialization, and more can introduce invalid values. The default case is meant to handle this, but Ada just goes "nah you're good bro, you handled everything you said would be passed to me".
Like I alluded to earlier, there's limited accessibility modifiers. It uses sections, which is fine, but not my preference. But it also only has three options and it's bizarre. One is publicly in the specification, just like "public" normally. One is in the "private" part of the specification, but this is actually just "protected/friend". And one is in the implementation, which is the actual" private". Now Ada doesn't use classes, so the accessibility blocks are in the package (namespace). So guess what? Everything in your type has exactly the same visibility! Better hope people don't modify things you wanted to keep hidden.
That brings me to another bad decision. There is no "read-only" protection. Granted this is only a compiler check and can be bypassed, but it still helps prevent a lot of errors. There is const and it works well, better than in most languages I feel. But if you want a field within a record to not be changeable? Yeah too bad.
And if you think properties could fix this? Yeah no. Transparent functions that do validation on superficial fields? Nah.
The community loves to praise the language for being highly resilient and "for serious engineers", but oh my god. These are awful decisions.
Now again there's a lot of reasons why I still like the language, but holy shit does it scare me when I see things like an auto maker switching over to it.
The leading Ada compiler is literally the buggiest compiler I've ever used in my life. The leading Ada IDE is literally the buggiest IDE I've ever used in my life. And they are written in Ada.
Side note: good resilient systems are a byproduct of knowledge, diligence, and discipline, not the tool you used. -
I really hate working with learning management systems (LMS).
I make training simulations for retail companies and some of these have the worst, backwards LMS's out there.
The providers who install and manage these LMSs for the companies always insist we make our training run inside their own environment, but we can't since it's a 3D training made in Unity that doesn't run well in a browser.
Luckily some of these are fine to figure out. Just a few API calls here and there for authorization and reporting progress, but some are an absolute nightmare.
Just now one of the providers provided me with a 2000 page documentation of all the functions of the LMS's API that our customer is using. All I need are like 5 pages that explain what URL to call with what data and the responses, but now I'm stuck spending days trying to find the 0.5% of this documentation that I need to communicate with their API.
And of course, the documentation is vague as all hell. minimal descriptions of what each endpoint does. Subjects names are super vague, as in do I look for course progress or lesson completion state. What the heck is a Learning Event, is it relevant to me?
And the errors in this document, too.
Bullet-point lists with duplicate items.
language errors everywhere.
Property lists where they copy-pasted the description of properties.
An entire EMPTY chapter, literally a page with only the chapter's title.
I just can't stand how these providers barely seem to know anything about the API of the LMS's they provide to customers.
(for clarity, the LMS is produced by some big tech company, it's installed and maintained by some 3rd party which is our main line of communication when rolling out trainings to these).
It always goes like: "Hey, we want to use your training." "Oh, that's great, we have our own, simple LMS where you can view your employee's progress." "Nah, we want to use our backwards LMS. Here's a giant manual about it's API, go figure it out!"
And then I'm left here tearing my hair out trying to figure out which 3 calls I need to send their API from the tons of extra stuff it can do which is completely unnecessary and being unable to rely on the provider because they lack the knowledge and have such thick skulls about the implementation of the LMS itself that they also seems completely unwilling to help to begin with!
Just another day at the office. -
AWS Contractor
I've been putting a web application together that I'm looking to have published on AWS. Not having too much experience with AWS, I am looking to hire a contractor. I've had a number of quotes from different AWS admin's ranging from $40 an hour to $200 an hour, from 1-days worth of work to 2-months worth of work!
I'm not really sure what to make of it or to whom to trust. I believe they’re using my ignorance to overcharge me. I've listed my requirements below, could you guys use your professional experiences to let me know what you think is reasonable charge and where best I could find someone to help me.
My application is a US shopping website where people can set up an online shop and upload their products and maintain an inventory of the items.
This is what I’m looking for setup and configuration with the following two areas:
1) AWS SYSTEMS…
* AIM - Set up my server admin users.
* EC2 - Web Hosting.
* RDS - Fast DB.
* SES - To send emails.
* S3 Buckets - Uploaded image hosting.
Route 53 - I don’t know but someone said I should have this.
* Elastic Load Balancing - For, well, load balancing.
2) SCRIPTS…
* A script that would back up the database once a day and save it to a private S3 Bucket.
* A script that will run once a day that calls an internal API, and POST a query to it.
* A script that runs once every 90 days, to refresh the SSL using ZeroSSL.com
Is there anything that I've missed such as security systems, firewalls, auto scaling and CDNs?
The quotes that I've received arranged from $320 to $64,000. I know I am being abused because of my ignorance. I would never overcharge someone because the customer doesn't know the efforts of the work. I hope someone here can help to understand the efforts needed and can tell me the true cost.
Thank you6 -
Worst enterprise software experience... I was fresh out of college, and needed money. I was working in a call center, fielding IT helpdesk calls for a major US telecom company, who had just acquired a competitor. One day I got to work and about ten of us were given a new desk, new phone number, an an email address at the newly acquired company. My manager said to us "We have no clue how any of their proprietary systems work, what servers they run on, or how to login to them. Your phones are ringing, make sure you take good notes so the Tier-1s can help out next week. Good luck."
Trial by shit-storm fire, all while trying to convince the caller that yes, I did know what I was talking about. It was a lot of cold calling random employees whose job title in the corporate directory looked even remotely close to somebody I could escalate a ticket to. They didn't use the same ticketing system we used, so it was a lot of copy/pasting between two ticketing systems. To this day, I still have no clue what happened to their original call center staff. I'm sure they must have had one, but it seemingly just dissolved overnight.
That job was the springboard to my development career. I left for a gig in software helpdesk, then to quality assurance, automated testing, and now I'm a senior DevOps engineer. It was worth it. -
So how do you find motivation to finish a work project which is supposed to "go long"?
So, umm, this is weird, but i have been in this situation a few times and i am not sure if i deal with them correctly.
- the company proposes a brand new feature : a feature which never existed in the product before.
- they have high level directions (both business logics and technical) on how its supposed to be build
-they set vague but comfortable timelines (20-30days) to complete it
- they align me as the main dev for frontend, some x guy for backend , some y guy for parallel frontend (ios/web) and kinda forget about us.
- the business requirements are evolved/cleared as we go on making the product, the backend keeps on providing evolving apis which get stable over time.
- the business ppl shows that yeah there is no pressure and we won't mind extending this for release as other systems will be "obviously" taking time.
- our (the folks on new feature) feature is sidelined .nd we are rarely talked about until we reach those deadlines and at that time we are questioned.
I... am not a powerful performer in these situations. adding a new feature required solving some major problems again and again , while solving smaller problems too, so as the product finally takes shape . for eg:
1. i will start fast by adding all the possible screens, their abstract code, their navigation logic, their xmls etc
2. then based on designs, i will try creating designs a bit
3. then once the apis arrive, i start adding them and modify the logic to handle those.
4. meanwhile many smaller problems come up , like when sending an image from one screen to the previous screen, the thumbnail don't show up, so i spend 5+ hours ensuring that it works precisely . or how i could make 3 api calls in async and make the upload flow better.
5. this goes on for days, until and i and other people start to realise that my project is not upto the point of completion
i keep getting distracted from the original goal of making a working poc first and then fixing the nuances2