Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "logging"
Fuck the memes.
Fuck the framework battles.
Fuck the language battles.
Fuck the titles.
Anybody who has been in this field long enough knows that it doesn't matter if your linus fucking torvalds, there is no human who has lived or ever will live that simultaneously understands, knows, and remembers how to implement, in multiple languages, the following:
- jest mocks for complex React components (partial mocks, full mocks, no mocks at all!)
- token cancellation for asynchronous Tasks in C#
- fullstack CRUD, REST, and websocket communication (throw in gRPC for bonus points)
- database query optimization, seeding, and design
- nginx routing, https redirection
- build automation with full test coverage and environment consideration
- docker container versioning, restoration, and cleanup
- internationalization on both the front AND backends
- secret storage, security audits
- package management, maintenence, and deprecation reviews
- integrating with dozens of APIs
- fucking how to center a div
and that's a _comically_ incomplete list; barely scratches the surface of the full range of what a dev can encounter in a given day of writing software
have many of us probably done one or even all of these at different times? surely.
but does that mean we are supposed to draw that up at a moment's notice some cookie-cutter solution like a fucking robot and spit out an answer on a fax sheet?
recruiters, if you read this site (perhaps only the good ones do anyway so its wasted oxygen), just know that whoever you hire its literally the luck of the draw of how well they perform during the interview. sure, perhaps some perform better, but you can never know how good someone is until they literally start working at your org, so... have fun with that.
Oh and I almost forgot, again for you recruiters, on top of that list which you probably won't ever understand for the entirety of your lives, you can also add writing documentation, backup scripts, and orchestrating / administrating fucking JIRA or actually any somewhat technical dashboard like a CMS or website, because once again, the devs are the only truly competent ones - and i don't even mean in a technical sense, i mean in a HUMAN sense of GETTING SHIT DONE IN GENERAL.
There's literally 2 types of people in the world: those who sit around drawing flow charts and talking on the phone all day, and those WHO LITERALLY FUCKING BUILD THE WORLD
why don't i just run the whole fucking company at this point? you guys are "celebrating" that you made literally $5 dollars from a single customer and i'm just sitting here coding 12 hours a day like all is fine and well
i'm so ANGRY its always the same no matter where i go, non-technical people have just no clue, even when you implore them how long things take, they just nod and smile and say "we'll do it the MVP way". sure, fine, you can do that like 2 or 3 times, but not for 6 fucking months until you have a stack of "MVPs" that come toppling down like the garbage they are.
How do expect to keep the "momentum" of your customers and sales (I hope you can hear the hatred of each of these market words as I type them) if the entire system is glued together with ducktape because YOU wanted to expedite the feature by doing it the EASY way instead of the RIGHT way. god, just forget it, nobody is going to listen anyway, its like the 5th time a row in my life
we NEED tests!
we NEED to know our code coverage!
we NEED to design our system to handle large amounts of traffic!
we NEED detailed logging!
we NEED to start building an exception database!
BILBO BAGGINS! I'm not trying to hurt you! I'm trying to help you!
Don't really know what this rant was, I'm just raging and all over the place at the universe. I'm going to bed.20
Worst hack/attack I had to deal with?
Worst, or funniest. A partnership with a Canadian company got turned upside down and our company decided to 'part ways' by simply not returning his phone calls/emails, etc. A big 'jerk move' IMO, but all I was responsible for was a web portal into our system (submitting orders, inventory, etc).
After the separation, I removed the login permissions, but the ex-partner system was set up to 'ping' our site for various updates and we were logging the failed login attempts, maybe 5 a day or so. Our network admin got tired of seeing that error in his logs and reached out to the VP (responsible for the 'break up') and requested he tell the partner their system is still trying to login and stop it. Couple of days later, we were getting random 300, 500, 1000 failed login attempts (causing automated emails to notify that there was a problem). The partner knew that we were likely getting alerted, and kept up the barage. When alerts get high enough, they are sent to the IT-VP, which gets a whole bunch of people involved.
VP-Marketing: "Why are you allowing them into our system?! Cut them off, NOW!"
Me: "I'm not letting them in, I'm stopping them, hence the login error."
VP-Marketing: "That jackass said he will keep trying to get into our system unless we pay him $10,000. Just turn those machines off!"
VP-IT : "We can't. They serve our other international partners."
<slams hand on table>
VP-Marketing: "I don't fucking believe this! How the fuck did you let this happen!?"
VP-IT: "Yes, you shouldn't have allowed the partner into our system to begin with. What are you going to do to fix this situation?"
Me: "Um, we've been testing for months already went live some time ago. I didn't know you defaulted on the contract until last week. 'Jake' is likely running a script. He'll get bored of doing that and in a couple of weeks, he'll stop. I say lets ignore him. This really a network problem, not a coding problem."
IT-MGR: "Now..now...lets not make excuses and point fingers. It's time to fix your code."
IT-VP: "I agree. We're not going to let anyone blackmail us. Make it happen."
So I figure out the partner's IP address, and hard-code the value in my service so it doesn't log the login failure (if IP = '10.50.etc and so on' major hack job). That worked for a couple of days, then (I suspect) the ISP re-assigned a new IP and the errors started up again.
After a few angry emails from the 'powers-that-be', our network admin stops by my desk.
D: "Dude, I'm sorry, I've been so busy. I just heard and I wished they had told me what was going on. I'm going to block his entire domain and send a request to the ISP to shut him down. This was my problem to fix, you should have never been involved."
After 'D' worked his mojo, the errors stopped.
Month later, 'D' gave me an update. He was still logging the traffic from the partner's system (the ISP wanted extensive logs to prove the customer was abusing their service) and like magic one day, it all stopped. ~2 weeks after the 'break up'.8
So... I'm pretty much dead inside.
But today I laughed in a meeting.
Nearly died of laughter.
We're currently understaffed for various reasons, especially the ongoing migrations etc.
So a lot of projects are currently in "maintenance" mode (e.g. no new features) - cause we lack the necessary man power.
The meeting was more or less:
Team: We had an ongoing discussion in the team regarding logging and possibilities of tracing and XY suggested we implement OpenTelemetry in *all* projects in the next weeks, can we do that?"
Sometimes I'm not sure If I'm in a sitcom for torture experts.4
Dear Atlassian Support,
In my life I had a lot of experiences...
But your software manages to replace all these experiences with a unique feeling of depression, hatred, anger... Only negative emotions.
Not once have I said anything good about your software - not once in > 5 years.
Whenever your chum bucket of mismanagement and misanthropy stops working, it's never the fault of the end user, the administrator or someone else.
It's entirely your fault.
Fucked up upgrades, lack of documentation, catastrophic handling of logging, lack of support of current database systems, lack of proper migration and clean up of plugins, ....
I could go on. But it's really just and endless tirade.
I wish I could stop management for even giving you money for the pile of poo you call software, but sadly they don't listen.
But there's hope on the horizon.
Thanks for making people go cloud only.
No one wants that.
It would mean entrusting that pile of poo to the craptastic hands of your irresponsible people.
No one really wants that.
Not even management who blindly paid the license fees all the times.
Thank you for your cloud only movement.
Maybe we can finally find an alternative and I can finally start a therapy for the PTSD I have thx to your software.3
I created our login system to be secure and reliable.
One coworker hardcoded the roles a person who is logging in receives and built a backend to just assign roles you want. He pushed this to prod...
X: Hi, regarding that ticket that you made...
You said "Implement logging to find out the culprit in site generation"...
What do you mean exactly?
Me: "Read the meeting notes, we had a full discussion on this 2 weeks ago".
X: "We don't understand it..."
Me: "As I said before, I have no experience in this tech stack... I'd expect bla to have a logging framework and I'd - for easier recognition - implement additional logging levels based on criteria <me just reading the meeting notes>"
X: But how do we do it?
I wish I had invented this discussion.
Because it hurts.
For the jolly of it, I had similar discussions today.
Three times to be exactly.
As I asked some dev what I should do next, put a foley catheter up his urethra or change the bed pan he wasn't amused.
Guess I'll get monday a call of HR.
So Monday I have less work to do, which is awesome.5
I'm getting annoyed with the increasing number of platforms that implement the "Oops, something went wrong" vague error message.9
Still on the primenumbers bender.
Had this idea that if there were subtle correlations between a sufficiently large set of identities and the digits of a prime number, the best way to find it would be to automate the search.
And thats just what I did.
I started with trace matrices.
I actually didn't expect much of it. I was hoping I'd at least get lucky with a few chance coincidences.
My first tests failed miserably. Eight percent here, 10% there. "I might as well just pick a number out of a hat!" I thought.
I scaled it way back and asked if it was possible to predict *just* the first digit of either of the prime factors.
That also failed. Prediction rates were low still. Like 0.08-0.15.
So I automated *that*.
After a couple days of on-and-off again semi-automated searching I stumbled on it.
[1144, 827, 326, 1184, -1, -1, -1, -1]
That little sequence is a series of identities representing different values derived from a randomly generated product.
Each slots into a trace matrice. The results of which predict the first digit of one of our factors, with a 83.2% accuracy even after 10k runs, and rising higher with the number of trials.
It's not much, but I was kind of proud of it.
I'm pushing for finding 90%+ now.
Some improvements include using a different sort of operation to generate results. Or logging all results and finding the digit within each result thats *most* likely to predict our targets, across all results. (right now I just take the digit in the ones column, which works but is an arbitrary decision on my part).
Theres also the fact that it's trivial to correctly guess the digit 25% of the time, simply by guessing 1, 3, 7, or 9, because all primes, except for 2, end in one of these four.
I have also yet to find a trace with a specific bias for predicting either the smaller of two unique factors *or* the larger. But I haven't really looked for one either.
I still need to write a generate that takes specific traces, and lets me mutate some of the values, to push them towards certain 'fitness' levels.
This would be useful not just for very high predictions, but to find traces with very *low* predictions.
Why? Because it would actually allow for the *elimination* of possible digits, much like sudoku, from a given place value in a predicted factor.
I don't know if any of this will even end up working past the first digit. But splitting the odds, between the two unique factors of a prime product, and getting 40+% chance of guessing correctly, isn't too bad I think for a total amateur.
Far cry from a couple years ago claiming I broke prime factorization. People still haven't forgiven me for that, lol.6
Ask me about that one time a motherfucking LOG STATEMENT caused the code to not work properly, breaking both the Test and QA environments, but failed in a way that made it maddening to figure out (in conjunction with the cloud-based hosting environment and the abomination that is centralized logging, which just makes EVERYTHING more difficult).
Actually, DON'T ask me about it, because it was today, it wasted most of my day, and I'm still salty as fuck about it.6
Supervisor: YOU NEED TO INCREASE THE COVERAGE OF YOUR UNIT TESTS! THE FILE logger.js DOESN'T HAVE >80% COVERAGE! IMAGINE PICKING THIS UP 6 MONTHS FROM NOW!
Bro. It's a Winston instance.
I am literally exporting a fucking Winston instance with 0 custom logic.
If 6 months from now I take a file and can't understand a Winston instance anymore, you're well within your right to fire me on the spot.2
We receive an email from Splunk when errors go above a certain threshold, and a particular service has been especially problematic this week (throwing hundreds of exceptions). Email response from the team mgr responsible for the service.
"We are working to address these errors. We don’t currently have a way to prevent a user who’s account is locked from logging into the service and performing work."
The exception? NullReferenceException: Object reference not set to an instance of an object.
The code? (paraphrasing)
var user = GetUser(request.Login);
if (user.CanPerformWork) ...
I'm doing my best not to reply .."Really? No way? You do realize we can read code, right?"3
I am building my portfolio website and added a contact section. In the API call to the backend, I am logging potential API failures to Firebase Analytics. Is it ethical to include the request data (content of the contact form) in log data?5
I dont understand the Log4j vulnerability.
Isnt the ability to execute code a feature they added so that you can add dynamic data to the logs?
If it is a feature then isnt it written in the documentation?
Is the problem that a lot of companies forgot to sanitize the input before logging it?23
God I fucking hate macs.
I got a mac at work. I tried to install ubuntu, with rather questionable results (unfortunately, I expected that) - so I tried to get mac work for me the way I like a system to work. I needed to download slack, simple enough, right? Ha, you wish. It's gotta be done through Apple store, so I went to create an Apple ID inside the Apple Store form. And, well, it just errored out on the submission. Great start. I went then to the settings and created an account there, great success, went back to Apple Store. Unfortunately being logged in at the system level doesn't mean you are logged in to the store. So, I went to log in to the store, simple enough, right? No, nothing's simple with Apple. After logging in I got a message that the Apple ID has not yet been used with Apple Store and that I need to review the account's setting. So, I click the "review" button and... I'm presented with a log in form. Yep, a perfect log in loop. I can't log in because I can't review the account but I can't review the account because I can't log in. Fun :)
You can't just go to the web admin panel for your account to review it for Apple Store, that would too be too easy. After a bit of searching I've found an answer on StackOverflow. You need to log in to iTunes. Through a fucking MUSIC APP. To install a free application from the store you need to log in to a music app. Yes, we're all mad here.
Then, after finding out that to be able to use side buttons on my mouse I need an app that I need to manually restart every time I restart the machine and that I need to have an app to fucking transfer files from an android I need another fucking app, because reading a storage of a linux-based system would be too standards compliant - something in me broke. I found out that installing windows on a mac is officially supported.
Supported doesn't mean that it's easy. I tried to install it trying different solutions from SO, but each time I would get an error that Windows couldn't modify the boot partition. Turns out that even wiping the drive and reinstalling OSX doesn't remove residual files on a boot partition and Windows installer is not allowed to modify them. It took me hunting into some shady looking site to actually find this answer. I have no fucking idea how long it all took me, but, finally, great success, Windows, WSL, side buttons working, I can even install slack from an installer. I just wish I could have those hours of my life back.19
A large pool of application instances' is writing logs to the same physical file. No way to distinguish which instance wrote which line.
Welcome to hell
We're being asked questions. We're replying that we cannot help unless logging is fixed. Noone's bothering to fix this mess and instead returns tickets with requests to investigate more.
Continuing to learn k8s ecosystem and to achieve acceptable level
With trying eventually Helm, Argo CD and even trying to use not managed setup for k8s.
Going though books to find out theory about being SRE.
And about data intensive apps.
Learning and trying Kafka
Learning and trying FastAPI and diving in generally to async python ecosystem
Learning few more books to increase code quality and its compositioning.
Getting more practice in monitoring and logging systems with applicating them to k8s.3
Microsoft Windows can burn.
I have this feature where I configure a remote API via some endpoints and the API pushes data back to some webhooks in my API.
Yesterday I set everything up for the final test; fired up my own API with some test data, added some configuration and started trace logging to ensure that everything works as expected when the remote site tries to send me data.
I was ready to collect ! Enter this morning: Windows have forcibly rebooted to install an update and shut everything down.
inb4 install Linux; No, I can not. Windows is company policy and I am required to use shit that is only designed for Windows.6
We had made an api which had endpoints for each different domain model, so /user, /company, the usual. Beyond being restful they all had basic filtering and pagination.
We also had an endpoint to return an entity from any set based on guid for when you needed to attach the related entity to notifications and logging and such.
We received a bug report on how you couldn't use filtering or pagination on this endpoint, and after weeks of asking what they need it for we just had to implement it.
You can imagine how non-trivial it is to "just" filter across different datasets, but we eventually got it working so now you can get a user via /user/123 or /entity?type=user&id=123. They only use it for one type and id at the time.2
> totally disable logging on console
> logging is redirected instead on some 1200 separated files (no, not "log files rotated through 1200 days"; every part of the program has its own log file, and there's 1200 of them)
> still, in all those files, none of them contains MY log; somehow it disappeared
Why some people even chose to become developers will always elude me2
Somebody: (whinwy) we need something to log into nonprivileged technical accounts without our rootssh proxy. We want this pammodule pam_X.so
me: this stuff is old (-2013) and i can't find any source for it. How about using SSSD with libsss_sudo? Its an modern solution which would allow this with an advantage of using the existing infrastructure.
somebody: NO I WANT THIS MODULE.
me: ok i have it packaged under this name. Could you please test it by manipulating the pam config?
Somebody: WHAT WHY DO I NEED TO MANIPULATE THE PAMCONFIG?
me: because another package on our servers already manipulates the config and i don't want to create trouble by manipulate it.
Somebody: why are we discussing this. I said clearly what we need and we need it NOW.
we have an package that changes the pam config to our needs, we are starting to roll out the config via ansible, but we still use configuration packages on many servers
For authentication as root we use cyberark for logging the ssh sessions.
The older solution allowed additionally the login into non-rootaccounts, but it is shut down in the next few weeks after over half an year of both systems active and over half an year with the information that the login into non-privileged accounts will be no more.7
I had a pretty good year! I've gone from being a totally unknown passionate web dev to a respected full stack dev. This will be a bit lengthy rant...
- Got my first full time employment dev role at a company after being self-taught for 8+ years at the start of the year. Finally got someone to take the risk of hiring someone who's "untested" and only done small and odd jobs professionally. This kickstarted my career, super grateful for that!
- Started my own programming consulting company.
- Gained enough confidence to apply to other jobs, snatched a few consulting jobs, nailed the interviews even though I never practiced any leet code.
- Currently work as a 99% remote dev (only meet up in person during the initialization of some projects.) I never thought working remotely could actually work this well. I am able to stay productive and actually focus on the work instead of living up to the 9-5 standard. If I want to go for a walk to think I can do that, I can be as social and asocial as I want. I like to sleep in and work during the night with a cup of tea in the dark and it's not an issue! I really like the freedom and I feel like I've never been more productive.
- Ended up with very happy customers and now got a steady amount of jobs rolling in and contracts are being extended.
- I learned a lot, specialized in graph databases, no more db modelling hell. Loving it!
- Got a job where I can use my favorite tools and actually create something from scratch which includes a lot of different fields. I am really happy I can use all my skills and learn new things along the way, like data analysis, databricks, hadoop, data ingesting, centralised auth like promerium and centralised logging.
- I also learned how important softskills are, I've learned to understand my clients needs and how to both communicate both as a developer and an entrepeneur.
- First job had a manager which just gave me the specifications solo project and didn't check in or meet me for 8 weeks with vague specifications. Turns out the manager was super biased on how to write code and wanted to micromanage every aspect while still being totally absent. They got mad that I had used AJAX for requests as that was a "waste of time".
- I learned the harsh reality of working as a contractor in the US from a foreign country. Worked on an "indefinite" contract, suddenly got a 2 day notification to sum up my work (not related to my performance) after being there for 7+ months.
- I really don't like the current industry standard when it comes to developing websites (I mostly work in node.js), I like working with static websites (with static website generators like what the Svelte.js driver) and use a REST API for dynamic content. When working on the backend there's a library for everything and I've wasted so many hours this year to fix bugs and create workarounds related to dependencies. You need to dive into a rabbit hole for every tool and do something which may work or break something later. I've had so many issues with CICD and deployment to the cloud. There's a library for everything but there's so many that it's impossible to learn about the edge cases of everything. Doesn't help that everything is abstracted away, which works 90% of the time but I use 15 times the time to debug things when a bug appears. I work against a black box which may or may not have an up to date documentation and it's so complex that it will require you to yell incantations from the F#$K
era and sacrifice a goat for it to work properly.
- Learned that a lot of companies call their complex services "microservices". Ah yes, the microservice with 20 endpoints which all do completely unrelated tasks?
I've been asked to release a project which has been written by someone else, then rewritten by another developer, and both have left the company.
I can't release it yet because there is an inconsistent bug throwing some values out.
We've got it running side by side with an older legacy system which it's going to replace. Before the 2nd developer left they added some logging to our live system to record both values so that they could be monitored to make sure there was no inconsistency.
There are some inconsistencies... however, when I run the same data through the new system and the legacy system in a test environment they both come out correct.
I've considered quitting...2
How do I deal with this;
Edge case hiccup on production, no errors in the available logs(very shallow logging), no access to the production server, issue unreproducable on staging and a manager that want me to fix it AFTER I already said that im kind of sailing blind and can't do much without logs or access, and already looked at it with another dev who also has no idea what is going on3