Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "kafka"
Fucking HR cunts. I found a nice computer vision vacancy and decided to apply, look what these motherfuckers require :
- Amazon Web Services
- Spring Boot
- Web Sockets
- Grafana - Prometheus
- Java EE
- Graph QL
FUCK OFF FUCK OFF FUCKING FuCK OFF you monkeys. Shove your vacancy up your ass.
Another vacancy, another instance, I'm going to apply at employers site, (they sell some sort of CRM crapshit) forms not working, bc not fucking connected to anything. Just a fucking HTML and JS popup notification for submission laying around with its legs wide spread. Like its copied from w3schools or something.
Looked more on the site it is fucked from all sides like it's made by a school kid.
So I fired an email to the HR, something like "I went to apply but your forms are not working, I found the issue, it's that. Also I found some glory holes in your site, hope your I.T. finds them too. Anyways here is my CV, have a nice day"
Day later I'm shortlisted.36
After our Head Of Software has terminated.
I started to take control over our development crew. And in this year I did more then the old head in the last 6 years.
- Swapped from plain old SVN to Gitlab.
- Build a complete autonomous deployment with Gitlab.
- Introduced code reviews.
- Started to refactor the legacy product with 500.000 lines of code...
- learned how to use confluent apache kafka and kubernetes to split the legacy project in many small and maintainable one.(not done yet)
- Last 3 weeks I learned how to use elasticstack with kibana and co. That we aren't blind anymore. Big dashboards are now shown in the middle of the room :) and maybe convinced my coworker that we use unity3d for our business application cause of support for all devices and same design on them. And offline capabilities. (Don't know if this was my best idea)
When I look back, I'm proud to did that much in one year alone. And my coworkers are happy too that they have less work with deployments and everything.
But I can't decide what's the title for this. System or Software Architect cause I litterallity did both :/7
First of all, I hate crammers so much. These people kill the industry without even understanding it. They turned interviews into exams, missed the point of hiring, and saw no distinction between knowledge and information all the time. They don't understand that if you can google an answer in five seconds, it's not knowledge. It's information.
They don't understand that questions like 'what will Python do if you delete an item from a dict while iterating over it' are complete nonsense. They don't understand that it's not 'dig deep'; it's just a bad practice that leads to errors, thus must be avoided. The fact of remembering 'RuntimeError: dictionary changed size during iteration' means that you haven't been avoiding it enough.
One more example. Which signature is correct?
Second. What's the point of forcing you to write compilable code in google docs? Do they really expect that one could possibly remember 'import org.springframework.beans.factory.annotation.Autowired;'? Seriously?
Third. Why do they expect me to know Spark, Java, J2EE, Spring Boot, Python, Kafka, Postgres, React/Redux, TypeScript, and work for miserable 70K EUR?
What's wrong with the European IT job market? Are they fucking nuts?9
I was looking for a part time job when I was 15, knew nothing about coding, got a developer job cause I said I'm keen to learn xD .. 5 years later.. I'm still learning but I have a better knowledge now!4
Excited for the following upcoming product releases:
Dark Mode GitHub
Dark Mode Jira
Dark Mode Kubernetes
Dark Mode Kafka Streams
Dark Mode WebEx13
The wife is asleep. The two year old wrecking ball of a boy is asleep. Wanted to finish a library to run kafka/grpc for nodejs microservices with plug and play functionality.
Make some tea. Figured the caffeine will keep me up. Maybe work late night. It'll be fun.
Then diarrhea hits. Now, staying up late because apparently I ate something I shouldn't have. Don't you hate it when that happens?12
A lot of engineering fads go in circle.
Architecture in the 80s: Mainframe and clients.
Architecture in the 90s: Software systems connected by an ESB.
Architecture in the 2000s: Big central service and everyone connects to it for everything
Architecture in the 2010s: Decentralized microservices that communicate with queues.
Current: RabbitMQ and Kafka.
... Can't we just go back to the 90s?
I hate fads.
I hate when I have to get some data, and it's scattered on 20 different servers, and to load a fucking account page, a convoluted network of 40 apps have to be activated, some in PHP, others in JS, others on Java, that are developed by different teams, connected to different tiny ass DBs, all on huge clusters of tiny ass virtual machines that get 30% load at peak hours, 90% of which comes from serializing and parsing messages. 40 people maintaining this nightmare, that could've been just 7 people making a small monolithic system that easily handles this workload on a 4-core server with 32GB of RAM.
Tripple it, put it behind a load balancer, proper DB replication (use fucking CockroachDB if you really want survivability), and you've got zero downtime at a fraction of the cost.
Just because something's cool now, doesn't mean that everybody has to blindly follow it for fucks sake!
Same rant goes for functional vs OOP and all that crap. Going blindly with any of these is just a stupid fad, and the main reason why companies need refactoring of legacy code.13
Fun day, lots of relief and catharsis!
Client I was wanting to fire has apparently decided that the long term support contract I knew was bullshit from go will instead be handled by IBM India and it's my job to train them in the "application." Having worked with this team (the majority of whom have been out of university for less than a year), I can say categorically that the best of them can barely manage to copy and paste jQuery examples from SO, so best of fucking luck.
I said, "great!," since I'd been planning on quitting anyways. I even handed them an SOW stating I would train them for 2 days on the application's design and structure, and included a rider they dutifully signed that stated, "design and structure will cover what is needed to maintain the application long term in terms of its basic routing, layout and any 'pages' that we have written for this application. The client acknowledges that 3rd party (non-[us]) documentation is available for the technologies used, but not written by [us], effective support of those platforms will devolve to their respective vendors on expiry of the current support contract."
Contract in hand, and client being too dumb to realize that their severing of the maintenance agreement voids their support contract, I can safely share what's not contractually covered:
- Stream based programming
- Angular 9
- Any of the APIs
- Dotnet core
- pretty much anything not in a commit
I'm a little giddy just thinking about the massive world of hurt they've created for themselves. Couldn't have happened to nicer assholes.6
Pet peeve #91847 - when your non technical manager routinely forwards you articles about technical subjects, usually written by non technical idiots, and says "please see if this is something we should be using".
Yeah, I get that your business manager friend has heard Blockchain is amazing, Rsocket is revolutionary, and everyone should now be using Kafka, but none of that makes any sense for our use case.
The clincher had to be telling me to look at AWS groundstation though. And no, we don't have anything to do with satellites...2
Inspired by @Billgates
everyone around is hyped about new tech they get to use, new toys to tinker with, I can see their eyes shining when they hear "let's try and introduce kafka" - they would wiggle their tails all day long if they had ones!
And me? Well, a new potential employer got me so excited I couldn't wash a smile off my face for a few days! You know what they said? "we don't use any frameworks, we focus on clean code, solid, kiss and we write with tdd". Bare java - that's the best position I've heard of in years!
I guess I'm oldschool. But I truly believe their approach is the right one. Not trashing the code with spring [which is turning into smth what systemd is for linux/unix], hibernate and what not.
Just good old java code. Db, multithreading, request-mapping -- all plain, manual and simple.
I work in a contract position and reviewed the code of a senior engineer recently. Regretfully I can't provide context to preserve anonymity.
- handled a single DOM element with 2 different frontend libraries
- used the logical operator && to 'chain' two methods (it didn't work) instead of returning a boolean value,
- broke everything down into minute detail (a comment box had 7 components!),
- API calls were made for every component update instead of maintaining local component state where it made sense, which meant UI updates were slow,
- animated EVERYTHING, which made my Firefox on Xubuntu i7 64bit with 16GB RAM beg for mercy.
I had a rough couple of months with interviews, with 2nd stage technical interviewers throwing impossible tasks at me.
3. Hack a website from the browser's address bar using parameters ( what?!! ),
Obviously, the next time I meet a 'senior', I'm going to tell him talk is cheap;
'SHOW ME YOUR CODE.'3
It's a job description for an entry level Software Developer role at Apple.
Basically they want
- A Front end dev
- A back end dev
- An iOS expert
- A Gimp/photoshop designer
- A streaming data expert
- An Oracle/Teradata expert
- A Kafka expert ( Not shown in the picture)
And on top of that the guy should be able to learn new tech faster.
Do they want a developer or a fucking terminator?5
Set it up, seems to work fine.
Oh no...! Take a broker down, then messages go missing - hmm, that's not right. Fine, I'll just look into... Ah, bad replication factor, my fault. So then it's all fixed! Woop. Wait, no. Some messages still going missing occasionally. Oh, only set to "at most once" delivery. My bad, fix that, and... now everything is out of order. Oh, ok, partitions setup wrongly. Wtf, now the whole thing stalls when there's a network blip until a restart. Right, ok, looks like commits have to receive acks in the library I'm using before continuing. Switch to a library that uses CommitWithoutReply. Brilliant....
Apart from said library seems to have commits failing all over the place because it keeps trying to commit during a rebalance 🙄😒😤
The frustrating thing is I KNOW for a fact that Kafka is a fault tolerant, resilient, horizontally scalable thing capable of handling stupid amounts more than I'm throwing at it without missing a beat. But damn,configuring it, and checking you've configured it sanely is a royal, monumental PITA.5
If you plan on using Apache Kafka for whatever:
Don't do it. You are on the gates of hell.
If you are using Apache Kafka:
I feel sorry for you.
If you decided not to use Apache Kafka:
You chose wisely.12
A 2d simulation of the mars rover curiosity and its commandControl station. Pure java. The two components talk to each other via apache-kafka. Rover has its own operating system (kernel) and resource management. Hooked up some sensors to NASA API.2
We were talking about Kafka's "metamorphosis" and a friend said "what if the parents were programmers and did not want to have a bug in the family?"
When you hear the data engineers talk about ambari, storm, sqoop, kafka & phoenix and you are not sure if they are talking about frameworks or pokemons..2
Working with acutal BigData. Will be ''promoted'' to a new team where I will work on a system wrote in php+mysql with literal millions of requests and database rows. We are currently seeing server crashes around once a week on peak usage. Stack is a vps 64gb ram server + i dont know how many cpu/ cores. Apache, php, mysql.Best ways to optimise and adapt in this case? Kafka? Rabbitmq?ngnix? More hardware?21
For all the hate that Java gets, this *not rant* is to appreciate the Spring Boot/Cloud & Netty for without them I would not be half as productive as I am at my job.
Just to highlight a few of these life savers:
- Spring security: many features but I will just mention robust authorization out of the box
- Netflix Feign & Hystrix: easy circuit breaking & fallback pattern.
- Spring Data: consistent data access patterns & out of the box functionality regardless of the data source: eg relational & document dbs, redis etc with managed offerings integrations as well. The abstraction here is something to marvel at.
- Spring Boot Actuator: Out of the box health checks that check all integrations: Db, Redis, Mail,Disk, RabbitMQ etc which are crucial for Kubernetes readiness/liveness health checks.
- Spring Cloud Stream: Another abstraction for the messaging layer that decouples application logic from the binder ie could be kafka, rabbitmq etc
- SpringFox Swagger - Fantastic swagger documentation integration that allows always up to date API docs via annotations that can be converted to a swagger.yml if need be.
- Last but not least - Netty: Implementing secure non-blocking network applications is not trivial. This framework has made it easier for us to implement a protocol server on top of UDP using Java & all the support that comes with Spring.
For these & many more am grateful for Java & the big big community of devs that love & support it.
Accounting software- and it’s even not the developers fault but because law is changing so often these days they barely keep up.
So I run sole business and visit my accountant from time to time to chat and give my documents.
Sometimes I also help with accounting software like finding why it’s doesn’t sending this crap and doing what it’s expected to do only in some cases.
It usually takes an hour to find out why something doesn’t work.
Also once I was sending some companies fiscal year summary documents cause no one was able to figure out how to sign those documents and how to fill the form so it’s accepted by the “system”.
Based on how I see bureaucracy is increasing cause of technology instead of decreasing and how stupid are those protocols that are required for sending some financial documents over internet. Seeing that those protocols are changing every year if not half a year and software to send those documents mostly doesn’t work.
I’d say any accounting software is temple of doom.
It’s classic “The trial” by Kafka.2
In a time where a web dev is expected to know, well.. everything... Backend -JAVA, python, nodejs and C++ would be great.
Front- angular, react, other 10 libs
DBs -sql, mongo, redis, elastic, kafka, rebbitmq
Also be devops on the side with AWS and docker kubernetis and more stuff
How the f is that possible?
In my real job for the last couple of years and different companies, I usually use 1 language/framework & 1 main DB.. and although it's possible in some companies, but in mine, ppl dont get access to AWS etc..
So let's say there's me.. a server side dev for years.
So I decide to be better and learn Golang.. cool lang, never needed in my job, after few days of not using it I forgot all I learned and that was it.
Then I realized I gotta know some frontend cause everyone want a fullstack ninja nowadays.. so I tried Vuejs.. it was amazing .. never got to use it at work, cause i was a backend, and we didnt use frameworks on our products back then..
Then I decided to learned nodejs, because this is the coolest thing ever.. hated it, but whatever... Never got to use it at work, cause everything was written in other lang which the whole team knew... Forgot the little i knew.
Then I decided, its time to see what Angular is, cause everyone started using it... similar idea to vuejs which i barely remembered, but wow it's a lot of code to remember, or I'll have to google everything.. so I went over it, but can't say i even learned it.
Now Im trying to move on to python, which, I really am learning in depth.. however, since I dont have real experience with it, no one gives me a shot at being a python dev, so again i feel like I'm trying to memorize syntax and wasting my time..
Tired of seeing React in all job ads, i decided to have a look what's that all about.. and whadoyaknow... It's fucking the same idea as vue/angular with again different syntax..
THIS IS CRAZY!
in how many syntaxes do i need to know how to make a fucking crud api, and a page with same fucking post form, TO BE A GOOD PROGRAMMER?!?6
Just came across this wonderful video about Apache Kafka for building something real time and event based project.
Kafka lead after a perf test - we have amazing performance, processing 43 records a second😄
Me - WTF!!!! 🤬5
Why are some people incapable of reading documentation? THE "DUPLICATE RECORDS" IN OUR KAFKA TOPIC ARE BECAUSE ITS AN EVENT STREAM AND NOT A DATABASE. THIS IS LITERALLY ON THE FIRST PAGE OF THE GUIDE YOU ABSOLUTE MORONS.3
How many sh*t days does it need to make me down?
I hate my company, for making everything overcomplicated and annoying.... I have to discuss with 3 peoples for 3 days to getting some gitlab premium licenses (20$ per month for 10 licenses)... Why do you need it? Why we can't use the free version? Why Why Why... It's not enough to tell them it will save us much times and improves the quality of development.....
Also I wanted to ask if we can to Jaxb or another Dev Conference this year... Then I got the information that we have about 2000 Euro for 10 people for training.......... What should we do if everyone buys a book this budget is out .... f*ck company....
Second day, half of the day was taken for fixing the live db on the fly cause of a bad structure of tables... at least fixed some other inconsistence too... later the day fixed a freaking shitty bug with Spring Devtools and 2 Classloader to make the product that I'm presenting in 2 days running.
Today next shitty day with discussion that everything I did last half year (introducing Microservices, Kubernetes, Kafka and other DevOps things) could be maybe useless when the external company will say that they use another ecosystem -..- for their microservices...
Someone looking for a disappointed java developer? I just want to develop the best product ever... I'm happy with every area... Frontend, Backend, DevOps, Fullstack, Architect in some kinds depends on the wishes and technologies.1
Being a Tech Co-Founder is both boon and a bane.
Boon: You know how to build it.
Bane: You just start building without thinking.
We often crave I will be using Redis😍😍, Kafka, Load Balancer, Muti Layer Neural Net
"Dude", Whatever you are building does anyone want it or not. Tell me that first.1
Today our customer tried to change the log dir of Kafka. Too bad he did not change the intended log4j.log, instead he changed the log.dir where kafka writes his data.
He basically tried to change the tires at 180km/h at the Autobahn. Only god knows how he managed to get the tires off. After that he slided a kilometer on his rims and wondered why the car wouldn't drive anymore.
Consumers fucked, Topics fucked and Producers fucked2
Development: we need Nginx installed on *insert server list*
Me: ok, let me get in tough with the platform team.
Platform team: This should be installed in the userspace, Unix teams don't support this.
And here I am, trying to get a reverse proxy running on servers on which I do not have sudo rights.
Since it doesn't work, it's my fault, both sides block the door.
I installed it locally on a virtual machine, but the compiled or installed code doesn't work once copied.
The joy of being an "application engineer". This job title means nothing!12
Well I was not actually in this meeting but I had to prepare everything so the CEO could show the new software to the employees and when I saw him clicking on stuff and it wasn't working... I was like: it works on my machine! After a little while I found out that I didnt push it to demo so it was still not updated..
Oh that embarrassing moment when I had to go to the room and say: refresh the page please!!
What is this format? Spaces are separated by + sign and columns are separated by & sign and it’s value is followed by = sign. Can I directly convert it to Spark dataframe?8
- Finish "Introduction to algorithms"
- Learn some genetic algorithms
- Get my hands dirty on reinforcement learning
- Learn more about data streaming application (My currently app is still using plain stupid REST to transport image). I don't know, maybe Kafka and RabbitMQ.
- Learn to implement some distributed system prototypes to get fitter at this topic. There must be more than REST for communicating between components.
- Implementing a searching module for my app with elastic search.
- Employ redis at sometime for background tasks.
- Get my handy dirty on some operating system concepts (Interprocess Communication, I am looking at you)
- Take a look at Assembly (I dont want to do much with Assembly, maybe just want to implement one or two programs to know how things work)
- Learn a bit of parallel computing with CUDA to know what the hell Tensorflow is doing with my graphic card.
- Maybe finishing my first research paper
- Pass my electrical engineering exam (I suck at EE)1
ONE WEEK i've been working on this migration from Mongo to Kafka for a service that will probably handle 5 requests A MONTH for this thing that launch an entire JVM and take 10 seconds just to create a so-called topic
From civil background, not sure about internal system architectures.
Question: how does cpu switches between processes ?
Ans: how the f I know? Do I really need to know, internally!
Explained a bit.
He was like you are the one who are going to use kafka, not the one who are going to write kafka.
I was like devasted.5
I’ve been at this issue at work for four days now and no progress and I feel really bad because we have important stories to pick up and I feel I’m wasting my capacity like this because I haven’t fixed it. Basically, only in our QA environment (one before production) our services is not acknowledging duplicate events posted by Kafka, thus keeps reprocessing them. I’ve spent so long trying to diagnose the code, which is the same in all envs currently, seeing how this suddenly occurred, restarted things, went through complications of using different tools, asked for help from others a lot but IVE gotten NOWHERE. Idr wanna say to my team that I should prioritise other things because we have deadlines but I feel this issue is important to fix but I just can’t figure out how. Now I’m worried this whole sprint will go without me doing anything and then fingers pointed at me later6
I am working on distributed message stream/message bus system. Somewhat similar to Kafka but uses Memcached protocol, so you can use any memcached client already out there to speak with it.
Originally I'm coming from Java , about 2 years ago, I switched Node with TypeScript and had a hard time getting accustomed to Promises. It was a big relief when I learned about async/await. Much cleaner code, no brainfuck anymore when thinking about how to handle stuff that requires multiple async values and so on.
Now I'm working on a clients project as a Java dev again. SOA, Spring Framework, Kafka and MongoDB, nothing too complicated... if they wouldn't use reactor to bring reactive functionalities to Java.
It feels like I'm back in Promise Hell...2
I am using python scripts for mathematical computations and deep learning wheres spring to build the server and now want to handle the scripts from spring guess what scripts were in python 3.6 and fucking jython does not support it.. so tried finding the different libs like jpy , jpe and analysisRPC.. Everything seems just bullshit .
Finally using Kafka to have my interprocess communication..
Anyone have experience / opinions on Kafka? We currently don't have a messaging system like this, and I'm thinking it would be the best choice for things like providing webhooks because its persistent nature, instead of having a "synchronous" message queue that I push into and hope it gets handled. Might also want to store attempted webhook calls in a db but I don't know if that's overkill and could be done with just transforming messages (if they fail) and trying to handle them again.3
When to use Kafka?
Based on my understanding, We can use it if we want to do more than one thing from a single input.
Is this the correct approach?5
The day I'm fucking around with Kafka Streams API and had to work with Java after swearing it off 5 years ago for JS bae...
Is the day that Apache's everything is down. Can't even hit their fucking website. I just want a fat jar. Just one.
I NEED HELP with Kafka
I'm working a thesis. I developed 4 different microservices (REST APIs). I would like to use Kafka to support large number of users. I may also place the microservices behind a HAPI Proxy. How can I use Kafka to stream requests and respond accordingly. I'm using Node.js. I think I haven't grasped Kafka. My Prof, suggested I try it to act as a broker but I'm blank right now. How do I tell Kafka I want it do a POST or GET etc?2
My team is currently using a confluent cloud kafka architecture via protobuf and wordpress as website.
We realized this is quite impractical when you want login functionality and such and would like to make a page from scratch now.
problem is we are all java backend developers with minor php knowledge, and php doesnt work well with protobuf or kafka.
how would you get such a website started. what language and frameworks are easy to get going?
I'm looking into an app for work. It needs to connect to both a mongo and Kafka server using SSL or something, but different keys basically.
So need to load a jks file in the main() I think except I have 2 files.
I don't remember exactly how it works but I guess need to combine them into 1?
How do I load each though in app? Need to tell mongo, kafkaClient with one to use?1
My Spark Program to process changes from Kafka topic to DB table is working end-to-end. But when it is manually terminated when an insert/update/delete is in progress, the whole tables data gets dropped. How can I handle this in a safe way?3