Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "rabbitmq"
Today my manager asked me about my research into using RabbitMQ as a backup in case Azure Service Bus ever goes down.
Me: "Good. The way we designed the framework, all we have to do is drop the DLLs into the directory, update the config, and the services will start using RabbitMQ."
Mgr: "Excellent. Probably should be looking into using RabbitMQ as a permanent replacement for Azure"
Me: "What? The whole reason we moved to Azure was to eliminate the problems with having an on prem service bus. Since we've switched, there has been zero downtime."
Mgr: "That's what VP-Joe is afraid of. If Azure ever goes down, he won't know how to explain Azure to the president as to why we're not taking orders or can't ship packages."
Me: "That makes no sense. What did VP-Joe tell the president when a database goes down or a server mis-configuration?"
Mgr: "President understands internal outages, its just the whole 'cloud' thing he doesn't understand."
Me: "Um..then VP-Joe needs to explain it to him?"
Mgr: "The decision has already been made. Are you on board? Lets look at this move as a cost savings."
Me: "You mean the $10 a month? How much hardware will we need to support RabbitMQ?"
Mgr: "Yea, nobody probably thought of that."
Me: "I'm on board with whatever decision, but I'd like a little more than VP-Joe being afraid of the president."
Mgr: "I'm sure its not being afraid."
Mgr: "OK, lets wait and see if VP-Joe forgets about this and moves on to something new."5
Experience that made me feel like a dev badass?
Users requested the ability to 'send' information from one application to another. Couple of our senior devs started out saying it would be impossible (there is no way to pass objects across a machine's memory boundary), then entertained the idea of utilizing the various messaging frameworks such as Microsoft's ServiceBus and RabbitMQ, but came up with a plan to use 2 WebAPI services (one messenger, one receiver) along with a homegrown messaging API (the clients would 'poll' the services looking for message) because ServiceBus, RabbitMQ, etc might not be able to scale to our needs. Their initial estimates were about 6 months development for the two services, hardware requirement for two servers, MSSQL server licenses, and padded an additional 6 months for client modifications. Very...very proud of their detailed planning.
I thought ...hmmm...I've done memory maps and created simple TCP/IP hosts that could send messages back and forth between other apps (non-UI), WPF couldn't be that much different.
In an afternoon, I came up with this (see attached), and showed the boss. Guess which solution we're going with.
The two devs are still kinda pissed at me. One still likes say as I walk in the room "our hero returns"....frack him.11
So, I was gonna rant about how it can be difficult to design event-based Microservices.
I was gonna say some shit about gateways APIs and some other stuff about data aggregation and keeping things idempotent.
I was going to do all this but then as I was stretching out the old ranting fingers I decided to draw a diagram to maybe go along with the rant.
Now I’m not here to really rant about all that Jazz...
I’m here to give you all a first class opportunity to tear apart my architecture!
A few things to note:
Using a gateway API (Kong) to separate the mobile from the desktop.
This traffic is directed through to an in intermediate API. This way the same microservices can provide different data, and even functionality for each device.
Most Microservices currently built in golang.
All services are event based, and all data is built on-the-fly by events generated and handled by each Microservices.
RabbitMQ used as a message broker.
And finally, it is hosted in Google Cloud Platform.
The currently hosted form is built with Microservices but this will be the update version of things.
So, feel free to rip it apart or add anything you think should change.
Also, feel free to tell me to fuck right off if that’s your cup of tea as well.
A lot of engineering fads go in circle.
Architecture in the 80s: Mainframe and clients.
Architecture in the 90s: Software systems connected by an ESB.
Architecture in the 2000s: Big central service and everyone connects to it for everything
Architecture in the 2010s: Decentralized microservices that communicate with queues.
Current: RabbitMQ and Kafka.
... Can't we just go back to the 90s?
I hate fads.
I hate when I have to get some data, and it's scattered on 20 different servers, and to load a fucking account page, a convoluted network of 40 apps have to be activated, some in PHP, others in JS, others on Java, that are developed by different teams, connected to different tiny ass DBs, all on huge clusters of tiny ass virtual machines that get 30% load at peak hours, 90% of which comes from serializing and parsing messages. 40 people maintaining this nightmare, that could've been just 7 people making a small monolithic system that easily handles this workload on a 4-core server with 32GB of RAM.
Tripple it, put it behind a load balancer, proper DB replication (use fucking CockroachDB if you really want survivability), and you've got zero downtime at a fraction of the cost.
Just because something's cool now, doesn't mean that everybody has to blindly follow it for fucks sake!
Same rant goes for functional vs OOP and all that crap. Going blindly with any of these is just a stupid fad, and the main reason why companies need refactoring of legacy code.13
Fun day, lots of relief and catharsis!
Client I was wanting to fire has apparently decided that the long term support contract I knew was bullshit from go will instead be handled by IBM India and it's my job to train them in the "application." Having worked with this team (the majority of whom have been out of university for less than a year), I can say categorically that the best of them can barely manage to copy and paste jQuery examples from SO, so best of fucking luck.
I said, "great!," since I'd been planning on quitting anyways. I even handed them an SOW stating I would train them for 2 days on the application's design and structure, and included a rider they dutifully signed that stated, "design and structure will cover what is needed to maintain the application long term in terms of its basic routing, layout and any 'pages' that we have written for this application. The client acknowledges that 3rd party (non-[us]) documentation is available for the technologies used, but not written by [us], effective support of those platforms will devolve to their respective vendors on expiry of the current support contract."
Contract in hand, and client being too dumb to realize that their severing of the maintenance agreement voids their support contract, I can safely share what's not contractually covered:
- Stream based programming
- Angular 9
- Any of the APIs
- Dotnet core
- pretty much anything not in a commit
I'm a little giddy just thinking about the massive world of hurt they've created for themselves. Couldn't have happened to nicer assholes.6
“I need one fullstack engineer”
“Ok, what exactly do you need?”
Don’t we all just love these kind of discussions?1
Working with acutal BigData. Will be ''promoted'' to a new team where I will work on a system wrote in php+mysql with literal millions of requests and database rows. We are currently seeing server crashes around once a week on peak usage. Stack is a vps 64gb ram server + i dont know how many cpu/ cores. Apache, php, mysql.Best ways to optimise and adapt in this case? Kafka? Rabbitmq?ngnix? More hardware?21
When you realize that you have only couple of gigs left on your SSD and the culprit is 41 GB log file.2
For all the hate that Java gets, this *not rant* is to appreciate the Spring Boot/Cloud & Netty for without them I would not be half as productive as I am at my job.
Just to highlight a few of these life savers:
- Spring security: many features but I will just mention robust authorization out of the box
- Netflix Feign & Hystrix: easy circuit breaking & fallback pattern.
- Spring Data: consistent data access patterns & out of the box functionality regardless of the data source: eg relational & document dbs, redis etc with managed offerings integrations as well. The abstraction here is something to marvel at.
- Spring Boot Actuator: Out of the box health checks that check all integrations: Db, Redis, Mail,Disk, RabbitMQ etc which are crucial for Kubernetes readiness/liveness health checks.
- Spring Cloud Stream: Another abstraction for the messaging layer that decouples application logic from the binder ie could be kafka, rabbitmq etc
- SpringFox Swagger - Fantastic swagger documentation integration that allows always up to date API docs via annotations that can be converted to a swagger.yml if need be.
- Last but not least - Netty: Implementing secure non-blocking network applications is not trivial. This framework has made it easier for us to implement a protocol server on top of UDP using Java & all the support that comes with Spring.
For these & many more am grateful for Java & the big big community of devs that love & support it.
Manager said we need to use Queue. Several meetings after then I looked at prototype by 6 senior devs:
A QueueListener connects to RabbitMQ check for payload then *disconnects*;
A TaskProvider in ASP.Net.MVC.Core(whatever it is) listening http and dependency inject that QueuePoller;
A Visual Cron timer calls that http url every 5 minutes.
Wait for it: a set of database tables to store messages for another MessageProcessor.
It’s a XML to CSV file conversion project consists of 43 unique projects under a solution. I did it within 500 lines of Node with ElasticSearch and told we don’t use fancy new stuffs here.1
L1 support requested to terminate an EC2 instance on which one of our apps seemed to be misbehaving. The node was terminated after few min.
L1 later realized that that instance didn't belong to that app, but instead it was one of the RabbitMQ nodes.
Then, after some panicking we remembered that HA was enabled, so nothing should've been lost.
Later, we realized that the recent RMQ upgrade necessitated a new cluster on which HA was NOT enabled!1
Thank you, company forced windows update! My 60 minutes reconfiguring rabbitmq and postgres were well invested instead of investigating the memory leak fucking hibernate causes.
Working on a project in college
we thought a messenger would be cool. (Language: C# )
We decided to use a microservice architecture, rabbitMQ, .Net Core and some frameworks like EF.
My project leader changed the whole project because he is convinced that we need bleeding edge technologies if we want to create a great product. He added over 15 frameworks like MassTransit which released 10 days ago (.Net Core support).
Now we have to put a lot of effort into learning how these frameworks work instead of implementing.
How would you handle this?4
Everyone I tell this to, thinks it’s cutting edge, but I see it as a stitched together mess. Regardless:
A micro-service based application that stages machine learning tasks, and is meant to be deployed on 4+ machines. Running with two message queues at its heart and several workers, each worker configured to run optimally for either heavy cpu or gpu tasks.
The technology stack includes rabbitmq, Redis, Postgres, tensorflow, torch and the services are written in nodejs, lua and python. All packaged as a Kubernetes application.
Worked on this for 9 months now. I was the only constant on the project, and the architecture design has been basically re-engineered by myself. Since the last guy underestimated the ask.2
I work as android dev for the past year and a half, this week in my startup I started doing some backend (it consists of microservices, codebase is written in python. were using rabbitmq for queuing and for http our framework is falcon).
At first I was very adamant to learn new stuff but now that Im getting deeeper into backend I started to really enjoy it!
Its still lots of new information but at the same time it feels soo refreshing to have two experienced mentors who can guide you, since Ive spent last year and half working on android completely by myself and relying only on myself. Also im very lucky that codebase is clean.
Then suddenly my boss wants me to handle servers and the environment etc... can anybody suggest me some site tutorials about Docker, Dockerhub, Celery and RabbitMQ?7
This is what happens when you don't use compression on messages in a real time backend (with RabbitMQ)5
This is true I swear... I once worked on part of a project "optimization" that required, running a job on sidekiq in the background that spawns multiple threaded RPC calls on RabbitMQ (and be I/O blocking) till the jobs are done (or failed) so that it updates the status of the master object (that has the associated objects processed) and sends an email to the ops manager (just a summary email)... instead of using database locks... or dropping the email requirement...
I did it without arguing because I've already quit the job a while ago...
Today I learned if you ack a message twice, RabbitMQ shits the bed. Dang. Another day wasted on a dumb refactoring regression.2
- Finish "Introduction to algorithms"
- Learn some genetic algorithms
- Get my hands dirty on reinforcement learning
- Learn more about data streaming application (My currently app is still using plain stupid REST to transport image). I don't know, maybe Kafka and RabbitMQ.
- Learn to implement some distributed system prototypes to get fitter at this topic. There must be more than REST for communicating between components.
- Implementing a searching module for my app with elastic search.
- Employ redis at sometime for background tasks.
- Get my handy dirty on some operating system concepts (Interprocess Communication, I am looking at you)
- Take a look at Assembly (I dont want to do much with Assembly, maybe just want to implement one or two programs to know how things work)
- Learn a bit of parallel computing with CUDA to know what the hell Tensorflow is doing with my graphic card.
- Maybe finishing my first research paper
- Pass my electrical engineering exam (I suck at EE)1
In my office,
My fren was facing problem installing rabbitMq so he contacted the IT guy. He told please try installing tortoiseGit..🤣😂2
Wanted to install rabbitmq 🐰 on my machine but manager said he doesn't allow it.
I just wanted to do some r&d for potential uses for the company but instead got shut down for no freaking reason. 🔥1
Programming is such:
1. Just want to talk to RabbitMQ from Python
2. End up reading the AMQP specification, looking for gotchas
"Microservices" implemented on a PHP symfony/doctrine/rabbitMQ stack, whereby the app was daemonized. Lots of fun.1
The company firewall blocking rabbitmq messages?!
Developing inside company structures is the pure fuck-up!