My company decided to reinvent the wheel by writing its own queue system instead of using the existing message queue service.

And it uses plain PHP with exec() to run the workers.

Where do we store the job? We use mongoDB which is already used in our existing projects. We can query the collection/table each time the queue service start, execute the jobs, and let it exit if there's no job anymore. Don't worry, systemd will start the queue service again once it exits.

How to monitor the workers? Yep, we use ps and grep to check if the worker's PID still exists in the OS.

What about error stack traces? Nice question, we redirect the stdout and stderr when exec()-ing into a file.

What about timeout? We don't need it, let's just assume no one is going to write while(true).

It works flawlessly! /s

  • 1
    Holy mother of god.. burn it with fire
  • 3
    Can't you write a cron that queries the next job in queue every 10min and kills then restarts the process if it didn't change?
  • 0
    @homo-lorens No, we need near realtime processing
  • 1
    cool, now build a frontend so you can monitor the queue, kick jobs and manage your workers!
  • 2
    This got worse with every sentence
  • 1
    Did somebody get burned by Kafka so they just decided to avoid it all costs?
  • 2
    @SuspiciousBug probably at some point of time if they got overwhelmed by failures

    @devphobe No one has any experince on Kafka, I believe some simpler MQ like RabbitMQ works well
  • 0
    @curiousjoe whenever I hear “real-time processing” I start looking for anything with the pub/sub model instead of queues. Reinventing the wheel sucks and is dangerous to your overall velocity as a company. I can understand a few choices like that, but if that’s the norm, then personally I’d be on my way out 😢
Add Comment