I just realized how weird my searches would seem to an outsider

"python twisted vs falcon celery"

"python hug vs falcon"

I do like naming stuff as animals though. Maybe we should sort all of the python web servers by speed and name them accordingly.

Falcon is obviously the fastest, we can name twisted cheetah, and all the way at the end Django becomes elephant. Big and powerful but soooooooooo slow.

C: what technologies are you using in this product?
Me: we are using celery which uses kombu to communicate through redis. Additionally, we have beat tasks that run and check the state of workers
C: was someone hungry when they named these frameworks?
Me: Possibly, but I just got hungry naming them.

Fuck this I need to ventilate.

Thinking about job change because maintaining and extending 3 years old codebase (flask project) is FUCKIN exhausting. It was badly written since start by someone who obviously didn't know much about python. (Going by commit history.)

Examples:
- if var != None / if var == None
- if var is not None / if var is None (well..)
- Returning self-parsed obscure JSONs from dict variable
- Serializing dictionaries into database by str() (both sqlalchemy and mysql support JSON format) - THEY ARE ALMOST UNUSABLE OTHER WAY AROUND (luckily, python can deal even with that)
- celery tasks, the way they are called they BLOCK the whole flask (not bad in itself, but if connection breaks there are no errors, nothing it just hangs)
- obscure generator/yielding that contains return of flask's response in itself
- creating fifteen thousands of variables one by one where they would look so nicely as dict keys, and hey they are then both MANUALLY SERIALIZED into returning dict by "%s" (string formatting) [okey, some of them are objecst like datetime but MATE WTF]
- many, many more, PEP lint shall not pass

I would rather deal with fresh startup owners wanting me to program unicorns in one week then trying to extend and manage zombie-like projects.

Nothing personal against the firm I actually like the place.

Celery is awesome.

Installed celery...
ran my code...
error...
module 'celery' not found...
wondered why...
reviewed...
didn't install it properly in Docker...

end of story

I’ve been bashing my head against a project for the past 8 weeks. The project creates a PDF pulling data from multiple APIs, scrapes and private DBs and plots charts using Plotly. We built it with Python, wkhtmltopdf and Celery+Redis. The input is an excel with a list of up to 5 influencers to analyse and compare. Runs on demand on a Linux machine and each report takes around 20 mins to generate. The project has no unittests so the only way I can check everything works is by running a bunch of different inputs. Even though you test 10 inputs (taking you more than a day), there is a high chance something goes wrong on the 11th input. I’m thinking that the only way to fix this mess is to go back to the drawing board and plan yet another refactoring to add unittests everywhere. What do you guys think?

Never in my life have i seen such a convoluted and complicated framework as Celery. Celery sucks donkey ass. That is all.

For fuck, fucking sake I literally spent 10 hours trying to setup Celery logging with Django but no, ohhh no Celery has some ninja handlers that don't even show up in logging_tree. If I try to change date format for Celery then I get big "fuck off" in my face. One more hour and I will explode.

Then suddenly my boss wants me to handle servers and the environment etc... can anybody suggest me some site tutorials about Docker, Dockerhub, Celery and RabbitMQ?

We use celery at work, and one of the issues we face is that we use Django logging.
I'm not sure how it happened, however we only get 1 level of tracebacks from it now.

This has made debugging painstakingly difficult, since we have to manually traverse the code every time.

(we're in the process of moving to sentry, and we'll get our full logs back soon)

Airflow… airflow… I hate you so freaking much, you are a bloated piece of software that packages wayyy too much and increase the complexity of any solution we built on top of you. Plus unit-testing and integration tests are wayyy too difficult with you. But man… your recent UI changes are a massive welcome.

A year ago I was tasked with either upgrading airflow to version 2 or to migrate the code to another tool. Naively I thought “well might as well upgrade to avoid a rewrite”. Little did I know that the reason airflow didn’t scale out well for us, was due to people over the years not having a grasp on airflow primitives like “pools”, “workers” and “operators”. Ended up refactoring the entire codebase for both infra and DAGs anyhow AND upgrading the beast AND lower cost by a factor of 2 (from $100-$150 daily to $50-$70 daily).

But seriously feels like I could’ve solved the scheduling issue with literally any message queue+decent library (like celery or Faust) and I’d have half the headache.

What did celery say when he broke up with his girlfriend? She wasn't right for me, so I really don't carrot all.

Working on a feature which heavies relies on a data pipeline. I noticed it is a couple of lambda functions calling each other ( Fuck you to the guy who made it). The best way to get sanity back is build a proper etl pipeline. Any suggestions for building a etl in python with reliability.
Options already considered
1. Celery tasks - Worked well but no overview of the single task progress across celery tasks
2. Airflow - Gives good overview but the docs make less sense than a 10 yr talking. Mostly because they introduced a new syntax and not everything has migrated fully yet. Also no support for reusing dags