To all the data engineers in here: WTF is going on in your field?

I've worked closely with a dozen data engineers in the last 5 years (and talked to friends and internet strangers about this and get similiar responses), mine if them seem to know how to use a computer!

They don't understand git, ORMs, best practices, how to use a terminal, DAGs (important for using modern ETL scheduling tools like airflow and prefext), etc

Guys with 10 years of experience on their resume and they can't wrap a model into a flask app with 1 endpoint. They'll reference local files on their machine in w jupyter notebook and are shocked it won't work on other computers!

  • 6
    One of these days I'm going to proof read my posts before submitting.

    I hate that editing your post is a paid feature on devrant...
  • 3
    @lungdart Yeah, here have an upvote to get you closer to free premium.
  • 4
    Not everyone has the same expertise in all fields.

    I work as a kind of data scientist but most definitely I'm not one.

    However, I *am* able to set the project up, elicit requirements, do DevOps, set up CI/CD, code the many boilerplate needed to expose this stuff to normies, etc.

    And I can still deliver on the ML front. Sure, I might not be as good as a true data scientist, but when budget is tight and you need an all rounder, I end up giving you the most bang for your buck.
  • 2
    @Oktokolo is there a free premium? :o
  • 5
    There's been a sudden increase in demand for folk who can handle data to any level of competence, as companies realise how much value there is in getting data systems up and running.You can get a well-paid job without having a truly full-range skillset, extending to Flask and so on, because you'll be able to add a lot of value without that. This doesn't excuse an individual not understanding that his/her local files aren't available to all; nonetheless, there are acceptable and useful levels of knowledge which are well above that and yet still short of your own level (and expectations).
  • 0
    Geez, thanks mate.
  • 2
    Apparently none of the skills listed are even necessary to be a data engineer/scientist.
  • 0
    Worked with some data engineers as well. For me they seem like a freelancer who actually works maybe 20 hours a month and rest of the time they are kept on a retainer. Idk what they do with rest of the time but clearly its not learning or building skills, most likely youtube or something.
  • 1
    Well, I'm a lot happier in data. You can get stuff done, which has become progressively more of a very, very slow, frustrating grind in web dev. I was handed a project this morning, will deliver that on Monday and then go back to doing easy, stress-free data shaping and problem solving again. Really though, I agree - there are more jobs than capable people in data at the moment.
  • 0
    @spongegeoff I would love to see a capable data guy. Maybe I should get into it, i could be a 100x data engineer with 10h/wk...
  • 1
    @lungdart It's actually very nice to be using tools like Power BI, Tableau and, well, even Excel. You generally get more out of them than you put in - usually, a lot of result for little effort. I got my first data job because I could build a database-driven web app. Thankfully now no web stuff, pulling reports together for an airline.
  • 0
    @spongegeoff WHERE? US or EU?
  • 0
    tbh, I had to google how testing of ETL/ELT pipelines looks like.

    What if they are "incremental"?
    What do unit tests look like?
    What do integration tests look like?

    It's "straightforward" in non-"data" software, but in data engineering? Idk!

    I am checking avro next
  • 0
    @MLPops I'm in UK. I don't really 'test' ETL in any way beyond the obvious - we pass the data through and have a look. Can nearly always just assume that things are doing what you set them to do and see them doing. If you can pull accurate and reasonably well-presented results, rationalise or replace existing (frequently somewhat shaky, often semi-manual) systems, you'll be appreciated.
  • 0
    @spongegeoff do you have an opinion on dbt-core for making data engineering projects more git-friendly and testable?

    I'm an MLOps guy who might be looking into moving to data engineering
  • 0
    @MLPops I'm not a big fan of over-complicating anything really.
  • 0
    So what? My employer doesn't even understand two-step authentication and has me guess a clients secret question.

    It's not what you know

    It's who you know

    And how much bullshit

    You are willing to peddle
Add Comment