Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
-
@lungdart Yeah, here have an upvote to get you closer to free premium.
-
Not everyone has the same expertise in all fields.
I work as a kind of data scientist but most definitely I'm not one.
However, I *am* able to set the project up, elicit requirements, do DevOps, set up CI/CD, code the many boilerplate needed to expose this stuff to normies, etc.
And I can still deliver on the ML front. Sure, I might not be as good as a true data scientist, but when budget is tight and you need an all rounder, I end up giving you the most bang for your buck. -
@Oktokolo is there a free premium? :o
-
There's been a sudden increase in demand for folk who can handle data to any level of competence, as companies realise how much value there is in getting data systems up and running.You can get a well-paid job without having a truly full-range skillset, extending to Flask and so on, because you'll be able to add a lot of value without that. This doesn't excuse an individual not understanding that his/her local files aren't available to all; nonetheless, there are acceptable and useful levels of knowledge which are well above that and yet still short of your own level (and expectations).
-
@spongegeoff
Geez, thanks mate. -
Well, I'm a lot happier in data. You can get stuff done, which has become progressively more of a very, very slow, frustrating grind in web dev. I was handed a project this morning, will deliver that on Monday and then go back to doing easy, stress-free data shaping and problem solving again. Really though, I agree - there are more jobs than capable people in data at the moment.
-
@spongegeoff I would love to see a capable data guy. Maybe I should get into it, i could be a 100x data engineer with 10h/wk...
-
@lungdart It's actually very nice to be using tools like Power BI, Tableau and, well, even Excel. You generally get more out of them than you put in - usually, a lot of result for little effort. I got my first data job because I could build a database-driven web app. Thankfully now no web stuff, pulling reports together for an airline.
-
@spongegeoff WHERE? US or EU?
-
@MLPops I'm in UK. I don't really 'test' ETL in any way beyond the obvious - we pass the data through and have a look. Can nearly always just assume that things are doing what you set them to do and see them doing. If you can pull accurate and reasonably well-presented results, rationalise or replace existing (frequently somewhat shaky, often semi-manual) systems, you'll be appreciated.
-
@spongegeoff do you have an opinion on dbt-core for making data engineering projects more git-friendly and testable?
I'm an MLOps guy who might be looking into moving to data engineering -
@MLPops I'm not a big fan of over-complicating anything really.
To all the data engineers in here: WTF is going on in your field?
I've worked closely with a dozen data engineers in the last 5 years (and talked to friends and internet strangers about this and get similiar responses), mine if them seem to know how to use a computer!
They don't understand git, ORMs, best practices, how to use a terminal, DAGs (important for using modern ETL scheduling tools like airflow and prefext), etc
Guys with 10 years of experience on their resume and they can't wrap a model into a flask app with 1 endpoint. They'll reference local files on their machine in w jupyter notebook and are shocked it won't work on other computers!
rant