Ranter
Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Comments
-
😆 this sounds to for me as a non dev even so stupid that it hurts.
Convince the client to Get graphQL then he can select the data beforehand and gets exactly what he needs. -
@asgs it's GZipped. But still.
AND all of that data is present on a WEB page ! Even my core i7 (Granted a bit dated, only i7-3770k) have troubles to process this web page. -
@heyheni it's in a web page context (to make things even worse)
We already have a filtering system. They can pin point exactlly what they a reelooking for. -
Hazarth94344yI'm currently limited to a 8Mbit/s link, speed testers show about 130kb/s on average so it's possibly slightly more
so this website would currently load for about 78.8 minutes on my link
which ofc would timeout way before that
why did you make this?! -
@Hazarth because "We need to make this client happy and we don't care that they don't use product properly" :(
Trust me, nothing will be more fun for me than delete the whole API -
@AtuM nop, just one of very big accountant firms :) Can't give country/name for obvious reasons tho. (I can say it's not USA)
-
...si-s-s-s-six hundred megs of plaintext JSON.
I think I'm gonna get the strong Vodka for this one -
Hazarth94344y@drewbie he said they use gzip, but even with gzip 600Mb of json just doesn't turn into anything reasonable
-
@drewbie it is compressed by Gzip, but it doesn't change the fact that web browser still need to decompress it and process 600 MB json lol
-
@kateliuyi Imagine, 300 emploeeyes worjkin on a total of 8000 projetcs. Each projects has an average of 8 task.
Now Imagine it on a web page. -
Shit in -> shit out.
Why even bother making an API? Just send them the whole database if they like to work "just like with Excel". -
jaylord4544yCompletely ridiculous of course. But did he say he wanted to see everything at the same time? 🤔
Yes it could still be very challenging to solve, but that would make it less impossible right 😅
But it doesn't sound like the client gave you a whole lot of time to build this properly.. -
@foox the thing is.. it should be supported by ALL major browsers before puting in production. It's not the case.
Belive me, I would love to use that ! -
maladiec294y+1 for stupidity of decisions!
Also would ask sysadmin to limit the connection speed for that specific endpoint! -
@maladiec naaaan too much work that.
It would require : 1+ Azure service for load balancing
+1 Fixed endpoint (ouy pay for that)
+1 App service firewall
+1 Rule of acess
+1 rule of "fail" (if speed > XX send email/page what ever)
Basiclly the 10 GBit link on our end still holds. We'll just work with thewm to see how we can fuckiong NOT send 347 MB (Now, down from 600). -
@hjk101 Well, I made properties names shorter lol. Not like 1 letter short, but very short.
There is a way do optimize it more, but it will require to send arrays without properties, so it's annoying to work with on front end side -
eeee31504yLet me guess. To update / patch a single value, you have to upload all those MBs again? I hope your client is on a limited internet plan with a fair use policy.
-
hjk10157314y@NoToJavaScript that's minification for ya. Still insane. I would not optimize it for them. If they want a shitty solution they can get it. Sure it has to run on all browsers, can't run on any normal device though...
-
v3ctor344ySo pagination and datatables are completely out of the picture?
If they are not I would still press for a better pagination style like a load more.
I'm assuming that this data is a super long list right? And no way they can possibly "see everything" without scrolling. So a load more button thingy can be added to ease this nightmarish situation. -
White104yWhat about something like https://npmjs.com/package/... or something equivalent? Then you could display "everything" at once without actually having everything loaded at once
-
@v3ctor that's FOR ONE page of data :) total dataset is around 5 pages like that
-
@AtuM
can't. this particualr API uses around 15 differents tables, Multi thread for agregation AND some inProc calculations. It's not matter of DB acess. We get DB dataset from all of these 15 tables in around 2 seconds.
Post processing takes around 15 seconds (Now 5 seconds, Improuvement were made !!).
I have no idea hhow much time serialization take. My next thing to look :) -
imaji1104yJust like excel, they say. So why they're not asking for .xlsx file generate feature then? So they can download the spreadsheet file and open it in excel.
We have an API which returns 600 MB of JSON.
Because client "Wants to see everything first and then apply filters, just like Excel".
FML
Edit and ofc thier laptops with core i3 and 4GB of ram can't even process that.
rant