Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "file stream"
-
One of our web developers reported a bug with my image api that shrunk large images to a thumbnail size. Basically looked like this img = ResizeImage(largeImage, 50); // shrink the image by 50%
The 'bug' was when he was passed in the thumbnail image and requesting a 300% increase, and the image was too pixelated.
I tried to explain that if you need the larger image, use the image from disk (since the images were already sized optimally for display) and the api was just for resizing downward.
Thinking I was done, the next day I was called into a large conference room with the company vice-president, two of the web-dev managers, and several of the web developers.
VP: "I received an alarming email saying you refused to fix that bug in your code. Is that correct?"
Me: "Bug? No, there is no bug. The image api is executing just as it is supposed to."
MGR1: "Uh...no it isn't. Images using *your* code is pixelated and unfit for our site and our customers."
MGR2: "Yes, I looked at your code and don't understand what the big deal is. Looks like a simple fix."
<web developers nodding their heads>
Me: "OK, I'll bite. What is the simple fix?"
<MGR2 looks over at one of the devs>
Dev1: "Well, for example, if we request an image resize of 300, and the image is only 50x50, only increase the size by 10. Maybe 15."
Me: "Wow..OK. So what if the image is, for example, 640x480?"
MGR1: "75. Maybe 80 if it's a picture of boots."
VP: "Oh yes, boots. We need good pictures of boots."
Me: "I'm not exactly sure how to break this to you, but my code doesn't do 'maybe'. I mean, you have the image from disk.
You obviously used the api to create the thumbnail, but are trying to use the thumbnail to go back to the regular size. Why not use the original image?"
<Web-Dev managers look awkwardly towards the web devs>
Dev3: "Yea, well uh...um...that would require us to create a variable or something to store the original image. The place in the code where we need the regular image, it's easier to call your method."
Me: "Um, not really. You still have to resolve the product name from the URL path. Deriving the original file name is what you are doing already. Just do the same thing in your part of the code."
Dev2: "But we'd have to change our code"
Mgr2: "I know..I know. How about if we, for example, send you 12345.jpg and request a resize greater than 100, you go to disk and look for that image?"
<VP, mgrs, and devs nod happily>
Me: "Um, no that won't work. All I see is the image stream. I have no idea what file is and the api shouldn't be guessing, going to disk or anything like that."
Dev1: "What if we pass you the file name?"
<VP, mgrs, and devs nod happily again>
Me: "No, that would break the API contract and ...uh..wait...I'm familiar with your code. How about I make the change? I'm pretty sure I'll only have to change one method"
VP: "What! No...it’s gotta be more than that. Our site is huge."
<Mgrs and devs grumble and shift around in their chairs>
Me: "I'm done talking about this. I can change your code for you or you can do it. There is no bug and I'm not changing the api because you can't use it correctly."
Later I discovered they stopped using the resize api and wrote dynamic html to 'resize' the images on the client (download the 5+ meg images, and use the length and width properties)23 -
Me: I have an input stream!
Library: I want a file.
Me: I can see your code, you will convert it to an input stream! Don't any of your constructors take an input stream?
Library: No. I want a file. Okay, you can also include an encoding.
Me: I don't care about encoding.
Library: Fine, just give me the file then.
Me: You mean I have to somehow convert my input stream into an actual file so I can give it to you?
Library: You're gradually catching on, yes.
Me: Can I add a new constructor myself that takes an input stream?
Library: Sure! Good luck in getting it approved by the maintainers and the new jar distributed everywhere, including Maven Central before your deadline.
Me: Fine, I'll just rearchitect everything so I can give you an actual file.
Library: And then everything will be fine.
Me:3 -
I'm the leader of a team developing a project at my company. Recently I had an accident which required me to have a surgery, and I was out for the last month.
Since it's an important project, our CTO reassigned a Xamarin dev to help my team (in JEE) while I'm out, you know, cause "C# looks like Java". The guy had to write a file stream handler, and - though the guy does a really good job - it's funny to see how desperately he's trying to understand how Java works with streams. Censored because of reasons.7 -
I swear I work with mentally deranged lunatics.
Dev is/was using TFS's web api to read some config stuff..
Ralph: "Ugh..this is driving me crazy. I've spent all day trying to read this string from TFS and it is not working"
Me: "Um, reading a string from an web api is pretty easy, what's the problem?"
Ralph: "I'm executing the call in a 'using' statement and cannot return the stream."
Me: "Why do you need to return a stream? Return the object you are looking for."
Ralph: "Its not that easy. You can return anything from TFS. All you get back is a stream. Could be XML, JSON, text file, image, anything."
Me: "What are you trying to return?"
Ralph: "XML config. If I use XDoc, the stream works fine, but when I step into each byte from the stream, I the first three bytes have weird characters. I shouldn't have to skip the first three bytes to get the data. I spent maybe 5 hours yesterday digging around the .Net stream readers used in XDoc trying to figure out how it skips the first few bytes."
Me: "Wow...I would have used XDoc and been done and not worried about that other junk."
Ralph: "But I don't know the stream is XML. That's what I need to figure out."
Me: "What is there to figure out? You do know. Its your request. You are requesting a XML config."
Ralph: "No, the request can be anything. What if Sam requests an image? XDoc isn't going to work."
Me: "Is that a use-case? Sam requesting an image?"
Ralph: "Uh..I don't know...he could"
Me: "Sounds like your spending a lot of time doing premature optimization. You know what your accessing TFS for, if it's XML, return XML. If it's an image, return an image. Something new comes along, modify the code to handle it. Eazy peezy."
<boss walks in from a meeting>
Boss: "Whats up guys?"
Ralph: "You know the problem with TFS and not being able to stream the data I had all day yesterday? I finally figured it out. I need to keep this TFS reader simple. I'll start with the XML configs and if we more readers later, we can add them."
Boss: "Oh yea, always start simple and add complexity only when you need it."
Frack...Frack..Frack...you played some victim complaining to anyone who would listen yesterday (which I mostly ignored) about reading data from TFS was this monumental problem no one could solve, then you start complaining to me, I don't fall for the BS, then tell the boss the solution was your idea?
Lunatic or genius? Wally would be proud.4 -
I watched today one of our devs working in Windows with a Docker Environment.
I think I'm pretty insensitive regarding pain, horror and morbid stuff.
But damn. I really needed to turn off the stream or else I'd walk to the company and rip his fucking workstation out of the server rack to put it out of his misery...
Errors? ignore them....
Weird python messages? Ignore them...
wild copy pasta between notepad++ containing shell commands and a git bash... Per mouse context. Yes. Move the cursor, mark the text, right click, copy, go to terminal, right click, paste.
Understanding of whats happening. Zero. Like literal zero.
He was wondering why there were strange characters when he pasted log output in a text file...
My question: How do you think colored text works in a terminal environment?
was answered by : "Don't know, never thought about it. But don't think this has something to do with the weird characters?"
I don't wanna talk about the rest.
Retarded humanity can please kindly kill itself so the intelligent above average nice people can live in peace...
The meeting was 2 hours. I drank 5 bottles of beer after it in1 hour and I'm please to announce I'm forgetting large parts of what has happened.
Cheers.8 -
Diary of an insane lead dev: day 447
pdf thumbnails that the app generates are now in S3 instead of saved on disk.
when they were on disk, we would read them from disk into a stream and then create a stream response to the client that would then render the stream in the UI (hey, I didn't write it, I just had to support it)
one of my lazy ass junior devs jumps on modifying it before I can; his solution is to retrieve the file from the cloud now, convert the stream into a base64 encoded string, and then shove that string into an already bloated viewmodel coming from the server to be rendered in the UI.
i'm like "why on earth are you doing that? did you even test the result of this and notice that rendering those thumbnails now takes 3 times as long???"
jr: "I mean, it works doesn't it?"
seriously, if the image file is already hosted on the cloud, and you can programmatically determine its URL, why wouldn't you just throw that in the src attribute in your html tag and call it a day? why would you possibly think that the extra overhead of retrieving and converting the file before passing it off to the UI in an even larger payload than before would result in a good user experience for the client???
it took me all of 30 seconds to google and find out that AWS SDK has a method to GetPreSignedURL on a private file uploaded to s3 and you can set when it expires, and the application is dead at the end of the year.
JFC. I hate trying to reason with these fuckheads by saying "you are paid for you brain, fucking USE IT" because, clearly these code monkeys do not have brains.3 -
Java dev here. I rewrote an app and replaced a system call to ssh with a modern jaxrs post for uploading a file and (new) some additional data.
I even used a stream.
1 hour in production, first client doesn't get his file. Log says OutOfMemoryError: heap.
Me: wtf? I already use streams.
Looking at the Jersey library. Docs say nothing. An issue from 2013 says: oh if you silly don't use the Apache httpclient addon, we disable chunking and buffer the whole body, because our tests fail with the jdk included http client otherwise.
Me: meh.
No warning in the logs. Thank you soooooo much! Who could have known?4 -
In fact I'm a sinful dev, so that I can't easily decide which one is worst. From indenting with tabs, or using nano instead of vim/emacs, to hardcoding database credentials on server, to many hacks and workarounds I use as actual "fixes" when the deadline is upon me and I've tried all I could. But it always led only to my own regret. For instance, my latest sin was that I prefered Debian over Arch and used proprietary graphic drivers to speed up my new setup. But ended up with a curse from St. Ignucius. (check my last rant)
But my worst sin probably goes to when I was "printf-debugging" some issue for a GSM controller on a raspberry pi. I forgot to remove one little print line and deployed the new "fixed" version. I didn't follow that project after that for like a month or so, when the client posted back the device and said that "it just doesn't work anymore". It seemed that raspbian didn't boot beacause the sd card was curroptted. I dd'ed through the card and I noticed that there are billions of lines of "DEBUG:: reading stream from 192.some.shitty.ip", took almost all over the 32G sdcard. Just as I suddenly remembered the cursed line I just added a month ago, I declared the sd card dead with no hesitation, dunce-commented the line (so the history would remember), implemented a time out for the thread containing it, setup a journald unit for my service and removed the redirection of process output to a log file, found a new sd card and installed everything again, and finally posted back the new "fix" to the client.
Moral: Never comfort yourself for the sins you have commited in the past kids, they certainly will come back to you. And also not to do any io especially write to a file on an SD card with ext fs, in a potentially infinite loop with no timeout.
P.S: I'd posted my last rant just before the new week rant last nigh. I really liked the St. Ignucius meme so decided to create a new one. He's very adorable :)1 -
Buffer usage for simple file operation in python.
What the code "should" do, was using I think open or write a stream with a specific buffer size.
Buffer size should be specific, as it was a stream of a multiple gigabyte file over a direct interlink network connection.
Which should have speed things up tremendously, due to fewer syscalls and the machine having beefy resources for a large buffer.
So far the theory.
In practical, the devs made one very very very very very very very very stupid error.
They used dicts for configurations... With extremely bad naming.
configuration = {}
buffer_size = configuration.get("buffering", int(DEFAULT_BUFFERING))
You might immediately guess what has happened here.
DEFAULT_BUFFERING was set to true, evaluating to 1.
Yeah. Writing in 1 byte size chunks results in enormous speed deficiency, as the system is basically bombing itself with syscalls per nanoseconds.
Kinda obvious when you look at it in the raw pure form.
But I guess you can imagine how configuration actually looked....
Wild. Pretty wild. It was the main dict, hard coded, I think 200 entries plus and of course it looked like my toilet after having an spicy food evening and eating too much....
What's even worse is that none made the connection to the buffer size.
This simple and trivial thing entertained us for 2-3 weeks because *drumrolls please* none of the devs tested with large files.
So as usual there was the deployment and then "the sudden miraculous it works totally slow, must be admin / it fault" game.
At some time it landed then on my desk as pretty much everyone who had to deal with it was confused and angry, for understandable reasons (blame game).
It took me and the admin / devs then a few days to track it down, as we really started at the entirely wrong end of the problem, the network...
So much joy for such a stupid thing.18 -
It's rant time again. I was working on a project which exports data to a zipped csv and uploads it to s3. I asked colleagues to review it, I guess that was a mistake.
Well, two of my lesser known colleague reviewed it and one of the complaints they had is that it wasn't typescript. Well yes good thing you have EYES, i'm not comfortable with typescript yet so I made it in nodejs (which is absolutely fine)
The other guy said that I could stream to the zip file and which I didn't know was possible so I said that's impossible right? (I didn't know some zip algorithms work on streams). And he kept brushing over it and taking about why I should use streams and why. I obviously have used streams before and if had read my code he could see that my code streamed everything to the filesystem and afterwards to s3. He continued to behave like I was a literall child who just used nodejs for 2 seconds. (I'm probably half his age so fair enough). He also assumed that my code would store everything in memory which also isn't true if he had read my code...
Never got an answer out of him and had to google myself and research how zlib works while he was sending me obvious examples how streams work. Which annoyed me because I asked him a very simple question.
Now the worst part, we had a dev meeting and both colleagues started talking about how they want that solutions are checked and talked about beforehand while talking about my project as if it was a failure. But it literally wasn't lol, i use streams for everything except the zipping part myself because I didn't know that was possible.
I was super motivated for this project but fuck this shit, I'm not sure why it annoys me so much. I wanted good feedback not people assuming because I'm young I can't fucking read documentation and also hate that they brought it up specifically pointing to my project, could be a general thing. Fuck me.3 -
So I'm making a file uploader for a buddy of mine and I got an error that I had never seen before. Suddenly I had C++ code and some other weird shite in my terminal. Turns our that I got a memory leak and the first thing that sprung to mind was "Fuck yes, I get to do some NCIS ass debugging".
Now the app worked fine for smaller files, like 5MB - 10MB files, but when I tried with some Linux ISO's it would produce the memory leak.
Well I opened the app with --inspect and set some breakpoints and after setting some breakpoints I found it. Now, for this app I needed to do some things if the user uploads an already existing file. Now to do that I decided to take the SHA string of the file and store it in a database. To do this I used fs.readFile aaaaaaaaaand this is where it went wrong. fs.readFile doesn't read the file as a stream.
Well when I found that, boy did I feel stupid :v5 -
Ahh.. there is nothing like the joyous feeling of writing a working piece of code for your own personal projects.
I spent several weeks and a few hours today to finally get my Python automation script working and I am very proud of myself.
Here's what it does:
* open a text file, extract a specific string from it using rather complicated xpath
* open another text file and do the same
* replace result 1 with result 2
* log results
* close file
* automate the process
Even though it looks easy, I had to mess around with a lot of problems such as permissions, indentation, stream writing, file status, etc.
Now, instead of having to manually do this job, I can just let my machine do it!1 -
The guy who wrote a ton of legacy code at my company apparently had no idea that you need to close a file stream. And also that IO may throw exceptions.
Wooo1 -
I am a Technical Lead in the department in my company that writes code for our clients that have money but doesn't have the technical expertise to handle the complexities of our own software.
Part of my tasks involve taking care of a few projects written by employees that have left after using third-party tools rather than using our own software. No one else in this department knows these third-party tools, they only know our own, and my *still limited* web development experience means I get dumped these things in my lap.
And I'm SO pissed at these projects and their authors and the manager that let these ex-employees write these things. There is this one project that was managed by two different "developers" (I don't know they deserve this title) at two different times, and it is so riddled with different technologies it makes me want to throw up almost daily.
Don't believe me? Here is a complete list of the dependencies listed in the package.json of this project: babel-polyfill, body-parser, cookie-parser, debug, edge, edge-sql, excel-to-json, exceljs, express, html-inline, jade, morgan, mssql, mysql, pug, ramda, request, rotating-file-stream, serve-favicon, webpack, xlsx, xml2js
What this doesn't even show, is that one part of this project (literally one page) is made using react, react-dom, react-redux, and jade. The other part (again literally one page) is made using Angular and Pug. In case you missed it while picking up your jaw, there's also mssql, mysql, edge and edge-sql. excel-to-json, exceljs, xlsx.
Oh you want *more* juicy details? This project takes the entire data object used by the front-end, stringifies it into JSON, and shoves it into the database *as a single field*. And instead of doing WHERE clauses in the SQL queries, it grabs the entire table, loops, parses the json, and does a condition on it. If even one of those JSON entries gets corrupted, the entire solution breaks because these "developers" don't know what try/catch is.
The client asked for a very simple change in their app, which was to add a button that queries the back-end for a URL, shows it in a modal dialog, after which a button is clicked to verify the link by doing a second query to the back-end before modifying a couple of fields in the page.
This. Took. Me. Two. Months*. Save me. Please, save me.
*between constant context switches between this and other projects that were continuously failing because of their mistakes.4 -
This literally happened in my current team, and I'm not even an experienced dev yet.
Incident happened like this :
Our team is working on a RCP based on eclipse plugins, which has a headless mode and a GUI mode. Now, in the GUI mode, my manager cum architect thought there are no need of user log files (long story) because the user can see the info on screen, whereas in the headless mode, she wanted me to print the logs onto the console and a log file as well.
Now it just so happened that our team had got a recent addition as a replacement to our lead developer (she left the company) who claimed she had 3 years of expertise and a masters degree, and she was assigned a task. The task was to format a custom file we were generating out of the product (basically dumping info in a file) in a human-readable format. Miss new-addition-masters-degree decided it would be a very good idea to redirect the standard java output stream to a file output stream ( which she used for generating the formatted file ) but somehow never realized that she needed to reset the output stream back to standard output.
Consequences were devastating. I wrote the logic for the logger ( yes, apparently any available logging mechanism won't do it, again, long story ) and had it printing to a file in tmp directory. The logs seemed to be working fine initially but after a few logs, specifically from the point where the formatter started working, all the logs got printed in the formatted file. And this file was supposed to be used by our clients to develop something on top of it. Naturally, I got the heat of it and then naturally, worried and nervous and curious and in a frenzied state of mind, I started debugging.
When I got to the actual fault, I seriously could not decide whether to cry or laugh or call up miss masters and scream at her. I decided to ask her about what the hell she had written and her answer was most of it was written by the developer she replaced, so she didn't know it would cause this much problem. Anyway, I fixed the leak after that and averted the catastrophe.
And that, fellow devs, is the story of how I solved a crisis in my first year at corporate.1 -
I'm thinking of self hosting all my small web projects,
I have this old laptop running ubuntu server heedlessly I used to store and stream pirated movies, after multiple embarrassing moments with free backend/platform as a service options and not finding a cheap VPS, this seems like the way to go. I don't get much traffic on these sites i just want them to be available when i need to present them.
then there's tons of other features that are locked behind a paywall,
I once had to store images in the database because heroku wont accept file uploads and the project hadn't been paid, in short, I was dead broke9 -
I downloaded a youtube stream with a tool i found on reddit. When I try to play it with VLC it tells me that the moov atom is missing. The file is ~110Mb so there is video data in it, I just don't know how to play it. Can anyone help?7
-
So i'm trying to upload a file to an SSH server using node. First I try the obvious putFile method provided by the obvious node-ssh package. On any other server this would work fine but this server doesn't have sftp installed so that doesn't work.
OK, so next I learn how scp works (it runs the command "scp -t" on the remote server, and sends to stdin a command like "C0666 1234 file.txt" and then sends the contents of the file) and I write some javascript code to do this. It's pretty finicky, the first few tries I forget to close the stream right or detect the program finishing. I add some logging and that helps me figure out what the problem is, and finally I get it to not output any errors.
So I log into the server and check and the file isn't there. I try again several times, file still isn't there. I try running scp -t manually on the server, typing in exactly what my program is sending, and it works. This goes on for a while until I realize that I've been sending a file to one server and logging into a different server to check if the file was sent. grrr6 -
Wasted 5 hours to implement video recording with cv2 in python. Followed the tutorials and searched SO for help. The video file I need to produce just doesn't get produced. It's a camera stream from ROS. Yet no errors that might help. Or even any errors. Thanks.
It's for my bachelor thesis. Fuck.4 -
Warning: fopen(welcome.txt) [function.fopen]: failed to open stream:
No such file or directory in C:\webfolder\test.php on line 21 -
I was once trying to create a video player since the format was not supported by the browsers today, so i started to download the video files from the server. Only to discover a bug in my code was trying to download a video file from the server. This was way to much for the server to handle and caused it to crash. People where running in the building to get the stream back up, i sat there silently and killing my browsers process. Since then i always test my request from local if they are okay before downloading a file from a server
-
My current "file/media server" is a crappy old falling apart windows box with a stupid mismash of internal and external drives with no redundancy. That sucks for a number of reasons, so planning on dropping around a grand or so (including drives) on doing it properly.
Space requirement would be around 20TB-ish of usable space, with 1 disk's worth of redundancy. That can include a newish 5TB drive I have lying around however. Would also run either Plex / Jellyfin, so some horsepower for transcoding would be nice (but no need for more than a single 1080 stream at once.) 24/7 operation, so don't want anything too power hungry.
Current (loose) thinking on the hardware side is an AM4 board and a reasonably low end CPU, 3x8TB WD golds. Software side, probably CentOS, then mergerfs + snapraid. Anyone got any insight as to other options? Hardware not my speciality in particular, so open to suggestions.14 -
Something interesting i learned today about the html5 video tag is that even if preload is set it's up to the player the render engine is using to fetch the index of the file first as with mp4 this is usually at the end of the file.
This means that for Blink and Gecko most likely fetch this first themselves. But for webkit it opens in quicktime on mobile devices which you cannot pass parameters to and flat out waits for the entire stream to start playing.1 -
So how does an experienced programmer break his own code in a such a terrible manner.
And why would someone try to make me embrace a bogus thing I KNOW doesn't work to make sure its broken ?
See this is what I hate about this shit.
No programmer would make the mistake that was made. They converted all their classes that interpret STRING VALUES using a parse to BitConverter.
(The input is a byte array) for like, ALL THEIR CLASSES.
Again this seems fucking intentional.
Instead of letting it remain a goddamn class that handed around an open file stream with a binaryreader around it which was simple, they tried some fucking fancy shit and throw readonlyspan<byte> at a read method which is where the code WILL fuck up ! wtf is wrong with you fucking dumb bastards ? IF YOU DONT WANT THE CHOMS GETTING AHEAD SHOOT THEM JESUS CHRIST !
Maybe this is some special interests shit to make sure ArcGis remains the dominant gis package and no ordinary lowly programmer can have a reader to get started with their own manipulations.
YOu fucking bastards screwed up the world and i want to eat your fucking hearts from your chests and wear your fucking scalps and ears on a fucking necklace while burning down your fucking houses !
AND FIX THAT GODDAMN STEAM RELEASE VALVE !!!1