Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "flat file"
-
CLIENT "So my nephew who does stuff with computers built it and we are ok with how it all works so don't worry about changing that. "
DEV "so like you have a public form with no input filtering, spam mitigation let alone sanitization or remote concern for security. Basically you have a Json flat file that is 34mbs of links to, viagra, replica watches, nock off name brands and one real estate company. It is getting about 15 submissions an hour. Since you don't want me changing how it works are you happy to just leave all that ?"
CLIENT "no no we don't want all that but we have no route to delete it, can you just stop all the spam and let us continue on?"
DEV "ok so back to my first question can we rebuild all of this properly, or do you really want to just leave it all"
:/ FML3 -
'Sup mates.
First rant...
So Here's a story of how I severely messed up my mental health trying to fit in university.
But the bonus: Found my passion.
Her we go,
Went to university thinking it'll be awesome to learn new stuff.
1st sem was pure shock - Programming was taught at the speed of V2 rockets.
Everything was centred around marks.
Wanted to get a good run in 2nd sem, started to learn Vector design, but RIP- Hospitalized for Staph infection, missed the whole sem and was in recovery for 3 months.
So asked uni for financial assistance as I had to re-register the courses the next semester. They flat out refused, not even in this serious of a case.
So, time to register courses for third semester, turns out most of the 2nd year courses are full, I had to take 3rd year courses like:
Social and Informational Networks
Human Computer Interaction
Image processing
And
Parallel and Distributed Computing (They had no prerequisites listed, for the cucks they are: BIG MISTAKE)
Turns out the first day of classes that I attend, the Image proc. teacher tells me that it's gonna be difficult for 2nd years so I drop it, as the PDC prof. also seconds that advice.
Time travel 2 months in: The PDC prof is a bitch, doesn't upload any notes at all and teaches like she's on Velocity-9 while treating this subject like a competition on who learns the most rather than helping everyone understand.
Doesn't let students talk to each other in lab even if one wants to clear their friend's doubt, "Do it on your own!" What the actual fuck?
Time for term end exams and project submission: Me and 3 seniors implement a Distributed File System in python and show it to her, she looks satisfied.
Project Results: Everyone else got 95/100
I got 76.
She's so prejudiced that she thinks that 2nd years must have been freeloaders while I put my ass on turbo for the whole sem, learning to code while tackling advanced concepts to the point that I hated to code.
I passed the course with a D grade.
People with zero consideration for others get absolutely zero respect from me.
Well it's safe to say that I went Nuclear(heh.. pun..) at this point, Mentally I was in such a bad place that I broke down.... Went into depression but didn't realise it.
But,
I met a senior in my HCI class that I did a project with, after which I discovered we had lots of similar interests.
We became good friends and started collaborating on design projects and video game prototyping.
Enter the 4th sem and holy mother of God did I got some bad bad profs....
Then it hit me
I have been here for two years, put myself through the meat grinder and tore my soul into shreds.
This Is Not Me
This Wont Be The End Of Me
I called up my sister in London and just vented all my emotions in front of her.
Relief.
Been a long time since I felt that.
I decided to go for what I truly feel passionate about: Game Design
So I am now trying to apply for Universities which have specialised courses for game design.
I've got my groove again, learnt to live again.
Learning C# now.
:)
It's been a long hello, and If you've reached till here somehow, then damn, you the MVP.
Peace.9 -
Last Monday I bought an iPhone as a little music player, and just to see how iOS works or doesn't work.. which arguments against Apple are valid, which aren't etc. And at a price point of €60 for a secondhand SE I figured, why not. And needless to say I've jailbroken it shortly after.
Initially setting up the iPhone when coming from fairly unrestricted Android ended up being quite a chore. I just wanted to use this thing as a music player, so how would you do it..?
Well you first have to set up the phone, iCloud account and whatnot, yada yada... Asks for an email address and flat out rejects your email address if it's got "apple" in it, catch-all email servers be damned I guess. So I chose ishit at my domain instead, much better. Address information for billing.. just bullshit that, give it some nulls. Phone number.. well I guess I could just give it a secondary SIM card's number.
So now the phone has been set up, more or less. To get music on it was quite a maze solving experience in its own right. There's some stuff about it on the Debian and Arch Wikis but it's fairly outdated. From the iPhone itself you can install VLC and use its app directory, which I'll get back to later. Then from e.g. Safari, download any music file.. which it downloads to iCloud.. Think Different I guess. Go to your iCloud and pull it into the iPhone for real this time. Now you can share the file to your VLC app, at which point it initializes a database for that particular app.
The databases / app storage can be considered equivalent to the /data directories for applications in Android, minus /sdcard. There is little to no shared storage between apps, most stuff works through sharing from one app to another.
Now you can connect the iPhone to your computer and see a mount point for your pictures, and one for your documents. In that documents mount point, there are directories for each app, which you can just drag files into. For some reason the AFC protocol just hangs up when you try to delete files from your computer however... Think Different?
Anyway, the music has been put on it. Such features, what a nugget! It's less bad than I thought, but still pretty fucked up.
At that point I was fairly dejected and that didn't get better with an update from iOS 14.1 to iOS 14.3. Turns out that Apple in its nannying galore now turns down the volume to 50% every half an hour or so, "for hearing safety" and "EU regulations" that don't exist. Saying that I was fuming and wanting to smack this piece of shit into the wall would be an understatement. And even among the iSheep, I found very few people that thought this is fine. Though despite all that, there were still some. I have no idea what it would take to make those people finally reconsider.. maybe Tim Cook himself shoving an iPhone up their ass, or maybe they'd be honored that Tim Cook noticed them even then... But I digress.
And then, then it really started to take off because I finally ended up jailbreaking the thing. Many people think that it's only third-party apps, but that is far from true. It is equivalent to rooting, and you do get access to a Unix root account by doing it. The way you do it is usually a bootkit, which in a desktop's ring model would be a negative ring. The access level is extremely high.
So you can root it, great. What use is that in a locked down system where there's nothing available..? Aha, that's where the next thing comes in, 2 actually. Cydia has an OpenSSH server in it, and it just binds to port 22 and supports all of OpenSSH's known goodness. All of it, I'm using ed25519 keys and a CA to log into my phone! Fuck yea boi, what a nugget! This is better than Android even! And it doesn't end there.. there's a second thing it has up its sleeve. This thing has an apt package manager in it, which is easily equivalent to what Termux offers, at the system level! You can install not just common CLI applications, but even graphical apps from Cydia over the network!
Without a jailbreak, I would say that iOS is pretty fucking terrible and if you care about modding, you shouldn't use it. But jailbroken, fufu.. this thing trades many blows with Android in the modding scene. I've said it before, but what a nugget!8 -
Alright since I have to deal with this shit in my part time job I really have to ask.
What is the WORST form of abusing CSV you have ever witnessed?
I for one have to deal with something like this:
foo,1,2,3,4,5
0,2,4,3,2,1
0,5,6,4,3,1
bar,,,,,
foobar,,,,,
foo can either be foo, or a numeric value
if it is foo, the first number after the foo dictates how many times the content between this foo and the next bar is going to be repeated. Mind you, this can be nested:
foo,1,,,,
1,2,3,4,5,6
foo,10,,,,
6,5,4,3,2,1,
bar,,,,,
1,2,3,4,5,6
bar,,,,,
foobar,,,,,
foobar means the file ends.
Now since this isn't quite enough, there's also SIX DIFFERENT FLAVOURS OF THIS FILE. Each of them having different columns.
I really need to know - is it me, or is this format simply utterly stupid? I was always taught (and fuck, we always did it this way) that CSV was simply a means to store flat and simple data. Meanwhile when I explain my struggle I get a shrug and "Just parse it, its just csv!!"
To top it off, I can not use the flavours of these files interchangably. Each and everyone of them contains different data so I essentially have to parse the same crap in different ways.
OK this really needed to get outta the system6 -
¡rant|rant
Nice to do some refactoring of the whole data access layer of our core logistics software, let me tell an story.
The project is around 80k lines of code, with a lot of integrations with an ERP system and an sql database.
The ERP system is old, shitty api for it also, only static methods through an wrapper to an c++ library
imagine an order table.
To access an order, you would first need to open the database by calling Api.Open(...file paths) (yes, it's an fucking flat file type database)
Now the database is open, now you would open the orders table with method Api.Table(int tableId) and in return you would get an integer value, the pointer.
Now for the actual order. first you need to search for it by setting the search parameter to the column ID of the order number while checking all calls for some BS error code
Api.SetInt(int pointer, int column, int query Value)
Then call the find method.
Api.Find(int pointer)
Then to top this shitcake of an api of: if it doesn't find your shit it will use the "close enough" method of search.
And now to read a singe string 😑
First you will look in the outdated and incorrect documentation given to you from the devil himself and look for the column ID to find the length of the column.
Then you create a string variable with ALL FUCKING SPACES.
Now you call the Api.GetStr(int pointer, int column, ref string emptyString, int length)
Now you have passed your poor string to the api's demon orgy by reference.
Then some more BS error code checking.
Now you have read an string value 😀
Now keep in mind to repeat these steps for all 300+ columns in the order table.
News from the creators: SQL server? yes, sql is good so everything will be better?
Now imagine the poor developers that got tasked to convert this shitcake to use a MS SQL server, that they did.
Now I can honestly say that I found the best SQL server benchmark tool. This sucker creams out just above ~105K sql statements per second on peak and ~15K per second for 1.5 second to read an order. 1.5 second to read less than 4 fucking kilobytes!
Right at that moment I released that our software would grind to an fucking halt before even thinking about starting it. And that me & myself and I would be tasked to fix it.
4 months later and two weeks until functional beta, here I am. We created our own api with the SQL server 😀
And the outcome of all this...
Fixes bugs older than a year, Forces rewriting part of code base. Forces removal of dirty fixes. allows proper unit and integration testing and even database testing with snapshot feature.
The whole ERP system could be replaced with ~10 lines of code (provided same relational structure) on the application while adding it to our own API library.
Best part is probably the performance improvements 😀. Up to 4500 times faster and 60 times less memory usage also with only managed memory.3 -
I've decided to, as an educational exercise, implement DEFLATE compression / decompression and zip file format, and eventually tackling Excel format (which is just a .zip) so I can generate true excel spreadsheets (instead of .csv files) client-side using JavaScript.
Are there already libraries that do this? Yes, but then I don't get to try to implement these interesting algorithms. Is it currently 1 AM? Yes. Do I have work tomorrow? Also yes.
If I don't just fall flat on my face, I'll post updates!1 -
Best
typescript - I needed to learn it for a project and I like it, I know java and javascript and it is something in between of those two that makes writing enterprise web applications easier, it’s nice that you can debug it directly in chrome, it makes things easier
Worst
docker, Dockerfiles - devops tools - amount of shell commands inside them and mangled && to make everything running in one file layer makes those unreadable mess that you need to think twice to understand, there is no debugger for it, you do everything with try and see what happens, there is actually no real dev toolset for devops and that sucks, since you got builder images that makes things more mangled than before, it’s clearly missing some external officially approved scripting language or at least
FUNCTION and
WITH LAYER and indentation / parentheses syntax and they still trying to make it flat, why are you doing that ?
as a result next to Dockerfile cause you can’t import multiple ones you get bunch bash scripts with mangled syntax and other crap that is glued together to make a monster - and this runs most of current software on this planet2 -
Aside from simple programs I wrote by hand-transcribing code from the "Basic Training" section of 3-2-1 Contact magazine when I was a kid in the '80s, I would say the first project I ever undertook on my own that had a meaningful impact on others was when I joined a code migration team when I was 25. It was 2003.
We had a simple migration log that we would need to fill out when we performed any work. It was a spreadsheet, and because Excel is a festering chunk of infected cat shit, the network-shared file would more often than not be locked by the last person to have the file open. One night after getting prompted to open the document read-only again, I decided I'd had it.
I went to a used computer store and paid $75 out of pocket for an old beater, brought it back to the office, hooked it to the network, installed Lunar Linux on it, and built a simple web-based logging application that used a bash-generated flat file backend. Two days later, I had it working well enough to show it to the team, and they unanimously agreed to switch to it, rather than continue to shove Excel's jagged metal dick up our asses.
My boss asked me where I was hosting it, as such an application in company space would have certainly required his approval to procure. I showed him the completely unauthorized Linux machine(remember, this was 2003, when fortune 500 corporations, such as my employer, believed Ballmer's FUD-spew about Linux being a "virus" was real and not nonsense at all), and he didn't even hesitate to back me up and promise to tell the network security gestapo to fuck off if they ever came knocking. They never did.
I was later informed that the team continued to use the application for about five years after I left. -
personal projects, of course, but let's count the only one that could actually be considered finished and released.
which was a local social network site. i was making and running it for about three years as a replacement for a site that its original admin took down without warning because he got fed up with the community. i loved the community and missed it, so that was my motivation to learn web stack (html, css, php, mysql, js).
first version was done and up in a week, single flat php file, no oop, just ifs. was about 5k lines long and was missing 90% of features, but i got it out and by word of mouth/mail is started gathering the community back.
right as i put it up, i learned about include directive, so i started re-coding it from scratch, and "this time properly", separated into one file per page.
that took about a month, got to about 10k lines of code, with about 30% of planned functionality.
i put it up, and then i learned that php can do objects, so i started another rewrite from scratch. two or three months later, about 15k lines of code, and 60% of the intended functionality.
i put it up, and learned about ajax (which was a pretty new thing since this was 2006), so i started another rewrite, this time not completely from scratch i think.
three months later, final length about 30k lines of code, and 120% of originally intended functionality (since i got some new features ideas along the way).
put it up, was very happy with it, and since i gathered quite a lot of user-generated data already through all of that time, i started seeing patterns, and started to think about some crazy stuff like auto-tagging posts based on their content (tags like positive, negative, angry, sad, family issues, health issues, etc), rewarding users based on auto-detection whether their comments stirred more (and good) discussion, or stifled it, tracking user's mental health and life situation (scale of great to horrible, something like that) based on the analysis of the texts of their posts...
... never got around to that though, missed two months hosting payments and in that time the admin of the original site put it back up, so i just told people to move back there.
awesome experience, though. worth every second.
to this day probably the project i'm most proud of (which is sad, i suppose) - the final version had its own builtin forum section with proper topics, reply threads, wysiwyg post editor, personal diaries where people could set per-post visibility (everyone, only logged in users, only my friends), mental health questionnaires that tracked user's results in time and showed them in a cool flash charts, questionnaire editor where users could make their own tests/quizzes, article section, like/dislike voting on everything, page-global ajax chat of all users that would stay open in bottom right corner, hangouts-style, private messages, even a "pointer" system where sending special commands to the chat aimed at a specific user would cause page elements to highlight on their client, meaning if someone asked "how do i do this thing on the page?", i could send that command and the button to the subpage would get highlighted, after they clicked it and the subpage loaded, the next step in the process would get highlighted, with a custom explanation text, etc...
dammit, now i got seriously nostalgic. it was an awesome piece of work, if i may say so. and i wasn't the only one thinking that, since showing the page off landed me my first two or three programming jobs, right out of highschool. 10 minutes of smalltalk, then they asked about my knowledge, i whipped up that site and gave a short walkthrough talking a bit about how the most interesting pieces were implemented, done, hired XD
those were good times, when I still felt like the programmer whiz kid =D
as i said, worth every second, every drop of sweat, every torn hair, several times over, even though "actual net financial profit" was around minus two hundred euro paid for those two or three years of hosting. -
Been working on trying to get JMdict (relatively comprehensive Japanese dictionary file) into a database so I can do some analysis on the data therein, and it's been a bit of a pain. The KANJIDIC XML file had me thinking it'd be fairly straightforward, but this thing uses just about every trick possible to complicate what one would think would be a straightforward dictionary file:
* Readings and Spellings/Kanji usage are done in a many-to-many manner, with the only thing tying them together being an arbitrary ID. Not everything is related, however, as there can be certain readings that only apply to specific spellings within the group and vice versa. In short, there's no way to really meaningfully establish a headword fora given entry.
* Definitions are buried within broader Sense groups, which clumsily attach metadata and have the same many-to-many (except when not) structure as the readings/spellings.
Suffice to say, this has made coming up with a logical database schema for it a bit more interesting than usual.
It's at least an improvement over the original format, however, which had a couple different ways of setting up the headword section and could splatter tagging information across any part of a given entry. Fine if you're going to grep the flat file, but annoying if you're looking for something more nuanced.
Was looking online last night to see if anyone had a PHP class written to handle entries and didn't turn anything up, but *did* find this amusing exchange from a while back where the creator basically said, "I like my idiosyncratic format and it works for me. Deal with it!": https://sci.lang.japan.narkive.com/...
Grateful to the creator for producing the dictionary I've used most in my studies over the years, but still...3 -
In the past, apps I've written have used a flat file backend. It's very fast, but obviously clunky to have a big structure of flat files for an app. It ran circles around framework-based RDBMS backends, as performance is concerned, but again, it was clunky. Managing backups and permissions on tens or hundreds of thousands of small files was no fun. Optimizing code for scaling was fun- generating indexes, making shortcuts -but something was still missing. Early in 2017 I discovered redis. A nosql backend that just stores variables and lives almost entirely in memory. Excellent modules and frameworks for every language. It was EXACTLY what I'd needed, even though I didn't know I did. I spent a good deal of time in 2017 converting apps from flat files to redis, and cackled with glee as they became the apps I wanted them to be. Earlier this week, I started building my first app that started with redis, instead of flat files, and I can't stop gushing to anyone who will listen. Redis for president!
-
When you receive a psd file with a million layers, when all you really need from it is the colour values and what font/size they've used and some measurements. You'll just do it all properly with CSS. Better off with a flat png or 2 with transparency and some original vectors.
-
Several years ago I spent over two months working out how to integrate Text To Speech and Speech To Text (TTS/STT) into any windows program I wrote in Delphi, originally for a powerful flat-file search engine. Does anyone know if TTS/STT is useful on windows 10+ or have any use?
I was thinking about redeveloping the search engine into a stand alone program which can be used as a fast and light query tool with trigger functions, it can be made into a "reply bot" or used with a server like Apache, but without the old IBM mainframe mentality being readopted as "AI" and "social media" everywhere today. low-level Independent and secure droid like systems sound more fun to develop. -
is there a Go module for editing Flat Open Document Text? I'll like to edit placeholder text from a fodt file and send the results to thecodingmachine/gotenberg to convert the file to a PDF
-
R&D Lead Architect: "We want this next gen platform to be all AWS."
Us: "Alright, can we talk about automated testing?"
R&D Lead Architect: "Sure, for automated tests you'll want to just dump events from your system into a flat file on S3. It's readable with Microsoft Excel."
Us now: still here.
R&D Lead Architect now: not here.
:) -
Visual Studio is a fucking shitheap of an IDE and everyone who worked on it should be fucking incinerated.
I've been trying to get Unity to build my game for about a fucking hour and a half now, only to realize that it was a warning from a script that was causing it to fall flat on it's face.
So I deleted the script because it was a shitty script anyways, not much was being lost here, and I started building the game, and lo and behold, it was actually fucking doing something.
I went to go get a drink, only to come back to see that this stupid fucking engine gave me yet ANOTHER error that wasn't even from a script anywhere in my game's files.
It was fucking Visual Studio. It didn't even give me that concise of a fucking error, just "this file doesn't exist" or whatever hypercomplex bullshit it spat out at me.
So, I took to google, and found that I should open the solution file hidden within the uncompleted build, and upon doing so Visual Studio told me it needed to install some more shit in order to do so.
I decided to let it do it's thing, and you wanna know what the real kicker is?
I started writing this rant when it was at 25%.
I had started talking to my friend about how absolutely fucking garbage and slow this IDE is at around the point where it started downloading. It took fifteen fucking minutes for it to get to 25%.
I could uninstall and reinstall both Destiny 2 and Killing Floor 2, twice, in the time takes for this shitty fucking program to install its tumor of an update onto my system.
FUCK Visual Studio.
Fuck the person who conceived the idea of it.
And fuck every single person who supports it.
Every single person that thinks this fucking anathema of an IDE was a good idea should be incinerated.12 -
Can we say that "a basic website is a type of app that serves documents", implying a website is more specific than an app, and the most common type of data served is in the form of a document (html, pdf, json, xml).
I'm trying to see how this argument does for/against flat-file cms'es in specific/ general cases per type of app/ website.1 -
Something interesting i learned today about the html5 video tag is that even if preload is set it's up to the player the render engine is using to fetch the index of the file first as with mp4 this is usually at the end of the file.
This means that for Blink and Gecko most likely fetch this first themselves. But for webkit it opens in quicktime on mobile devices which you cannot pass parameters to and flat out waits for the entire stream to start playing.1 -
Internal: a program that takes files (or a zip/7z file with files in it) and loads them into a database/table in file created order per a json-based map file because flat file loading in SSIS is horrible.
Public: ChromeKiosk https://github.com/benjaminkwilliam...