AboutCollecting the data to train our future overlords. Writing pipelines to process bullshit data with the help of rust and shell.
SkillsA lot of programming languages, a high bullshit-tolerance, the ability to pun-ish people who put credentials in code and the power to procastinate.
LocationSoon near you
Joined devRant on 2/14/2018
Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
My current task involves processing the commoncrawl web archive, and it's like a box of junk you buy at a flea market. You find so much useless stuff, broken stuff, stuff that makes you question people...
I queried a few commoncrawl files and amassed 4GB of this spam. Every time I look at it it gets weirder. There is an italian article about malware in there too.
Here's a text sample:
"Not from her bedroom, she her stream view and meet new experience. In hd india, because swimsuit still laws exist no interaction or frigthened and."1
I'm back from the dead to rant again. This time it's punycode.
My job has to do with processing the commoncrawl web archives, and for some reason one in 20.000.000 archived webpages crashed my program. After some debugging I found this issue that seems to be the reason my code crashes https://github.com/servo/rust-url/...
To summarize the issue: Since punycode unicode characters can be encoded into domain names. But not every character is allowed. Not only do these invalid domains get registered, I need an in-depth knowledge about unicode to understand what is wrong here.
How did we turn domain names into something so complicated?3
Goodbye devrant, maybe I'll come back in a few months. Or maybe I'll go forth to the next community instantly.15
On the one hand my 1820-char rant did as good as my comment comparing facebook users to monkeys, which is really annoying me.
On the other hand dfox ++'d my rant, so maybe i'll let this rant live a little while longer.
I don't know what you did yesterday, but i did make my company throw away 2 months of progress.
It all started in the beginning, since that i've made numerous complaints about the workflow or code and how to improve it. I've been told off every time, and every time i either told the boss who agreed in the end or wrote code to prove myself. Everything was a hassle and my tasks weren't better.
Team lead: you'll do X now, please do that by making Y.
Me: but Y is insecure, we should do Z.
Team lead: please do Y
Later it turns out Y is impossible and we do Z in the end...
Team lead: please do W now
Me, a few days later: i've tried and their server doesn't give http cors headers, doing W in the browser is impossible
Team lead, a few days later: have you made progress on W?
Me: * tells again it's impossible and uploads code to prove it *
Team lead: * no response *
After that i had enough. Technically i still was assigned to do W, but i used my time to look over the application and list all the things wrong with it. We had everything, giant commits, commented out code, unnecessary packages, a new commit introduced packages that crashed npm install on non-macs, angularjs-packages even though we use angular, weird logic, a security bug, all css in one file even though you can use component-specific css files...
I sent that to my boss, telling him to let the backend-guys have a look at it too and we had a meeting about this. I couldn't attend but they agreed with me completely. They decided to throw away what we have already and to let one of the backend-guys supervise our team. I guess there will be another talk with the team lead, but time will tell.
It feels so good having hope to finally escape this hellish development cycle of badly defined task, bad communication and headache-inducing merges.
My laptop now finally refuses to connect to the company wifi, and here i am browsing bitbucket on my phone
I didn't expect me to be productive today anyway, but that's another rant.11
Fucking shit for brains authors that think the digital world is a fantasy realm where everything can happen just to aid their story. Out of boredom i watched "scorpion" today, a tv series about a group of geniusses which are a special case task force.
They got a visitor from the government saying the servers from the federal reserve bank were encrypted with ransomware. I already twitched when they said the economic system would collapse if the servers were left inoperational for a few days. Then one guy got to his desk and "hacked" the fed network to check... he then tried to remove the malware but "it changed itself when observed". But they got the magical fingerprint of the device that uploaded it. In the end some non-programmers created the malware, but it is super fast and dangerous because it runs on a quantum computer which makes it hyper fast and dangerous. They got to the quantum computer which was a glowing cube inside another cube with lasers going into it and they had to use mirrors to divert the lasers to slow down that quantum thingy. And be careful with that, otherwise it explodes. In the end the anti-malware battled the malware and won, all in a matter of minutes.
This is a multimillion hollywood production. How can a movie this abusive to computer science even air on television? Shit like this is the reason people still think the cyberworld is some instable thing that can explode any second. It's not, it's an instable thing that can break down any second. I remember "ghost in the wires" and people had surreal imaginations about the internet already. Shit like this is why people stay dumb and think everything can be done in seconds. If i ever should encounter one of these idiots i tell him i have an app that can publish his browser history by taking a picture of his phone and watch his reaction.
Time to shuw down the tv and learn vim again.11
Fucking experimental technologies. I feel like doing webassembly stuff is like buying a smart device, it's not worth any of the trouble for now.
I wanted to do some webassembly-stuff with rust and yew (basically react for rust). I was really hyped because it all looked promising and i found this cool band "heilung" whose music made me my coding feel like black magic with complex incantations and shit.
A basic webassembly setup did work, but everything afterwards was pure shit. Crate installation didn't go as expected, i get weird errors even though i simply copied the example (and checked the versions). The best i got was when i tried to compile and rust told me to go fuck myself because i cant use feature XY in a package in the stable environment. Why the hell would someone even publish said package then? After losing half a day because of this i give up for now. I don't feel like a badass magician anymore anyways, more like the guy that puts mentos into coke and gets hit by the foam.
Any good rust-IDE's out there? There's an atom-plugin that's ok imo and i found an eclipse-based IDE, but for that i have to set up a shitton of things.
How do you develop in rust?13
If it were possible i would make windows punish itself by playing this: https://youtu.be/th4Czv1j3F8 . But this goddamn piece of garbage software refuses to connect to my Ian-network since an hour ago. Why? I have no fucking idea. There are a switch and a repeater between the router but that never stopped wanblows before. And the devices located deeper in my network topology are fine, i also reset everything from the physical connection to "network adapters" in the UX horror they call system settings.
And i'm pretty sure it'll work again in an hour or so for no appearent reason just so this steaming pile of shit code could ruin my afternoon.
Are there any valid reasons laptops turn on by themselves?
I just watched my mothers laptop turn on by itself and i could have sworn it was turned off (not hibernating)9
For work i'll have to use an API whose server doesn't support OPTIONS-requests. All would be ok if the request wouldn't be made FROM A FUCKING BROWSER.
How old is CORS? Have you been living under a rock since then? (Well, maybe. Because they're using IIS7)
I'm currently taking another dive into haskell. I didn't know you needed a masters degree in theoretical something to print text on stdout.
Note to myself: make dog-emoji-based language huskyell2
Using an api: ok, this url (.../xml/endpoint) gives me an xml-document. Oh, and there is a node whose text contains html markup, interesting.
Using the same endpoint, but requesting json: yep, that's the same data, there even is this big html string and... why is this string in a json object wrapped inside "<![CDATA[...]]>"?
If i ever see a courtroom from the inside i'll plead insanity.2
Is it just me or has german tv become more and more ad-ridden? A few years ago it was considered a dick move to play a single ad during an episode.
Now some channels continue the tv program but make it smaller so they can show an ad on the sides. And whenever i switch to the most popular channels i get an overlay that is basically an ad too.
I switched to livestreams and web-based tv for a reason, and that's because tv ads get more intrusive with every year. And don't get me started on the bullshit that smart tv's do nowadays, by that i mean sending data home.
I can't wait for tv to die out or to be replaced by an ip-based protocol just like telephones did.6
I'm currently in the progress of deleting whatsapp and migrating to signal.
The hardest part of it is dealing with friends and family. I informed them about the incoming whatsapp-deletion tomorrow and the results were mixed. One friend told me she will not use signal, but i haven't talked with her that much anyway so... My mother asked me "can you not do this because i don't have space left on my phone?", my father told me "can you tell me about [...] before you go offline?", 2 people don't seem to care, and only my cousin contacted me on signal yet.
I have signal for 3 years now and even invited people to it, but i got the expected response "but all my friends are on whatsapp". Until recently i was the one with the shitload of messengers on my phone but some people can't be bothered to install a second one because i want to take one step (out of many to follow) to widen my privacy.
I'm really pissed by now and will declare any contact lost due to this as collateral damage.29
Had to fix an git-error today when the coworker checket out the repository. The culprit was a "\r" (CR) in a filename. None of us knew how this could have happened, but we think it happened when files were copied from windows to google drive and from there to a mac.
I should've become a gardener, or a butler, or someone else who secretly kills people.3
First rant from my new job.
I got a position as backend-dev in a startup and for now i'm learning angular. Yes, you read that correctly, because the frontend-team is short-staffed i decided to switch teams. We are 3 people and neither one has sufficient angular-experience (the framework was a management decision).
First of all i got confused because we use slack and trello but the frontend-lead decided to do some stuff via google-spreadsheet too. Then we didn't have any code in our repository until yesterday. I tried to check out the repository after that, did an npm-install but when running ng serve i got an error "css-file not found". It turns out you had to download some files from the official website and put them in the unversioned node_modules directory. It was the teamlead's decision to do so and me and my coworker got really annoyed when we tried to set up everything on our end. But that's not all, yesterday the other dev's merged their first versions of the project. But not via git, that is way to mainstream. The coworker had to upload his code into the cloud and the teamlead copied the files into the project folder.
Aside from that the code already isn't the best, some things should be done differently imo and we have credentials in the code (not in some separate files, but in an if-else-clause that checks node.env.production).
We'll have a discussion about this tomorrow, let's hope things can be straightened out.3
This is all i can say about the recent re-surfacing of dark-theme-discussions.
(s/hipster/dark theme/ig for comedic effect)9
Do things in JS they said, it'll be easier they said...
(After a few WTF's i found the problem, arr.map passes more parameters to parseInt than just the strings. It also passes an array index that gets interpreted as radix)18
Movie idea: a plane in mid air catches the wannacrypt virus and refuses manual control. The plane flies straight forward but they only have 2h until they're out of fuel and crash. The only way to pay the ransom is to get enough bitcoins but a recent price-fluctuation made the amount of bitcoins to pay way too high. The only way to resolve this is to create a tumoil on social media causing the bitcoin price to go down.
Visit your local cinema this summer to see 200 passengers and a group of devrant-guest-starrings use nothing but their brains, geniuety and arsenal of devices. Will they find the guy that blocks the wifi by watching 4k porn? Will alice and alexdelarge have to resort to building a fuel-powered mining-rig? Or will linux and linuxxx compile an open-source cockpit program before they run out of time? If so, will they even be able to decide on a linux distro to install on the cockpit?
Coming out in <% new Date().getFullYear() + 1 %>22
Because of reasons we had to use the reactabular-module in our react application. There were problems giving relative values to the table-columns so a coworker decided to install a resize-listener in the window and calculate the relative values itself when the window width changes.
But i have to defend the guy who did this. Given the other things to do and our (limited) knowledge of the css-display-attribute and the reactabuler-module this was the best thing to do. Or we just wamted to finish this because we already wasted hours on this.1
We had to add licensing to a program of us. In the end we chose a small java-library for that and i wrote a convenience script that creates a valid license.
But the script got its input from static strings and that was its doom.
My boss cloned the repo with the script (and jars), replaced the strings with real world data and pushed.
For his conveinience, because there were several clients, he copied the data-section, commented out the first one and put another data into the second section. This happened a few times and HE PUSHED AGAIN.
Now this repository contains a fine record of everyones licenses and their passwords. I know it shouldn't bother me, but it still gets my eye twitching, just like md5-hashing on passwords (which actually happens on that licensed project)2
DO NOT, i repeat, DO NOT USE "scapy.all" in python3.
I spend hours figuring this one out. In one commit i added tests and other tests not connected to my tests started failing. I ran the tests several times and also checked the rng-states, but everything was the same as on the commit before...
There was one additional error message i decided to check out, which was the result of "import scapy.all" (that's a module that contains all the scapy-exports). I removed that import and used the right packages and suddenly: all tests passed.
Fuck this inconsistent piece of crapware that has its python2-files in the pip3-repo and gave me that hell to debug.2
What docker means:
Sorry pal, service names mustn't contain spaces
What docker says:
desc = name must be valid as a DNS component1
No actual data loss here, but the feeling of data loss.
After having my data scattered across several devices i decided to get a grip on it use a cloud. I'm too paranoid for a real cloud so i used a local nextcloud installation. That was done via docker and with a 2TB raid1-array.
I noticed that after restarting the server the cloud was somehow reset and pointed me to the setup-page, afterwards my files were already there. It did strike me as odd but i figured "maybe don't restart the server in the next time".
But i did restart it. And this time i had to setup the cloud again, but my files were gone. I got close to a heart attack, even though all those files weren't that valuable. I ripped one disk from the usb hub, connected it to my laptop and tried to mount it, but raid array. Instead i started photorec and recovered a bunch of files, even though their names were some random hex and i knew i'd spend my next weeks sorting my files. While photorec ran i inspected the docker container and saw that there were only 10GB of space available. After a while and one final df i found the culprit: the raid. For some reason the raid wasn't mounted at boot and docker created the volumes on the servers hard disk, same goes for the container data. After re-adding the disk to the hub i mounted the raid and inspected everything again. All my files were still there.
At no point did i lose my data, but the thought was shocking enough. It'd be best not to fiddle with this server in the next time.
What are good resources to keep up with the latest developments on the field of computer security?1