AboutLinux/FOSS, cyber sec, privacy and programming guy. Hardstyle/rawstyle freak.
Joined devRant on 5/14/2016
Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
Nearly, nearly ready to deploy self-hosted maps, fucking awesome!
Aaaand now comes the goddamn styling since these will be vector tiles..
I fucking hate designing/styling and I fucking suck at it too. 10 attempts which all pretty much look like shit.
Frontender, social media manager and windows (server) admin!
Frontender; being paid to do something I couldn't care less about and find very frustrating (as for developing it)
Social media manager; being paid to use mass surveillance engines 😷
Windows (server) admin; I don't think I have to explain this one...7
A little late but whatever.
About half a year ago, I started working on setting up self hosted (slippy) maps. For one, because of privacy reasons, for two, because it'd be in my own control and I could, with enough knowledge, be entirely in control of how this would work.
While the process has been going on for hours every day for about half a year (with regular exceptions), I'll briefly lay out what I've accomplished.
I started with the OpenMapTiles project and tried to implement it myself. This went well but there were two major pitfalls:
1. It worked postgres database based. This is fine but when you want to have the entire world.... the queries took insanely long (minutes, at lower zoom levels) and quite intimate postgres/tooling knowledge was required, which I don't have.
2. Due to the long queries and such, the performance was so bad that the maps could take minutes to render and when you'd want that in production... yeah, no.
After quite some time I finally let that idea sail and started looking into the MBTiles solution; generating sqlite databases of geojson features. Very fast data serving but the rendering can take quite some time.
After some more months, I finally got the hang of it to the point that I automated 50-70 percent of the entire process. The one problem? It takes a shitload of resources and time to generate a worldwide mbtiles database.
After infinite numbers of trial and error, I figured out that one can devide a 'render' (mbtiles aka sqlite database) into multiple layers (one for building data, one for water, one for roads and so on), so I started doing renders that way.
Result? Styling became way more easy and logical and one could pick specific data to display; only want to display the roads? Its way more simple this way. (Not impossible otherwise but figuring out how that works... Good luck).
Started rendering all the countries, continents and such this way and while this seemed like a great idea; the entire world is at 3-4 percent after about a month. And while 40-70 percent generates 10 times as fast, that's still way too slow.
Then, I figured out that you can fetch data per individual layer/source. Thus, I could render every layer separately which is way faster.
Tried that with a few very tiny datasets and bam, it works. (And still very fast).
So, now, I'm generating all layers per continent. I want to do it world based but figured out that that's just not manageable with my resources/budget.
Next to that, I'm working on an API which will have exactly the features I want/need!14
6 hours into a 124gb upload and then the internet goes out due to a malfunctioning modem.
Just. Fucking. Great.9
Rain monitoring/alert api's/services/websites/apps.
Why the fuck are those fuckers (in my case) nearly always right when I couldn't care less and very fucking wrong when I couldn't care more?
Today, morning before leaving:
Nah mate you're good for the next x minutes (more than enough time for me to bike to the station and get on the train).
Reality: suddenly getting two gallons of water a second on me in the form of rain, arrived completely soaked and headphones broke.
Before biking home from the station;
*its raining hard as hell*
Rain alert/monitoring app/radar thingy: nah mate no worries, no rain last half hour and you're good for another half hour!
Reality: arrived home soaked a-fucking-gain.
So motherfucking annoying and frustrating. And yes, I do have an umbrella and such but I hate carrying that stuff around when 'not needed'.5
"Let's quickly make sure that the docker container can access a host database"
I guess I can put that quickly up my ass.
How motherfucking difficult can this be?!18
"Let's create a docker-swarm cluster thingy for this application with horizontal scaling to learn docker and run this application better!"
This stuff is so overwhelming and I don't understand half of how to possibly set this up 😅32
Colleague got a few statical electricity (or however you say that) shocks.
After yet another one:
Colleague: seriously, again?
Me: damn, that's quite shocking!
C: *blank stare* bro.... 😐
Never did programming professionally so none yet and I don't think that's going to change anytime soon 😄1
Spent multiple weeks preparing data for rendering. Part of this process is reprojecting coordinates of huge ass GeoJSON files (some files are 100gb+).
Compared one normal and one reprojected file with the head command and noticed no difference, then found out that this data already comes in the right projection....
Full-speed extraction it is!3
Wanted to automate some stuff through bash scripts but did want to separate tasks so I wrote two bash scripts for data processing.
Hmm.. two scripts..... how could I even further automate thi....
*writes script which executed those two scripts*
The company behind ads on dutch national news sites/tv stations is stopping with tracking ads.
The interesting reason behind that IMO is that they have this very simple 'banner' which asks in a very understandable way whether people want (or not) tracking shit/cookies placed/loaded on their browser.
Apparently 90% of all visitors go for the "No" option so they don't see the point in continuing to try it anymore.
Build a global, fast maps/tile hosting "platform".
Already working on this like mad and with a little bit of luck, I'll have a first version ready within a week.
Also learning Rust, becoming even better at/with Linux....10
Ordered a new high storage dedicated server (rented online) today.
The fucker wasn't delivered as soon as I hoped so decided to finally go to bed at 2am.
Aaaaaand then, of fucking-course, I got the email with the deployment notification literally the second before I went to sleep.
No sleep yet anyways, I guess 😅11
Finally fucking managed to setup quite fast map tile hosting including the tile generation after ages of research and trial and error.
I love this open (source) maps (openstreetmaps) project but man, figuring out what to do from a gazillion sources can be rather hard.
Now I'm just having some styling issues and the filesize is fucking insane (only the Netherlands with all data, 20gb+ if I remember correctly) so I'm just generating road maps for now. If someone knows some more about the styling as for the maps, please let me know!
Yeah, this is fucking satisfying.2
That moment that you realize that you forgot one critically important data tag in a very heavy multi-day render.
University of Maastricht right now:
(Klop (clop as pun in this context) is knock in Dutch)5
Biggest terminal hack I've discovered till now which is so fucking obvious I can't believe I literally just started using it:
Executing the date command right before and after another command/series of commands to see how long shit takes.
$ date; command or commands; date
This is incredibly useful when rendering loads of data in screen sessions!
I actually feel rather retarded for only thinking of this now.17
Many people want a cashless society. This is especially useful when:
- you just lost your bank card.
- the banks mobile payment app isn't working at all.
- its Saturday evening.
I'm so fucking glad that I've got some cash left at home and that cash is still a valid payment method 😅34
Is anyone here familiar with hosting maps through vector tiles and displaying them on webpages?
I'm fucking stuck on this and the forums aren't really replying/answering anything.
Any tips/help would be appreciated!11
Not entirely dev related but the year that the mass surveillance companies go away (Google, Facebook (and WhatsApp, Instagram etc), Microsoft and a few others), I'll call that the best fucking year ever.
I hope I'll get to experience that in my lifetime!6
Finally found a quite reliable way of generating mbtiles 'archives' for tile hosting!
Now I yet have to find a reliable way to fucking actually HOST it and a way to get it fucking displayed.
Spend my entire goddamn day on this and I'm hardly a step closer, for fucks sake.2
HOW COCKSUCKING DIFFICULT CAN IT BE TO CHANGE THE FUCKING MEMORY LIMIT FOR NPM PACKAGES?!
HOLY MOTHERFUCKING SHIT.6
Holy fuck, muscle memory just saved my ass.
At a train station wanting to do some work on my mini laptop which has disk encryption (LUKS I think). Realised that I forgot the password partly 😬
Few tries.... incorrect. FUCK.
*hey, let's try to let my hands do the work based on muscle memory!*
*starts typing the password (its insanely long) and presses enter*
*succesfully unlocked, booting...*
Facebook publicly announced that it won't build a backdoor into its services for the intelligence agencies as for the latest requests to weaken/remove the encryption.
I can only imagine the intelligence agencies going like this now:
NSA director: Alright, as expected they said no so they won't have more damage to their public image, lets go for plan A 2.0!
NSA employee: Aaaand that is?
NSA director: Serve them a FISA court order requiring them to do this shit anyways but also serve a gag order so they can't tell legally.
NSA employee: Ahh, fair enough, I'll get that rolling. By the way, how did we do this with WhatsApp's encryption again?
NSA director: Oh that one was simple. There's a backup function which nearly everyone uses on either Android/iOS which does plaintext backups to Google Drive/iCloud.
NSA employee: Oh, okay. How do we access that data again?
NSA director: PRISM/XKeyScore!
NSA employee: Right, but then still the issue of how we even collect the encrypted messages from Facebo...
NSA director: PRISM/XKeyScore as well, don't worry about that.
NSA employee: But, how'd we justify this....?
NSA director: We probably never have to as these programs operate outside of the public view but otherwise just call terrorism/pedophelia... BAM, done.
NSA employee: Gotya, let's put this into motion!29
A lot of times during holiday parties, I meet new people.
Afterwards, often a quick group chat is created to share pictures with all the people there.
Every goddamn time we come at this point again where this is done and since WhatsApp is nearly considered a requirement to function in society (Netherlands) and I refuse to use it and other mainstream media...
The moment that I have to explain why I don't use WhatsApp arrives, yet again. I don't find it awkward anymore but it doesn't improve the atmosphere and getting called paranoid for the rest of the party also isn't the most fun thing ever!
In the end I usually get the pictures emailed.19
The Dutch minister of justice and (national) security is joining the countries that are in favor of weakening encryption algorithms so law enforcement "can combat child pornography and terrorism better".
Oh yeah no fucking great, I was hoping that the Dutch ministers weren't as incompetent on this one as the ones from the US and Australia who want this as well (and probably even more countries).
Let's fuck over an entire country with false arguments while it won't help for shit!
If you ever completely want to waste your goddamn time, call Keolis (company behind Arriva I think) with a question regarding their app. They'll be able to provide you with no fucking usable help/information at all and will gladly guide you through the login process twenty-motherfucking-billion times again and have no single solution as to why its not working.
As a bonus, they'll ask for your details so they can investigate and call you back a little later to take you through the exact goddamn process you just did two-hundred-trillion times before with the exact same completely fucking useless result.11
I wrote a random string/int/other stuff API somewhere this year which I still regularly use because I'm a lazy fuck.
Never posted anything about it on here and the documentation isn't entirely complete (and not all the endpoints are extensively bug-tested yet) but if someone is interested I'll see if I can patch some stuff and put it on here as I find it useful!3