16
linuxxx
51d

A little late but whatever.

About half a year ago, I started working on setting up self hosted (slippy) maps. For one, because of privacy reasons, for two, because it'd be in my own control and I could, with enough knowledge, be entirely in control of how this would work.

While the process has been going on for hours every day for about half a year (with regular exceptions), I'll briefly lay out what I've accomplished.

I started with the OpenMapTiles project and tried to implement it myself. This went well but there were two major pitfalls:
1. It worked postgres database based. This is fine but when you want to have the entire world.... the queries took insanely long (minutes, at lower zoom levels) and quite intimate postgres/tooling knowledge was required, which I don't have.
2. Due to the long queries and such, the performance was so bad that the maps could take minutes to render and when you'd want that in production... yeah, no.

After quite some time I finally let that idea sail and started looking into the MBTiles solution; generating sqlite databases of geojson features. Very fast data serving but the rendering can take quite some time.

After some more months, I finally got the hang of it to the point that I automated 50-70 percent of the entire process. The one problem? It takes a shitload of resources and time to generate a worldwide mbtiles database.
After infinite numbers of trial and error, I figured out that one can devide a 'render' (mbtiles aka sqlite database) into multiple layers (one for building data, one for water, one for roads and so on), so I started doing renders that way.
Result? Styling became way more easy and logical and one could pick specific data to display; only want to display the roads? Its way more simple this way. (Not impossible otherwise but figuring out how that works... Good luck).

Started rendering all the countries, continents and such this way and while this seemed like a great idea; the entire world is at 3-4 percent after about a month. And while 40-70 percent generates 10 times as fast, that's still way too slow.

Then, I figured out that you can fetch data per individual layer/source. Thus, I could render every layer separately which is way faster.

Tried that with a few very tiny datasets and bam, it works. (And still very fast).

So, now, I'm generating all layers per continent. I want to do it world based but figured out that that's just not manageable with my resources/budget.

Next to that, I'm working on an API which will have exactly the features I want/need!

Comments
  • 10
    For the record, I'm gonna write a blog post about the entire project and how to do everything once I've got some free time (this might be a while).

    Also looking into the possibility to somehow (low-cost) monitize this but not sure how yet.
  • 2
    What was your blog's address again?
  • 2
    @PrivateGER https://much-security.nl

    Don't know yet if I'll post it there, though!
  • 5
    @linuxxx well if you do figure out how to monetize it, sign me the fuck up!
  • 1
    @ArcaneEye For what exactly? 😄
  • 2
    @linuxxx Just be sure to accept crypto or random envelopes stuffed with cash!
  • 0
    @Root Will do! Just not sure how fast I can even monitize this as the quality of the maps isn't the best ever (not bad, just not as good as I hoped) due to specs (money) limitations right now 😅
  • 0
    @linuxxx Will it support directions? Maybe a "what's here?" showing businesses/etc. on the map?

    I'd pay for that.
  • 0
    @Root Its literally just hosted map tiles. Wanna build in directions? Build a directions applications with my map tiles. I think businesses are in there as far as they're integrated in openstreetmaps (their data, that is) and otherwise its not as hard to include that data afterwards anyways!

    Amenities are included already for sure.
  • 1
    @linuxxx what specs are the hardware that's doing the rendering?

    If you'll keep it open source, it may be possible for me to render things on our school servers (we have massive storage limitations, but not compute limitations, so if you can partition the stuff to render into small chunks, I could feed them to the servers here).

    If not that, I have a bunch of old PC lying around here that could chip in, fairly cold here so I'd appreciate the room heating.

    Also if you want a dev to contribute to this during summer, I'll be up for that.

    I'd also suggest that you don't do the entire world in one go. Maybe start with just the Netherlands, get everything working, and then scale up. Deploying a maps system at scale is pretty difficult.
  • 1
    @RememberMe Let me think on that one! The thing is that I often can't predict size outcomes.

    I'm currently rendering all layers of each continent, which is going quite okay but too slow IMO.

    I'll let you know and thanks for the offer!
  • 0
    @RememberMe Size predictability example by the way:

    North America:

    Boundaries. GeoJSON data: about 6gb. Rendered: 134gb.

    Transportation. GeoJSON data: 5.4gb. Rendered: +-5gb.

    (Rendered zoom levels 1 to 15 for the record)
  • 1
    @linuxxx https://much-security.nl/?page=-1 clicked on previous at the first page and it brought me to -1
  • 1
    @JKamue Thank you! Temporary patch for now as I'm working on many different things but it should be working normally at least!
Add Comment