39

So i wrote a Tool that Indexes the ENTIRE darkweb (yes it’s possible)...made it look and feel like a mix of Shodan and Google... when I show and tell people they simply shrug like this was easy😪
I feel like a tree that fell in the woods when no one was around

Comments
  • 5
    Oooh, nice. I haven't been around darknet for years. I remember I liked to learn IT stuff there around the time Hidden Hosting went down.

    Suggestion: try finding keywords in those sites and posting them alongside your results. Such analysis would help to not accidentally click one wrong link...
  • 4
    @haze it does that, I actually save the entire HTML for future data fun
  • 2
    I wonder what kind of strange things you can find with that
  • 5
    Repo?
  • 0
    📌
  • 0
    Adding a tactical dot, JIC
  • 1
    @inhamul
    Profiting from your tactical dot.
  • 4
    How'd you do that? For example, I frequently use Tor based chatting software which launches a hidden service to connect to other hidden services to chat as per your decision but I'm not sure how on earth you'd index that...
  • 0
  • 0
  • 0
  • 0
  • 0
  • 0
  • 0
  • 0
  • 0
  • 1
    I doubt your success at indexing too.
    Tactical dot deployed.
    .
  • 0
    📍
  • 0
  • 0
    🍦
  • 2
    The entire darkweb? Sorry but I'm skeptical at best, also pinning to see.

    📌
  • 0
    dot
  • 0
  • 0
    Ooh, sounds neat
  • 2
    @Noobish your skepticism is plausible, the indexer has not stopped yet, by this many possible addresses it can take years.
    HOWEVER that being said I get about 23 new onion urls a minute (which isn’t bad) I wrote this thing in nodejs before and I got like 1 every minute (this one is written in go) using colly for scraping
  • 9
    It works by employing the six degrees of separation technique. (You know ? That fun Kevin Bacon Wikipedia Game😀)
    We use Golang colly to scrape pages and collect all links. We then queue them and scrape.
    It works really well since people in darkweb forums tend to post links and or advertise their sites as well as other indexers. So once you hit a Site like that you get a few hundred new paths and so on
    Surprisingly you’d think that you can’t find new onion services, but turns out we got a bunch of plain new nginx sites that shouldn’t be referenced anywhere.
    To be quite frank I am surprised it works as well as it does.
    I didn’t want to open source the code because I don’t want people abusing this and putting too much pressure on the tor network.
    A buddy and I are working on a front end so everyone can use it soon
    Here is a sneak peek:
  • 0
  • 4
    That's just a normal web scraper though.
    Following redirects is nothing special.
    You won't get "special" URLs like that.
  • 1
    @PrivateGER it’s a spider, working efficiently enough to scrape many millions of sites in a relative short time.
    And yea I am pretty damn surprised on how great it grabs special urls myself I couldn’t believe it

    What’s even crazier is although it’s a „simple“ scraper, no one has officially done this on the darkweb like this.
    If you think about it mobile apps are just like, small websites on tiny phones, yet there is so much more to it. The point is , simple things can be impressive.
  • 4
    @angrysnekguy but it's not the ***ENTIRE*** dark web. That was just clickbait
  • 4
    @Diabetus well in a rant you tend to exaggerate sometimes.
    That being said, it’s still running, still grabbing new sites by the minute and one day it will be ❤️

    Wasn’t trying to clickbait people, just ranting and letting off steam you know ?
  • 5
    @angrysnekguy no matter what you're doing, just keep it up and all the best, hope you achieve what you want! Just some random words of encouragement to hopefully make your day
  • 2
    🔪
  • 0
    Do you have a public repo for this? Teach me the ways sensei!
  • 0
  • 0
    Don't listen to the haters my man. Just because the abstract solution is technically speaking simple doesn't mean it isn't really cool! Now implement a pagerank algorithm for it too!
  • 0
  • 0
    Awsome, there used to be one called harry-somenumber, like harry79 or something, and also Fresh Onions, but they are gone, and I have been looking for a replacement! Thx
Add Comment