11
party101
224d

Parallel downloading module for python

Project Type
Existing open source project
Summary

Parallel downloading module for python

Description
I have been designing a python package that allows you to download file parallely via multithreading (something i have been missing a lot). I believe I have written enough to make it usable and needed help to make it more resilient and flexible.
Tech Stack
Python
Current Team Size
1
URL
Comments
  • 0
    What does it mean for it to be parallel?
  • 3
  • 0
    @aprzn123 Generally requests is used for requesting stuff from the internet. This package stores the data in memory itself. This can cause issue if the file size is large. For this it provides a streaming option to download the file in chunks. But this only uses one connection therefore the download speeds tend to be slow. This package develops upon this functionality and uses multiple threads to break the download into multiple parts and each thread downloads one part and then these are merged together to form the requested file. Since the threads are downloading the file "parallely" it tends to faster.
  • 0
    Implemented exactly this already, but with a more specific approach on the problem I had to solve. Would love to contribute as far as my time allows me to do so!
  • 0
    I'd love to contribute, I'll check it out after work.
  • 0
    Would this actually work, from what I understand multithreading in python itsnt actually multithreading because of global interpreter lock
  • 0
    @Axis It offers betters than what exists right now🙄. Moreover the GIL that you talk about is implementation specific for cython.
  • 1
    @party101 fair enough can't say I know much about it but I have heard about it thanks for the info
  • 0
    @Axis The GIL is only an issue for CPU bound tasks. Downloading files is an IO-bound task.
  • 0
    @daegontaven well actually it is causing an issue for python 2 while updating the status information but not for python3 due to the GIL enhancements or maybe i wrote bad code. Either way i was hoping if someone could take a look at it and help.
Your Job Suck?
Get a Better Job
Add Comment