Ranter
Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Comments
-
@aprzn123 Generally requests is used for requesting stuff from the internet. This package stores the data in memory itself. This can cause issue if the file size is large. For this it provides a streaming option to download the file in chunks. But this only uses one connection therefore the download speeds tend to be slow. This package develops upon this functionality and uses multiple threads to break the download into multiple parts and each thread downloads one part and then these are merged together to form the requested file. Since the threads are downloading the file "parallely" it tends to faster.
-
onfi457yImplemented exactly this already, but with a more specific approach on the problem I had to solve. Would love to contribute as far as my time allows me to do so!
-
Axis8447yWould this actually work, from what I understand multithreading in python itsnt actually multithreading because of global interpreter lock
-
@Axis It offers betters than what exists right now🙄. Moreover the GIL that you talk about is implementation specific for cython.
-
@daegontaven well actually it is causing an issue for python 2 while updating the status information but not for python3 due to the GIL enhancements or maybe i wrote bad code. Either way i was hoping if someone could take a look at it and help.
Parallel downloading module for python
Parallel downloading module for python