Found a use for multiprocessing in my final project work, from 15 seconds to 5 seconds on 128 images. Not bad. (4 processes used).

Wish I bothered to do it sooner as now I have to do tens of thousands and was sick of waiting.

  • 0
    Processes? Why not pthreads?
  • 0
    @netikras I haven't touched pthreads in python before, is it a better solution for running simultaneous processes that don't care about outside variables?
  • 1
    @SoulSkrix it is usually a better choice to use threads over subprocesses in general. Only use subprocesses where you are sure you need subprocesses and you know why threads dont work for your case. In threads:
    - you are very lightweight on memory: you are reusing parent processes' heap and only spawn your own stack;
    - you reuse libs, configs, etc loaded in memory
    - you can spawn *much* faster
    - impossible to lose control of threads; subprocesses can be detached from parent, they can block parebt from exitting, they can remain running after parent's death, etc
    - *much* easier to synchronize actions, distribute jobs, reuse children inf. amount of times; this is much harder w/ subprocesses

    imo threads should always be considered first and only in cases they do not fit your bill subprocesses could be used
  • 1
    @netikras Thanks for all the info, I haven't really ever had to write performant code outside of using appropriate algorithms and data structures (just at University). I wrote some threaded code in C for a robotics project but that's about it; I hope when I'm not trying to finish my exam/course work period to educate myself more on threading/multiprocessing.

    Cheers again
  • 1
    @SoulSkrix Might not want to use threads in python for performance, as it doesn’t increase the performance very much.
  • 0
    @-vim- yeah that makes sense, read up more on it and it's not really good for CPU intensive tasks. More to make use of idle time.
Your Job Suck?
Get a Better Job
Add Comment