Not sure if many people heard about nltk in python but I'm currently using a lot now for research.

So one day I was doing multiprocessing while using lemmatizer in nltk, for those who don't know, lemmatizer is a thing that change the word to its base form. So it is like, ran to run, bitches to bitch.

Anyway, the nltk package, to ensure it does not take too much memory, here's what it does: it loads a data file, and once it is loaded and accessed for the first time, it breaks the data file into CSV file. And since I was doing multiprocessing, the data file is accessed for multiple time while it can only be loaded once, hence error happened.

Instead of changing my code, which I think is good already, I went to the package directory of nltk and directly changed the source code from there and now the code works perfectly.

I'm very proud of my self at the moment, this is a very good lesson that I've learned: always look for alternatives. And suck it, nltk.