Spent 2 days installing different versions of nvidia-driver, nvidia-cuda-toolkit, cudnn.
Disassembled pc due to some ram memory problems and pc not starting after freeze.
Looks like sometimes plug out and plug in various components on your motherboard fix pc lol. Destroyed PC casing that collapsed due to me sitting on it.

All of this to just find out tensorflow error and all crashes are from graphics card overheating after keeping it for some time at 82 celsius.

Added time.sleep to python code and looks like it's working, keeping temperature below 65 celsius.

Still ~100 times faster than cpu training so I can live with that.