I remember when I first heard about the general concept of a stack, I wasn't quite sure how you would use it.

Now im learning about stacks in java algorithms class as well as learning about how the memory stack works in assembly class. This probably seems small to you pro devs, but ngl, it kinda blew my mind how useful this structure can be.

I feel you man. There is this point in the journey of learning something when it just clicks! You grab the intuition behind the existence of that algorithm/ technique, etc.
And that's when all the time you have devoted pays back!
The best part? It becomes an integral part of your thinking. You KNOW that stuff now, and can use it as a puzzle piece to solve problems.