Load tests:

I'm used to do load tests in Visual Studio where it gives which line is exactly your bottleneck. But now I'm using VS Code (visual studio requires enterprise license for load tests :\ no longer have one)

Anyways long story short, what are the best practices for load tests? For me what I'm testing is how much can a given hardware specs handle and when test fails I go back and check if code can be optimized, is this the correct way to do this?

We use HP LoadRunner at work, sometimes combined w/ yourkit profiler. Monitoring is yet another area of toolsets.

Too bad all these are commercial and far from cheap. Assuming you need a non-commercial solution, I could suggest going w/ jmeter. Build your tests w/ UI and launch them via CLI (don't run "good" load tests via UI).

Never use a single machine to produce load. Use as many as you can. to distribute load generation across.

Always set up a strictly controlled environment. Always compare apples-to-apples, do not assume that your changes should not have had any effect on the part you are testing. Do not do shortcuts. Try to have as clean runs as possible, i.e. reboot/flush/etc everything before each run.

Do not use profilers for bare performance testing (i.e. for comparision). Only use profiles when you are planning to analyze what took that long, since profilers and other attached agents have effect on test results.

Oh, and you might want to use warm-up runs. To load all .txt from disks, prepare loadbalancers, prefill caches, etc.

Do use warmups, porribly use warmdowns

@netikras Thank you so much for your detailed info, though one thing I would like to ask you about, you said:
Never use a single machine to produce load. Use as many as you can. to distribute load generation across.

By this you mean I have multiple machines one for initiating tests and one for having service running?

Would you recommend I spread tests across multiple virtual machines on the same physical computer? Or this will give me wrong results?

@gitpush if you have multiple instances of service app, try to deploy them in a way so that one overloaded instance would not affect other instances. That means avoid deploying your apps on the same machine.

As for your first qry - I meant load generators. Os/hw has some limitations and can only submit/process that many simultaneous requests. Distributing load generation across multiple systems would minimize effects of those limitations on test results. That being said you'd have more realistic results

@netikras I understand you now, well then no wonder why my results are confusing, I'm running in an Ubuntu Desktop VM with 8GB RAM and the service is on the same VM where tests are run *runs away before you shoot me*

I think I'll rent a small VPS but I'm afraid I get detected as DDoS attack XD

@gitpush how about a free-tier AWS? won't t2.micro be enough for your app? And how about GCloud? It gives you 300$ worth of services for a year.

AWS for one doesn't mind HI loads. Just make sure you are not hitting network limitations - that'd also affect your results

@netikras Thanks man really appreciate your help, I'll consider your solutions and lets hope I get the grasp of it