**A thief never says that he has stealed**
~You can relate norton AV's warning and this~

I had spent the last year working on a online store power by woocommerce with over 100k products from various suppliers. This online store utilized a custom API that would take the various formats that suppliers offer their inventory in and made them consistent. Now everything was going swimmingly initially, but then I began adding more and more products using a plug-in called WP all import. I reached around 100k products and the site would take up to an entire minute to load sometimes timing out. I got desperate so I installed several caching plugins, but to no avail this did not help me. The site was originally only supposed to take three to four months but ended up taking an entire year. Then, just yesterday I found out what went wrong and why this woocommerce website with all of these optimizations was still taking anywhere from 60 to 90 seconds to load, or just timing out entirely. I had initially thought that I needed a beefier server so I moved it to a high CPU digitalocean VM. While this did help a little bit, the site was still very slow and now I had very high CPU usage RAM usage and high disk IO. I was seriously stumped the Apache process was using a high amount of CPU and IO along with MYSQL as well. It wasn't until I started digging deeper into the database that I actually found out what the issue was. As I was loading the site I would run 'show process list' in the SQL terminal, I began to notice a very significant load time for one of the tables, so I went to go and check it out. What I did was I ran a select all query on that particular table just to see how full it was and SQL returned a error saying that I had exceeded the maximum packet size. So I was like okay what the fuck...
So I exited my SQL and re-entered it this time with a higher packet size. I ran a query that would count how many rows were in this particular table and the number came out to being in the millions. I was surprised, and what's worse is that this table belong to a plugin that I had attempted to use early in the development process to cache the site. The plugin was deactivated but apparently it had left PHP files within the wp content directory outside of the actual plugin directory, so it's still executing scripts even though the plugin itself was disabled. Basically every time I would change anything on the site, it would recache the whole thing, and it didn't delete any old records. So 100k+ products caching on saves with no garbage collection... You do the math, it's gonna be a heavy ass database. Not only that but it was serialized data, so when it did pull this metric shit ton of spaghetti from the database, PHP then had to deserialize it. Hence the high ass CPU load. I had caching enabled on the MySQL end of things so that ate the ram. I was really desperate to get this thing running.
Honest to God the main reason why this website took so long was because the load times made it miserable to work on. I just thought that the hardware that I had the site on was inadequate. I had initially started the development on a small Linux VM which apparently wasn't enough, which is why I moved it to digitalocean which also seemed to not be enough, so from there I moved to a dedicated server which still didn't seem to be enough. I was probably a few more 60-second wait times or timeouts from recommending a server cluster to my client who I know would not be willing to purchase it. The client who I promised this site to have completed in 3 months and has waited a year. Seriously, I would tell people the struggles that I would go through with this particular site and they would just tell me to just drop the site; just take the money, just take the loss. I refused to, this was really the only thing that was kicking my ass. I present myself as this high-and-mighty developer like I'm just really good at what I do but then I have this WordPress site that's just beating the shit out of me for a year. It was a very big learning experience and it was also very humbling as well, it made me realize that I really don't know as much as I think I might. It was evidence that there is still so much more to learn out there, I did learn a lot from that experience especially about optimizing websites the different types of methods to do that particular lonely on the server side and I'll be able to utilize this knowledge in the future.
I guess the moral of the story is, never really give up. Ultimately things might get so bad that you're running on hopes and dreams. Those experiences are generally the most humbling. Now I can finally present the site that I am basically a year late on to the client who will be so happy that I did not give up on the project entirely. I'll have experienced this feeling of pure euphoria, and help the small business significantly grow their revenue. Helping others is very fulfilling for me, even at my own expense.

Anyways, gonna stop ranting. Running out of characters. If you're still here... Ty for reading :')

I just realized one of today's emails is asking to review again that spaghetti program and this time figure out how to optimize performance because it is getting flagged for high cpu and database usage. A Niagra of If-statements and nobody-cares-for-comments-and-technical-documentation.

*bleep* *bleep* *bleep* previous programmer.

I'll deal with this torture on Monday.

FUUCCKKKK!! I need to hit smth. Or rant..

So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.

What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.

Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.

I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!

So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians

and I get a reply: 'we will apply cpu usage limitations to fix the issue'

wait... Cpu% limitations will do nothing but highlight the underlying problem...

'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'

oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:

'yes, cpu utilization is the problem. We will limit it and solve the problem'

I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....

Fuck that's frustrating....

*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>

Is your code green?

I've been thinking a lot about this for the past year. There was recently an article on this on slashdot.

I like optimising things to a reasonable degree and avoid bloat. What are some signs of code that isn't green?

* Use of technology that says its fast without real expert review and measurement. Lots of tech out their claims to be fast but actually isn't or is doing so by saturation resources while being inefficient.
* It uses caching. Many might find that counter intuitive. In technology it is surprisingly common to see people scale or cache rather than directly fixing the thing that's watt expensive which is compounded when the cache has weak coverage.
* It uses scaling. Originally scaling was a last resort. The reason is simple, it introduces excessive complexity. Today it's common to see people scale things rather than make them efficient. You end up needing ten instances when a bit of skill could bring you down to one which could scale as well but likely wont need to.
* It uses a non-trivial framework. Frameworks are rarely fast. Most will fall in the range of ten to a thousand times slower in terms of CPU usage. Memory bloat may also force the need for more instances. Frameworks written on already slow high level languages may be especially bad.
* Lacks optimisations for obvious bottlenecks.
* It runs slowly.
* It lacks even basic resource usage measurement.

Unfortunately smells are not enough on their own but are a start. Real measurement and expert review is always the only way to get an idea of if your code is reasonably green.

I find it not uncommon to see things require tens to hundreds to thousands of resources than needed if not more.

In terms of cycles that can be the difference between needing a single core and a thousand cores.

This is common in the industry but it's not because people didn't write everything in assembly. It's usually leaning toward the extreme opposite.

Optimisations are often easy and don't require writing code in binary. In fact the resulting code is often simpler. Excess complexity and inefficient code tend to go hand in hand. Sometimes a code cleaning service is all you need to enhance your green.

I once rewrote a data parsing library that had to parse a hundred MB and was a performance hotspot into C from an interpreted language. I measured it and the results were good. It had been optimised as much as possible in the interpreted version but way still 50 times faster minimum in C.

I recently stumbled upon someone's attempt to do the same and I was able to optimise the interpreted version in five minutes to be twice as fast as the C++ version.

I see opportunity to optimise everywhere in software. A billion KG CO2 could be saved easy if a few green code shops popped up. It's also often a net win. Faster software, lower costs, lower management burden... I'm thinking of starting a consultancy.

The problem is after witnessing the likes of Greta Thunberg then if that's what the next generation has in store then as far as I'm concerned the world can fucking burn and her generation along with it.

So just sitting here watching my windows VM's cpu usage to monitor was seems to be causing some stuttering and then out of fucking nowhere the default photo's app pops and and claims 96% of the CPU...

I understand computers are weird but how the hell does an app randomly start and jump to such a high CPU usage?

Fuck Antimalware Service Executable.
All my homies hate Antimalware Service Executable.

I hope I can hop into Linux before this happens to my personal computer.

!rant #tip

Windows 10 - service host high cpu usage

Stop the superfetch service and it would be down to normal.

You should checkout the description of the superfetch service... Lol ):D

I hate Intellij Idea but it's best option available to develop in Scala. Improvements in VSCode/Metals is my last hope.

The (few) things I NEED from Intellij:
* Very good autocompletion
* Refactoring tools (renaming, auto imports)
* Search tools (find usages, sub/super-types)

The (many) things I hate of Intellij:

* Layout with panel sizes doesn't behave properly and it scales instead of remaining fixed.

* Tedious 2-hands shortcuts makes the right hand to move a lot from the mouse

* Delays and lag in the UI, freezes on garbage collection

* High memory consumption, high CPU usage and generally slow and cumbersome

* The delay in the UI between commands is so that it's accidentally possible to introduce typos

* Can't move tabs around and organize them as I like

* Ugly font rendering and missing typography settings

* Multi-caret implementation as a different editing mode is annoying because requires frequent switching

* Unnatural code folding regions, why method arguments are not folded with the method?

* Unhelpful support forum, sometimes dismissive answers

* Highlighting of current word under the caret doesn't work

* Very slow editor, can't keep spacebar pressed to move text or it hangs!

* Several settings reset at every update. Like the auto fetch of git

* New features are added and enabled by default which is very invasive

* Some of the features mentioned above are really annoying and it's not possible/not trivial to disable them

* It uses its own compile and several times it highlights false positives

Somebody please explain to Microsoft Win 10 team that normal usage of a computer should just work, home users don't have hours on end to spend on dealing with:

- BS sound drivers
- high CPU usage & diagnosing, log tracing from system processes
- many other crap you need to invest time constantly fixing or you don't have a useful machine

Windows 10 is a piece of shit