Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

So, you start with a PHP website.

Nah, no hating on PHP here, this is not about language design or performance or strict type systems...

This is about architecture.

No backend web framework, just "plain PHP".

Well, I can deal with that. As long as there is some consistency, I wouldn't even mind maintaining a PHP4 site with Y2K-era HTML4 and zero Javascript.

That sounds like fucking paradise to me right now. 😍

But no, of course it was updated to PHP7, using Laravel, and a main.js file was created. GREAT.... right? Yes. Sure. Totally cool. Gotta stay with the times. But there's still remnants of that ancient framework-less website underneath. So we enter an era of Laravel + Blade templates, with a little sprinkle of raw imported PHP files here and there.

Fine. Ancient PHP + Laravel + Blade + main.js + bootstrap.css. Whatever. I can still handle this. 🤨

But then the Frontend hipsters swoosh back their shawls, sip from their caramel lattes, and start whining: "We want React! We want SPA! No more BootstrapCSS, we're going to launch our own suite of SASS styles! IT'S BETTER".

OK, so we create REST endpoints, and the little monkeys who spend their time animating spinners to cover up all the XHR fuckups are satisfied. But they only care about the top most visited pages, so we ALSO need to keep our Blade templated HTML. We now have about 200 SPA/REST routes, and about 350 classic PHP/Blade pages.

So we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA 😑

Now the Backend grizzlies wake from their hibernation, growling: We have nearly 25 million lines of PHP! Monoliths are evil! Did you know Netflix uses microservices? If we break everything into tiny chunks of code, all our problems will be solved! Let's use DDD! Let's use messaging pipelines! Let's use caching! Let's use big data! Let's use search indexes!... Good right? Sure. Whatever.

OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA + Redis + RabbitMQ + Cassandra + Elastic 😫

Our monolith starts pooping out little microservices. Some polished pieces turn into pretty little gems... but the obese monolith keeps swelling as well, while simultaneously pooping out more and more little ugly turds at an ever faster rate.

Management rushes in: "Forget about frontend and microservices! We need a desktop app! We need mobile apps! I read in a magazine that the era of the web is over!"

OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + GraphQL + React + SPA + Redis + RabbitMQ + Google pub/sub + Neo4J + Cassandra + Elastic + UWP + Android + iOS 😠

"Do you have a monolith or microservices" -- "Yes"

"Which database do you use" -- "Yes"

"Which API standard do you follow" -- "Yes"

"Do you use a CI/building service?" -- "Yes, 3"

"Which Laravel version do you use?" -- "Nine" -- "What, Laravel 9, that isn't even out yet?" -- "No, nine different versions, depends on the services"

"Besides PHP, do you use any Python, Ruby, NodeJS, C#, Golang, or Java?" -- "Not OR, AND. So that's a yes. And bash. Oh and Perl. Oh... and a bit of LUA I think?"

2% of pages are still served by raw, framework-less PHP.

I’m a senior dev at a small company that does some consulting. This past October, some really heavy personal situation came up and my job suffered for it. I raised the flag and was very open with my boss about it and both him and my team of 3 understood and were pretty cool with me taking on a smaller load of work while I moved on with some stuff in my life. For a week.

    Right after that, I got sent to a client. “One month only, we just want some presence there since it’s such a big client” alright, I guess I can do that. “You’ll be in charge of a team of a few people and help them technically.” Sounds good, I like leading!

    So I get here. Let’s talk technical first: from being in a small but interesting project using Xamarin, I’m now looking at Visual Basic code, using Visual Studio 2010. Windows fucking Forms.

    The project was made by a single dev for this huge company. She did what she could but as the requirements grew this thing became a behemoth of spaghetti code and User Controls. The other two guys working on the project have been here for a few months and they have very basic experience at the job anyways. The woman that worked on the project for 5 years is now leaving because she can’t take it anymore.

    And that’s not the worse of it. It took from October to December for me to get a machine. I literally spent two months reading on my cellphone and just going over my shitty personal situation for 8 hours a day. I complained to everyone I could and nothing really worked.

    Then I got a PC! But wait… no domain user. Queue an extra month in which I could see the Windows 7 (yep) log in screen and nothing else. Then, finally! A domain user! I can log in! Just wait 2 extra weeks for us to give your user access to the subversion rep and you’re good to go!

    While all of this went on, I didn’t get an access card until a week ago. Every day I had to walk to the reception desk, show my ID and request they call my boss so he could grant me access. 5 months of this, both at the start of the day and after lunch. There was one day in particular, between two holidays, in which no one that could grant me access was at the office. I literally stood there until 11am in which I called my company and told them I was going home.

    Now I’ve been actually working for a while, mostly fixing stuff that works like crap and trying to implement functions that should have been finished but aren’t even started. Did I mention this App is in production and being used by the people here? Because it is. Imagine if you will the amount of problems that an application that’s connecting to the production DB can create when it doesn’t even validate if the field should receive numeric values only. Did I mention the DB itself is also a complete mess? Because it is. There’s an “INDEXES” tables in which, I shit you not, the IDs of every other table is stored. There are no Identity fields anywhere, and instead every insert has to go to this INDEXES table, check the last ID of the table we’re working on, then create a new registry in order to give you your new ID. It’s insane.

   

    And, to boot, the new order from above is: We want to split this app in two. You guys will stick with the maintenance of half of it, some other dudes with the other. Still both targeting the same DB and using the same starting point, but each only working on the module that we want them to work in. PostmodernJerk, it’s your job now to prepare the app so that this can work. How? We dunno. Why? Fuck if we care. Kill you? You don’t deserve the swift release of death.

    Also I’m starting to get a bit tired of comments that go  ‘THIS DOESN’T WORK and ‘I DON’T KNOW WHY WE DO THIS BUT IT HELPS and my personal favorite  ‘??????????????????????

So my manager (a 29 y/o, who hardly can use a mac) walks towards me with a hint of panic in his eyes.

Manager: Hey commander keen, do you know how to use vertical look up in excel, I've tried, and looked at tutorials.

Me: yeah I really don't know excel (and not willing to learn, especially on the fly), I don't even have excel installed, I can write I script that does what you want.

Manager: No you have enough on your plate

3 hours later

Manager: hey I still can't figure it out, could you solve it with a script, won't that take to long

Me: no send me the files, Ill do it with a script.

I start writing 2 for loops and wait for the file, 10 ish minutes later its basicly done, just need to put in the column indexes.

I send a message on both slack channels (hey are you going to email or slack me the file)

After a hour I walk to his desk and again ask him for the file.

Manager a good 2 hours later on slack: Hey I just send you the file, I hope its not to much work, it has to happen asap.

So if you have kids, and they are not that bright by some kind of birth defect, don't worry, they can always become a manager.

But you can't get me down today. I hit 2000 upvotes and the employer is unknowingly a proud sponsor for reading and writing all these rants and comments :-) thnx devrant

So I'm back from vacation! It's my first day back, and I'm feeling refreshed and chipper, and motivated to get a bunch of things done quickly so I can slack off a bit later. It's a great plan.

First up: I need to finish up tiny thing from my previous ticket -- I had overlooked it in the description before. (I couldn't test this feature [push notifications] locally so I left it to QA to test while I was gone.)

It amounted to changing how we pull a due date out of the DB; some merchants use X, a couple use Y. Instead of hardcoding them, it would use a setting that admins can update on the fly.

Several methods deep, the current due date gets pulled indirectly from another class, so it's non-trivial to update; I start working through it.

But wait, if we're displaying a due date that differs from the date we're actually using internally, that's legit bad. So I investigate if I need to update the internals, too.

After awhile, I start to make lunch. I ask my boss if it's display-only (best case) and... no response. More investigating.

I start to make a late lunch. A wild sickness appears! Rush to bathroom; lose two turns.

I come back and get distracted by more investigating. I start to make an early dinner... and end up making dinner for my monster instead.

Boss responds, tells me it's just for display (yay!) and that we should use <macro resource feature> instead.

I talk to Mr. Product about which macros I should add; he doesn't respond.

I go back to making lunch-turn-dinner for myself; monster comes back and he's still hungry (as he never asks for more), so I make him dinner.

I check Slack again; Mr. Product still hasn't responded. I go back to making dinner.

Most of the way through cooking, I get a notification! Product says he's talking it through with my boss, who will update me on it. Okay fine. I finish making dinner and go eat.

No response from boss; I start looking through my next ticket.

No response from boss. I ping him and ask for an update, and he says "What are you talking about?" Apparently product never talked to bossmang =/ I ask him about the resources, and he says there's no need to create any more as the one I need already exists! Yay!

So my feature went from a large, complex refactor all the way down to a -1+2 diff. That's freaking amazing, and it only took the entire day!

I run the related specs, which take forever, then commit and push.

Push rejected; pull first! Fair, I have been gone for two weeks. I pull, and git complains about my .gitignore and some local changes. fine, whatever. Except I forgot I had my .gitignore ignored (skipped worktree). Finally figure that out, clean up my tree, and merge.

Time to run the specs again! Gems are out of date. Okay, I go run `bundle install` and ... Ruby is no longer installed? Turns out one of the changes was an upgrade to Ruby 2.5.8.

Alright, I run `rvm use ruby-2.5.8` and.... rvm: command not found. What. I inspect the errors from before and... ah! Someone's brain fell out and they installed rbenv instead of the expected rvm on my mac. Fine, time to figure it out. `rbenv which ruby`; error. `rbenv install --list`; skyscraper-long list that contains bloody everything EXCEPT 2.5.8! Literally 2.5 through 2.5.7 and then 2.6.0-dev. asjdfklasdjf

Then I remember before I left people on Slack made a big deal about upgrading Ruby, so I go looking. Dummy me forgot about the search feature for a painful ten minutes. :( Search found the upgrade instructions right away, ofc. I follow them, and... each step takes freaking forever. Meanwhile my children are having a yelling duet in the immediate background, punctuated with screams and banging toys on furniture.

Eventually (seriously like twenty-five minutes later) I make it through the list. I cd into my project directory and... I get an error message and I'm not in the project directory? what. Oh, it's a zsh thing. k, I work around that, and try to run my specs. Fail.

I need to update my gems; k. `bundle install` and... twenty minutes later... all done.

I go to run my specs and... RubyMine reports I'm using 2.5.4 instead of 2.5.8? That can't be right. `ruby --version` reports 2.5.8; `rbenv version` reports 2.5.8? Fuck it, I've fought with this long enough. Restarting fixes everything, right? So I restart. when my mac comes back to life, I try again; same issue. After fighting for another ten minutes, I find a version toggle in RubyMine's settings, and update it to 2.5.8. It indexes for five minutes. ugh.

Also! After the restart, this company-installed surveillance "security" runs and lags my computer to hell. Highest spec MacBook Pro and it takes 2-5 seconds just to switch between desktops!

I run specs again. Hey look! Missing dependency: no execjs. I can't run the specs.

Fuck. This. I'll just push and let the CI run specs for me.

I just don't care anymore. It's now 8pm and I've spent the past 11 hours on a -1+2 diff!

What a great first day back! Everything is just the way I left it.

I've optimised so many things in my time I can't remember most of them.

Most recently, something had to be the equivalent off `"literal" LIKE column` with a million rows to compare. It would take around a second average each literal to lookup for a service that needs to be high load and low latency. This isn't an easy case to optimise, many people would consider it impossible.

It took my a couple of hours to reverse engineer the data and implement a few hundred line implementation that would look it up in 1ms average with the worst possible case being very rare and not too distant from this.

In another case there was a lookup of arbitrary time spans that most people would not bother to cache because the input parameters are too short lived and variable to make a difference. I replaced the 50000+ line application acting as a middle man between the application and database with 500 lines of code that did the look up faster and was able to implement a reasonable caching strategy. This dropped resource consumption by a minimum of factor of ten at least. Misses were cheaper and it was able to cache most cases. It also involved modifying the client library in C to stop it unnecessarily wrapping primitives in objects to the high level language which was causing it to consume excessive amounts of memory when processing huge data streams.

Another system would download a huge data set for every point of sale constantly, then parse and apply it. It had to reflect changes quickly but would download the whole dataset each time containing hundreds of thousands of rows. I whipped up a system so that a single server (barring redundancy) would download it in a loop, parse it using C which was much faster than the traditional interpreted language, then use a custom data differential format, TCP data streaming protocol, binary serialisation and LZMA compression to pipe it down to points of sale. This protocol also used versioning for catchup and differential combination for additional reduction in size. It went from being 30 seconds to a few minutes behind to using able to keep up to with in a second of changes. It was also using so much bandwidth that it would reach the limit on ADSL connections then get throttled. I looked at the traffic stats after and it dropped from dozens of terabytes a month to around a gigabyte or so a month for several hundred machines. The drop in the graphs you'd think all the machines had been turned off as that's what it looked like. It could now happily run over GPRS or 56K.

I was working on a project with a lot of data and noticed these huge tables and horrible queries. The tables were all the results of queries. Someone wrote terrible SQL then to optimise it ran it in the background with all possible variable values then store the results of joins and aggregates into new tables. On top of those tables they wrote more SQL. I wrote some new queries and query generation that wiped out thousands of lines of code immediately and operated on the original tables taking things down from 30GB and rapidly climbing to a couple GB.

Another time a piece of mathematics had to generate all possible permutations and the existing solution was factorial. I worked out how to optimise it to run n*n which believe it or not made the world of difference. Went from hardly handling anything to handling anything thrown at it. It was nice trying to get people to "freeze the system now".

I build my own frontend systems (admittedly rushed) that do what angular/react/vue aim for but with higher (maximum) performance including an in memory data base to back the UI that had layered event driven indexes and could handle referential integrity (overlay on the database only revealing items with valid integrity) or reordering and reposition events very rapidly using a custom AVL tree. You could layer indexes over it (data inheritance) that could be partial and dynamic.

So many times have I optimised things on automatic just cleaning up code normally. Hundreds, thousands of optimisations. It's what makes my clock tick.

Biggest challenge I overcame as dev? One of many.

Avoiding a life sentence when the 'powers that be' targeted one of my libraries for the root cause of system performance issues and I didn't correct that accusation with a flame thrower.

What the accusation? What I named the library. Yep. The *name* was causing every single problem in the system.

Panorama (very, very expensive APM system at the time) identified my library in it's analysis, the calls to/from SQLServer was the bottleneck

We had one of Panorama's engineers on-site and he asked what (not the actual name) MyLibrary was and (I'll preface I did not know or involved in any of the so-called 'research') a crack team of developers+managers researched the system thoroughly and found MyLibrary was used in just about every project. I wrote the .Net 1.1 MyLibrary as a mini-ORM to simplify the execution of database code (stored procs, etc) and gracefully handle+log database exceptions (auto-logged details such as the target db, stored procedure name, parameter values, etc, everything you'd need to troubleshoot database errors). This was before Dapper and the other fancy tools used by kids these days.

By the time the news got to me, there was a team cobbled together who's only focus was to remove any/every trace of MyLibrary from the code base. Using Waterfall, they calculated it would take at least a year to remove+replace MyLibrary with the equivalent ADO.Net plumbing.

In a department wide meeting:

DeptMgr: "This day forward, no one is to use MyLibrary to access the database! It's slow, unprofessionally named, and the root cause of all the database issues."
Me: "What about MyLibrary is slow? It's excecuting standard the ADO.Net code. Only extra bit of code is the exception handling to capture the details when the exception is logged."
DeptMgr: "We've spent the last 6 weeks with the Panorama engineer and he's identified MyLibrary as the cause. Company has spent over $100,000 on this software and we have to make fact based decisions. Look at this slide ... "
<DeptMgr shows a histogram of the stacktrace, showing MyLibrary as the slowest>
Me: "You do realize that the execution time is the database call itself, not the code. In that example, the invoice call, it's the stored procedure that taking 5 seconds, not MyLibrary."
<at this point, DeptMgr is getting red-face mad>
AreaMgr: "Yes...yes...but if we stopped using MyLibrary, removing the unnecessary layers, will make the code run faster."
<typical headknodd-ers knod their heads in agreement>
Dev01: "The loading of MyLibrary takes CPU cycles away from code that supports our customers. Every CPU cycle counts."
<headknod-ding continues>
Me: "I'm really confused. Maybe I'm looking at the data wrong. On the slide where you highlighted all the bottlenecks, the histogram shows the latency is the database, I mean...it's right there, in red. Am I looking at it wrong?"
<this was meeting with 20+ other devs, mgrs, a VP, the Panorama engineer>
DeptMgr: "Yes you are! I know MyLibrary is your baby. You need to check your ego at the door and face the facts. Your MyLibrary is a failed experiment and needs to be exterminated from this system!"

Fast forward 9 months, maybe 50% of the projects updated, come across the documentation left from the Panorama. Even after the removal of MyLibrary, there was zero increases in performance. The engineer recommended DBAs start optimizing their indexes and other N+1 problems discovered. I decide to ask the developer who lead the re-write.

Me: "I see that removing MyLibrary did nothing to improve performance."
Dev: "Yes, DeptMgr was pissed. He was ready to throw the Panorama engineer out a window when he said the problems were in the database all along. Didn't you say that?"
Me: "Um, so is this re-write project dead?"
Dev: "No. Removing MyLibrary introduced all kinds of bugs. All the boilerplate ADO.Net code caused a lot of unhandled exceptions, then we had to go back and write exception handling code."
Me: "What a failure. What dipshit would think writing more code leads to less bugs?"
Dev: "I know, I know. We're so far behind schedule. We had to come up with something. I ended up writing a library to make replacing MyLibrary easier. I called it KnightRider. Like the TV show. Everyone is excited to speed up their code with KnightRider. Same method names, same exception handling. All we have to do is replace MyLibrary with KnightRider and we're done."
Me: "Won't the bottlenecks then point to KnightRider?"
Dev: "Meh, not my problem. Panorama meets primarily with the DBAs and the networking team now. I doubt we ever use Panorama to look at our C# code."

Needless to say, I was (still) pissed that they had used MyLibrary as dirty word and a scapegoat for months when they *knew* where the problems were. Pissed enough for a flamethrower? Maybe.

Student: what are indexes on mysql tables?

Teacher: I don't know, nor do I recommend it's usage.

PHP arrays.

The built-in array is also an hashmap. Actually, it's always a hashmap, but you can append to it without specifying indexes and PHP will use consecutive integers. Its performance characteristics? Who knows. Oh, and only strings, ints and null are valid keys.

What's the iteration order for arrays if you use them as hashmaps (string keys)? Well, they have their internal order. So it's actually an ordered hashmap that's being called an array. And you can produce an array which has only integer keys starting with 0, but with non-sequential internal (iteration) order.

This array weirdness has some non-trivial implications. `json_encode` (serializes argument to JSON) assumes an array corresponds to a JSON array if its keys are consecutive integers in increasing order starting with 0, otherwise the array becomes a JSON object. `array_filter` (filters arrays/hashmaps using callback predicate) preserves keys, so it will punch holes in the int key sequence if non-last items are removed, thus turning arrays into hashmaps and changing your JSON structure if you forget to discard keys before serialization.

You may wonder how JSON deserialization works, then? There's a special class for deserialized JSON objects, `stdClass`. It's basically a hashmap too, but it's an object, not an array, and all functions that would normally accept arrays won't work with it. So basically its only use is JSON (de)serialization. You can even cast arrays to objects, producing `stdClass`.

Bonus PHP trivia:

Many functions return nonsensical values. `preg_match`, the regex matching function, returns 1 for success, 0 for no matches and false for malformed regular expression. PHP supports exceptions, so it could just throw one on errors. It would even make more sense to return true, false and null for these three cases. But no, 1, 0 and false. And actual matches are returned by output arg.

`array_walk_recursive`, a function supposed to recursively apply callback to each element of an array. That's what docs say. It actually applies it to leafs only. It will also silently accept object instead of array and "walk" it, but without recursing into deeper objects.

Runtime type enforcing is supported for function arguments and returned values. You can use scalar types, classes, array, null and a few special keywords. There's also a `mixed` keyword, which is used in docs and means "anything". It's syntactically valid, the parser will accept it, but it matches no values in runtime. Calling such function will always cause a runtime error.

Strings can be indexed with negative integers. Arrays can't.

ReflectionClass::newInstanceWithoutConstructor: "Creates a new class instance without invoking the constructor". This one needs no commentary.

`array_map` is pretty self-explanatory if you call it with a callback and an array. Or if you provide more arrays of equal length via varargs, callback will be called with more arguments, one from each array. Makes sense so far. Now, you can also call `array_map` with null instead of callback. In that case it treats provided arrays as rows of a matrix and returns that matrix, transposed.

Rant!

Been working on 'MVP' features of a new product for the past 14 months. Customer has no f**king clue on how to design for performance. An uncomfortable amount of faith was placed on the ORM (ORMs are not bad as long as you know what you are doing) and the magic that the current framework provides. (Again, magic is good so long as you understand what happens behind the smoke and mirrors - but f**k all that... coz hey, productivity, right?). Customer was so focussed on features that no one ever thought of giving any attention to subtler things like 'hey, my transaction is doing a gazillion joins across trizillion tables while making a million calls to the db - maybe I should put more f**king thought into my design.' We foresaw performance and concurrency issues and raised them way ahead of the release. How did the customer respond? By hiring a performance tester. Fair enough - but what did that translate into? Nothing. Nada. Zilch. Hiring a perf tester doesn't automagically fix issues. The perf tester did not have a stable environment, a stable build or anything that is required to do a test with meaningful results. As the release date approached, the customer launched a pilot and things started failing spectacularly with the system not able to support more than 15 concurrent users. WTF! (My 'I told you so' moment) Emails started flying in all directions and the hunt for the scapegoat was on (I'm a sucker for CYA so I was covered). People started pointing in all directions but no one bothered to take a step back and understand what was causing the issues. Numero uno reason for transaction failure was deadlocks. We were using a proprietary DB with kickass tooling. No one bothered to use the tooling to understand what was the resource in contention let alone how to fix the contention. Absolute panic - its like they just froze. Debugging shit and doing the same thing again and again just so that management knew they were upto something. Most of the indexes had a fragmentation of 99.8% - I shit you not. Anywho, we now have a 'war room' where the perf tester needs to script the entire project by tonight and come up with some numbers that will amount to nothing while we stay up and keep profiling the shit out of the application under load.

Lessons learnt - When you foresee a problem make a LOT of noise to get people to act upon it and not wait till it comes back and bites you in the ass. Better yet, try not to get into a team where people can't understand the implications of shitty design choices. War room my ass!

>Be client
>Have an issue with incredibly slow webpage load time
>Blame memcache issues

So... I look into the problem. Yes, the page either loads up fast, or times out. So, into the logs I go. Webserver is fine (except the timeout), PHP though... Error log is fine (just notices), but slow log shows the issue is the database (of course... its always the database... ugh)

So, checking the database, there is one ugly query that seems to be an issue. 5 joins and a huge where condition.

So I run EXPLAIN on the query and... Proceed to bang my head against the wall.

OF COURSE ITS SLOW YOU FU******, NONE OF YOUR TABLES HAVE ANY INDEXES.

What do they expect when the database has to always go down the whole table and do everything in memory, until it runs out and has to dump it all on disk and work with it there.

Ugh... Some clients...

I don't understand how my managers suddenly forgot that my "down weeks" we're due to technical debt I inherited. The whole on boarding hasn't been in my favor. I've stayed at work everyday til long after work hours, digging through code, trying to get JIRA tickets done, encountering issues specific to our code base that no one would ever discover on their own without docs/help from the original dev. The whole time, I was told that they know what's going on and apologize. I constantly expressed that plenty of what we were doing was building on antipatterns. They acknowledged. When a ticket wasn't done, they always knew the very specific reason and I wasn't faulted. 6 months in, I receive a great annual review. 7 months in? I receive an email titled "Performance Discussion," detailing 4 of those incidents where a ticket was pushed back -- with inaccurate depictions of what actually went down. They actually wrote that I didn't communicate. One part of the report expressed that there were "bugs found in production due to inadequate test coverage." WTF!! Everything made it past code review and QA. What are you talking about?? In fact, the person who wrote that merged my code in each time!!!! Insane!! Anyway, Q2 is partly about cleaning up technical debt, which is a responsibility I have been vested (fantastic). I've deleted about 800 lines of code in the last 2 weeks and added plenty of doc strings. Two of the most important modules our application works from are about 1000 lines of JavaScript each without any comments/docs. I'm changing that, but I don't know if my managers truly know the significance. Someone was recently promoted to my position but manually wrote out a sorting algorithm (specified numeric indexes and all); didn't do shit to earn it but breathe. And while they get more and more praise and responsibility, I'm over here stuck trying to prove myself and live up to why I assume they hired me. It's ridiculous. I love the company, but I'm not getting any sleep and I'm stressed out. It's only been about 7 months and I've been doing everything I can. Why is this happening? What am I doing wrong? I've been developing a recurring (physical) headache and ticks. My heart/chest area sometimes feels like it's lifting weights. I sound like an idiot, pushing so hard for a company that isn't mine, but I take so much pride in being in this position, and I'm so set on proving myself this early in my career (I'm 25).

I’m fucking lost.

So, situation. I have a SQL table with about 3M rows (not a lot).

I have indexes. Indexes are used. BUT when I add where clause (On indexed column), it’s super slow. Around 10 seconds.

If I do select * (ALL 3M rows) and THEN I filter then on webserver side, it takes 0.5 seconds.

HOW my manual filtering is faster than DB filtering with indexes? I even tried bubble sort. Bubble sort is faster than SQL ‘where’. HOW ?!

I do not understand….

And if I add group by….. WELL, 25 seconds SQL time. 2 Seconds if I do select all and group by in code manually.

Does not make ANY sense to me.
What am I missing ?

The saddest and funniest side of our industry is (atleast in India): someone works hard and makes it to the best colleges, do great projects on AI, ML; get a good score on Leetcode, codechef; gets a job in FAANG-like companies...

Changes colors in CSS and texts in HTML.

And, why is there so much emphasis on Data Structures and Algorithms? I mean, a little bit is fine, but why get obsessed with it when you never write algorithms in production code?

Now, don't tell me that, we use libraries and we should know what we are doing, no, we don't use algorithms even in libraries.

Now, before you tell me that MySQL uses B-tree for maintaining indexes, you really don't need to solve tricky questions to be able to understand how a B-tree works.

It's just absurd.

I know how to little bit on how design scalable systems.
I know how to write good code that is both modular and extensible.
I know how to mentor interns and turn them into employees.
I know how to mentor junior engineers (freshers) and help them get started.
Heck I can even invert a binary tree.

But some FAANG company would reject me because I cannot solve a very tricky dynamic programming question.

v0.0005a (alpha)
- class support added to lua thanks to yonaba.
- rkUIs class created
- new panel class
- added drawing code for panel
- fixed bug where some sides of the UI's border were failing to drawing (line rendering quark)

v0.0014a (alpha) 11.30.2023 (~2 hours)
- successfully retrieving basic data from save folder, load text into lua from files
- added 'props' property to Entity class
- added a props table to control what gets serialized and what doesn't
- added a save() base method for instances (has to be overridden to be useful beyond the basics)
- moved the lume.serialize() call into the :save() method on the base entity class itself
- serialized and successfully saved an entities property table.
- fixed deserializion bugs involving wrong indexes (savedata[1] not savedata[2])
- moved deserialization from temp code, into line loading loop itself (assuming each item is on one line)
- deser'd test data, and init()'d new player Entity using the freshly-loaded data, and displayed the entity sprite

All in all not a bad session. Understanding filing handling and how to interact with the directory system was the biggest hurdle I was worried about for building my tools.

Next steps will be defining some basic UI elements (with overridable draw code), and then loading and initializing the UI from lua or json.

New projects can be set as subfolders folders in appdata, using 'Setidentity("appname/projectname") to keep things clean.

I'm not even dreading writing basic syntax highlighting!

Idea is to dogfood the whole process. UI is in-engine rendered just like you might see with godot, unity, or gamemaker, that way I have maximum flexibility to style it the way I want. I'm familiar enough with constructing from polygons, on top of stenciling, on top of nine-slicing, on top of existing tweening and special effects, that I can achieve exactly what I want.

Idea is to build a really well managed asset pipeline. Stencyl, as 'crappy' as it appeared, and 'for education' was a master class in how to do things the correct way, it was just horribly bloated while doing it.

Logical tilesets that you import, can rearrange through drag-n-drop, assign custom tile shapes to, physics materials, collisions groups, name, add tag data to, all in one editor? Yes please.
Every other 2D editor is basic-bitch, has you importing images, and at most generates different scales and does the slicing for you.

Code editor? Everything behavior was in a component, with custom fields. All your code goes into a list of events, which you can toggle on and off with a proper toggle button, so you can explicitly experiment, instead of commenting shit out (yes git is better, but we're talking solo amateurs here, they're not gonna be using git out the gate unless they already know what they're doing).

Components all have an image assignable to identify them, along with a description field, and they're arranged in a 2d grid for easy browsing, copying, modifying.

The physics shape editor, the animation editor, the map editor, all of it was so bare bones and yet had things others didn't.

I want that, except without the historic ties to flash, without the overhead of java, and with sexier fucking in-engine rendering of the UI and support for modding and in-engine custom tools.

Not really doing it for anyone except myself, and doubt I'll get very far, but since I dropped looking for easy solutions, I've just been powering through all the areas I don't understand and doing the work.

I rediscovered my love of programming after 3-4 years of learning to hate it, and things are looking up.

Math question time!

Okay so I had this idea and I'm looking for anyone who has a better grasp of math than me.

What if instead of searching for prime factors we searched for a number above p?
One with a certain special property. BEAR WITH ME. I know I make these posts a lot and I'm a bit of a shitposter, but I'm being genuine here.

Take this cherry picked number, 697 for example.

It's factors are 17, and 41. It's trivial but just for demonstration.

If we represented it's factors as a bit string, where each bit represents the index that factor occurs at in a list of primes, it looks like this

1000001000000

When converted back to an integer that number becomes 4160, which we will call f.

And if we do 4160/(2**n) until the result returns
a fractional component, then N in this case will be 7.

And 7 is the index of our lowest factor 17 (lets call it A, and our highest factor we'll call B) in our primes list.

So the problem is changed from finding a factorization of p, to finding an algorithm that allows you to convert p into f. Once you have f it's a matter of converting it to binary, looking up the indexes of all bits set to 1, and finding the values of those indexes in the list of primes.

I'm working on doing that and if anyone has any insights I'm all ears.

Working on an application - everywhere an enum should be is a database table instead.

Me: What happens when someone changes "rejected" to "approved" in your status table?

Me: What happens when you re-seed your database and the indexes for your types are different?

All problems, with no time or scope to fix!
USE A FUCKING ENUM

It's a shame that people don't want to use F# but prise C# for how cool it became and continue becoming. At the same time, little do they know that many of the features were simply drawn from F#.

It's just rediculous how far this OO and C-Style syntax crap has progressed. They keep copying things from functional langugages, making the initial language to be a monstrocity like C++ is now, insted of just using languages like C#. I mean, it was right there before C#: async/task, immutablility, records, indexes, lambdas, non-null by default, who the hell knows what else.
Besides, many people (in my company at least) are just blindly overengineering with patterns and shit, where a simple function would be just enogh.

Watch some some NDC talks about F#, in particular those of Scott Wlaschin. It's just better in so many ways: less noice (I'm looking at you, brackets, commas and semicolons), the whole LOT of type inference and less duplication (just look at the C# signatures of linq methods - it's difficult to read them), immutability by default, non-nullable by default, ADTs and pattern matching, some neat features like type providers (how many times have used "paste special" or an online tool to create C# classes from a JSON/XML file, and how many times have your regenrated it because of schema changes?) and units of measure.

Of course, in some cases it's not optimal, in some cases mutable datastructures of C# are better for performance. But dude, how many performance critical systems have you wrote in C#? I mean, if it comes to performance you should use Rust or C++ or C after all.

*sighs*

Manjaro has some quirks that annoy me(no MST timezone, spotty support for my WD NVME), so I decided that since I'm not interested in any pre-configured graphical desktop of any kind, I should just dive into Arch, since it increasingly felt like that's what I was doing anyway but with Manjaro to dull the blow. So I did, and I am over the moon for doing so. Lots of gnashed teeth, but DDG indexes an answer to every question I've had, and it always makes sense when I find it. I've enjoyed having to dive into systemd in a much more low-level way than ever before-- to actually LEARN what it's doing, how, and why.

But one by one, I have been faced with some issue that I need to resolve, and one by one, I've knocked them off. The result now is the best work and gaming desktop I have ever used.

Arch is not for geniuses or wizards. Just patient people who are willing to read. The payoff is staggering, and many times over worth the effort.

Chinese remainder theorem

So the idea is that a partial or zero knowledge proof is used for not just encryption but also for a sort of distributed ledger or proof-of-membership, in addition to being used to add new members where additional layers of distributive proofs are at it, so that rollbacks can be performed on a network to remove members or revoke content.

Data is NOT automatically distributed throughout a network, rather sharing is the equivalent of replicating and syncing data to your instance.
Therefore if you don't like something on a network or think it's a liability (hate speech for the left, violent content for the right for example), the degree to which it is not shared is the degree to which it is censored.

By automatically not showing images posted by people you're subscribed to or following, infiltrators or state level actors who post things like calls to terrorism or csam to open platforms in order to justify shutting down platforms they don't control, are cut off at the knees. Their may also be a case for tools built on AI that automatically determine if something like a thumbnail should be censored or give the user an NSFW warning before clicking a link that may appear innocuous but is actually malicious.

Server nodes may be virtual in that they are merely a graph of people connected in a group by each person in the group having a piece of a shared key.
Because Chinese remainder theorem only requires a subset of all the info in the original key it also Acts as a voting mechanism to decide whether a piece of content is allowed to be synced to an entire group or remain permanently.

Data that hasn't been verified yet may go into a case for a given cluster of users who are mutually subscribed or following in a small world graph, but at the same time it doesn't get shared out of that subgraph in may expire if enough users don't hit a like button or a retain button or a share or "verify" button.

The algorithm here then is no algorithm at all but merely the natural association process between people and their likes and dislikes directly affecting the outcome of what they see via that process of association to begin with.

We can even go so far as to dog food content that's already been synced to a graph into evolutions of the existing key such that the retention of new generations of key, dependent on the previous key, also act as a store of the data that's been synced to the members of the node.

Therefore remember that continually post content that doesn't get verified slowly falls out of the node such that eventually their content becomes merely temporary in the cases or index of the node members, driving index and node subgraph membership in an organic and natural process based purely on affiliation and identification.

Here I've sort of butchered the idea of the Chinese remainder theorem in shoehorned it into the idea of zero knowledge proofs but you can see where I'm going with this if you squint at the idea mentally and look at it at just the right angle.

The big idea was to remove the influence of centralized algorithms to begin with, and implement mechanisms such that third-party organizations that exist to discredit or shut down small platforms are hindered by the design of the platform itself.

I think if you look over the ideas here you'll see that's what the general design thrust achieves or could achieve if implemented into a platform.

The addition of indexes in a node or "server" or "room" (being a set of users mutually subscribed to a particular tag or topic or each other), where the index is an index of text audio videos and other media including user posts that are available on the given node, in the index being titled but blind links (no pictures/media, or media verified as safe through an automatic tool) would also be useful.

PHP dev help/advice needed!
We have problems with mysql. Still stuck with mariaDB, I'm using indexes (correct ones) and we have problems with scaling. we have a few tables with over 100mil rows, 1 of them is being read every morning with a subselect that counts unique rows, and fails every time because of timeout/lock, the temp table size was increased and helped for a little while but as time goes on the table grows and the problem reappears. I'm reading from a slave server that was purposely created for read only, yet we still have problems. We're using managed dedicated servers for out hosting and they aren't willing to optimise the database configs for our needs. What are the easiest options for scaling at this point? Going fully dedicated server and perconaDB? NOsql? Sharding the server? Anyone got any good blogposts or something to read about this? your own experience?

I've been freelancing lately with an agency to develop an android app for their client and at the same time another person is developing the website .

The story begins when I first contacted the web dev to give me access to the database (because he started before me ).It turns out that this guy purchased an almost ready cms template with a shitty data structure that has no relations between object .This database has no primary keys , no foreign keys , no indexes ... no nothing . Adding to that the web dev refused that I rewrite a new data structure claiming that he has done a good progress on the website .
Forward couple of weeks , I managed to create the api and develop an alpha for the app and sent it to the agency manager .
This bastard told me that the website and design have changed and the app shouldn't be like that .He told me to contact the other bastard the web dev to seen what the changes are . I'm waiting for the response about the new updates and I'm praying that they'll be just minor colors updates or something not a whole concept update .

My problem here is I'm stuck with this fucking agency cuz they paid half of the payment when I started .

Damn I must learn to say no to people .

JPA my friend ... JPA why are you like this? JPA why do hate me so much? JPA, let's have a word ...

How come you are so far away from real-world problems, so cumbersome to use, so ugly (criteria API), so wrong and inconsistent?
Oh, what it's all your parents fault? Oh come, on that can't be, right? Did you have a bad childhood?

Your parent's were fucking crack-smoking maniacs which didn't know a single bit about actual databases?
They design you as an API without actually trying you out in the wild? And then they patched up together with some essential DB stuff, like friggin indexes? Not even tried to make this API consistent nor really functional?

Oh poor, you little JPA ...