Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "what are indexes"
Oh, man, I just realized I haven't ranted one of my best stories on here!
So, here goes!
A few years back the company I work for was contacted by an older client regarding a new project.
The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.
Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.
Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.
Long story short, a lot bigger than what we were expecting based on the initial brief.
The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.
They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.
Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.
So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.
So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.
We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.
As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.
So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...
Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.
Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!
[Continues in the comments]27
So, you start with a PHP website.
Nah, no hating on PHP here, this is not about language design or performance or strict type systems...
This is about architecture.
No backend web framework, just "plain PHP".
That sounds like fucking paradise to me right now. 😍
But no, of course it was updated to PHP7, using Laravel, and a main.js file was created. GREAT.... right? Yes. Sure. Totally cool. Gotta stay with the times. But there's still remnants of that ancient framework-less website underneath. So we enter an era of Laravel + Blade templates, with a little sprinkle of raw imported PHP files here and there.
Fine. Ancient PHP + Laravel + Blade + main.js + bootstrap.css. Whatever. I can still handle this. 🤨
But then the Frontend hipsters swoosh back their shawls, sip from their caramel lattes, and start whining: "We want React! We want SPA! No more BootstrapCSS, we're going to launch our own suite of SASS styles! IT'S BETTER".
OK, so we create REST endpoints, and the little monkeys who spend their time animating spinners to cover up all the XHR fuckups are satisfied. But they only care about the top most visited pages, so we ALSO need to keep our Blade templated HTML. We now have about 200 SPA/REST routes, and about 350 classic PHP/Blade pages.
So we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA 😑
Now the Backend grizzlies wake from their hibernation, growling: We have nearly 25 million lines of PHP! Monoliths are evil! Did you know Netflix uses microservices? If we break everything into tiny chunks of code, all our problems will be solved! Let's use DDD! Let's use messaging pipelines! Let's use caching! Let's use big data! Let's use search indexes!... Good right? Sure. Whatever.
OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + React + SPA + Redis + RabbitMQ + Cassandra + Elastic 😫
Our monolith starts pooping out little microservices. Some polished pieces turn into pretty little gems... but the obese monolith keeps swelling as well, while simultaneously pooping out more and more little ugly turds at an ever faster rate.
Management rushes in: "Forget about frontend and microservices! We need a desktop app! We need mobile apps! I read in a magazine that the era of the web is over!"
OK, so we enter the Era of Ancient PHP + Laravel + Blade + main.js + bootstrap.css + hipster.sass + REST + GraphQL + React + SPA + Redis + RabbitMQ + Google pub/sub + Neo4J + Cassandra + Elastic + UWP + Android + iOS 😠
"Do you have a monolith or microservices" -- "Yes"
"Which database do you use" -- "Yes"
"Which API standard do you follow" -- "Yes"
"Do you use a CI/building service?" -- "Yes, 3"
"Which Laravel version do you use?" -- "Nine" -- "What, Laravel 9, that isn't even out yet?" -- "No, nine different versions, depends on the services"
"Besides PHP, do you use any Python, Ruby, NodeJS, C#, Golang, or Java?" -- "Not OR, AND. So that's a yes. And bash. Oh and Perl. Oh... and a bit of LUA I think?"
2% of pages are still served by raw, framework-less PHP.35
I’m a senior dev at a small company that does some consulting. This past October, some really heavy personal situation came up and my job suffered for it. I raised the flag and was very open with my boss about it and both him and my team of 3 understood and were pretty cool with me taking on a smaller load of work while I moved on with some stuff in my life. For a week.
Right after that, I got sent to a client. “One month only, we just want some presence there since it’s such a big client” alright, I guess I can do that. “You’ll be in charge of a team of a few people and help them technically.” Sounds good, I like leading!
So I get here. Let’s talk technical first: from being in a small but interesting project using Xamarin, I’m now looking at Visual Basic code, using Visual Studio 2010. Windows fucking Forms.
The project was made by a single dev for this huge company. She did what she could but as the requirements grew this thing became a behemoth of spaghetti code and User Controls. The other two guys working on the project have been here for a few months and they have very basic experience at the job anyways. The woman that worked on the project for 5 years is now leaving because she can’t take it anymore.
And that’s not the worse of it. It took from October to December for me to get a machine. I literally spent two months reading on my cellphone and just going over my shitty personal situation for 8 hours a day. I complained to everyone I could and nothing really worked.
Then I got a PC! But wait… no domain user. Queue an extra month in which I could see the Windows 7 (yep) log in screen and nothing else. Then, finally! A domain user! I can log in! Just wait 2 extra weeks for us to give your user access to the subversion rep and you’re good to go!
While all of this went on, I didn’t get an access card until a week ago. Every day I had to walk to the reception desk, show my ID and request they call my boss so he could grant me access. 5 months of this, both at the start of the day and after lunch. There was one day in particular, between two holidays, in which no one that could grant me access was at the office. I literally stood there until 11am in which I called my company and told them I was going home.
Now I’ve been actually working for a while, mostly fixing stuff that works like crap and trying to implement functions that should have been finished but aren’t even started. Did I mention this App is in production and being used by the people here? Because it is. Imagine if you will the amount of problems that an application that’s connecting to the production DB can create when it doesn’t even validate if the field should receive numeric values only. Did I mention the DB itself is also a complete mess? Because it is. There’s an “INDEXES” tables in which, I shit you not, the IDs of every other table is stored. There are no Identity fields anywhere, and instead every insert has to go to this INDEXES table, check the last ID of the table we’re working on, then create a new registry in order to give you your new ID. It’s insane.
And, to boot, the new order from above is: We want to split this app in two. You guys will stick with the maintenance of half of it, some other dudes with the other. Still both targeting the same DB and using the same starting point, but each only working on the module that we want them to work in. PostmodernJerk, it’s your job now to prepare the app so that this can work. How? We dunno. Why? Fuck if we care. Kill you? You don’t deserve the swift release of death.
Also I’m starting to get a bit tired of comments that go ‘THIS DOESN’T WORK and ‘I DON’T KNOW WHY WE DO THIS BUT IT HELPS and my personal favorite ‘??????????????????????15
So my manager (a 29 y/o, who hardly can use a mac) walks towards me with a hint of panic in his eyes.
Manager: Hey commander keen, do you know how to use vertical look up in excel, I've tried, and looked at tutorials.
Me: yeah I really don't know excel (and not willing to learn, especially on the fly), I don't even have excel installed, I can write I script that does what you want.
Manager: No you have enough on your plate
3 hours later
Manager: hey I still can't figure it out, could you solve it with a script, won't that take to long
Me: no send me the files, Ill do it with a script.
I start writing 2 for loops and wait for the file, 10 ish minutes later its basicly done, just need to put in the column indexes.
I send a message on both slack channels (hey are you going to email or slack me the file)
After a hour I walk to his desk and again ask him for the file.
Manager a good 2 hours later on slack: Hey I just send you the file, I hope its not to much work, it has to happen asap.
So if you have kids, and they are not that bright by some kind of birth defect, don't worry, they can always become a manager.
But you can't get me down today. I hit 2000 upvotes and the employer is unknowingly a proud sponsor for reading and writing all these rants and comments :-) thnx devrant8
One fine Friday night in early '97 while drinking with my buddies I got a page from work. Called the office to understand what the problem is.
*shit I can't fix this over the phone, and buddy here doesn't have a PC so I can't dial-in via PCAnywhere*
Told told the users "Ok I'll be there in an hour and a half. Stop all the running jobs and start the backup"
*figures I still have 1hr to spare so continues to down fair amounts of O-be-joyful with buddies then hailed a cab to office*
I arrived in office 1.5hrs later (2am) exactly as I predicted and went straight to work. Initial checks confirmed my suspicion of the issue so I wrote the appropriate SQL to get started:
'drop table foobar'
***The specified table (foobar) is not in the database***
I looked at foobar and figured out immediately why I got the error, then corrected the SQL and ran again:
'drop database foobar'
*What the FUCK!!! You fucking drunk!!! What did you fucking do? What if I disappear to another country, work as a waiter or something*
After a few moments of panic and a good deal of 'What ifs' I calmed down, looked to the users and made up some bullshit "Some of the indexes are corrupted, we need to restore from the backup"
I wrote most of my '94 midterm project during weekends where me and my buddies were drunk
So I'm back from vacation! It's my first day back, and I'm feeling refreshed and chipper, and motivated to get a bunch of things done quickly so I can slack off a bit later. It's a great plan.
First up: I need to finish up tiny thing from my previous ticket -- I had overlooked it in the description before. (I couldn't test this feature [push notifications] locally so I left it to QA to test while I was gone.)
It amounted to changing how we pull a due date out of the DB; some merchants use X, a couple use Y. Instead of hardcoding them, it would use a setting that admins can update on the fly.
Several methods deep, the current due date gets pulled indirectly from another class, so it's non-trivial to update; I start working through it.
But wait, if we're displaying a due date that differs from the date we're actually using internally, that's legit bad. So I investigate if I need to update the internals, too.
After awhile, I start to make lunch. I ask my boss if it's display-only (best case) and... no response. More investigating.
I start to make a late lunch. A wild sickness appears! Rush to bathroom; lose two turns.
I come back and get distracted by more investigating. I start to make an early dinner... and end up making dinner for my monster instead.
Boss responds, tells me it's just for display (yay!) and that we should use <macro resource feature> instead.
I talk to Mr. Product about which macros I should add; he doesn't respond.
I go back to making lunch-turn-dinner for myself; monster comes back and he's still hungry (as he never asks for more), so I make him dinner.
I check Slack again; Mr. Product still hasn't responded. I go back to making dinner.
Most of the way through cooking, I get a notification! Product says he's talking it through with my boss, who will update me on it. Okay fine. I finish making dinner and go eat.
No response from boss; I start looking through my next ticket.
No response from boss. I ping him and ask for an update, and he says "What are you talking about?" Apparently product never talked to bossmang =/ I ask him about the resources, and he says there's no need to create any more as the one I need already exists! Yay!
So my feature went from a large, complex refactor all the way down to a -1+2 diff. That's freaking amazing, and it only took the entire day!
I run the related specs, which take forever, then commit and push.
Push rejected; pull first! Fair, I have been gone for two weeks. I pull, and git complains about my .gitignore and some local changes. fine, whatever. Except I forgot I had my .gitignore ignored (skipped worktree). Finally figure that out, clean up my tree, and merge.
Time to run the specs again! Gems are out of date. Okay, I go run `bundle install` and ... Ruby is no longer installed? Turns out one of the changes was an upgrade to Ruby 2.5.8.
Alright, I run `rvm use ruby-2.5.8` and.... rvm: command not found. What. I inspect the errors from before and... ah! Someone's brain fell out and they installed rbenv instead of the expected rvm on my mac. Fine, time to figure it out. `rbenv which ruby`; error. `rbenv install --list`; skyscraper-long list that contains bloody everything EXCEPT 2.5.8! Literally 2.5 through 2.5.7 and then 2.6.0-dev. asjdfklasdjf
Then I remember before I left people on Slack made a big deal about upgrading Ruby, so I go looking. Dummy me forgot about the search feature for a painful ten minutes. :( Search found the upgrade instructions right away, ofc. I follow them, and... each step takes freaking forever. Meanwhile my children are having a yelling duet in the immediate background, punctuated with screams and banging toys on furniture.
Eventually (seriously like twenty-five minutes later) I make it through the list. I cd into my project directory and... I get an error message and I'm not in the project directory? what. Oh, it's a zsh thing. k, I work around that, and try to run my specs. Fail.
I need to update my gems; k. `bundle install` and... twenty minutes later... all done.
I go to run my specs and... RubyMine reports I'm using 2.5.4 instead of 2.5.8? That can't be right. `ruby --version` reports 2.5.8; `rbenv version` reports 2.5.8? Fuck it, I've fought with this long enough. Restarting fixes everything, right? So I restart. when my mac comes back to life, I try again; same issue. After fighting for another ten minutes, I find a version toggle in RubyMine's settings, and update it to 2.5.8. It indexes for five minutes. ugh.
Also! After the restart, this company-installed surveillance "security" runs and lags my computer to hell. Highest spec MacBook Pro and it takes 2-5 seconds just to switch between desktops!
I run specs again. Hey look! Missing dependency: no execjs. I can't run the specs.
Fuck. This. I'll just push and let the CI run specs for me.
I just don't care anymore. It's now 8pm and I've spent the past 11 hours on a -1+2 diff!
What a great first day back! Everything is just the way I left it.6
I've optimised so many things in my time I can't remember most of them.
Most recently, something had to be the equivalent off `"literal" LIKE column` with a million rows to compare. It would take around a second average each literal to lookup for a service that needs to be high load and low latency. This isn't an easy case to optimise, many people would consider it impossible.
It took my a couple of hours to reverse engineer the data and implement a few hundred line implementation that would look it up in 1ms average with the worst possible case being very rare and not too distant from this.
In another case there was a lookup of arbitrary time spans that most people would not bother to cache because the input parameters are too short lived and variable to make a difference. I replaced the 50000+ line application acting as a middle man between the application and database with 500 lines of code that did the look up faster and was able to implement a reasonable caching strategy. This dropped resource consumption by a minimum of factor of ten at least. Misses were cheaper and it was able to cache most cases. It also involved modifying the client library in C to stop it unnecessarily wrapping primitives in objects to the high level language which was causing it to consume excessive amounts of memory when processing huge data streams.
Another system would download a huge data set for every point of sale constantly, then parse and apply it. It had to reflect changes quickly but would download the whole dataset each time containing hundreds of thousands of rows. I whipped up a system so that a single server (barring redundancy) would download it in a loop, parse it using C which was much faster than the traditional interpreted language, then use a custom data differential format, TCP data streaming protocol, binary serialisation and LZMA compression to pipe it down to points of sale. This protocol also used versioning for catchup and differential combination for additional reduction in size. It went from being 30 seconds to a few minutes behind to using able to keep up to with in a second of changes. It was also using so much bandwidth that it would reach the limit on ADSL connections then get throttled. I looked at the traffic stats after and it dropped from dozens of terabytes a month to around a gigabyte or so a month for several hundred machines. The drop in the graphs you'd think all the machines had been turned off as that's what it looked like. It could now happily run over GPRS or 56K.
I was working on a project with a lot of data and noticed these huge tables and horrible queries. The tables were all the results of queries. Someone wrote terrible SQL then to optimise it ran it in the background with all possible variable values then store the results of joins and aggregates into new tables. On top of those tables they wrote more SQL. I wrote some new queries and query generation that wiped out thousands of lines of code immediately and operated on the original tables taking things down from 30GB and rapidly climbing to a couple GB.
Another time a piece of mathematics had to generate all possible permutations and the existing solution was factorial. I worked out how to optimise it to run n*n which believe it or not made the world of difference. Went from hardly handling anything to handling anything thrown at it. It was nice trying to get people to "freeze the system now".
I build my own frontend systems (admittedly rushed) that do what angular/react/vue aim for but with higher (maximum) performance including an in memory data base to back the UI that had layered event driven indexes and could handle referential integrity (overlay on the database only revealing items with valid integrity) or reordering and reposition events very rapidly using a custom AVL tree. You could layer indexes over it (data inheritance) that could be partial and dynamic.
So many times have I optimised things on automatic just cleaning up code normally. Hundreds, thousands of optimisations. It's what makes my clock tick.4
The built-in array is also an hashmap. Actually, it's always a hashmap, but you can append to it without specifying indexes and PHP will use consecutive integers. Its performance characteristics? Who knows. Oh, and only strings, ints and null are valid keys.
What's the iteration order for arrays if you use them as hashmaps (string keys)? Well, they have their internal order. So it's actually an ordered hashmap that's being called an array. And you can produce an array which has only integer keys starting with 0, but with non-sequential internal (iteration) order.
This array weirdness has some non-trivial implications. `json_encode` (serializes argument to JSON) assumes an array corresponds to a JSON array if its keys are consecutive integers in increasing order starting with 0, otherwise the array becomes a JSON object. `array_filter` (filters arrays/hashmaps using callback predicate) preserves keys, so it will punch holes in the int key sequence if non-last items are removed, thus turning arrays into hashmaps and changing your JSON structure if you forget to discard keys before serialization.
You may wonder how JSON deserialization works, then? There's a special class for deserialized JSON objects, `stdClass`. It's basically a hashmap too, but it's an object, not an array, and all functions that would normally accept arrays won't work with it. So basically its only use is JSON (de)serialization. You can even cast arrays to objects, producing `stdClass`.
Bonus PHP trivia:
Many functions return nonsensical values. `preg_match`, the regex matching function, returns 1 for success, 0 for no matches and false for malformed regular expression. PHP supports exceptions, so it could just throw one on errors. It would even make more sense to return true, false and null for these three cases. But no, 1, 0 and false. And actual matches are returned by output arg.
`array_walk_recursive`, a function supposed to recursively apply callback to each element of an array. That's what docs say. It actually applies it to leafs only. It will also silently accept object instead of array and "walk" it, but without recursing into deeper objects.
Runtime type enforcing is supported for function arguments and returned values. You can use scalar types, classes, array, null and a few special keywords. There's also a `mixed` keyword, which is used in docs and means "anything". It's syntactically valid, the parser will accept it, but it matches no values in runtime. Calling such function will always cause a runtime error.
Strings can be indexed with negative integers. Arrays can't.
ReflectionClass::newInstanceWithoutConstructor: "Creates a new class instance without invoking the constructor". This one needs no commentary.
`array_map` is pretty self-explanatory if you call it with a callback and an array. Or if you provide more arrays of equal length via varargs, callback will be called with more arguments, one from each array. Makes sense so far. Now, you can also call `array_map` with null instead of callback. In that case it treats provided arrays as rows of a matrix and returns that matrix, transposed.5
Been working on 'MVP' features of a new product for the past 14 months. Customer has no f**king clue on how to design for performance. An uncomfortable amount of faith was placed on the ORM (ORMs are not bad as long as you know what you are doing) and the magic that the current framework provides. (Again, magic is good so long as you understand what happens behind the smoke and mirrors - but f**k all that... coz hey, productivity, right?). Customer was so focussed on features that no one ever thought of giving any attention to subtler things like 'hey, my transaction is doing a gazillion joins across trizillion tables while making a million calls to the db - maybe I should put more f**king thought into my design.' We foresaw performance and concurrency issues and raised them way ahead of the release. How did the customer respond? By hiring a performance tester. Fair enough - but what did that translate into? Nothing. Nada. Zilch. Hiring a perf tester doesn't automagically fix issues. The perf tester did not have a stable environment, a stable build or anything that is required to do a test with meaningful results. As the release date approached, the customer launched a pilot and things started failing spectacularly with the system not able to support more than 15 concurrent users. WTF! (My 'I told you so' moment) Emails started flying in all directions and the hunt for the scapegoat was on (I'm a sucker for CYA so I was covered). People started pointing in all directions but no one bothered to take a step back and understand what was causing the issues. Numero uno reason for transaction failure was deadlocks. We were using a proprietary DB with kickass tooling. No one bothered to use the tooling to understand what was the resource in contention let alone how to fix the contention. Absolute panic - its like they just froze. Debugging shit and doing the same thing again and again just so that management knew they were upto something. Most of the indexes had a fragmentation of 99.8% - I shit you not. Anywho, we now have a 'war room' where the perf tester needs to script the entire project by tonight and come up with some numbers that will amount to nothing while we stay up and keep profiling the shit out of the application under load.
Lessons learnt - When you foresee a problem make a LOT of noise to get people to act upon it and not wait till it comes back and bites you in the ass. Better yet, try not to get into a team where people can't understand the implications of shitty design choices. War room my ass!3
I’m fucking lost.
So, situation. I have a SQL table with about 3M rows (not a lot).
I have indexes. Indexes are used. BUT when I add where clause (On indexed column), it’s super slow. Around 10 seconds.
If I do select * (ALL 3M rows) and THEN I filter then on webserver side, it takes 0.5 seconds.
HOW my manual filtering is faster than DB filtering with indexes? I even tried bubble sort. Bubble sort is faster than SQL ‘where’. HOW ?!
I do not understand….
And if I add group by….. WELL, 25 seconds SQL time. 2 Seconds if I do select all and group by in code manually.
Does not make ANY sense to me.
What am I missing ?23
>Have an issue with incredibly slow webpage load time
>Blame memcache issues
So... I look into the problem. Yes, the page either loads up fast, or times out. So, into the logs I go. Webserver is fine (except the timeout), PHP though... Error log is fine (just notices), but slow log shows the issue is the database (of course... its always the database... ugh)
So, checking the database, there is one ugly query that seems to be an issue. 5 joins and a huge where condition.
So I run EXPLAIN on the query and... Proceed to bang my head against the wall.
OF COURSE ITS SLOW YOU FU******, NONE OF YOUR TABLES HAVE ANY INDEXES.
What do they expect when the database has to always go down the whole table and do everything in memory, until it runs out and has to dump it all on disk and work with it there.
Ugh... Some clients...
Math question time!
Okay so I had this idea and I'm looking for anyone who has a better grasp of math than me.
What if instead of searching for prime factors we searched for a number above p?
One with a certain special property. BEAR WITH ME. I know I make these posts a lot and I'm a bit of a shitposter, but I'm being genuine here.
Take this cherry picked number, 697 for example.
It's factors are 17, and 41. It's trivial but just for demonstration.
If we represented it's factors as a bit string, where each bit represents the index that factor occurs at in a list of primes, it looks like this
When converted back to an integer that number becomes 4160, which we will call f.
And if we do 4160/(2**n) until the result returns
a fractional component, then N in this case will be 7.
And 7 is the index of our lowest factor 17 (lets call it A, and our highest factor we'll call B) in our primes list.
So the problem is changed from finding a factorization of p, to finding an algorithm that allows you to convert p into f. Once you have f it's a matter of converting it to binary, looking up the indexes of all bits set to 1, and finding the values of those indexes in the list of primes.
I'm working on doing that and if anyone has any insights I'm all ears.9
Working on an application - everywhere an enum should be is a database table instead.
Me: What happens when someone changes "rejected" to "approved" in your status table?
Me: What happens when you re-seed your database and the indexes for your types are different?
All problems, with no time or scope to fix!
USE A FUCKING ENUM9
PHP dev help/advice needed!
We have problems with mysql. Still stuck with mariaDB, I'm using indexes (correct ones) and we have problems with scaling. we have a few tables with over 100mil rows, 1 of them is being read every morning with a subselect that counts unique rows, and fails every time because of timeout/lock, the temp table size was increased and helped for a little while but as time goes on the table grows and the problem reappears. I'm reading from a slave server that was purposely created for read only, yet we still have problems. We're using managed dedicated servers for out hosting and they aren't willing to optimise the database configs for our needs. What are the easiest options for scaling at this point? Going fully dedicated server and perconaDB? NOsql? Sharding the server? Anyone got any good blogposts or something to read about this? your own experience?11
Manjaro has some quirks that annoy me(no MST timezone, spotty support for my WD NVME), so I decided that since I'm not interested in any pre-configured graphical desktop of any kind, I should just dive into Arch, since it increasingly felt like that's what I was doing anyway but with Manjaro to dull the blow. So I did, and I am over the moon for doing so. Lots of gnashed teeth, but DDG indexes an answer to every question I've had, and it always makes sense when I find it. I've enjoyed having to dive into systemd in a much more low-level way than ever before-- to actually LEARN what it's doing, how, and why.
But one by one, I have been faced with some issue that I need to resolve, and one by one, I've knocked them off. The result now is the best work and gaming desktop I have ever used.
Arch is not for geniuses or wizards. Just patient people who are willing to read. The payoff is staggering, and many times over worth the effort.4
Heres the initial upgraded number fingerprinter I talked about in the past and some results and an explanation below.
Note that these are wide black images on ibb, so they appear as a tall thin strip near the top of ibb as if they're part of the website. They practically blend in. Right click the blackstrip and hit 'view image' and then zoom in.
Hastebin wouldn't save for some reason so paste.ofcode.org it is.
Not much to look at, but I was thinking I'd maybe mark the columns where gaps occur and do some statistical tests like finding the stds of the gaps, density, etc. The type test I wrote categorizes products into 11 different types, based on the value of a subset of variables taken from a vector of a couple hundred variables but I didn't want to include all that mess of code. And I was thinking of maybe running this fingerprinter on a per type basis, set to repeat, and looking for matching indexs (pixels) to see what products have in common per type.
Or maybe using them to train a classifier of some sort.
Each fingerprint of a product shares something like 16-20% of indexes with it's factors, so I'm thinking thats an avenue to explore.
What the fingerprinter does is better explained by the subfunction findAb.
The code contains a comment explaining this, but basically the function destructures a number into a series of division and subtractions, and makes a note of how many divisions in a 'run'.
Typically this is for numbers divisible by 2.
So a number like 35 might look like this, when done
p = 35
And we'd represent that as
ab(w, x, y, z)
Where w is the starting value 35 in this case,
x is the number to divide by at each step, y is the adjustment (how much to subtract by when we encounter a number not divisible by x), and z is a string or vector of our results
which looks something like
ab(35, 2, 1, [1, 4])
because we were only able to divide by 2 once, before having to subtract 1, and repeat the process. And then we had a run of 4 divisions.
And for the fingerprinter, we do this for each prime under our number p, the list returned becoming another row in our fingerprint. And then that gets converted into an image.
And again, what I find interesting is that
unknown factors of products appear to share many of these same indexes.
What I might do is for, each individual run of Ab, I might have some sort of indicator for when *another* factor is present in the current factor list for each index. So I might ask, at the given step, is the current result (derived from p), divisible by 2 *and* say, 3? If so, mark it.
And then when I run this through the fingerprinter itself, all those pixels might get marked by a different color, say, make them blue, or vary their intensity based on the number of factors present, I don't know. Whatever helps the untrained eye to pick up on leads, clues, and patterns.
If it doesn't make sense, take another look at the example:
This is semi-unique to each product. After the fact, you can remove the variable itself, and keep just the structure in question, replacing the first variable with some other number, and you get to see what pops out the otherside.
If it helps, you can think of the structure surrounding our variable p as the 'electron shell', the '-1's as bandgaps, and the runs of '2's as orbitals, with the variable at the center acting as the 'nucleus', with the factors of that nucleus acting as the protons and neutrons, or nougaty center lol.
Anyway I just wanted to share todays flavor of insanity on the off chance someone might enjoy reading it.1
I've been freelancing lately with an agency to develop an android app for their client and at the same time another person is developing the website .
The story begins when I first contacted the web dev to give me access to the database (because he started before me ).It turns out that this guy purchased an almost ready cms template with a shitty data structure that has no relations between object .This database has no primary keys , no foreign keys , no indexes ... no nothing . Adding to that the web dev refused that I rewrite a new data structure claiming that he has done a good progress on the website .
Forward couple of weeks , I managed to create the api and develop an alpha for the app and sent it to the agency manager .
This bastard told me that the website and design have changed and the app shouldn't be like that .He told me to contact the other bastard the web dev to seen what the changes are . I'm waiting for the response about the new updates and I'm praying that they'll be just minor colors updates or something not a whole concept update .
My problem here is I'm stuck with this fucking agency cuz they paid half of the payment when I started .
Damn I must learn to say no to people .1
JPA my friend ... JPA why are you like this? JPA why do hate me so much? JPA, let's have a word ...
How come you are so far away from real-world problems, so cumbersome to use, so ugly (criteria API), so wrong and inconsistent?
Oh, what it's all your parents fault? Oh come, on that can't be, right? Did you have a bad childhood?
Your parent's were fucking crack-smoking maniacs which didn't know a single bit about actual databases?
They design you as an API without actually trying you out in the wild? And then they patched up together with some essential DB stuff, like friggin indexes? Not even tried to make this API consistent nor really functional?
Oh poor, you little JPA ...