What’s the most efficient algorithm to find duplicates in a set?

An SO moderator

Best "error" I've ever seen today.

We have a very large database with millions of people, some of those records are duplicates. So I had a long project to write something that would automatically merge duplicate records while allowing employees to review them. Today we had a duplicate show up in the list which should not have.

Same name (apart from one letter), address, employee ID (off by 1 number), same manager, title, phone number, birthdate. But we figured out it wasn't the same person and therefore wasn't a good match.

Turns out they are twin sisters who live together and we're hired by the same manager for the same position at the same time. What are the odds...

Another stack overflow rant.

I had a disagreement with a self proclaimed "high repper" last night. We exchanged words in the comments of one of my questions.

Later (about ten mins) i see that another one of my questions has been closed and marked as duplicate - by this same fuck-knuckle. He has obviously gone to my profile and then gone out of his way to harass / bully me by doing this.

The 2 questions are absolutely not duplicates and he has marked them as identical.

I go to his profile and his headline thing is

"Low reppers hate closers but they need to go bitch about it elsewhere"

If anyone on here doesn't understand why SO gets a bad rap, it's specifically due to complete cunts like this guy.

If you happen to be on here and recognise yourself from the really cringy "low reppers" comment on your profile, then all I have to say to you is that you are a complete an utter ballbag; a tool; an arsehole of the highest order.

Fuck you and all your spawn.

Ticket: Add <feature> to <thing>. It works in <other things> so just copy it over. Easy.

Thing: tangled, over-complicated mess.
Feature: tangled and broken, and winds much too deep to refactor. Gets an almost-right answer by doing lots of things that shouldn't work but somehow manage to.

I write a quick patch that avoids the decent into madness and duplicates the broken behavior in a simple way for consistency and ease of fixing later. I inform my boss of my findings and push the code.

He gets angry and mildly chews me out for it. During the code review, he calls my patch naive, and says the original feature is obviously not broken or convoluted. During the course of proving me wrong, he has trouble following it, and eventually finds out that it really is broken -- and refuses to admit i was right about any of it. I'm still in trouble for taking too long, doing it naively, and not doing it correctly.

He schedules a meeting with product to see if we should do it correctly. He tells product to say no. Product says no. He then tells me to duplicate the broken behavior. ... which I already did.

At this point I'm in trouble for:
1) Taking too long copying a simple feature over.
2) Showing said feature is not simple, but convoluted and broken.
3) Reimplementing the broken feature in a simpler way.
4) Not making my new implementation correct despite it not working anywhere else, and despite how that would be inconsistent.

Did everything right, still in the wrong.

Also, they decided I'm not allowed to fix the original, that it should stay broken, and that I should make sure it's broken here, too.

You just have to admire the sound reasoning and mutual respect on display. Best in class.

Client:
"Ok,. so your saying that its gonna take you 63 hrs to create a simplified CRM with basic functionality and auto fill docs or automated work flow docs as an added feature?"

My response (after already under-quoting and planning on cutting some corners because he has a smaller budget than normally necessary):
"It sounds simpler than it is. There are a lot of things I need to take into account that you wouldn't even think about.

For instance:

Making sure your emails don't go to the client's spam folder. This requires the sending domain to be verified via DNS settings. I have to ensure your email content passes a spam test (link to text ratio needs to be good). I assumed you'd want an email that has your logo and looks good. This means testing the design in Outlook to make sure it's not broken.

What if the email doesn't send due to an invalid email address, or bounces back? You'll need to be notified.

What if the client list for the week contains duplicates? You need them merged or ignored.

Generating a PDF from HTML can be tricky because the conversion isn't apples to apples so there are things I need to adjust to make them as close as possible.

Making a site completely mobile friendly (the tier 3 option) can be very time consuming as well. It's not about whether or not it fits on a mobile phone, it's about whether or not it's intuitive and useful. You're essentially getting a mobile app without paying for separate development of an app.

If I took everything into consideration and built this to be 100% bullet proof, it would cost tens of thousands.

I'm doing my best to leverage your needs with the probability of running into an issue. I'm not going waste my time/your money on something that will likely never happen."

My biggest enemy is what i call "development fork bomb".
My boss duplicates code on a daily basis. Instead of creating subclasses he simply creates new files by copying lines from other files. The projects class hierarchy is as flat as holland.
You can take a comment, do a project-wide search and you will find 3+ matches, an ugly hack i wrote exists 4 times in the project, and so on.

Worst of all, we spend more time on bugfixes than refactoring. With my power i could add a commit-hook to block or lessen this behaviour, but i cant. There's no program that can detect this reliably and sometimes it needs to be done.

This is a curse i'm stuck with appearently.

New job was killing me. remote team has an 8 hour time difference to us, and no understanding of it. Constant last minute invites to meetings at 10pm my time. Made worse by the fact that many were unnecessary, duplicates or just plain pointless. So for the last few weeks of last year I made in my mission to clean it up.

New plan: move my hours around on Mondays, stay later, move all the meetings back to back and get everything out of the way for the rest of the week.

First day back and heres how the new plan is shaping out:

- 5:30pm meeting organiser decided we actually need 2 almost identical meetings instead. Sends out a big team meeting for the same time as my 1st meeting at 5pm, as well as the existing one for 5:30pm. Already agreed on by everyone else, so had to go.
- Cancelled my original 5pm meeting for today, said we'll re-arrange it for earlier going forward (not enough time for notice for remote team).
- Went into my new 5pm meeting, turns out we don't need 2. Got everything done by 5:30pm.
- Just to be safe though, a new invite will be sent around for the hour of 5 - 6 "Just in case".
- My 6pm meeting just got cancelled as she has a conflict (despite setting it up 2 months ago)
- Now I have to wait around, after hours for my 6:30 meeting.

..... believe it or not, this is progress.

Happy new year!

It works.

How I hate that sentence.

Whenever that sentence pops up, I wanna take a frying pan, make some bacon, eat the bacon and slam the still hot pan with grease through someone's face till the skull breaks.

Why has he so many anger issues, one might ask.

Usually the sentence "It works" means that after looking at "working thing" it works wrong in 95 % of all cases, but hey - for 5 % it at least does *something* right. Not everything, don't get ya hope up.

We had this fun topic happening again today and I'm still too angry to sleep.

Lucene analysis of texts in Elasticsearch.

Stopword list? Multiple word n-grams per line, duplicates, not lower cased, not properly encoded.

Tokenizers? Duh. Why should one put them in proper order.... Or more realistic: There is an order in tokenizers necessary *devs with shocked faces*.

Language specific details... UHM. Wait. Languages are different? There are edge cases in languages? *more shocked faces*.

Even more shocking that if an text processing pipeline is implemented horribly wrong, it delivers wrong results. *mind blown*.

But our unit tests (this goes out to @kiki) were working.

Yeah. You dumb nuggets who even an amoeba would be ashamed of, when you only do positive tests in unit tests with the most obvious working examples, then your unit tests are just useless waste of nibbles.

Some of the devs are really a fucking waste of genetic information, should have probably ended better in a sock.

If this sounds too harsh, they had 2 weeks.
In just 3 hours I found out that they can redo that with supervision.

-.-

I'm getting too old for that shit. Seriously.

So one rant reminded me of a situation I whent through like 10 years ago...
I'm not a dev but I do small programs from time to time...
One time I was hired to pass a phone book list from paper to a ms Access 97 database...
On my old laptop I could only add 3 to 5 records cause MS access doesn't clean after itself and would crash...

So I made an app (in vb6) , to easily make records, was fast, light and well tabbed.
But now I needed a form to edit the last record when I made a mistake...
Then I wanted a form to check all the records I made.
Well that gave me an idea and presented the software to the client... A cheesy price was agreed for my first freelance sell...
After a month making it perfect and knowing the problems the client would had I made a admin form to merge all the databases and check for each record if it would exist.... I knew the client would have problems to merge hundreds of databases....
When it was done... The client told me he didn't need the software anymore.... So I gave it to a friend to use as an client dabatase software... It was perfect for him.
One month later the client called me because he couldn't merge the databases...
I told him I was already working in a company. That my software was ready to solve his problem, but I got mad and deleted everything...
He had to pay almost 20 times more for a software company to make the same software but worst... Mine would merge and check all the databases in a folder... Their's had to pick one by one and didn't check for duplicates... So he had to pay even more for another program to delete duplicates...

That's why I didn't follow programming as a freelance... Lots of regrets today...
Could be working at home, instead had a burn out this week cause of overwork...

Sorry for the long rant.

No idea if duplicates exist here, but anyway, from smbc-comics.com -> firstRant

I wish there was some kind of "Close as Duplicate" function for rants like there is on Stack Overflow... tends of be a lot of reposts of the same memes.

Was interviewing someone for a role, asked them a basic question in Python (before anyone gets on my case about interview coding questions, it's removing duplicates and the answer is to just cast to set, I'm just checking that they actually know Python). Perusing Stack Overflow while I wait for their answer (it's a remote call and I give them a bit of time to calmly deliberate). The exact prompt I gave them pops up as a question, the asker is registered to their profile.

Not only did they not get the job, but I downvoted the question and marked it as duplicate. Rejection and unemployment can be temporary, but StackOverflow reputation is FOREVER.

Ok, so our team is responsible for writing an app that consumes an API written by the client's team (I refuse to call it a "REST" API, despite their claims). On one of the clarification meetings we are discussing an endpoint that accepts a (logically) unique field multiple times, even though an entity is already registered in the system with that unique identifier. Our proposal would be that this API of theirs should not happily accept duplicates as many times as there are bits on a 4TB hard drive, rather it should signal an error.

The response we got is this: Due to the Separation of Concerns principle they thought that it should be our app's responsibility to not send a request if an entity with said field is already in the system. Thus there's no need for the backend to validate this.

I didn't hear the next part, because I had to collect my headphones from the other side of the room where they were flung in rage.

Go to fucking hell SO!

Question: "How to do X?"
Answer: "You can solve your particular problem without doing X."

God fucking damnit, yeah some noob tried to use X to solve a completely unrelated problem and thank god somebody pointed out a better solution. But since all other questions about doing X are labelled as duplicates of this question, could you dickheads at least provide an actual answer to the question instead of an answer which only works for that particular problem and has nothing to do with X?

I'm pretty sure my clients would fail the marshmallow test 9/10 times if not 10/10 times. We have a certain time period of the day set aside for me to look into new reported bugs but besides that I'm supposed to work on regular tasks. Of course, they ask me five hours after that time period is done, whether I can look into a new (non-urgent) bug. At the cost of the new thing they want to launch in 2 weeks. 🤔 I would love it if we actually had time to fix every single bug in the codebase but what typically happens is I get about 15 bug reports (most duplicates) and I'm expected to fix all of them in a span of 2 hours.

Guy rage quit over someone resolve duplicating his bugs (when they were really duplicates) but he kept reopening them repeatedly so the other guy kept resolving them as such till he lost his mind on work chat. A few days later he is now casually sending an email (after being told to do so by management) to apologize but he skirted around apologizing and instead excuses himself by saying "we are all fellow autists to a certain extent."

Terminator's broadcast feature is nice.
However, it's fragile as crap, and oftentimes sends duplicate characters, or adds garbage.

Quite awhile ago I posted a PSA on how to fix it (by killing ibus) but now it's broken for me again. And due to just how it broke, I've given up on fixing it. Instead of broadcasting one or two duplicate characters like before, it now hangs for many seconds because it's broadcasting THREE BLOODY MEGABYTES WORTH OF DUPLICATES with every keystroke.

Like, seriously?
How do you fail that hard?

MTP is utter garbage and belongs to the technological hall of shame.

MTP (media transfer protocol, or, more accurately, MOST TERRIBLE PROTOCOL) sometimes spontaneously stops responding, causing Windows Explorer to show its green placebo progress bar inside the file path bar which never reaches the end, and sometimes to whiningly show "(not responding)" with that white layer of mist fading in. Sometimes lists files' dates as 1970-01-01 (which is the Unix epoch), sometimes shows former names of folders prior to being renamed, even after refreshing. I refer to them as "ghost folders". As well known, large directories load extremely slowly in MTP. A directory listing with one thousand files could take well over a minute to load. On mass storage and FTP? Three seconds at most. Sometimes, new files are not even listed until rebooting the smartphone!

Arguably, MTP "has" no bugs. It IS a bug. There is so much more wrong with it that it does not even fit into one post. Therefore it has to be expanded into the comments.

When moving files within an MTP device, MTP does not directly move the selected files, but creates a copy and then deletes the source file, causing both needless wear on the mobile device' flash memory and the loss of files' original date and time attribute. Sometimes, the simple act of renaming a file causes Windows Explorer to stop responding until unplugging the MTP device. It actually once unfreezed after more than half an hour where I did something else in the meantime, but come on, who likes to wait that long? Thankfully, this has not happened to me on Linux file managers such as Nemo yet.

When moving files out using MTP, Windows Explorer does not move and delete each selected file individually, but only deletes the whole selection after finishing the transfer. This means that if the process crashes, no space has been freed on the MTP device (usually a smartphone), and one will have to carefully sort out a mess of duplicates. Linux file managers thankfully delete the source files individually.

Also, for each file transferred from an MTP device onto a mass storage device, Windows has the strange behaviour of briefly creating a file on the target device with the size of the entire selection. It does not actually write that amount of data for each file, since it couldn't do so in this short time, but the current file is listed with that size in Windows Explorer. You can test this by refreshing the target directory shortly after starting a file transfer of multiple selected files originating from an MTP device. For example, when copying or moving out 01.MP4 to 10.MP4, while 01.MP4 is being written, it is listed with the file size of all 01.MP4 to 10.MP4 combined, on the target device, and the file actually exists with that size on the file system for a brief moment. The same happens with each file of the selection. This means that the target device needs almost twice the free space as the selection of files on the source MTP device to be able to accept the incoming files, since the last file, 10.MP4 in this example, temporarily has the total size of 01.MP4 to 10.MP4. This strange behaviour has been on Windows since at least Windows 7, presumably since Microsoft implemented MTP, and has still not been changed. Perhaps the goal is to reserve space on the target device? However, it reserves far too much space.

When transfering from MTP to a UDF file system, sometimes it fails to transfer ZIP files, and only copies the first few bytes. 208 or 74 bytes in my testing.

When transfering several thousand files, Windows Explorer also sometimes decides to quit and restart in midst of the transfer. Also, I sometimes move files out by loading a part of the directory listing in Windows Explorer and then hitting "Esc" because it would take too long to load the entire directory listing. It actually once assigned the wrong file names, which I noticed since file naming conflicts would occur where the source and target files with the same names would have different sizes and time stamps. Both files were intact, but the target file had the name of a different file. You'd think they would figure something like this out after two decades, but no. On Linux, the MTP directory listing is only shown after it is loaded in entirety. However, if the directory has too many files, it fails with an "libmtp: couldn't get object handles" error without listing anything.

Sometimes, a folder appears empty until refreshing one more time. Sometimes, copying a folder out causes a blank folder to be copied to the target. This is why on MTP, only a selection of files and never folders should be moved out, due to the risk of the folder being deleted without everything having been transferred completely.

(continued below)

Wk78

My worst dev experience was my first software as freelance...
1 month codding
When delivering the app the client didnt want it anymore...
Two years latter the client calls me because he had a problem... Merging hundreds of access 97 databases... Exactly what my software did (besides editing, filters and remove duplicates)
Told him I got mad, deleted the source code and was already working on a company...
He had to pay for a software company to do the same 10 times the cost

-click-

MySQL: 'the table does not exist'

I just fucking made the temp table dude, that's what you literally do in step 1 .... how could it not....

-click-

MySQL 'Records: 1 Duplicates: 0 Warnings: 0'

Me: "Oh there we g---"

-click-

MySQL: 'the table does not exist'

Me: "Hey you just worked!"

-click-

MySQL: 'the table does not exist'

GOD DAMN IT

-click-
-click-
-click-
-click-
-click-
-click-

MySQL 'Records: 1 Duplicates: 0 Warnings: 0'

Me: Uh you're working now?

-click-

MySQL 'Records: 1 Duplicates: 0 Warnings: 0'

-click-

MySQL 'Records: 1 Duplicates: 0 Warnings: 0'

-click-

MySQL 'Records: 1 Duplicates: 0 Warnings: 0'

Guess that API just needed breaking in....

Ok now I'm gonna tell you about my "Databases 2" exam. This is gonna be long.

I'd like to know if DB designers actually have this workflow. I'm gonna "challenge" the reader, but I'm not playing smartass. The mistakes I point out here are MY mistakes.

So, in my uni there's this course, "Databases 2" ("Databases 1" is relational algebra and theoretical stuff), which consist in one exercise: design a SQL database.

We get the description of a system. Almost a two pages pdf. Of course it could be anything. Here I'm going to pretend the project is a YouTube clone (it's one of the practice exercises).

We start designing a ER diagram that describes the system. It must be fucking accurate: e.g. if we describe a "view" as a relationship between the entities User and Video, it MUST have at least another attribute, e.g. the datetime, even if the description doesn't say it. The official reason?

"The ER relationship describes a set of couples. You can not have two elements equal, thus if you don't put any attribute, it means that any user could watch a video only once. So you must put at least something else."

Do you get my point? In this phase we're not even talking about a "database", this is an analysis phase.

Then we describe the type dictionary. So far so good, we just have to specify the type of any attribute.

And now... Constraints.
Oh my god the constraints. We have to describe every fucking constraint of our system. In FIRST ORDER LOGIC. Every entity is a set, and Entity(e) means that an element e belongs to the set Entity. "A user must leave a feedback after he saw a video" becomes like

For all u,v,dv,df,f ( User(u) and Video(v) and View(u, v, dv) and feedback(u, v, f) ) ---> dv < df

provided that dv and df are the datetimes of the view and the feedback creation (it is clear in the exercise, here seems kinda cryptic)

Of course only some of the constraints are explicitly described. This one, for example, was not in the text. If you fail to mention any "hidden" constraint, you lose a lot of points. Same thing if you not describe it correctly.

Now it's time for use cases.
You start with the usual stickman diagram. So far so good.
Then you have to describe their main functions.

In first order logic. Yes.

So, if you got the point, you may think that the following is correct to get "the average amount of feedback values on a single video" (1 to 5, like the old YT).
(let's say that feedback is a relationship with attribute between User and Video

getAv(Video v): int
Let be F = { va | feedback(v, u, va) } for any User u
Let av = (sum forall f in F) / | F |
return av

But nope, there's an error here. Can you spot it (I didn't)?

F is a set. Sets do not have duplicates! So, the F set will lose some feedback values! I can not define that as a simple set!

It has to be a set of couples, like (v, u), where v is the value and u the user; this way we can have duplicate feedback values in our set.

This concludes the analysis phase. Now, the design.

Well we just refactor everything we have done until now. Is-a relations become relationships, many-to-many relationships get an "association entity" between them, nothing new.
We write down on paper every SQL statement to build any table, entity or not. We write down every possible primary key or foreign key. The constraint that are not natively satisfied by SQL and/or foreign keys become triggers, and so on.

This exam is considered the true nightmare at our department. I just love it.

Now my question is, do actually DB designers follow this workflow? Or is this just a bloody hard training in Pai Mei style?

FUCK YOU, UNITY.

The fucking UI lag causes mouse events in the editor to aggregate and act fucking weird when it's randomly reloading scripts and shit.

I just needed to move some assets into another folder. I control clicked a few scripts at the same time Unity decided to recompile them. It ended up thinking I was making a box selection and then, weirdly, some mouse combo that duplicates scripts (didn't even know that existed).

Okay, whatever, nbd. I just control clicked the other scripts (all with `1` postfixes) to delete them. No problemo, right click, delete.

FUCKING UNITY LAGGED, THOUGHT I BOX-SELECTED AGAIN, AND DELETED EVERYTHING IN THE GOD DAMNED FUCKING FOLDER. HOURS OF WORK. JUST GONE.

FUCK YOU UNITY. YOU'VE CRASHED THREE TIMES THIS WEEK AND NOW THIS. FUCK YOU.

Anyone knows why I can copy paste text with color??
OS: Android 6.0.1
BTW: It's a regex for searching frequently repeating duplicates. Have been trying this for a while but still not working perfect...

Was offered a really good price to set up a webshop, and he would pay me the same amount for two literal duplicates of the site.
"Oh btw I need it so no one can see who logged in to the website, there must be no traces of activity"

IOW - he makes shitloads on illegal horse trade while I take the blame for the website.

Today was successful. I deployed an app to a dev environment that worked perfectly on local. When I asked if it was ready for QA I said "no, there is issues that need attention"

I am now the proud owner of 75 QA emails of things that do not work. Luckily they're all duplicates of the same issue.

Ffffffffffuuuuuuuuuuuuu

One thing when working with a ton of data:
If there is a slight, infinitesimal probability that something will be wrong, then it will 100% be wrong.

Never make assumptions that data is consistent, when dealing with tens of gigabytes of it, unless you get it sanitized from somewhere.

I've already seen it all:
* Duplicates where I've been assured "these are unique"
* In text fields that contain exclusively numeric values, there will always be some non-numeric values as well
* There will be negative numbers in "number sequences starting with 1"
* There will be dates in the future, and in the far far future, like 20115 in the future.
* Even if you have 200k customers, there will be a customer ID that will cause an integer overflow.

Don't trust anything. Always check and question everything.

Holly crap..... Just fixed a 7 months old bug in a system i worked on and had to leave it for 7 months coz i just wasn't seeing what i did wrong and today..... everything simply unraveled

It's a flask app and i got stuck on updating a column in the database and also it created duplicate configs on one of the configuration functionality

Figured out what was causing duplicates
A different function was creating new configs instead of updating...bypassing duplicate checking in the config function

Do you think it’s appropriate to use the phrase "dried" in commit messages to refer to removing duplicate code? (DRY = don’t repeat yourself)

I just used it and I’m not sure if it’s ok because some devs might not understand it and the the original letters from DRY went away and became "dried" so it might be even more cryptic.

On the other hand it’s so much more concise having "dried type X" compared to "refactored the code so that it doesn’t contain duplicates of type X"

Tl;dr: I'm a shitty PHP programmer, but even a shittier JS programmer.

I made a whole image upload system which has an elaborate login/sign up facility, checks the dimensions of an image, it's size, checks if it's broken, compares it to previous uploads to prevent duplicates, take comments, sorts them per user and stores it efficiently in a db and accesses it using PDO, not mySQL, displays it as a gallery, and even lets the users transfer images amongst them.
But I can't make a simple shopping cart in js.

Why? Just, why?

"It's very unfortunate that someone has to sit 10 hours a day on a computer screen but that's what we are paying for."
Working with an EU client.
Task is in stages beginning from assessment of an 9 year old Salesforce instance, cleanup and then transition to lightning.
The deliverabale in the first stage is excel sheet - objects(2400+), fields, apps, packages, profiles(110k), users, perm sets, apex classes, triggers, pages, s-controls and insert each and everything that a Salesforce instance could have.
Each and every data needs to be, analyzed and documented with our recommendations before being sent over. (Finding duplicates in 110k profiles??)
Oh, did I tell you, this was to be done four weeks? Weekend goes to hell.
That's when this beautiful motivational line comes up from the bridge, "It's very unfortunate that someone has to sit 10 hours a day on a computer screen but that's what we are paying for."
Fortunately(un?), that part of project is done and over with.

Now comes the clean up, identify packages not being used, remove them, qa and then push for deployment.
Mind you, this project is to be 2 weeks long. Its Friday of the first week today. And I am still working weekends.
Can I say, FML?

I was given two lists of values in an excel spreadsheet and I needed to compare/find duplicates. I wrote some JavaScript to handle it. I don't know how to use fucking Excel.

So I've been working a project while now. last week we got a lot of changes from the client and the boss suggest we pull one of the senior devs from another project to help out. All good...until I checked the code...WTF!
For ex we have a method that checks and update weather info, if required, and returns a view(100 lines of code). so the client wants the weather to display differently in certain areas. exactly same data and everything just the view to look different. easy right..? Mr "senior" dev duplicates the method each time and just change the return statement to a different view...Fuck me right? Oh and 90% of CSS statements ends with !important. senior my fucking ass!

TL;DR: TIL for heavy queries use PDO and not some frameworks DB class

ffs I was trying to save 300k+ lines at once with Laravel for weeks. Mind you from a text file. 1gb ram on the vps so while trying to prepare the text to save: Fatal Error: Allowed Memory Size of bla bla Bytes Exhausted

ok so lets put it in a loop: Fatal error: Maximum execution time of 30 seconds exceeded (set_time_limit(0); lol)

optimising, varying the code got me into a situation when the content got saved in the BD but inconsistent (duplicates) and the table had often more than 1,5M rows. That was what told me its not a performance issue, my code is the issue. (dah)

I was starting to think it would be easier to export a prepared query to a sql file and load the file into the db as thats the fastest possible option...I even started to think about switching to python, then it hit me, Laravel has a shitload of routes to the DB so I switched to PDO

benchmark on 1vCPU, 1GB RAM VPS with SSD
379k lines with 11 columns in less than 10 sec with a loop of saving every ~6000 rows (if i tried choking it to save the whole thing at once it went up to 16-17sec)

!@#GDSGA!Q@#@

In my absolute lack of genius, I just lost way too much time finally realizing that in C# linq-to-objects removes duplicates when you use Union().

At one point I knew this, but I forgot about it because I never want duplicates, slowly forgetting what Union() was doing.

*This time* however, the duplicates are required and it all goes to crap without them.

Save me, Concat()!

You can run but you can't hide from noobery.

Blahb;asdfabnahsdfa.

Outlook - " You have many duplicated contacts. Want to merge ?"

Me - ( inside mind thinking .... wtf, there are no duplicate contacts ) Ok do it.

pressed 'merge' for every contact ( Yeah, 'merge all' wasnt even there. Fucker designed the application which ask user to merge every contact, one by one ).

End Result - Brain Fucking piece of shitty microsoft' outlook android application, created 5 duplicates of all the contacts. Cant be more happy. Now i have 1000 contacts.

Microshit at its best again
Microshit managed to keep my trust on its products. Always, performs like Tatti ( Shit in hindi )

Edit - A wise man honoured Microshit with 'Macroshit'

I had a problem with too many backups from our personal data (Photos,videos etc.)
Always I had 1-2 hard drives to backup all important files every time.
Too many duplicates!!

So I created a batch file that for every image-video file type in my backup , will move only one in a new folder, sorted by daytime taken and in a folder of that year, then it renamed all with the datetime of the file.

Now I have a great backup sorted by year in folders since 2003. Just saved me from 2 terabyte duplicates and I have now 600gb sorted backup files!

Once took over a bunch of webservices from a coworker who had found another job.
Worst bunch of copy-pasted, anti-pattern, duplicates-ridden, dead-code infected spaghettimess I ever saw.
I spent a year getting that shit into a semblance of order. You could tell for certain the guy had no professional pride or sense of maintainability.

The kicker is, after that year of getting everything in shape, I got laid off due to cutbacks.

'Hey, great work, Refactor Ninja!
Now leave, you must be needed elsewhere!'

🙄

Argghhhhhh!

Non-unique Multi-Index killing me...

Even when there is no duplicates found in the data frame.
What the hell am I missing..

I can now appreciate some design decisions behind react-redux after witnessing some angular OOP clusterfuck.

I am sure there is some clean/correct way to code in angular, but everyone is treating angular as java.

Some angular application (the one I have to work with) is littered with network calls. It's difficult to spot duplicates. People usually resolve promises everywhere. In services, in a top-level component, or in for loops. In react, people use apollo/redux-query or redux-saga to handle network calls. Since these libraries prevent duplicate network calls internally and reassigning apollo network call function or redux action function is always useless, it's easy to spot all network calls in a component tree.

In angular, it's difficult to trace data mutations when data can be updated everywhere. In react, you can easily find UI state updates by tracing state hooks/dispatch/apollo usages.

In angular, it's difficult to trace data pipeline. Since everything is imperative by default, people need to add update functions in data subscriptions. With all the littered mutations. Soon you will lose track of what the fuck is going on.

I hope angular get the agonizing death it deserves and fuck everyone who codes JS OOP clusterfuck UI.

It reaaaally annoys me when my business logic is sound but the data is corrupted.

For example, find duplicates in a HashMap<String>.. but I didn't take into account the input could contain a space either before or after.. so I end up wondering: if a HashMap only contains unique keys, how come the count of items in the map is the same as the count of the input keys?! Well.. spaces were the culprit.

"12345" != "12345 ".. and therefore the Map sees it as two distinct keys..

What an annoying bug.

Lesson learned: 1) Sanitize input first and never trust it. 2) Never make assumptions

We noticed that in our landing directory we were receiving duplicate files.

I asked the source to investigate.

He told me that the issue was not at his end. He asked me to mark the issue has been resolved from his end. I refused.

We get on a call to debug the issue. After 30mins he is extremely frustrated. As he was sharing his screen, he runs the command `ls -ltr | uniq -cd` on his server which sends the files and then screams at me "Where are the duplicates? Show me. Check the output. There are no duplicates.".

I first muted the call. Had a good laugh. Made him repeat it to show my team mates. They had a good laugh too.

I then asked him to call it a day. And once you cool down, think about what you just showed me.

Made a simulation/game in java using swing that runs on this algorithm:

-2D array was made (kinda like a chessboard).
-Random living cell was placed on the board.
Repeat:
-If a cell has X or less cells around it (living) it duplicates.
-If a cell has Y or more cells around it, it dies.

I was amazed at the types of shapes this created. There were so many variables I could play with, and probably spent hours just experiementing. I was really satisfied in the end. 😄

I'm interning at a mech eng company. Our products have many possible permutations that customers can choose from a spec sheet.

The backend for us mechanical designers is equivalent to copying and pasting the same code (with slight changes) into a massive switch statement depending on the program's options. So many near duplicate drawings. Each with individual settings that need to be tweaked and linked to other new duplicates every time a new order comes in.

As a programmer it drives me absolute bonkers! I've talked to them about automating it but "we've just always done it this way, so it probably won't change". Well, as soon as I'm done grinding this current project, I'm hoping to put together a practical demo to change their minds.

So I have a question to anyone familiar with the General Transit Feed Specification...

Why is the data provided in text files? Is there not a way to format the data to allow for random access to it?

Like I'm currently writing a transit app for a school project, and as far as I can tell, the only way to get all specific stops for a route, is to first look up all trips in a route, then look up all the stopids that are associated with a trip in stoptimes.txt (while also filtering out duplicates since the goal is to get stop ids, not specifically stop times) and then look up those stop ids in the stops.txt file.

The stoptimes file alone is over 500000 lines long, unless there is a way better way to be parsing the data that I'm not aware of? Currently I'm just loading the entire stoptimes file into a data structure in memory because the extra bit of ram used seems negligible compared to the load times I'm saving...

Would it be faster if I just parsed all the data once and threw it into a database? (And then updated the database once a month when the new data comes in?)

I have to build a database migration that generates user handles. The user handles are unique within an organization. The user can change them. The auto generated handles are either the first name + last name, or the business name depending on which user type it is. Unless it would be a duplicate. Duplicates auto increment if the handle is taken. The character limit for a user handle is the same length as first name plus last name so I have to check for possible overflow if I add digits. I also have to see if the generated name is in the DB already because a user could have custom entered the result of the auto generation.

This has to be programmed async. The DB driver is using a transaction but multiple calls have to be made to check if the generated handle exists for that organization. Also I have to check the migration script itself for possible duplicates. 3/4 of the users have a handle and with the scale there will definitely be duplicate names.

My idea is if there is a collision, use a UUID and let the users pick something nicer next time they log in. Business says “Reeeeeee!!!! The users shouldn’t see a UUID!!! You can do this!!!” Absurd uniqueness requirements. Absurd backfill procedure. Absurd business rules.

We had an issue where a query to a db replica set was returning duplicates randomly when paging. Aka each HTTP call for next N results was hitting different dbs with same/copy data.

No one could figure out why... I look at the query and ask where's the ORDER BY ID?

These guys were interviewing ppl last week and saying how even they could solve algo questions they were asking candidates.

And so to explain the problem, I'm like "tell me what's the difference between a list and a sorted list?"

#why algo questions suck at predicting job performance

My task for today: RDP into a vm of windows server 2008, use winSCP to ftp into the client's server, pull down approx 900 images, renaming the 200 or so duplicate filenames then upload them to a file sharing service...

THEN I have to pull them down onto my machine, optimise them, and then reverse the entire process, again manually renaming the duplicates.

As I sit here trying not to do the same shit as these idiots keep going crazy and driving me the same place I thought I’d browse the news and I saw a repeat in this precise life stealing scenario where people who are long past their expiration date pretend to be doing me a favor by destroying the evidence that they wronged me and everyone else that was an honest person and now can’t explain them fucking selves

60 year old 50 year olds and such

And a closed network of assholes that repeat the same shit and keep me trapped

I’m trying to live
To work
To progress
They made the country a prison withholding media news releases and keeping morons wandering around in a “stabilizing” way

Now they’re talking about a. Very real farce that is occurring in dc yet again
Open to half capacity after the farsical COVID scare again

How are so many people such brainwashed trash ?

Thinking they’re either winning or nothing better could happen while the next generation of dupes is inserted into the mix

It makes me angry

Meanwhile my fucked up chomo father and his surveillance duplicates... well it’s possible my actual father is dead at this point

Maybe

Still go about business as usual and I find myself wondering when younger people will wake up and when I’ll see some new fucking tv or movies at the very least since seemingly holding onto photos and videos and writing and personal effects seems a tad difficult

I swear to god I’ll be dead before I’m officially 50 !

Struggled whole day to remove duplicate objects from a list in java. Those objects (of a custom class) contain a unique id, the remaining properties can be identical across list items.
Wondered why hashset.add() always returns true and thus duplicates are still inserted.
Little did I forget about things like equals() and hashCode() 🤦

“Wet code”

Code that contains as many duplicates as possible.

Oh, now I understand why my past self had some pain in the neck in front of his computer. Twas his future self (the latest version of me writing this post) strangling him for the terabyte of duplicates of so many shit spread around my external storages.

Over 1 Terabyte of duplicate data among the 2.5 Tb in use, in the 3.5 Tb of maximum storage I have.

Question about linux iptables. I am currently blocking all access and whitelisting only when my users launch my software. When software is launched a socket client is also launched, it connects to socket server, identifies itself with a password and disconnects. If given password by socket client is correct, then socket server whitelists the users IP by executing the following command: " iptables -I INPUT -s userIP -j ACCEPT".

My problem is that now I have lots of duplicates of IP's whitelisted and as far as I've heard I should not go over 25k iptable rules.

So my question is how to check if ip is already whitelisted, in order to avoid duplicate iptable rules for for same IP?

Obvious solution would be to store whitelist somewhere (mysql/txt) and double check before whitelisting ip, but maybe there is an easier way to do this?

So how many generations of the same people are you raising simultaneously or are you people killing them in between passing all of us off as younger than we are ? Just curious
Wonder just how much of a group of monsters you all are since I’ve seen counting on 5 duplicates this far with periods of amnesia in between

Context: data warehouse

A colleague gets notified that his historisation produces duplicates. He checks the tables, sees that something has gone wrong ... and asks me for help.

I'm so sorry but I had to laugh so hard. Turns out everything crashed cause the value to be historised had changed back to an earlier version (which can happen). And now I'm here wondering how someone who calls himself a senior dev can create a historisation without taking care of gaps and islands. Seriously, how can you not think about that?