!Rant

Wrote a crawler and now has 18 million records in the queue. About 500.000 files with metadata.
1 month until deadline and we have to do shit many things.

Now we discover we have a flaw in our crawler ( I don't see it as a bug ).. We don't know how much metadata we missed, but now we have to write a script that scrapes every webpage that we've already visited and get that metadata..

What's the flaw you ask? Some people find it funny to put capital letters in their attribute names.. *kuch* Microsoft.com!! *kuch*

And what didn't we do? We didn't lower case each entire webpage and then, only then, search the webpage for data..