Recently we got a new project assigned and as always you are hyped, really really hyped...........

We were supposed to find all kind of driver updates (especially bios ones) for all devices the company owns. So first of all we thought:

EAAAASY! A little bit of web crawling, regex, etc.
.
.
.
.
B
U
U
U
U
T
!

We were sooooo soooo wrong these fucking manufacturer websites are absolutely awful to crawl or parse and nowadays there are no proper FTP Servers or something else anymore you could use to get the information. Every subsite is little bit different...
While coding and literally brute forcing possible urls (there was some kind of vague pattern) we learned AGAIN to appreciate proper developed and designed websites. Especially by devs who may have some more usage scenarios in mind for their site than simple human clients.

So thank you to all of you awesome web developers who design proper websites and web tools!

All in all it took us 2 weeks to come up with a proper solution (by the way we are a smal team of 3 devs) which somewhat works reliable and can deal with site changes etc.