Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "puppeteer"
-
rant? rant!
I work for a company that develops a variety of software solutions for companies of varying sizes. The company has three people in charge, and small teams that each worked on a certain project. 9 months ago I joined the company as a junior developer, and coincidentally, we also started working on our biggest project so far - an online platform for buying groceries from a variety of vendors/merchants and having them be delivered to your doorstep on the same day (hadn't been done to this scale in Estonia yet). One of the people from management joined the team working on that. The company that ordered this is coincidentally being run by one of the richest men in Estonia. The platform included both the actual website for customers to use, a logistics system for routing between the merchants, the warehouse, and the customers, as well as a bunch of mobile apps for the couriers, warehouse personnel, etc. It was built on Node.js with Hapi (for the backend stuff), Angular 2 (for all the UIs, including the apps which are run through a WebView wrapper), and PostgreSQL (for the database). The deadline for the MVP we (read: the management) gave them, but we finished it in about 7 months in a team of five.
The hours were insane, from 10 AM to 10 PM if lucky. When we weren't lucky (which was half of the time, if not more), we had to work until anywhere from 12 PM to 3 AM, sometimes even the whole night. The weekends weren't any better, for the majority of the time we had to put in even more extra hours on the weekends. Luckily, we were paid extra for them, but the salary was no way near fair (the majority of the team earned about 1000€/mo after taxes in a country where junior developers usually earn 1500€/month). Also because of the short deadline given to us, we skipped all the important parts like writing tests, doing CI, code reviews, feature branching/PR's, etc. I tried pushing the team and the management to at least write tests and make feature branches/PRs, but the management always told me that there wasn't enough time to coordinate and work on all that, that we'll do that after launching the MVP, etc. We basically just wrote features, tested them by hand, and pushed into the "test" branch which would later get tested and merged into master.
During development, one of the other juniors managed to write the worst kind of Angular code you could imagine - enormous amounts of duplication, no reusable components (every view contained the everything used in the view, so popups and other parts that should logically be reusable were in every view separately), fuck - even the HTML was broken (the most memorable for me were the "table > tr > div > td" ones, but that's barely scratching the surface). He left a few months into the project, and we had to build upon his shit, ever so slightly trying to fix the shit he produced. This could have definitely been avoided if we did code reviews.
A month after launching the MVP for internal testing, the guy working on the logistics system had burned out and left the company (he's earning more than twice the salary he got here, happy for him, he is a great coder and an even better team player). This could have been avoided if this project had been planned better, but I can't really blame them, since it was the first project they had at this scale (even though they had given longer deadlines for projects way smaller than this).
After we finished and launched the MVP, the second guy from management joined, because he saw we needed extra help. Again I tried to push us into investing the time to write tests for the system (because at this point we had created an unstable cluster fuck of a codebase), but again to no avail. The same "no time, just test it manually for now, we'll do that later when we have time" bullshit from management.
Now, a few weeks ago, the third guy from management joined. He saw what a disaster our whole project was. Him joining was simply a blessing from the skies. He started off by writing migrations using sequelize. I talked to him about writing tests and everything, and he actually listened. He told me that I'm gonna be the one writing them, and also talked to the rest of management about it. I was overjoyed. I could actually hear the bitterness in the voices of the rest of management when they told me how to write the tests, what to test, etc. But I didn't give a flying rat's ass, I was hapi.
I was told to start off by writing a smoke test for the whole client flow using Puppeteer. I got even happier, since I was finally able to again learn new things (this stopped at about 4 or 5 months into the project).
I'm using jest as the framework and started writing the tests in TypeScript. Later I found a library called jest-extended, but it didn't have type defs, so I decided to write them and, for the first time in my life, contribute to the open source community.19 -
That feeling when you’re scraping a website to build an API and your script downloads 4049/6170 pages before failing and you have to rewrite it so that puppeteer hits the next button 4049 times before executing the script. 😅
This database is so frustrating.
I hate this website (the one I’m scraping).
It’s going to be so satisfying when this is finished.6 -
Joined a startup which does webscraping. I never knew that webscraping can be so much monotonous. I feel like I am doing a damn support job. Apart from that websites keep blocking the scrapers, and that is another thing to be fixed. jesus christ, non devs keep asking for ETA about how long would it take to complete a certain task. Thanks to remote work, I have been burnt out every day, and have been working for 14+ hours everyday!!!!! God help me!5
-
Created a simple bot for an online game using puppeteer.
After an evening (and night) of dev and debugging (quite some rejected promise errors), it worked fine and was ready for a 10-minutely cron job.
Fixed a couple bugs in the first three hours. Then started playing minecraft, which lagged like hell.
Opened task manager and saw a list of about 25 headless chrome processes. They had not been closed because of unhandled errors before the close method call 😵
Now added some basic error handling ☺2 -
Wanted to scrub my presence off of Facebook, but wanted to keep the account to stay in touch with friends.
That's why I built a small command-line tool to automate the deletion of my Facebook posts using Node.js & Puppeteer, in order not to resort to using third-party apps that you hand over your credentials to. What do you guys think?
https://github.com/ar-maged/...2 -
So, in Germany apprentices at companies need to file a "Berichtsheft".
It's a thing where you have to file, for each day that is, what you did at work or in job college and how long you did it.
Basically every company keeps records of their employees activities in their CRM or other management system and all schools use services for keeping timetables that include lesson duration and activity.
So why the fuck do we apprentices have to write that shit ourselves when we could literally just acces the databases and SELECT THE SHIT FROM FILED_ACTIVITIES, I thought.
So I'm writing scripts to acces our CRM database and a puppeteer script now that scrapes the Untis (online timetable service for schools) timetables to extract everything, group it by date and format it nicely as CSV.
I'm sick of this: Digital system & Digital system = write it yourself bullshit.
Once I'm done I'll make a github repo for the Untis scraper.
Also, I'll be making the tools usable for the other apprentices at my company to spare them the suffering.9 -
Gotta love software engineers. I'm writing some python to generate PDFs, so start by looking at chrome command line, which will be some headache with temp files and other madness.
Then find puppeteer, which does exactly what I want, but requires NodeJS.
Then find the unofficial python port, with a 5 line example of pretty much what I need to do.5 -
Perfect my script to automate YouTube videos to tiktok. The script would take a YouTube url as input and then split the video into 3 minute long pieces ( the longest time allowed for tiktok ) and post it to tiktok. I’ve gotten it before but I had to do some fucked shit bc I couldn’t get puppeteer to bypass tiktok’s bot security so it’s super inconsistent1
-
Browser automation is a PITA. I’m going on my fourth side mission with this crap and I honestly still look like a newbie. I’ve tried Java Selenium with Chrome, Excel VBA with IE9, Vanilla JS in the browser console, and tonight I’m thinking to concoct some kind of hybrid CDP & Selenium approach in Chrome. Never used CDP before, not even sure where to start but I heard it sucks like anything else unless you get some extra libraries and plugins and stuff.
It doesn’t help that I can’t get just anything I want from our IT Department. It would be another PITA to ask for puppeteer. If puppeteer is totally legit please let me know.
Selenium sucks. The buttons don’t click, the waits don’t wait. Its unusable. Iframes are annoying as all hell but I can deal with that. HTML Tables suck too. It doesn’t help I have to restart my whole java program and whole Chrome every time an element doesn’t get picked correctly. Scripting one single element can take all fucking night.
Chrome dev tools what the fuck. Why the fuck is the DOM explorer in the same window as the web page I’m working on?? I can’t undock it. Am I supposed to use a fucking TV screen to work with this bastard?? If I use the remote chrome tools on port 9225 or whatever - It Still Renders The Whole Fucking Page Alongside The Console. Get Out Of My Way!!! The nested HTML CODE IS ONE CHARACTER WIDE ALL THE TIME. I can’t for the life of me figure out what the fuck I’m looking at. Haven’t you people ever heard of A HORIZONTAL SCROLL BAR at least.
Fuck I tried using getElementById, and the Xpath thing and its not all that great seeing I have seemingly 1000s of nested Divs all over the god damned place oftentimes containing a single element. I’m finally on chrome now should I learn Jquery now? I mean seriously wtf.
I use this one no code tool for dev it has web automation built in. As you can imagine its just as broken as anything else!! I have 10 screens to navigate it gets stuck on the second screen all the damn time. Fuck I love clicking the buttons when my script misses and playing catch up with it.
So as a work around to Selenium not waiting even 1 millisecond when I use explicit wait or implicit wait or fluent wait, I’m guessing maybe I can attach both Chrome Dev Tools Protocol (CDP as ive called it earlier) and selenium to the same browser and maybe I can use CDP to perform a Wait with any degree of success. Selenium will do nothing more than execute vanilla javascript Element.click(); This is the only way I know to even ACTUALLY use selenium beyond the simplest html documents possible. Hell I guess CDP can execute js idk.
I can’t get the new selenium that has CDP but I do have some buggy ass selenium from a few years back. Yeah, I remember reading there was a pretty impactful regression defect in the version I have. Maybe I’m being gaslighted by some shit copy of selenium?
The worst part is that I do seem to be having issues that the rest of the internet’s devs do not seem to be having. People act like browser automation is totally viable and pretty OK. How in the fuck hell is my Selenium Test Suite going to be more reliable my application under test?!!?? I’ll have more fucking bugs in my test suite than in my application. Today, I have less than half a test script and, I. already. fucking. do.
I am still SUPER PISSED at the months of 12 hour days (always 8 hours spent on normal sprint work btw only 4 to automation) I spent trying to automate our regression tests. I got NOWHERE.
I did learn a lot about HTML and JS though like I’m not that mad…but I’m just trying to emphasize my achievement on my task was zero.
The buttons don’t click. There are so many divs and I swear you sometimes need to select a div somewhere in the middle sometimes to get it working. The waits don’t wait. XHR requests are invisible. Java crashes 100 times before I find an xpath and thread.sleep() combo that works. I have no failure modes to use — Sometimes I click the same element 20x in a script because I have no way to know if it clicked the first time! Sometimes you gotta scroll the page to make the click work. So many click methods all broken. So many wait methods all broken. Its not just the elements don’t click! There are so many ways to click that almost work but surely they all fail the same in the end. ok at this point I’m just repeating myself…
there yet even more issues that I can’t remember…and will soon remember as I journey into this project yet again…
thanks for reading I hope I entertained and would love to hear your experience!5 -
Hi
any of you guys had problems with electron/puppeteer, especially trying to load pages using proxies
I'm using arch linux3 -
(Hackerman intensifies)
Sadly that's all I've implemented so far and I don't have a lot of time for these things :-/
But damn puppeteer is cool!
shit I mixed up kissmanga and kissanime... -
PHP gurus / masochists.
I've been using Symfony components for new, isolated features in a legacy php application for awhile now. the time has come to integrate using the kernel, and routing for new endpoints while existing endpoints use the existing apache means of loading pages.
It's not my first rodeo doing this, but I'd appreciate any wisdom/resources/patterns you followed for anyone who's had to do the same.
My clients don't have the means to do hire the appropriate ammount of devs to do a proper port, so this is a long path towards modernization by ceasing to bolt on features to existing code and instead, when working on something, updating it to the new design pattern and then extending that, with a spec, documentation and code coverage.3 -
For web automation, what do you prefer?
1. Playwright
2. Puppeteer
3. Selenium
4. Something else
&
1. Python
2. TypeScript
3. JavaScript
4. Something else9 -
Switched from recording based automation tool (ranorex) to script based (puppeteer) for a web app. Risky because everyone was against this and I had to deliver.
-
Puppeteer, where have you been all my life?
Also anyone get a chance to check out cypress.io and uilicious and if so, what were your experiences? what would you recommend? -
Should I switch to Chrome Headless/Puppeteer for webpage header+footer scraping or stick with Express+Cheerio?1