Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "puppeteer"
I work for a company that develops a variety of software solutions for companies of varying sizes. The company has three people in charge, and small teams that each worked on a certain project. 9 months ago I joined the company as a junior developer, and coincidentally, we also started working on our biggest project so far - an online platform for buying groceries from a variety of vendors/merchants and having them be delivered to your doorstep on the same day (hadn't been done to this scale in Estonia yet). One of the people from management joined the team working on that. The company that ordered this is coincidentally being run by one of the richest men in Estonia. The platform included both the actual website for customers to use, a logistics system for routing between the merchants, the warehouse, and the customers, as well as a bunch of mobile apps for the couriers, warehouse personnel, etc. It was built on Node.js with Hapi (for the backend stuff), Angular 2 (for all the UIs, including the apps which are run through a WebView wrapper), and PostgreSQL (for the database). The deadline for the MVP we (read: the management) gave them, but we finished it in about 7 months in a team of five.
The hours were insane, from 10 AM to 10 PM if lucky. When we weren't lucky (which was half of the time, if not more), we had to work until anywhere from 12 PM to 3 AM, sometimes even the whole night. The weekends weren't any better, for the majority of the time we had to put in even more extra hours on the weekends. Luckily, we were paid extra for them, but the salary was no way near fair (the majority of the team earned about 1000€/mo after taxes in a country where junior developers usually earn 1500€/month). Also because of the short deadline given to us, we skipped all the important parts like writing tests, doing CI, code reviews, feature branching/PR's, etc. I tried pushing the team and the management to at least write tests and make feature branches/PRs, but the management always told me that there wasn't enough time to coordinate and work on all that, that we'll do that after launching the MVP, etc. We basically just wrote features, tested them by hand, and pushed into the "test" branch which would later get tested and merged into master.
During development, one of the other juniors managed to write the worst kind of Angular code you could imagine - enormous amounts of duplication, no reusable components (every view contained the everything used in the view, so popups and other parts that should logically be reusable were in every view separately), fuck - even the HTML was broken (the most memorable for me were the "table > tr > div > td" ones, but that's barely scratching the surface). He left a few months into the project, and we had to build upon his shit, ever so slightly trying to fix the shit he produced. This could have definitely been avoided if we did code reviews.
A month after launching the MVP for internal testing, the guy working on the logistics system had burned out and left the company (he's earning more than twice the salary he got here, happy for him, he is a great coder and an even better team player). This could have been avoided if this project had been planned better, but I can't really blame them, since it was the first project they had at this scale (even though they had given longer deadlines for projects way smaller than this).
After we finished and launched the MVP, the second guy from management joined, because he saw we needed extra help. Again I tried to push us into investing the time to write tests for the system (because at this point we had created an unstable cluster fuck of a codebase), but again to no avail. The same "no time, just test it manually for now, we'll do that later when we have time" bullshit from management.
Now, a few weeks ago, the third guy from management joined. He saw what a disaster our whole project was. Him joining was simply a blessing from the skies. He started off by writing migrations using sequelize. I talked to him about writing tests and everything, and he actually listened. He told me that I'm gonna be the one writing them, and also talked to the rest of management about it. I was overjoyed. I could actually hear the bitterness in the voices of the rest of management when they told me how to write the tests, what to test, etc. But I didn't give a flying rat's ass, I was hapi.
I was told to start off by writing a smoke test for the whole client flow using Puppeteer. I got even happier, since I was finally able to again learn new things (this stopped at about 4 or 5 months into the project).
I'm using jest as the framework and started writing the tests in TypeScript. Later I found a library called jest-extended, but it didn't have type defs, so I decided to write them and, for the first time in my life, contribute to the open source community.20
That feeling when you’re scraping a website to build an API and your script downloads 4049/6170 pages before failing and you have to rewrite it so that puppeteer hits the next button 4049 times before executing the script. 😅
This database is so frustrating.
I hate this website (the one I’m scraping).
It’s going to be so satisfying when this is finished.6
Joined a startup which does webscraping. I never knew that webscraping can be so much monotonous. I feel like I am doing a damn support job. Apart from that websites keep blocking the scrapers, and that is another thing to be fixed. jesus christ, non devs keep asking for ETA about how long would it take to complete a certain task. Thanks to remote work, I have been burnt out every day, and have been working for 14+ hours everyday!!!!! God help me!5
Wanted to scrub my presence off of Facebook, but wanted to keep the account to stay in touch with friends.
That's why I built a small command-line tool to automate the deletion of my Facebook posts using Node.js & Puppeteer, in order not to resort to using third-party apps that you hand over your credentials to. What do you guys think?
Created a simple bot for an online game using puppeteer.
After an evening (and night) of dev and debugging (quite some rejected promise errors), it worked fine and was ready for a 10-minutely cron job.
Fixed a couple bugs in the first three hours. Then started playing minecraft, which lagged like hell.
Opened task manager and saw a list of about 25 headless chrome processes. They had not been closed because of unhandled errors before the close method call 😵
Now added some basic error handling ☺2
any of you guys had problems with electron/puppeteer, especially trying to load pages using proxies
I'm using arch linux3
Sadly that's all I've implemented so far and I don't have a lot of time for these things :-/
But damn puppeteer is cool!
shit I mixed up kissmanga and kissanime...
PHP gurus / masochists.
I've been using Symfony components for new, isolated features in a legacy php application for awhile now. the time has come to integrate using the kernel, and routing for new endpoints while existing endpoints use the existing apache means of loading pages.
It's not my first rodeo doing this, but I'd appreciate any wisdom/resources/patterns you followed for anyone who's had to do the same.
My clients don't have the means to do hire the appropriate ammount of devs to do a proper port, so this is a long path towards modernization by ceasing to bolt on features to existing code and instead, when working on something, updating it to the new design pattern and then extending that, with a spec, documentation and code coverage.3
Switched from recording based automation tool (ranorex) to script based (puppeteer) for a web app. Risky because everyone was against this and I had to deliver.
Puppeteer, where have you been all my life?
Also anyone get a chance to check out cypress.io and uilicious and if so, what were your experiences? what would you recommend?
Should I switch to Chrome Headless/Puppeteer for webpage header+footer scraping or stick with Express+Cheerio?1