Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "browser automation"
-
At the end of this exercise, you will have your very own ‘bot’, capable of collecting any kind of information you want from the internet, into your own database. Sounds like a great way to get sued for Copyright infringement! Let’s get started!
https://franciskim.co/how-to-data-c...2 -
Best part about the covid19 manufactured crisis?
Liquor stores deliver. Worst part about liquor stores delivering? Needing to use their shoddy websites.
I've been using a particular store (Total Wines) since they're cheaper than the rest and have better selection; it's quite literally a large warehouse made to look like a store.
Their website tries really hard to look professional, too, but it's just not. It took me two days to order, and not just from lack of time -- though from working 14 hour days, that's a factor.
Signing up was difficult. Your username is an email address, but you can't use comments because the server 500s, making the ajax call produce a wonderfully ambiguous error message. It also fades the page out like it's waiting on something, but that fade is on top of the error modal too. Similar error with the password field, though I don't remember how I triggered it.
Signing up also requires agreeing to subscribe to their newsletter. it's technically an opt-in, but not opting-in doesn't allow you to proceed. Same with opting-in to receiving a text notification when your order is ready for pickup -- you also opt-in to reciving SMS spam.
Another issue: After signing up, you start to navigate through the paginated product list. Every page change scrolls you to the exact middle of the next page. Not deliberatly; the UI loads first, and the browser gets as close as it can to your previous position -- which was below that as the pagination is at the bottom -- and then the products populate after. But regardless of why, there is no worse place to start because now you must scroll in both directions to view the products. If it stayed at the very bottom, it would at least mean you only need to scroll upwards to look at everything on the page. Minor, but increasingly irritating.
Also, they have like 198 pages of spirits alone because each size is unique entry. A 50ml, 350ml, 500ml, 750ml, 1000ml, and 1750ml bottle of e.g. Tito's vodka isn't one product, it's six. and they're sorted seemingly randomly. I think it's by available stock, looking back.
If you fancy a product, you can click on it for a detail page. Said detail page lists the various sizes in a dropdown, but they're not sorted correctly either, and changing sizes triggers a page reload, which leads to another problem:
if you navigate to more than a few pages within a 10 or so second window, the site accuses you of using browser automation. No captcha here, just a "click me for five seconds" button. However, it (usually) also triggers the check on every other tab you have open after its next nagivation.
That product page also randomly doesn't work. I haven't narrowed it down, but it will randomly decide to start failing, and won't stop failing for hours. It renders the page just fine, then immediately replaces it with a blank page. When it's failing, the only way to interact with the page is a perfectly-timed [esc], which can (and usually does) break all other page functionality, too. Absolutely great when you need to re-add everything from a stale copy of your signed-out cart living in another tab. More on that later. And don't forget to slow down to bypass the "browser automation" check, too!
Oh, and if you're using container tabs, make sure to open new tabs in the SAME container, as any request from the same IP without the login cookie will usually trigger that "browser automation" response, too.
The site also randomly signs you out, but allows you to continue amassing your cart. You'd think this is a good thing until you choose to sign in again... which empties your cart. It's like they don't want to make a sale at all.
The site also randomly forgets your name, replacing it with "null." My screen currently says "Hello, null". Hello, cruft!
It took me two days to order.
Mostly from lack of time, as i've been pulling 14 hour shifts lately trying to get everything done. but the sheer number of bugs certainly wasted most of what little time i had left. Now I definitely need a drink.
But maybe putting up with all of this is worthwhile because of their loyalty program? Apparently if you spend $500, you can take $5 off your next purchase! Yay! 1%! And your points expire! There are three levels; maybe it gets better. Level zero is for everyone; $0 requirement. There are also levels at $500 and $2500. That last one is seriously 5x more than the first paid level. and what does it earn you? A 'free' magazine subscription, 'free' classes (they're usually like $20-$50 iirc), and a 'free' grab bag (a $2.99 value!) twice per month. All for spending $2500. What a steal. It reminds me of Candy Crush's 3-star system where the first two stars are trivial, and the third is usually a difficult stretch goal. But here it's just thinly-veiled manipulation with no benefit.
I can tell they're employing some "smarketing" people with big ideas (read: stolen mistakes), but it's just such a fail.
The whole thing is a fail.8 -
PORTFOLIO INFLATION
when every junior is writing algorithms, the next step up, the only way to keep up is writing apps. When every junior is writing apps, the next leg up is writing an entire SN.
Eventually junior full stack devs are writing microservice streaming cloud backend content delivery optimized social networks wrapped in virtualization with load balancing, proper CI, public accessible analytics apis, written in custom webaseembly compiled scripting backend utilizing both the latest graphql and every single feature of postgres, while also being a web site builder, an in browser app, mobile optimized, designed to transmogrify your asset pipelines linearflow functional-oriented modular rust cratified turbencabulator while cooking your turducken with CPU cycles, diffusing your gpt, and finetunning your llama 69 trillion parameter AI model to jerk you off all at the same time.
And then the title "wizard" becomes a reality as the void of meaning in our lives occupied by the anxiety of trying to reduce the fear of rejection in job hunting, is subsumed by the brief accidental glance into the cthulian madness-inducing yawning abyss of the future which is all the rest of our lives we have to endure existing for until at last sweet sweet death consumes us and we go to annihilation never having to configure one more framework or devops deploy of another virtual environment.
And it dawns on us that we no longer develop or write code at all. No, everything has become a "service" in this new hellscape future. We slowly come to the realization that every job is really just Costco greeter, or eventually going to be reduced to something equivalent, all human creativity, free will and emotions now taken care of by the automation while we manage the human aspects, like sardines pushing against one another not realizing their doom has been sealed along with the airless can they have been packed into, to be suffocated by circumstance and a system designed to reduce everything to a competition of metrics designed by the devil, if the metrics were misery", and "torture", while we ourselves are driven by this ratfuck wheel to turn endlessly toward social cannibalism, like rats eating their babies, but for the amusement of wallstreet corporate welfare whores who couldnt turn a dime if it wasnt already stolen.
And on our gravestones, those immortal words are carved, by the last person who gave up the ghost, the last whose soul wasnt yey shovelled onto the coal fires driving the content machine consuming the world:
Welcome to costco. I love you.12 -
I'm a jr developer. I started off in automation testing and don't mind it but the testing codebase is cancer, doesn't follow basic Java conventions even basic naming conventions like camelcase, and the tests are super slow using hardcoded Thread.sleep(). Since the automation tests are not automated, I have to run manually. YES manually, every morning I wake up early at 7am to run the 2.5 hour long tests (7am because this before people get to work and when the application goes back online). I run this bitch and monitor them but most of them fail anyways. I also have to write a email report on the results which means I have to explain why shit is failing so I have to debug all this crap. This shit literally eats up an additional 2-3 hours of my work day everyday and the time is not even accounted for. ALSO, since it's running on my laptop, it makes my computer slow most of the day. If I have to debug, I can't have the browser be headless so fuckin chrome browsers be popping up every 2 minutes. I did this for legitimately 8 sprints until I decided enough was enough and bitched about it and the team told me I had no choice. I eventually got them to push towards automating it but it's still in progress so I'm still running this dumb shit. The contractors try to take advantage of me any way they can by giving me mindless bitch work they don't want and they know I don't usually say no since I'm a jr resource. I hate running the fucking automation tumor. Sometimes I go into the meeting rooms alone to scream.
I feel like I'm wasting my life away and not learning as much as I could somewhere else10 -
Product Management thought of automating an entire legacy product so they funded undisclosed amount to program management who in turn hired >20 contract devs managed by architects and dev managers with zero functional or technical knowledge of product and who in turn went ahead automating the product in selenium, end result of which was an useless automation framework with lot of browser specific dependencies and whuch could run only on one setup environment and migrating test cases to another environment and running is almost impossible and tyrannical to configure. The automation test cases are highly disorganized with all generic setup, DB configurations and business case test data mixed up in same config files and which need to be rewriten every time ported from one environment to other.To add misery to my woes as a dev working in that product I was told to utilize that framework and enhance the quality of my code by writing inline automation Cases for the same. I am left speechless thoughtless and emotionless after that decision.2
-
TLDR;
How much do you earn for your skill set in your country vs your cost of living?
BONUS;
See how much I & others earn.
Recently I became aware of just how massive the gap in developers earnings are between countries. I'd love to calculate a fixed score for income vs cost of living.
I know this stuff is sensitive to some so if you prefer just post your score (avg income p/m after tax / cost of living).
I'm not shy so I'll go first:
MY RATES
Normal Rate (Long term): $23
Consulting / Short term: $30-$74
Pen Test: $1500 once off.
Pen Test Fixes: consulting rate.
Simple work/websites: min $400+
Family & Friends: Dev friends are usually free (when mutually beneficial). Family and others can fuck off, even if they can pay (I pass their info to dev friends with fair warning).
GENERAL INFO
Experience: 9 years
Country: South Africa
Developer rareness in country: Very Rare (+-90 job openings per job seeker).
Middle class wage in country: $1550 p/m (can afford a new car, decent apartment & some luxuries like beer/eating out).
Employment type: Permanent though I can and do freelance occasionally.
Client Locality: Mostly local.
Developer Type: Web Developer (True web dev - I do anything web related from custom HTTP servers to sockets, services, advanced browser api's, apps & more).
STACKS / SKILLSETS
I'M PROFICIENT IN:
python, JavaScript, ASP classic, bash, php, html, css, sql, msql, elastic search, REST, SOAP, DOM, IIS, apache
I DABBLE WITH:
ASP.net, C++, ruby, GO, nginx, tesseract
MY SPECIALTIES:
application architecture, automation, integrations, db's, real time data, advanced browser apps/extensions (webRTC, canvas etc).
SUMMARY
Avg income p/m after tax: $2250
Cost of living (car+rent+food): $1200
Score: 1.85
*Note: For integrity when calculating my cost of living I excluded debt repayments and only kept my necessities which are transport, food & shelter.
I really hope you guy's post your results, it would be great to get an idea of which is really the worst / best country to be a developer in.20 -
hey is there a non code browser automation tool? one with a record and play button?
I have to "copy / create" 16 drupal pages with a gazillion modules manually by hand. And each page takes an hour of tedious click work.
Thank you for your help ❤️10 -
Considering applying to a regular administrative job, where I can use just 1% of my dev skills in BAT files, Excel macros, browser automation with Selenium, and people will be like "oh man, you are like a hacker!!!"1
-
A house.. or lego star destroyer..😝😆
Joke aside, I'd rewrite current project from scratch to get rid of all the automation and IE dependancies and make it cross browser and all the dead code...and all excess ORMs and especially EFCF..and also make support for db diff than oracle..and no need for ora client installation..
Yeah, I'm a work junky, I have no projects of my own.. one kid is demanding enough of my time.. 😉3 -
Good evening programmers, IT's, devranters and memeians.
I would like to use a little bit of your collective conciousness - the hive mind if you will.
I've been working on my automation system for quite a while and I've received some exposure from non-programmers - which resulted in more questions than suggestions.
I would like to ask you guys to give me some suggestions as to what I could add to my system.. that is, if you have time..
The program in short (if you don't want to read the readme file) is an automation system scriptable in pure Lua.
It utilizes Selenium for web automations, NAudio for audio operations and Moonsharp as an interpreter.
While my tester friends say that they use it for the actual testing, I myself found it very useful in writting bots (for browser games for example).
Here's the github link: https://bit.ly/2GDu92g
Thanks a ton!
PS. Here's an unrelated image to draw your attention.6 -
Browser automation is a PITA. I’m going on my fourth side mission with this crap and I honestly still look like a newbie. I’ve tried Java Selenium with Chrome, Excel VBA with IE9, Vanilla JS in the browser console, and tonight I’m thinking to concoct some kind of hybrid CDP & Selenium approach in Chrome. Never used CDP before, not even sure where to start but I heard it sucks like anything else unless you get some extra libraries and plugins and stuff.
It doesn’t help that I can’t get just anything I want from our IT Department. It would be another PITA to ask for puppeteer. If puppeteer is totally legit please let me know.
Selenium sucks. The buttons don’t click, the waits don’t wait. Its unusable. Iframes are annoying as all hell but I can deal with that. HTML Tables suck too. It doesn’t help I have to restart my whole java program and whole Chrome every time an element doesn’t get picked correctly. Scripting one single element can take all fucking night.
Chrome dev tools what the fuck. Why the fuck is the DOM explorer in the same window as the web page I’m working on?? I can’t undock it. Am I supposed to use a fucking TV screen to work with this bastard?? If I use the remote chrome tools on port 9225 or whatever - It Still Renders The Whole Fucking Page Alongside The Console. Get Out Of My Way!!! The nested HTML CODE IS ONE CHARACTER WIDE ALL THE TIME. I can’t for the life of me figure out what the fuck I’m looking at. Haven’t you people ever heard of A HORIZONTAL SCROLL BAR at least.
Fuck I tried using getElementById, and the Xpath thing and its not all that great seeing I have seemingly 1000s of nested Divs all over the god damned place oftentimes containing a single element. I’m finally on chrome now should I learn Jquery now? I mean seriously wtf.
I use this one no code tool for dev it has web automation built in. As you can imagine its just as broken as anything else!! I have 10 screens to navigate it gets stuck on the second screen all the damn time. Fuck I love clicking the buttons when my script misses and playing catch up with it.
So as a work around to Selenium not waiting even 1 millisecond when I use explicit wait or implicit wait or fluent wait, I’m guessing maybe I can attach both Chrome Dev Tools Protocol (CDP as ive called it earlier) and selenium to the same browser and maybe I can use CDP to perform a Wait with any degree of success. Selenium will do nothing more than execute vanilla javascript Element.click(); This is the only way I know to even ACTUALLY use selenium beyond the simplest html documents possible. Hell I guess CDP can execute js idk.
I can’t get the new selenium that has CDP but I do have some buggy ass selenium from a few years back. Yeah, I remember reading there was a pretty impactful regression defect in the version I have. Maybe I’m being gaslighted by some shit copy of selenium?
The worst part is that I do seem to be having issues that the rest of the internet’s devs do not seem to be having. People act like browser automation is totally viable and pretty OK. How in the fuck hell is my Selenium Test Suite going to be more reliable my application under test?!!?? I’ll have more fucking bugs in my test suite than in my application. Today, I have less than half a test script and, I. already. fucking. do.
I am still SUPER PISSED at the months of 12 hour days (always 8 hours spent on normal sprint work btw only 4 to automation) I spent trying to automate our regression tests. I got NOWHERE.
I did learn a lot about HTML and JS though like I’m not that mad…but I’m just trying to emphasize my achievement on my task was zero.
The buttons don’t click. There are so many divs and I swear you sometimes need to select a div somewhere in the middle sometimes to get it working. The waits don’t wait. XHR requests are invisible. Java crashes 100 times before I find an xpath and thread.sleep() combo that works. I have no failure modes to use — Sometimes I click the same element 20x in a script because I have no way to know if it clicked the first time! Sometimes you gotta scroll the page to make the click work. So many click methods all broken. So many wait methods all broken. Its not just the elements don’t click! There are so many ways to click that almost work but surely they all fail the same in the end. ok at this point I’m just repeating myself…
there yet even more issues that I can’t remember…and will soon remember as I journey into this project yet again…
thanks for reading I hope I entertained and would love to hear your experience!5 -
Does anyone have a handy macro sort of quick automation windows desktop tool they use / like?
Scenario: I fix a rando thing on a rando web app. Now I need to go through 4 clicks in a browser and enter some text to make the thing happen and see the result....I'd rather not do that 8 times as I iterate / try stuff.
I've been playing with some more automated testing stuff but I'm not there yet and the granularity of those tends not to be in the area of "make quick task and watch it happen" kinda thing.4