Was exploring google foobar upon how it works and found out this strange
div:rhtext

wonder how they insert this code in html randomly.
xpath and ids are all obfuscated,unreadable

Ahh.. there is nothing like the joyous feeling of writing a working piece of code for your own personal projects.

I spent several weeks and a few hours today to finally get my Python automation script working and I am very proud of myself.

Here's what it does:
* open a text file, extract a specific string from it using rather complicated xpath
* open another text file and do the same
* replace result 1 with result 2
* log results
* close file
* automate the process

Even though it looks easy, I had to mess around with a lot of problems such as permissions, indentation, stream writing, file status, etc.

Now, instead of having to manually do this job, I can just let my machine do it!

// First rant
So I've spent the last three days trying to send requests to a website in C# (never had done that, so I had to learn from scratch) and using XPath to select certain nodes from the html.
Today, I port it to a new UWP project and turns out it doesn't support XPath. I guess I should learn LINQ now...fml

Today I wanted to automate checking the status of running jobs. There's a UI but navigating it is tedious and I sort of wanted a Dashboard.

So I started writing a Selenium app to automate it. I'm writing the code, inspecting all the inputs. And I'm looking "woah, all of them have nice ids... This should be easy"

Login is ok, but then run into IFrames (can't find the input elements on the page itself).

Fine Google... find the answer and so try again. But wait... It doesn't work???

There's apparently 9 of them on the page. And there's no easy way to identify the correct one...

Spent an hr trying different ways of locating the right one... eventually settling on very complex XPath...

All in all though it took 2 hours... And still not done...

"DefiantJS enables you to perform lightning-fast searches on JSON using XPath expressions, and transform JSON using XSL."

True if lightning takes for ages. What a disappointment :(

New Tester trying to get the xpath back from a Webelement by casting to By.

Something that would make things so much easier would be a Chrome plug-in that spit out Selenium code while in the browser.

Level of fuckity fuck mood.

After changing dozens of build plans in Bamboo, the build system of poo...

How to verify that nothing has gone wrong?

Poking the database, you'll be surprised that Bamboo stores the buildplan definition as XML.

Another surprise: Some of the keys / values have typos.

Yeah. You read that right. There are typos inside the XML...

Now together with Postgres, we can use XPATH and have some fun.

UNNEST(COALESCE(XPATH('/configuration/buildTasks/taskDefinition[userDescription[contains(text(),"Bleep")]]', build_definition.xml_definition_data::xml)::varchar[], ARRAY['']))

Lovely wrapping via coalesce for some null safety.

Now we get da task definitions for fields having user description text containing bleep.

Wrapping it in two REGEXP_REPLACE to strip out stupid identifiers....

REGEXP_REPLACE(REGEXP_REPLACE(...., '<id>\d+</id>', ''), '<oid>\d+</oid>', ''))

Then wrap that in MD5.

Boom. Lots of MD5 sums to help you identify if the configs are identical for a task or not.

Now wrapping that in another select to group by the MD5 and filter out the non identical ones.

I hate it how sometimes one has to seemingly do a full 2 hour dance for something as stupid as validation.

I'm pretty glad though for XML and XPATH.

Cause otherwise that would have been a whole can of worms I don't wanna think about....

Is there a HTML scraper library for JS/NPM like Selenium or HtmlAgilityPack where I can find elements by ID, XPath, element type and their attached attributes?

Browser automation is a PITA. I’m going on my fourth side mission with this crap and I honestly still look like a newbie. I’ve tried Java Selenium with Chrome, Excel VBA with IE9, Vanilla JS in the browser console, and tonight I’m thinking to concoct some kind of hybrid CDP & Selenium approach in Chrome. Never used CDP before, not even sure where to start but I heard it sucks like anything else unless you get some extra libraries and plugins and stuff.

It doesn’t help that I can’t get just anything I want from our IT Department. It would be another PITA to ask for puppeteer. If puppeteer is totally legit please let me know.

Selenium sucks. The buttons don’t click, the waits don’t wait. Its unusable. Iframes are annoying as all hell but I can deal with that. HTML Tables suck too. It doesn’t help I have to restart my whole java program and whole Chrome every time an element doesn’t get picked correctly. Scripting one single element can take all fucking night.

Chrome dev tools what the fuck. Why the fuck is the DOM explorer in the same window as the web page I’m working on?? I can’t undock it. Am I supposed to use a fucking TV screen to work with this bastard?? If I use the remote chrome tools on port 9225 or whatever - It Still Renders The Whole Fucking Page Alongside The Console. Get Out Of My Way!!! The nested HTML CODE IS ONE CHARACTER WIDE ALL THE TIME. I can’t for the life of me figure out what the fuck I’m looking at. Haven’t you people ever heard of A HORIZONTAL SCROLL BAR at least.

Fuck I tried using getElementById, and the Xpath thing and its not all that great seeing I have seemingly 1000s of nested Divs all over the god damned place oftentimes containing a single element. I’m finally on chrome now should I learn Jquery now? I mean seriously wtf.

I use this one no code tool for dev it has web automation built in. As you can imagine its just as broken as anything else!! I have 10 screens to navigate it gets stuck on the second screen all the damn time. Fuck I love clicking the buttons when my script misses and playing catch up with it.

So as a work around to Selenium not waiting even 1 millisecond when I use explicit wait or implicit wait or fluent wait, I’m guessing maybe I can attach both Chrome Dev Tools Protocol (CDP as ive called it earlier) and selenium to the same browser and maybe I can use CDP to perform a Wait with any degree of success. Selenium will do nothing more than execute vanilla javascript Element.click(); This is the only way I know to even ACTUALLY use selenium beyond the simplest html documents possible. Hell I guess CDP can execute js idk.

I can’t get the new selenium that has CDP but I do have some buggy ass selenium from a few years back. Yeah, I remember reading there was a pretty impactful regression defect in the version I have. Maybe I’m being gaslighted by some shit copy of selenium?

The worst part is that I do seem to be having issues that the rest of the internet’s devs do not seem to be having. People act like browser automation is totally viable and pretty OK. How in the fuck hell is my Selenium Test Suite going to be more reliable my application under test?!!?? I’ll have more fucking bugs in my test suite than in my application. Today, I have less than half a test script and, I. already. fucking. do.

I am still SUPER PISSED at the months of 12 hour days (always 8 hours spent on normal sprint work btw only 4 to automation) I spent trying to automate our regression tests. I got NOWHERE.

I did learn a lot about HTML and JS though like I’m not that mad…but I’m just trying to emphasize my achievement on my task was zero.

The buttons don’t click. There are so many divs and I swear you sometimes need to select a div somewhere in the middle sometimes to get it working. The waits don’t wait. XHR requests are invisible. Java crashes 100 times before I find an xpath and thread.sleep() combo that works. I have no failure modes to use — Sometimes I click the same element 20x in a script because I have no way to know if it clicked the first time! Sometimes you gotta scroll the page to make the click work. So many click methods all broken. So many wait methods all broken. Its not just the elements don’t click! There are so many ways to click that almost work but surely they all fail the same in the end. ok at this point I’m just repeating myself…

there yet even more issues that I can’t remember…and will soon remember as I journey into this project yet again…

thanks for reading I hope I entertained and would love to hear your experience!

No description needed :-(

Probably wiring up a door chime for my smart things (because they don't sell one) that plays the classic 7-11 chime when any door is opened.

That or using xpath to write beautiful dynamic SQL.

Trying to test this React app my team is working on with selenium. There were no classes or ids I could've used to find elements.
I ended up looking for elements by attributes and contained text using XPath.

I've to say that javascript is no language I like because I'm more fastinated of building a nice and scalable backend then building a gui (in the browser).

The funny thing is that right now javascript helps me at my current project because many websites implement wide-opened apis for their js frontend and it just works like a charm to use them instead of parsing the whole html and do some XPath stuff to fetch information.

Finding out this morning that Firepath is not showing xpath nodes anymore. Why?? Is it me or last update screwed what it was a really good tool?