seleniumbase

Looks like that one hasn't received any major updates in awhile.

The secret ingredient is to use Playwright's `connect_over_cdp()` method to connect to an existing browser that is already stealthy. Playwright can then perform its usual actions without being detected by anti-bot services. Example setup: `browser = playwright.chromium.connect_over_cdp("http://localhost:9222")`

Mix in some Python, PyAutoGUI, SeleniumBase, ThreadPoolExecutor, and then you get some serious multi-threaded CAPTCHA-bypass.

You can set `browser="firefox"`, but there's no stealth mode for it.

Here's an example that bypasses Kasada on the Hyatt website: https://github.com/seleniumbase/SeleniumBase/blob/master/exa...

It's OK. No offense was taken. It almost looked like the conversation was expanding into a "Python vs Java" debate, but (thankfully) it did not. I've seen both worlds. I've seen advantages to both. I decided to stay in the Python world.

Thank you!

The biggest issue with going from a home machine to a server is that you may lose having a "residential IP address", which is something that you'll want to have in order to prevent automation from being blocked outright. Hence the popularity of residential proxies. However, some servers live in a residential IP space, which makes them optimal for running web automation in. As was partially covered in https://www.youtube.com/watch?v=Mr90iQmNsKM, GitHub Actions appears to live in a "Residential IP space", which makes it a good server choice for web automation.

The "Python vs Java" debate is probably one for a different Hacker News post. :)

SeleniumBase CDP Mode uses `DOM.scrollIntoViewIfNeeded` (https://chromedevtools.github.io/devtools-protocol/tot/DOM/#...), so it only scrolls when elements are offscreen, rather than always scrolling. This reduces the number of scrolls needed. Also, it seems that most anti-bot services are not looking at scrolling as a way of identifying users.

Patching chromedriver is a lot easier than patching the browser. Plus, if you're just using a regular Chrome browser for the automation, then there's nothing to patch. Automated CDP calls aren't detectable if they don't leave any trace of automation activity. However, since Google created CDP, they might have ways of detecting automated CDP in ways that other services cannot.

There are live demos on YouTube of SeleniumBase bypassing CAPTCHAs (if you want see first): https://www.youtube.com/watch?v=Mr90iQmNsKM

There's actually a lot of examples being used for testing (https://github.com/seleniumbase/SeleniumBase/tree/master/exa...), which are run regularly (locally and in GitHub Actions). Plus, a lot of major companies are using SeleniumBase: https://github.com/seleniumbase/SeleniumBase/blob/master/hel... (if something breaks, I find out quickly)

There's a reCAPTCHA on the Pokemon website. This SeleniumBase example bypassed it: https://github.com/seleniumbase/SeleniumBase/blob/master/exa...

Call it "legacy code" if you'd like. That specific part is from a less common feature for setting options when running on a Selenium Grid. The new CDP Mode isn't compatible with The Grid (since CDP Mode makes direct CDP API calls without making Selenium API calls).

That patches chromedriver, (which gets renamed to uc_driver), but patching by itself isn't enough to bypass bot-detection. SeleniumBase also sets specific Chrome options and modifies methods to use the Chrome Devtools Protocol.

With SeleniumBase, you can bypass CAPTCHAs with one line of code: `sb.uc_gui_click_captcha()`

That method came from code that I accepted in a PR from December 31, 2019: https://github.com/seleniumbase/SeleniumBase/pull/459 Not a true representation of most of the code today.

SeleniumBase modifies the webdriver so that it doesn't get detected when used alongside the CDP stealth mode and methods. It'll download chromedriver for you. Not sure what you mean by the multiple branches, as there's just the primary one. What 1000-line methods are you referring to? By "flags", do you mean the different command-line options available? As for Playwright, they aren't undetected: See https://github.com/microsoft/playwright/issues/23884#issueco... - "Playwright is an end-to-end testing framework, where we expect you test on your own environments. Bypassing any form of bot protection is not something we can act on. Thanks for your understanding." On the contrary, SeleniumBase is OK with bypassing bot detection: https://github.com/seleniumbase/SeleniumBase/blob/master/exa...

Here's the main website: https://github.com/seleniumbase/SeleniumBase

Link moved to: https://github.com/seleniumbase/SeleniumBase/blob/master/hel...

SeleniumBase gives you more consistent results than just using WebDriver alone, as SeleniumBase wraps WebDriver methods to improve the reliability of browser actions.

SeleniumBase has those retries built-in. The browser will automatically wait for page elements to fully load before interacting with them. In addition, SeleniumBase includes all the abilities of pytest, which transforms Selenium from a library into a complete test automation framework.

That makes setup super easy! :)

On the topic of convenience and flexibilty, I've made it easier to use the Selenium Grid Hub: https://github.com/seleniumbase/SeleniumBase/tree/master/int...

I built a Docker integration for SeleniumBase. You can find all the info in the Docker ReadMe: https://github.com/seleniumbase/SeleniumBase/blob/master/int...

I just looked at the Cloud SQL ReadMe at https://cloud.google.com/sql/ and it definitely looks cool. The MySQL component in SeleniumBase that you saw has been there from the early days, before I started working with Google Cloud. I've got a long list of features to add in the near future, and Cloud SQL can definitely be one. Thanks for the idea!