Major updates have arrived in 4.28.0 (mostly for UC Mode)

[mdmintz]

For anyone that hasn't been following #2842, CF pushed an update that prevented UC Mode from easily bypassing CAPTCHA Turnstiles on Linux servers. Additionally, uc_click() was rendered ineffective for clicking Turnstile CAPTCHA checkboxes when clicking the checkbox was required. I've been working on solutions to these situations.

As I mentioned earlier in #2842 (comment), if CF detects either Selenium in the browser or JavaScript involvement in clicking the CAPTCHA, then they don't let the click through. (The JS-detection part is new.) I read online that CF employees borrowed ideas from https://github.com/kaliiiiiiiiii/brotector (a Selenium detector) in order to improve their CAPTCHA. Naturally, I was skeptical at first, but I have confirmed that the two algorithms do appear to get similar results. (Brotector was released 6 weeks ago, while the Cloudflare update happened 2 weeks ago.)

The solution to bypassing the improved CAPTCHAs requires using pyautogui to stay undetected. There was also the matter of how to make pyautogui work well on headless Linux servers. (Thanks to some ideas by @EnmeiRyuuDev in #2842 (comment), that problem was overcome by setting pyautogui._pyautogui_x11._display to Xlib.display.Display(os.environ['DISPLAY']) on Linux in order to sync up pyautogui with the X11 virtual display.)

The improved SeleniumBase UC Mode will have these new methods:

driver.uc_gui_press_key(key)  # Use PyAutoGUI to press the keyboard key

driver.uc_gui_press_keys(keys)  # Use PyAutoGUI to press a list of keys

driver.uc_gui_write(text)  # Similar to uc_gui_press_keys(), but faster

driver.uc_gui_handle_cf(frame="iframe")  # PyAutoGUI click CF Turnstile

It'll probably be easier to understand how those work via examples. Here's one for uc_gui_handle_cf based on the example in #2842 (comment):

import sys
from seleniumbase import SB

agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/126.0.0.0"
if "linux" in sys.platform:
    agent = None  # Use the default UserAgent

with SB(uc=True, test=True, rtf=True, agent=agent) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    sb.uc_gui_handle_cf()  # Ready if needed!
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

Above, I deliberately gave it an incomplete UserAgent so that CAPTCHA-clicking is required to advance. On macOS and Windows, the default UserAgent that SeleniumBase gives you is already enough to bypass the CAPTCHA screen entirely. The uc_gui_handle_cf() method is designed such that if there's no CAPTCHA that needs to be clicked on the page you're on, then nothing happens. Therefore, you can add the line whenever you think you'll encounter a CAPTCHA or not. In case there's more than one iframe on a website, you can specify the CSS Selector of the iframe as an arg when calling uc_gui_handle_cf(). There will be new examples in the SeleniumBase/examples/ folder for all the new UC Mode methods. To sum up, you may need to use the newer uc_gui_* methods in order to get past some CAPTCHAs on Linux where uc_click() worked previously.

On the topic of Brotector, (which is the open source bot-detector library that CF borrowed ideas from), there is a huge opportunity: Now that effective bot-detection software is available to the general public (all the code is open source!), anyone can now build their own CAPTCHA services (or just add CAPTCHAs to sites without the "service" part). I've already jumped on this with the Brotector CAPTCHA: https://seleniumbase.io/apps/brotector. I've also created a few test sites that utilize it:

• https://seleniumbase.io/hobbit/login
• https://seleniumbase.io/antibot/login

I did make some improvements to the original Brotector algorithm in order to be suitable for CAPTCHAs: I needed a definite Allow/Block answer, rather than a number between 0 and 1 determining the likelihood of a bot, etc. I've been using these new test sites for testing the improved UC Mode.

That covers the major updates from 4.28.0 (with the exception of Brotector CAPTCHA test sites, which were already available to the public at the URLs listed above).

There will also be some other improvements:

• More sb methods added directly into the driver.
• An improvement to the Recorder to better handle autogenerated IDs in selectors.
• Python dependency updates.
• Some method simplifications.
• Some timing updates.
• Some updates to default settings.

Now, when using UC Mode on Linux, the default setting is NOT using headless mode. If for some reason you decide to use UC Mode and Headless Mode together, note that although Chrome will launch, you'll definitely be detected by anti-bots, and on top of that, pyautogui methods won't work. Use xvfb=True / --xvfb in order to be sure that the improved X11 virtual display on Linux activates. You'll need that for the uc_gui_* methods to work properly.

Much of that will get covered in the 3rd UC Mode video tutorial on YouTube (expected sometime in the near future).

In case anyone has forgotten, SeleniumBase is still a Test Automation Framework at heart, (which includes an extremely popular feature for stealth called "UC Mode"). UC Mode has gathered a lot of the attention, but SeleniumBase is more than just that.

[mdmintz]

4.28.0 has been released: https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.28.0

The pyautogui example for a Cloudflare page with UC Mode:

• SeleniumBase/examples/raw_pyautogui.py

Examples of bypassing the Brotector CAPTCHA with UC Mode:

• SeleniumBase/examples/raw_brotector_captcha.py
• SeleniumBase/examples/raw_hobbit.py
• SeleniumBase/examples/raw_antibot_login.py

Examples of how the Brotector CAPTCHA detects regular Selenium:

• SeleniumBase/examples/raw_detection.py
• SeleniumBase/examples/raw_block.py

[mdmintz]

Here's an example script for Linux to prove it's working:

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    print(sb.get_page_title())
    sb.uc_gui_handle_cf()  # Ready if needed!
    print(sb.get_page_title())
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

The second print() should show "Virtual Manager", which means that the automation was able to get past the Turnstile.

[vmolostvov]

@mdmintz same problem here on linux vds (ubuntu without gpu), seleniumbase became unable to bypass the CloudFlare challenge. Using latest sb version. On local macos and windows keep working without any problem.

✅ I can confirm that my issue on headless Linux Ubuntu was solved by 4.28.0

Appreciate your work sir @mdmintz

[goldananas]

Hey @mdmintz , do you think you will be working on making the uc_gui_handle_cf method compatible with the returnable Driver ? Works just fine on headless Linux with SB, but I can't find way to do the same with the returnable Driver instead.

[mdmintz]

@goldananas If using the Driver() format instead of SB(), you'll need need to spin up the special X11 virtual display yourself before launching the driver. (See #2842 (comment).)

With the SB() format, SB(uc=True, xvfb=True) does all that for you when running on Linux.

[mdmintz]

Windows users should upgrade to 4.28.3 or newer (Fixes #2889 on 4.28.2)

[JimKarvo]

Seems that the CF detected the new way of bypassing.

Sometimes the click works (not always)

but after that, the checkbox is failed

[mdmintz]

macOS: ✅
Windows: ✅
Linux with natural GUI on residential IP: ✅
Linux without GUI on non-residential IP: ❌
Linux without GUI on residential IP: ⚠️ / ❓

So much for the free pass on GitHub Actions CAPTCHA bypassing. 😄 I didn't expect that loophole to last long.

[JimKarvo]

@mdmintz I forgot to mention that I am running an Ubuntu server, no GUI

[mdmintz]

@JimKarvo Residential IP or non-residential?

[OpsecGuy]

@mdmintz In my case I also have some issues with the bypass.
We talk about the 4.28.3 version of the seleniumbase. On Windows there are no issues, however on my Linux (Ubuntu 20) VM with GUI #https://github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_pyautogui.py
In that script, I just edit the URL to the website that at first connect shows Cloudflare CF captcha. The same IP that successfully bypasses the captcha on Windows doesn't want to work on Linux with GUI. On the bare-metal server where I have Ubuntu 22 installed, I'm also stuck on the CF captcha page and experimenting with the reconnect timeout doesn't solve my issues.

My user agent on both Linux machines is: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Google Chrome version is 126.0.6478.126
I tested with my residential IP and the other residential proxies.

//
Tested original code from raw_pyautogui.py and looks like it worked, but on any other website I test I get this:

[mdmintz]

The last successful GitHub Actions run for bypassing Cloudflare's Turnstile was https://github.com/mdmintz/undetected-testing/actions/runs/9748457978/job/26903480495 8 hours ago. Likely their QA Team did not initially catch that their Turnstiles were getting bypassed on GitHub Actions until they came over to the SeleniumBase repo and read the notes.

[gabrielsim]

Linux without GUI on residential IP: ⚠️ / ❓

@mdmintz fyi, Linux without GUI on residential IPs still works for me

[mdmintz]

@gabrielsim That's good news: That means the algorithm works right now when the IP Address hasn't been blocked already. When it worked earlier on GitHub Actions, it was due to a bug on Cloudflare's end when then forget to check IP ranges for known non-residential server addresses. They finally fixed it: Likely after reading this thread and learning about the loophole.

No changes are needed for UC Mode at this time. However, Brotector still has some bot-checks that Cloudflare hasn't picked up yet. This would allow them to detect switching into an iframe, as well the JavaScript for making an element the active one. There's already a plan in place for that scenario, involving pyautogui for more things, and not just clicking the active element.

[OpsecGuy]

@mdmintz In my case I also have some issues with the bypass. We talk about the 4.28.3 version of the seleniumbase. On Windows there are no issues, however on my Linux (Ubuntu 20) VM with GUI #https://github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_pyautogui.py In that script, I just edit the URL to the website that at first connect shows Cloudflare CF captcha. The same IP that successfully bypasses the captcha on Windows doesn't want to work on Linux with GUI. On the bare-metal server where I have Ubuntu 22 installed, I'm also stuck on the CF captcha page and experimenting with the reconnect timeout doesn't solve my issues.

My user agent on both Linux machines is: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Google Chrome version is 126.0.6478.126 I tested with my residential IP and the other residential proxies.

// Tested original code from raw_pyautogui.py and looks like it worked, but on any other website I test I get this:

That's very strange, however any website that I test now, bypass seems to be working properly... Idk what CF team is doing, but I believe the prepare another big update for us.

[enricodvn]

Hey guys, just to add to this discussion, CF is now detecting residential proxies with ML:

https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning

The residential proxies I used became pretty much useless since mid June :/

[amberbor]

Hey guys

can somebody help me i am trying on macos to get the dexscreener and to bypass cloudflare but it doesnt work

import time

from seleniumbase import SB

with SB(uc=True, xvfb=True) as sb:
url = "https://dexscreener.com"
sb.uc_open_with_reconnect(url, 4)
print(sb.get_page_title())
sb.uc_gui_handle_cf() # Ready if needed!
print(sb.get_page_title())

time.sleep(70)

[JimKarvo]

Updates: Nothing to do with blocked IP or proxies.

I have scripts running on Windows machines (headed) on my home IP.
I forward all traffic from ubuntu server through my home IP. The first 3-5 pages are ok. After that pages CF appears to my browser. SeleniumBase still can't bypass it meanwhile at my windows pc, I can bypass with no problems.

[amberbor]

test.mov

[mdmintz]

@amberbor The best user-agent to use is the default one that SeleniumBase sets for you automatically.

[mdmintz]

In case anyone was wondering where my top stalkers are from:

---

Yeah... they're watching. 👀

[max-frai]

github.mov

I will not reveal all the cards, but it's possible. We managed to bypass everything. We don't disconnect CDP session, don't use image to find checkbox by pattern. Also we have custom recorded mouse movements which we slightly modify each time and replicate (but looks like it's not required for cloudflare, but useful for recaptcha).

The only problem is running chrome inside docker. It would be better to manage everything, but I wasn't able to understand what's exactly they are detecting and what's leaking.

So for me ubuntu + headfull + isp/residential ips works stable now.

[OpsecGuy]

github.mov
I will not reveal all the cards, but it's possible. We managed to bypass everything. We don't disconnect CDP session, don't use image to find checkbox by pattern. Also we have custom recorded mouse movements which we slightly modify each time and replicate (but looks like it's not required for cloudflare, but useful for recaptcha).

The only problem is running chrome inside docker. It would be better to manage everything, but I wasn't able to understand what's exactly they are detecting and what's leaking.

So for me ubuntu + headfull + isp/residential ips works stable now.

gj, but bypassing their captcha through their callback is still the best option :)

[max-frai]

@OpsecGuy Thanks, probably yes, but making universal logic is better for me, because it will require less changes when cloudflare changes something from their side. Also it can bypass other smart captchas.

[mdmintz]

Here's where we're at:

macOS with residential IP: ✅
macOS with non-residential IP: ⚠️ / ❓ / ❌
Windows with residential IP: ✅
Windows with non-residential IP: ⚠️ / ❓ / ❌
Linux with residential IP: ✅
Linux with non-residential IP: ⚠️ / ❓ / ❌

Based on my observations (and others above) there are no problems when using residential IPs. You may need to adjust some timings (eg. reconnect_time, etc), but things are definitely working for the current version (assuming Cloudflare hasn't flagged your IP address for unusual traffic).

My tests for non-residential IPs have only used GitHub Actions so far, and the CAPTCHAs haven't been bypassed since 3 days ago when CF made some changes. Some people above have claimed that those non-residential IP CAPTCHAs can be bypassed if using pyautogui mouse actions for clicking, rather than the spacebar. I have yet to observe that behavior myself, but I'm still investigating. I'll probably add a pyautogui mouse-clicking method in the next release, although I'm not sure it will be any better than the current uc_gui_handle_cf() method, which clicks on the active element with the spacebar.

Keep in mind some of the methods that are currently available:

uc_open_with_reconnect(url, reconnect_time=None)
uc_open_with_disconnect(url, timeout=None)
reconnect(timeout)
disconnect()
connect()
uc_click(selector, by="css selector")  # Partially obsolete due to uc_gui_handle_cf()
uc_gui_press_key(key)
uc_gui_press_keys(keys)
uc_gui_write(text)
uc_gui_handle_cf(frame="iframe")

[NCLnclNCL]

use https://github.com/kaliiiiiiiiii/Selenium-Driverless, https://github.com/g1879/DrissionPage or https://github.com/ultrafunkamsterdam/nodriver/issues
i tested with driverless and DrissionPage , it is passed successfully

[mdmintz]

@NCLnclNCL Show me the link to working code running successfully in GitHub Actions. If people make claims, they need to back up those claims with hard evidence. The non-residential IP space (eg. GitHub Actions) appears to be an issue for everyone because Cloudflare flags those IP ranges as non-human traffic, and then prevents bypass.

As for the residential IP space, SeleniumBase is bypassing CAPTCHAs successfully with the right code. (Examples are listed in this thread.)

[mdmintz]

Improvements and new methods are available in SeleniumBase 4.28.4.

In particular, there's sb.uc_gui_click_cf(), which works even from GitHub Actions!

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    print(sb.get_page_title())
    sb.uc_gui_click_cf()  # Ready if needed!
    print(sb.get_page_title())
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

https://github.com/mdmintz/undetected-testing/actions/runs/9817403347/job/27108729362

New Status:

macOS / Windows / Linux with residential IP: ✅
macOS / Windows / Linux with non-residential IP: ✅ (New!)

[jens4626]

Improvements and new methods are available in SeleniumBase 4.28.4.

In particular, there's sb.uc_gui_click_cf(), which works even from GitHub Actions!

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    print(sb.get_page_title())
    sb.uc_gui_click_cf()  # Ready if needed!
    print(sb.get_page_title())
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

https://github.com/mdmintz/undetected-testing/actions/runs/9817403347/job/27108729362

New Status:

macOS / Windows / Linux with residential IP: ✅ macOS / Windows / Linux with non-residential IP: ✅ (New!)

Thats what I like to see! I can confirm that sb.uc_gui_click_cf() does indeed work for Windows non-residential!

Thanks once again @mdmintz - and also @chlwodud77 regarding using clicks work rather than keyboard actions :-)

Bypass.mp4

[OpsecGuy]

Improvements and new methods are available in SeleniumBase 4.28.4.
In particular, there's sb.uc_gui_click_cf(), which works even from GitHub Actions!

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    print(sb.get_page_title())
    sb.uc_gui_click_cf()  # Ready if needed!
    print(sb.get_page_title())
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

https://github.com/mdmintz/undetected-testing/actions/runs/9817403347/job/27108729362
New Status:
macOS / Windows / Linux with residential IP: ✅ macOS / Windows / Linux with non-residential IP: ✅ (New!)

Thats what I like to see! I can confirm that sb.uc_gui_click_cf() does indeed work for Windows non-residential!

Thanks once again @mdmintz - and also @chlwodud77 regarding using clicks work rather than keyboard actions :-)

Bypass.mp4

Have you tried on the server with ubuntu and a fake display?

[jens4626]

Improvements and new methods are available in SeleniumBase 4.28.4.
In particular, there's sb.uc_gui_click_cf(), which works even from GitHub Actions!

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    print(sb.get_page_title())
    sb.uc_gui_click_cf()  # Ready if needed!
    print(sb.get_page_title())
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

https://github.com/mdmintz/undetected-testing/actions/runs/9817403347/job/27108729362
New Status:
macOS / Windows / Linux with residential IP: ✅ macOS / Windows / Linux with non-residential IP: ✅ (New!)

Thats what I like to see! I can confirm that sb.uc_gui_click_cf() does indeed work for Windows non-residential!
Thanks once again @mdmintz - and also @chlwodud77 regarding using clicks work rather than keyboard actions :-)
Bypass.mp4

Have you tried on the server with ubuntu and a fake display?

No, I only use it for windows non-residential IP :-)

[JimKarvo]

Confirm of working in headless Ubuntu with server IP (for testing)

[bjornkarlsson]

I am testing the uc_gui_click_cf function on the following websites: https://rateyourmusic.com

It seems to work, but there a couple of issues mostly related to the spinning bar image in the input box becomes it expects a user action:

1. Before the input box is clickable: it will attempt clicking the box even when the wheel is spinning and the input box is not ready to be clicked.

2. The retry flag also potentially has issue related to the above, if set it will attempt to repeat the process when the wheel is still spinning after it has been clicked.

The flow however seems to work if driven in debug mode triggering these actions manually, ideally there should be a wait condition in between that checks that the wheel is not spinning and the input box is empty/clickable, haven't figured out how to do it yet.

[max-frai]

Also looks like mouse movement is mandatory now, just clicking checkbox with system mouse isn't enough.

[JimKarvo]

CF Patched the solution for Ubuntu server (virtual monitor) systems

[misterpeople]

CF Patched the solution for Ubuntu server (virtual monitor) systems

Seems like Windows doesn't work as well anymore.

[max-frai]

Also turnstile iframe is behind shadow dom now, so I think patching required in this direction.

[mdmintz]

I'm aware of the new ShadowDOM situation causing failures. No need for people to keep posting about it.
I'll post another update within 4 hours. Please hold off on new comments for now so that I can focus...

[mdmintz]

Upgrade to 4.28.5 and retry your scripts. Old methods have been fixed.

Also added this new method: uc_gui_click_captcha(), which auto-detects the CAPTCHA and clicks it.

[mdmintz]

More info on the 4.28.5 update:

• uc_gui_handle_cf() is working again on macOS and Windows (residential IP address)
• uc_gui_click_cf() is working again on macOS, Windows, and Linux (residential & non-residential IP)

https://github.com/mdmintz/undetected-testing/actions/runs/9847111425/job/27186339885

You can also use uc_gui_click_captcha() to auto-detect between Turnstile and reCAPTCHA, and then click it. But there are no guarantees that it will work on reCAPTCHA, as it's far more advanced than other CAPTCHAs because they use advanced AI to detect bots (more than just detecting Selenium in the browser).

[bjornkarlsson]

The new method is capable to pass the verification also with non residential, however this only happens when the button it's been clicked right after the spinning wheel disappears (and empty box appears). Not always though the click goes through, some times the cursor is positioned to the input box but not click gets dispatched

[mdmintz]

@bjornkarlsson When you call sb.uc_open_with_reconnect(url, reconnect_time), make sure that the reconnect_time is high enough that the spinning wheel has stopped spinning first.

[robertmaceda]

Would this update be able to run headless on windows as well? or does with SB(uc=True, xvfb=True) as sb only works for headless linux servers?

[bjornkarlsson]

@bjornkarlsson When you call sb.uc_open_with_reconnect(url, reconnect_time), make sure that the reconnect_time is high enough that the spinning wheel has stopped spinning first.

Thanks that seems to do the trick most of the times, is there a way to wait for the input box to appear with the Shadowed DOM? I was able to implement that prior the changes of yesterday on the shadowed iframe

[mdmintz]

@robertmaceda The xvfb=True is only for Linux. Headless is not possible with pyautogui.

@bjornkarlsson There's no way to wait for it specifically, so make sure to wait long enough.

Separate from that, I had to temporarily disable interaction due to an overwhelming number of GitHub notifications (mostly off-topic), so opening new tickets and comments might not be possible at the moment. Refer to existing documentation.

[mdmintz]

If you're using any of the following methods...

sb.uc_gui_click_captcha()
sb.uc_gui_click_cf()
sb.uc_gui_handle_cf()

...then be sure to upgrade to seleniumbase 4.29.2 (or newer if available).

CF made a big change to Turnstiles today... and we were ready for them!
If shipping code fast was an Olympic sport, then you'd be seeing us in the Olympics!

[mdmintz]

If you missed #3080, upgrade to 4.30.1 (or newer if available). (CF updated their CAPTCHAs again).

And if you haven't seen Video 3 yet, here's the link: https://www.youtube.com/watch?v=-EpZlhGWo9k

[mdmintz]

In case you missed #3236, upgrade to 4.32.5 (or newer if available).
There were major changes in Chrome 130 that led to UC Mode issues.

Also, be sure to check out the new CDP Mode within UC Mode.