Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

twitter.com: ads #20792

Closed
9 of 10 tasks
S00pergang opened this issue Nov 20, 2023 · 48 comments
Closed
9 of 10 tasks

twitter.com: ads #20792

S00pergang opened this issue Nov 20, 2023 · 48 comments

Comments

@S00pergang
Copy link

S00pergang commented Nov 20, 2023

Prerequisites

  • This is NOT a YouTube, Facebook or Twitch report.
  • I read and understand the policy about what is a valid filter issue.
  • I verified that this issue is not a duplicate. (Use this button to find out.)
  • I forced an update of my filter lists. (Click the "Purge all caches" button while holding the 'Shift' key, then click the "Update now" button.)
  • I did not remove any of the default filter lists, or I have verified that the issue was not caused by removing any of the default lists.
  • I did not enable additional filter lists, or I have verified that the issue still occurs without enabling additional filter lists.
  • I do not have custom filters/rules, or I have verified that the issue still occurs without custom filters/rules.
  • I am not using uBlock Origin (uBO) along with other content blocker extensions.
  • I have verified that the web browser's built-in blocker or DNS blocking (standalone or through a VPN) is not causing the issue.
  • I did not answer truthfully to ALL the above checkpoints.

URL address of the web page

https://twitter.com/

Category

ads

Description

It seems twitter has done something so now ads are appearing in replies along with For You Page and Following.

Other extensions used

none

Screenshot(s)

Screenshot(s)

image

Configuration

uBlock Origin: 1.53.0
Chromium: 119
filterset (summary):
 network: 103611
 cosmetic: 43778
 scriptlet: 19072
 html: 0
listset (total-discarded, last-updated):
 default:
  user-filters: 4-0, never
  easylist: 75532-16, now
  easyprivacy: 32902-64, now
  plowe-0: 3718-1078, now
  ublock-badware: 7423-137, now
  ublock-filters: 35634-156, now
  ublock-privacy: 680-5, now
  ublock-quick-fixes: 114-37, now
  ublock-unbreak: 2155-32, now
  urlhaus-1: 9970-0, now
filterset (user): [array of 4 redacted]
trustedset:
 added: [array of 1 redacted]
 removed:
  chrome-scheme
userSettings: [none]
hiddenSettings: [none]
supportStats:
 allReadyAfter: 337 ms
 maxAssetCacheWait: 148 ms
popupPanel:
 blocked: 15
 network:
  twitter.com: 13
  ads-twitter.com: 1
  google-analytics.com: 1
@gorhill
Copy link
Member

gorhill commented Nov 20, 2023

I can reproduce. Investigating an optimal solution.

@polymorphicshade
Copy link

I forgot how cancerous the ads are on twitter...

@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

My suggestion (if filter authors agree):

twitter.com##article path[d^="M13.1"]:upward([data-testid="cellInnerDiv"])

For browsers which supports native :has(), it could be plain CSS selector:

twitter##[data-testid="cellInnerDiv"] article:has(path[d^="M13.1"])

cc @ryanbr


Since :has() is enabled by default in Firefox 121 (currently beta), I suggest we go with the second form).

@Hakkin
Copy link

Hakkin commented Nov 21, 2023

path[d^="M13.1"] seems to match the verified business icon? I'm seeing ads from accounts without that icon. [data-testid='cellInnerDiv']:has([data-testid='placementTracking']) seems to block it for me, though obviously placementTracking is some kind of debug tag so Twitter could change it at will.

@unbeatable-101
Copy link

I added twitter##[data-testid="cellInnerDiv"] article:has(path[d^="M13.1"]) to my filters and still am seeing ads, from accounts with verified business icons as well.

@SakhJack
Copy link

[data-testid='placementTracking'] seems to block it for me

placementTracking is used for media in regular tweets

example: https://twitter.com/AzaelOfficial/status/1726754958480232620

@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

## article:has(span:has-text(/^Ad$/))

Efficiency-wise, this is a terrible filter, span returns hundreds of matching elements, which means :has-text() will have to iterate through all those elements.

I advise people to not paste into my filters any such suggestion from non-experienced filter list authors. The goal here is to find the minimum set of efficient filters which fulfill the task.

For now I will add twitter##[data-testid="cellInnerDiv"] article:has(path[d^="M13.1"]) to the list, we can improve on it thereafter.

@Hakkin
Copy link

Hakkin commented Nov 21, 2023

twitter##[data-testid="cellInnerDiv"] article:has(path[d^="M13.1"]) blocks every tweet from any verified business account, which has nothing to do with ads.

@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

If you want to report issue with that filter, post your troubleshooting information, this is a necessary step, just saying "it doesn't work" doesn't work:

Go to twitter.com, click the uBO toolbar icon > Click the chat icon 💬 > Click "Troubleshooting Information" > Report content here (https://www.reddit.com/link/173jmog/video/bogwf92af9tb1/player)

@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

@Hakkin Please post examples of tweets from such "verified business" which is not an ad.

@Hakkin
Copy link

Hakkin commented Nov 21, 2023

@Hakkin Please post examples of tweets from such "verified business" which is not an ad.

All the tweets on https://twitter.com/ibm, for example. There are many other verified business Twitter accounts.

gorhill added a commit that referenced this issue Nov 21, 2023
This reverts commit 5e8064d.
@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

Ok I spotted one such verified business which was not an ad. Reverted.

@RambIing
Copy link

its worth noting all sponsored tweets have a div with the data-testid being top-impression-pixel. Could not find it on any other tweets. Not sure if this is reproducible with Twitter in other languages?
image

@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

@rambling Good observation.

So how about:

twitter.com##[data-testid="primaryColumn"] [data-testid="cellInnerDiv"]:has([data-testid="top-impression-pixel"])

Feedback needed.

gorhill added a commit that referenced this issue Nov 21, 2023
@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

Click to force an update of quick-fixes list to get the new filter:

https://ublockorigin.github.io/uAssets/update-lists.html?listkeys=ublock-quick-fixes&manual=1

@DaisukeDaisuke
Copy link

DaisukeDaisuke commented Nov 21, 2023

In rare cases, blank spaces or flickering occur. Is this the intended behavior?
The flickering is, in my opinion, due to reinsertion and is unavoidable

Images

Desktop Screenshot 2023 11 21 - 13 31 43 67

Troubleshooting Information
uBlock Origin: 1.53.0
Chromium: 119
filterset (summary):
 network: 172081
 cosmetic: 162815
 scriptlet: 43044
 html: 0
listset (total-discarded, last-updated):
 removed:
  urlhaus-1: null
 added:
  adguard-generic: 80226-489, 38m
  adguard-social: 22006-36, 38m
  adguard-spyware: 72539-1208, 38m
  adguard-spyware-url: 1327-5, 38m
  block-lan: 55-0, 38m
  fanboy-cookiemonster: 48925-226, 38m
  ublock-annoyances: 5875-97, 38m
  ublock-quick-fixes: 115-40, 38m
 default:
  user-filters: 1388-16, never
  JPN-1: 13788-6, 38m
  easylist: 75532-1693, 38m
  easyprivacy: 32902-16416, 38m
  plowe-0: 3718-1179, 38m
  ublock-badware: 7423-329, 38m
  ublock-filters: 35635-3949, 38m
  ublock-privacy: 680-184, 38m
  ublock-unbreak: 2155-148, 38m
filterset (user): [array of 1393 redacted]
trustedset:
 added: [array of 10 redacted]
switchRuleset:
 added: [array of 19 redacted]
userSettings:
 largeMediaSize: 1
hiddenSettings: [none]
supportStats:
 allReadyAfter: 1316 ms (selfie)
 maxAssetCacheWait: 444 ms
popupPanel:
 blocked: 12
 network:
  twitter.com: 10
  ads-twitter.com: 1
  google-analytics.com: 1
 extended:
  ##div[data-testid="trend"]:has(path[d$="10H8.996V8h7v7z"])
  ##[data-testid="primaryColumn"] [data-testid="cellInnerDiv"]:has…
  ##[data-testid="trend"]:has-text(/Promoted|Gesponsert|Promociona…
  ##html:style(overflow:auto!important)
  ##+js(trusted-set-cookie, d_prefs, MjoxLGNvbnNlbnRfdmVyc2lvbjoyL…

@uBlock-user
Copy link
Contributor

@AdamWr happening in AdGuard for Android too, need to add the filter in 4a7a3d5

@cocona20xx
Copy link

@rambling Good observation.

So how about:

twitter.com##[data-testid="primaryColumn"] [data-testid="cellInnerDiv"]:has([data-testid="top-impression-pixel"])

Feedback needed.

Gets rid of sponsored posts/ads but doesn't get rid of the trending sidebar and the such.
The picker auto-generates the following when manually removing the 'please buy premium we need money' plea, alongside recommended users to follow and the trending sidebar:

twitter.com##.r-1f1sjgu.r-ymttw5.r-uaa2di.r-eqz5dr.r-1habvwh.css-175oi2r
twitter.com##div.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r:nth-of-type(4)
twitter.com##div.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r:nth-of-type(5)
twitter.com##div.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r:nth-of-type(3)
twitter.com##.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r

@RambIing
Copy link

@rambling Good observation.
So how about:

twitter.com##[data-testid="primaryColumn"] [data-testid="cellInnerDiv"]:has([data-testid="top-impression-pixel"])

Feedback needed.

Gets rid of sponsored posts/ads but doesn't get rid of the trending sidebar and the such. The picker auto-generates the following when manually removing the 'please buy premium we need money' plea, alongside recommended users to follow and the trending sidebar:

twitter.com##.r-1f1sjgu.r-ymttw5.r-uaa2di.r-eqz5dr.r-1habvwh.css-175oi2r
twitter.com##div.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r:nth-of-type(4)
twitter.com##div.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r:nth-of-type(5)
twitter.com##div.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r:nth-of-type(3)
twitter.com##.r-14wv3jr.r-g2wdr4.r-1udh08x.r-1ifxtd0.r-rs99b7.r-1phboty.r-1867qdf.css-175oi2r

This isn't very stable of a filter, since the class names will update once they push a new frontend update

@huyszn
Copy link
Contributor

huyszn commented Nov 21, 2023

@DaisukeDaisuke I get the blank spaces / flickering when I refresh a Twitter page and then immediately scroll down.

ryanbr added a commit to easylist/easylist that referenced this issue Nov 21, 2023
@RambIing
Copy link

RambIing commented Nov 21, 2023

@gorhill noticed something that the new filter doesn't block. Ads still show if you open a tweet through the image.

Images ![image](https://github.com/uBlockOrigin/uAssets/assets/16966523/14da0e63-2c3b-43b0-a4c7-713cfd5591e9) ![image](https://github.com/uBlockOrigin/uAssets/assets/16966523/01de7736-a255-4181-afc8-f90f911757e0)

##[aria-label="Timeline: Conversation"] [data-testid="cellInnerDiv"]:has([data-testid="top-impression-pixel"])
or
##[aria-labelledby="accessible-list-9"] [data-testid="cellInnerDiv"]:has([data-testid="top-impression-pixel"])
seems to fix this.

Troubleshoot info (if not reproducible) uBlock Origin: 1.53.0 Chromium: 119 filterset (summary): network: 103496 cosmetic: 43782 scriptlet: 19068 html: 0 listset (total-discarded, last-updated): default: user-filters: 3-0, never easylist: 75532-16, 3h.20m easyprivacy: 32902-64, 3h.20m plowe-0: 3718-1078, 3h.20m ublock-badware: 7423-137, 3h.20m ublock-filters: 35635-156, 3h.20m ublock-privacy: 680-5, 3h.20m ublock-quick-fixes: 115-38, 3h.20m ublock-unbreak: 2155-32, 3h.20m urlhaus-1: 9855-0, 3h.20m filterset (user): [array of 3 redacted] trustedset: added: [array of 41 redacted] switchRuleset: added: [array of 3 redacted] userSettings: [none] hiddenSettings: [none] supportStats: allReadyAfter: 227 ms (selfie) maxAssetCacheWait: 110 ms

@DaisukeDaisuke
Copy link

DaisukeDaisuke commented Nov 21, 2023

@gorhill It also does not appear to block reply ads that appear with the image
Problematic URL: (Please click on the image)
https://twitter.com/gorhill/status/996016461943574528/photo/1

Ads have placementTracking, so in my opinion it would be a good idea to block this
Please note that there are 5 placementTrackings for some reason

I am a beginner in filters, so please do not add it to My Filters as is

##[data-testid="placementTracking"]

The html of the ad is as follows. McAfee is inserting html, but it's probably not relevant
ad.txt

Images

Desktop Screenshot 2023 11 21 - 19 13 26 16

@huyszn
Copy link
Contributor

huyszn commented Nov 21, 2023

How about this filter for blocking reply ads that appear when you click on images?
twitter.com##[data-viewportview="true"] [data-testid="cellInnerDiv"]:has([data-testid="top-impression-pixel"])

@DaisukeDaisuke
Copy link

DaisukeDaisuke commented Nov 21, 2023

Sorry for reposting(i am confused)
I was unable to reproduce the problem after forcefully updating the filter
I'm sorry

@Hakkin
Copy link

Hakkin commented Nov 21, 2023

Problematic URL: (Please click on the image)
https://twitter.com/gorhill/status/996016461943574528/photo/1

I was able to reproduce it using this URL using the latest filters, An ad will show in the rightmost column when you click to expand the main image. It's because that column doesn't have the primaryColumn testid.

Removing the [data-testid="primaryColumn"] selector from the current filter fixes it for me. (ie: twitter.com##[data-testid="cellInnerDiv"]:has([data-testid="top-impression-pixel"]))

@DaisukeDaisuke
Copy link

AdGuard – Ads solution seems to remove ads next to images
AdguardTeam/AdguardFilters#166718

AdguardTeam/AdguardFilters@b3a8e2d

@MasterKia
Copy link
Member

#20807:
https://x.com/HeyKinde/status/1724562915213373515?s=20
https://github.com/uBlockOrigin/uAssets/assets/17685483/76ae7203-370f-4d0f-9246-5cb7ac738f62

@gorhill
Copy link
Member

gorhill commented Nov 21, 2023

I don't see the label "Ad" in that tweet:

screenshot

image

And the 3-dot menu doesn't have an entry like "About this ad", it's a plain tweet on my side.


Also, the reporter's troubleshooting information doesn't show that the new filter was injected on the page:

 extended:
    ##[data-testid="trend"]:has-text(/Promoted|Gesponsert|Promociona…
    ##div[data-testid="trend"]:has(path[d$="10H8.996V8h7v7z"])

Twitter is a long-lived page, people have to make sure that the page is reloaded after they force an update of filter lists.

@uBlock-user
Copy link
Contributor

@DaisukeDaisuke I get the blank spaces / flickering when I refresh a Twitter page and then immediately scroll down.

Yep, saw the same thing after a while of browsing

@RambIing
Copy link

I don't see the label "Ad" in that tweet:

screenshot
And the 3-dot menu doesn't have an entry like "About this ad", it's a plain tweet on my side.

Also, the reporter's troubleshooting information doesn't show that the new filter was injected on the page:

extended: ##[data-testid="trend"]:has-text(/Promoted|Gesponsert|Promociona… ##div[data-testid="trend"]:has(path[d$="10H8.996V8h7v7z"])

Twitter is a long-lived page, people have to make sure that the page is reloaded after they force an update of filter lists.

Had that as well earlier but in the replies of a tweet with an image:

Screenshots

image
image
image

Seems to omit the tracking HTML found earlier. Hard to reproduce (took at minimum 20 tries to get the one in the screenshot), but the earliest case I can find is this: https://twitter.com/MattBinder/status/1700227953043337662

First thought is to check graphQL request for the json object promotedMetadata since the HTML is indistinguishable from an actual tweet

@D4niloMR
Copy link
Contributor

twitter.com##[aria-label^="Timeline:"] [data-testid="cellInnerDiv"] > div:has([data-testid="top-impression-pixel"]):remove()

This filter fix both: #20792 (comment) #20792 (comment)

Also if they add video ads this will prevent the video playing in the background.

Yuki2718 pushed a commit that referenced this issue Nov 22, 2023
* update

fix flickering/ads in opened images

* Update filters-2023.txt

* Update filters-2023.txt

* Update quick-fixes.txt
Yuki2718 added a commit that referenced this issue Nov 30, 2023
@RambIing RambIing mentioned this issue Jan 16, 2024
9 tasks
@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Mar 17, 2024

Anyone with dev build, can you test this filter? This one is to prune the ads post from the request using new update of json-prune

twitter.com##+js(json-prune-xhr-response, data.home.home_timeline_urt.instructions.[].entries.[-].content.itemContent.promotedMetadata, , propsToMatch, url:/api/graphql)

(Need to disable other cosmetic filters first for properly testing)


Also, this one is for the ads in the comment. Based on the report in https://www.reddit.com/r/uBlockOrigin/comments/1bgcmxr/ads_in_twitter_comments/

twitter.com##+js(json-prune-xhr-response, data.threaded_conversation_with_injections_v2.instructions.[].entries.[-].content.items.[].item.itemContent.promotedMetadata, , propsToMatch, url:/api/graphql)

@gwarser
Copy link
Contributor

gwarser commented Mar 17, 2024

@stephenhawk8054 it targets these?

image

Then looks like it works.

@stephenhawk8054
Copy link
Member

Yup, I meant the posts with Ad there.

@Yuki2718
Copy link
Contributor

Yuki2718 commented Mar 17, 2024

Looks working on home timeline and comments, but ofc not on search and promotion in right-side. It will be useful if we can use wildcard in the properties, or JSONPath specification to remove all promotedMetadata.

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Mar 17, 2024

It would be the same idea for search

twitter.com##+js(json-prune-xhr-response, data.search_by_raw_query.search_timeline.timeline.instructions.[].entries.[-].content.itemContent.promotedMetadata, , propsToMatch, url:/api/graphql)

Actually I would think clear separate json paths would be better for performance as they don't need to loop through many (nested) entries


I can't reproduce the right side tho, and I don't know how they look like. It just shows "Trends for you" and "Who to follow" on my side.

@Yuki2718
Copy link
Contributor

Yuki2718 commented Mar 17, 2024

As JSONPath is specifically made to search and extract in json, I guess it's good addition anyway.

I can't reproduce the right side tho, and I don't know how they look like. It just shows "Trends for you" and "Who to follow" on my side.

As with other ads, you have to refresh many times. Sometimes still may not be shown, then just test after a while.

@stephenhawk8054
Copy link
Member

I think JSONPath-related would take quite some time to implement (if we decide to).

It's not just this site, but I think the network-related scriptlets are mainly for co-operating with the cosmetic ones, to reduce the number of potential glitches/slipped caused by cosmetic filters, especially for dynamic big sites like this. Raw data are usually not manipulated as much as after they are represented in DOM (once they are in DOM, websites can do whatever they want via js and css), since those data could be used internally by the sites too.

I think having solutions for 2 approaches at the same time would be good.

@Yuki2718
Copy link
Contributor

Yeah, it's just if gorhill have time to implement. But Twitter/X has been changing structure and things like threaded_conversation_with_injections_v2 is likely to change in future, while promotedMetadata may still be constant. Similar to entity filters, wildcard or JSONPath will reduce the maintenance cost in the long run.

@gorhill
Copy link
Member

gorhill commented Mar 17, 2024

I wasn't aware of JSONPath, this will require some reading: https://www.ietf.org/archive/id/draft-ietf-jsonpath-base-03.html

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Mar 17, 2024

I think another aspect of JSONPath would be the selection based on the values in the objects? Which is not a concept in current objectPrune yet.

For example, pruning a branch if there's an object {"key": "Ad"} following.

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Mar 18, 2024

Btw, I think we can test the above scriptlets in dev build first?

@Yuki2718
Copy link
Contributor

Sure, go ahead.

stephenhawk8054 added a commit that referenced this issue Mar 18, 2024
stephenhawk8054 added a commit that referenced this issue Mar 31, 2024
stephenhawk8054 added a commit that referenced this issue Apr 8, 2024
stephenhawk8054 added a commit that referenced this issue Apr 11, 2024
@Yuki2718
Copy link
Contributor

Yuki2718 commented Oct 8, 2024

I guess AdguardTeam/AdguardFilters@019a78d is not covered?

@stephenhawk8054
Copy link
Member

I don't think so. Need to be able to reproduce first.

@stephenhawk8054
Copy link
Member

Ok, I think this works for me:

x.com##+js(json-prune-xhr-response, data.immersiveMedia.timeline.instructions.[].entries.[-].content.itemContent.promotedMetadata, , propsToMatch, url:/ImmersiveMedia)

stephenhawk8054 added a commit that referenced this issue Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests