UnicodeEncodeError Logger exception with iso8859_2 encoding (non UTF-8)

[vserbu]

I've got an error, what can I do? This is the error log:

2022-02-12 08:55:45,426 pleroma_bot ERROR: Exception occurred
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pleroma_bot/cli.py", line 363, in main
config = yaml.safe_load(stream)
File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 125, in safe_load
return load(stream, SafeLoader)
File "/usr/local/lib/python3.8/dist-packages/yaml/init.py", line 81, in load
return loader.get_single_data()
File "/usr/local/lib/python3.8/dist-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/usr/local/lib/python3.8/dist-packages/yaml/composer.py", line 58, in compose_document
self.get_event()
File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 118, in get_event
self.current_event = self.state()
File "/usr/local/lib/python3.8/dist-packages/yaml/parser.py", line 193, in parse_document_end
token = self.peek_token()
File "/usr/local/lib/python3.8/dist-packages/yaml/scanner.py", line 129, in peek_token
self.fetch_more_tokens()
File "/usr/local/lib/python3.8/dist-packages/yaml/scanner.py", line 223, in fetch_more_tokens
return self.fetch_value()
File "/usr/local/lib/python3.8/dist-packages/yaml/scanner.py", line 577, in fetch_value
raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
in "/usr/local/lib/python3.8/dist-packages/pleroma_bot/config.yml", line 2, column 17

[robertoszek]

Hi @vserbu !

Could you share your config.yml file's content (obscuring the sensitive data, like the tokens, etc.), it seems the YAML parser is complaining about the formatting but I'm not sure what specifically is causing it.

[robertoszek]

I'm wondering if you used the config.yml.sample file on this repo and maybe this line does not have the final ending quote ":

pleroma-bot/config.yml.sample

Line 41 in 97bded2

original_date_format: "%Y/%m/%d %H:%M"

Which would cause issues while trying to parse it.

[robertoszek]

@vserbu
I could get a similar exception by putting a colon : inside a value, make sure you don't have something like this on your config as well:

pleroma_base_url: https://pleroma.instance:

If you really need that colon in a mapping's value for whatever reason (specifying a port, for example), you could put it inside quotes:

pleroma_base_url: "https://pleroma.instance:8080"

[vserbu]

I must point out that I could not install pleroma bot with pip install pleroma-bot because i always got:

Could not find a version that satisfies the requirement pleroma-bot (from versions: ) No matching distribution found for pleroma-bot

So I installed it with pip3 install pleroma-bot.
This is my config.yml, I used minimal yml, is it ok? I'm using mastodon too, not pleroma.

[robertoszek]

@vserbu

Could you comment out using # the line:

 Change this to your target Fediverse instance

[robertoszek]

And by the way, if you're using Mastodon you need your account ID (it's a limitation with the API).
more info here

If you are an admin or a moderator on the instance you can find the ID of an user this way too:
https://discourse.joinmastodon.org/t/how-to-get-mastodons-user-id-from-mastodon-account-acct-username-server/1658/6

A moderator or admin of the instance of your account can go to Preferences->Moderation->Accounts and find your account. After opening the account details, the URL will reveal the ID of the user.

[vserbu]

I'm the admin at salocha.online and I've found account ID but everything is the same. I tried again but this time with complete and not minimal config.yml . (I used minimal yml because I thought that it is going to be easier).

I would like to copy/paste complete config.yml here but it is not possible, because the styling is messing with the text.

Here is the link to my config.yml - https://justpaste.it/5zr0e

I left XXXXX for all the tokens and keys for the privacy. Do I have to #comment the keys I don't use like:

consumer_key: xxxxxxxxxxxxxxxxxxxxxxxxx
consumer_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
access_token_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
access_token_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[robertoszek]

For pasting the content here, you can wrap it with backticks:

```
Config's content goes here
```

The minimal config should be fine, those keys you mentioned are only needed if the Twitter account you're mirroring has their tweets protected. They're optional but if they are present pleroma-bot will try to use them.

Start with something simple first:

pleroma_base_url: https://salocha.online
max_tweets: 40
twitter_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
users:
- twitter_username: SciFiNow
  pleroma_username: 1771
  pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

In the link you provided you seem to have repeated users with the same Twitter account too, which I'm not sure it is what you're trying to do.

If you get any errors please provide the output and I'll try to help out.

[robertoszek]

@vserbu
Going by the first error you posted, looks like your encoding is iso8859_2.
Does it make any difference if you run pleroma-bot like so?:

PYTHONIOENCODING=UTF-8 pleroma-bot

[vserbu]

I tried with again with minimal config.yml and with UTF-8 encoding, I think something changed:
...
2022-02-12 17:59:07,905 - pleroma_bot - INFO - config path: /usr/local/lib/python3.8/dist-packages/pleroma_bot/config.yml
ℹ 2022-02-12 17:59:07,906 - pleroma_bot - INFO - tweets temp folder: /usr/local/lib/python3.8/dist-packages/pleroma_bot/tweets
ℹ 2022-02-12 17:59:07,909 - pleroma_bot - INFO - ======================================
ℹ 2022-02-12 17:59:07,910 - pleroma_bot - INFO - Processing user: 1771
✖ 2022-02-12 17:59:08,490 - pleroma_bot - ERROR - Exception occurred (cli.py:502)
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pleroma_bot/cli.py", line 398, in main
user = User(user_item, config, base_path)
File "/usr/local/lib/python3.8/dist-packages/pleroma_bot/cli.py", line 207, in init
self._get_twitter_info()
File "/usr/local/lib/python3.8/dist-packages/pleroma_bot/_twitter.py", line 68, in _get_twitter_info
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.twitter.com/1.1/users/show.json?screen_name=SciFiNow

[vserbu]

Just to clarify, I'm trying to mirror twitter account @ScifiNow (I just picked it for testing reasons).
I'm admin at salocha.online mastodon instance and I created there a bot account named @[email protected] and it has ID 1771. From there I'm using bearer token and from twitter bearer token too. Salocha.online is hosted on digitalocean.com .

[robertoszek]

Does your Twitter API project has Elevated access?
You can request it here

It was introduced recently and it is needed for Twitter's API v1.1
#55

[vserbu]

No, maybe that is a problem. I thought that for this purpose I need only basic api?

[robertoszek]

That was the case until recently, when Twitter pushed out v2 out of early access and in the process limited v1.1 to Elevated:
https://blog.twitter.com/developer/en_us/topics/tools/2021/build-whats-next-with-the-new-twitter-developer-platform

Here's the breakdown of the levels:
https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-level

And we would happily migrate to v2, however there are missing features which are only available on v1.1 at the moment.
So because of that we need a Twitter bearer token of an application with access to both v1.1 and v2.

[vserbu]

Hmm...I'm not sure they will approve me v2 for Mastodon bots. Do you have bots working with this script and using v1.1 api?

[robertoszek]

You should have Essential access by default, which has access to Twitter's API v2.
What you may be missing is Elevated access (and by extension, to v1.1).

Other people were able to request it and be granted access in the past for this usecase:
#54 (comment)

[vserbu]

Aha, I understand. I'll try.

[vserbu]

Thank you!!
Looks like you were right, now with new key (elevated) it seems the script is working. Great! (I hope they won't strip me of elevated keys because they granted me automatically, my "why do you need it essay" was pretty basic)

[robertoszek]

No problem!
As long as you don't abuse it I think you're on the clear.
And even then, in the near future I hope to remove the remaining v1.1 bits we use, so you'll only need Essential access by then.

If you don't mind, I'll reuse this issue for the problem you had with the encoding (so you don't have to launch the bot using PYTHONIOENCODING=UTF-8 everytime).
It should be resolved by the next release 51174a5

[vserbu]

Of course, and thank you for the really fine script with lots of options included. I hope I'm not going to abuse it, my primary intention was to select few interesting sci-fi twitter accounts and bot them to mastodon.

I suppose I have to make a cron job to run the script every hour or so to pick up the new tweets?

[robertoszek]

Yeah, to abuse it I think you would have to be doing millions of requests a day, I'm pretty sure you won't run into issues on that front.

And yes, you can use cron or a systemd service to fetch new tweets periodically:
https://robertoszek.github.io/pleroma-bot/gettingstarted/automateit/#automate-it

[robertoszek]

Oh, and I forgot to add that twitter_username can be a list (which I could do a better job of explaining on the documentation, to be fair).
So you can add multiple Twitter accounts to be posted on the same Fediverse account (and the tweets will be ordered chronologically across all of them).
Your config would look something like this if you want to aggregate multiple Twitter accounts into one Mastodon one:

pleroma_base_url: https://salocha.online
max_tweets: 40
twitter_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
users:
- twitter_username:
    - SciFiNow
    - SFXmagazine
    - whateverTwitterAccount
  pleroma_username: 1771
  pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

You lose the ability of getting their profile info (because there's no "main" account) though.

[vserbu]

Is there another way to add another twitter username in series so I can get profile info and everything? Like two or more minimal config ymls merged. Are there any rules how to make that config?

[robertoszek]

I guess you could achieve that with something along the lines of:

pleroma_base_url: https://salocha.online
max_tweets: 40
twitter_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
users:
- twitter_username:
    - SciFiNow
    - SFXmagazine
    - whateverTwitterAccount
  pleroma_username: 1771
  pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
- twitter_username: SciFiNow
  pleroma_username: 1771
  pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

This would get the tweets from the 3 Twitter accounts (not updating the Fedi profile),
and then it would process the next config user, see that they're no new tweets to post and finally it would update the profile (bio, profile image and profile banner).

You can add as many users in series as you want.

[vserbu]

For some reason I get error for the first (multiple twitter acc:) and the second version (combination).

Here is the error log for the multiple twitter acc: (maybe there is another way to separate twitter accounts?)

. . .
ℹ 2022-02-12 22:42:25,155 - pleroma_bot - INFO - It seems like pleroma-bot is running for the first time for this Twitter user: -SciFiNow -SFXmagazine
✖ 2022-02-12 22:42:25,344 - pleroma_bot - ERROR - Exception occurred (cli.py:502)
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/pleroma_bot/cli.py", line 398, in main
user = User(user_item, config, base_path)
File "/usr/local/lib/python3.8/dist-packages/pleroma_bot/cli.py", line 188, in init
self.pinned_tweet_id = self._get_pinned_tweet_id()
File "/usr/local/lib/python3.8/dist-packages/pleroma_bot/_pin.py", line 136, in _get_pinned_tweet_id
response.raise_for_status()
File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.twitter.com/2/users/by/username/-SciFiNow%20-SFXmagazine?user.fields=pinned_tweet_id&expansions=pinned_tweet_id&tweet.fields=entities
root@mastodon313onubuntu1804-s-1vcpu-1gb-fra1-01:/usr/local/lib/python3.8/dist-packages/pleroma_bot#

[robertoszek]

It looks like it's taking the twitter_username value as a string, weird.
Are you sure there is an space between the dashes - and the values?

- twitter_username:
    - SciFiNow
    - SFXmagazine
    - whateverTwitterAccount

instead of:

- twitter_username:
    -SciFiNow
    -SFXmagazine
    -whateverTwitterAccount

If that's not it, maybe sharing your current config could help us find what the issue is.

[vserbu]

You were right, one again, the space between the dashes were missing. Tnx.

[robertoszek]

Hey @vserbu !
Hopefully the latest version v1.0.2 fixed the UnicodeEncodeError you encountered in the past.
If not (or if you run into any other bugs), feel free to open a new issue, have a nice day!