Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for GoToSocial #115

Open
ynakao opened this issue Jan 22, 2023 · 9 comments
Open

Add support for GoToSocial #115

ynakao opened this issue Jan 22, 2023 · 9 comments
Assignees
@robertoszek
Labels
enhancement New feature or request

Comments

@ynakao
Copy link

ynakao commented Jan 22, 2023

GoToSocial is another Mastodon API compatible ActivityPub server implementation, but not a fork of Mastodon or others.

GoTosocial is still in alpha stage, so they hasn't yet implemented Mastodon/ActivityPub API completely, but I think it works like Mastodon server.

I tried to import tweets from twitter-archive.zip to GoToSocial account with minimal config, but KeyError exception occurred.

  • config.yml
pleroma_base_url: https://gts.example.com
software: mastodon
users:
- twitter_username: nakaoyuji
  # I got an account ID by following the document.
  # curl 'https://gts.example.com/api/v2/search?q=<username>&resolve=true&limit=5' -H 'Authorization: Bearer xxxxx'
  # ID on GoTosocial seems to be 26 alphanumeric charactors.
  pleroma_username: 02G7NUT8E2RBPBPZS4OXQ89T1T
  pleroma_token: xxxxx
  archive: /path/to/twitter-archive.zip
  • error.log with minimal config above
2023-01-22 08:46:33,872 pleroma_bot ERROR: Exception occurred for user, skipping...
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pleroma_bot/cli.py", line 556, in main
    user = User(user_item, config, base_path, posts_ids)
  File "/usr/lib/python3.10/site-packages/pleroma_bot/cli.py", line 289, in __init__
    self.mastodon_enforce_limits()
  File "/usr/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 547, in mastodon_enforce_limits
    if len(self.display_name[t_user]) > 30:
KeyError: 'nakaoyuji'

I also tried to change config.yml. For example, removing software section to make pleroma-bot assume GtS server is like default pleroma server, and changing pleroma_username section to real GoTosocial username, but all had no effects.

  • error.log when removing software section
2023-01-22 09:37:14,673 pleroma_bot ERROR: Exception occurred for user, skipping...
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pleroma_bot/cli.py", line 705, in main
    user.check_pinned(posted)
  File "/usr/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 340, in check_pinned
    ).format(str(self.pinned_tweet_id)))
AttributeError: 'User' object has no attribute 'pinned_tweet_id'

  • error.log when changing pleroma_username to actual GoTosocial username was the same error with the first one echoing KeyError.

  • version info
    pleroma-bot: v1.2.0 installed from AUR
    GoToSocial: v0.6.0
@robertoszek
Copy link
Owner

Hi!

I'll look into it by finding a GoToSocial instance where I can test the bot with, I've been following the development on and off and waiting for the beta before trying it out.

So far, I see that these endpoints are the ones missing that could cause issues with the bot:

/api/v1/statuses/:id/pin POST (Pin a status to profile)
/api/v1/statuses/:id/unpin POST (Unpin a status from profile)

(and /api/v1/statuses POST (Create a new status) is missing the poll parameter)

Regarding the first config you shared:

pleroma_base_url: https://gts.example.com
software: mastodon
users:
- twitter_username: nakaoyuji
  # I got an account ID by following the document.
  # curl 'https://gts.example.com/api/v2/search?q=<username>&resolve=true&limit=5' -H 'Authorization: Bearer xxxxx'
  # ID on GoTosocial seems to be 26 alphanumeric charactors.
  pleroma_username: 02G7NUT8E2RBPBPZS4OXQ89T1T
  pleroma_token: xxxxx
  archive: /path/to/twitter-archive.zip

I'm curious about the error, perhaps it is unrelated to the target instance being GoToSocial.

Could you set up a virtual environment, install 1.2.1rc9 on it and run the config with it?:

$ python3 -m venv testgts
$ source testgts/bin/activate
(testgts) $ pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc9
(testgts) $ pleroma-bot --config /path/to/config

I'm curious if it would fail in some other unexpected way.

@robertoszek robertoszek self-assigned this Jan 22, 2023
@robertoszek robertoszek added the enhancement New feature or request label Jan 22, 2023
@ynakao
Copy link
Author

ynakao commented Jan 23, 2023

Thank you for your reply.

I installed v1.2.1rc9 and ran pleroma-bot as you described. Now the error message changed to AttributeError: 'User' object has no attribute 'pinned_tweet_id'. So, is this because GoToSocial hasn't implement relevant API yet?

2023-01-23 20:40:15,025 pleroma_bot ERROR: Exception occurred for user, skipping...
Traceback (most recent call last):
  File "/home/ynakao/test/pleroma-bot/pleroma-bot-v1.2.1-rc9/lib/python3.10/site-packages/pleroma_bot/cli.py", line 709, in main
    user.check_pinned(posted)
  File "/home/ynakao/test/pleroma-bot/pleroma-bot-v1.2.1-rc9/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 340, in check_pinned
    ).format(str(self.pinned_tweet_id)))
AttributeError: 'User' object has no attribute 'pinned_tweet_id'

@robertoszek
Copy link
Owner

So, is this because GoToSocial hasn't implement relevant API yet?

Most likely, perhaps we can simply skip the pinning functionality completely if the target is GoToSocial for now.

Any luck with 1.2.1rc10?:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc10

@ynakao
Copy link
Author

ynakao commented Jan 24, 2023

Yes! It looks working well! I didn't check whether all imports are properly done though, since it will take over 14 hours to finish.

@ynakao ynakao mentioned this issue Jan 24, 2023
Closed
@robertoszek
Copy link
Owner

robertoszek commented Jan 27, 2023 •
edited
Loading

@ynakao did you run into any new issues during your import?

I spun up a local GoToSocial instance to test with and I found that the response code for attachments being too big is different than on Mastodon/Pleroma (HTTP 400 instead of 413).
So I made some changes to take that into account on 1.2.1rc16

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc16

Everything else seemed to work as I expected.
(except pinning posts and polls, which are not implemented in GoToSocial yet, so we skip them)

  • Posting images:

image

  • Posting videos:

image

  • Updating the profile:

image

I'm going to keep trying to find edgecases but let me know if you ran into anything else weird.

@ynakao
Copy link
Author

ynakao commented Jan 28, 2023

I encountered the following TypeError on a newly created GoToSocial test account. Is there a way to spot which tweet causes this issue easily?

(pleroma-bot-v1.2.1-rc16) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc16]% ./bin/pleroma-bot --archive twitter-archive-2022-12-27.zip
...
⚠ 2023-01-28 13:36:58,595 - pleroma_bot - WARNING - Raising max_tweets to the maximum allowed value (_utils.py:610)
ℹ 2023-01-28 13:37:05,061 - pleroma_bot - INFO - tweets gathered:        23497
Processing tweets... : 100%|█████████████████████████████████████████████████████████████| 23497/23497 [04:53<00:00, 80.19it/s]
ℹ 2023-01-28 13:41:58,113 - pleroma_bot - INFO - tweets to post:         23497
Posting tweets... :  82%|██████████████████████████████████████████████████▋           | 19205/23497 [2:54:53<39:28,  1.81it/s]✖ 2023-01-28 16:36:52,759 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:721)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/cli.py", line 688, in main
    post_id = user.post(
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/_utils.py", line 794, in post
    post_id = self.post_pleroma(tweet, poll, sensitive, media, cw=cw)
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc16/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 310, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable

@robertoszek
Copy link
Owner

It looks like it failed to upload an attachment.
I've added a handler for the error that should display the Tweet ID, the response from the GoToSocial instance and which file was trying to upload on 1.2.1rc18:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.2.1rc18

Hopefully that would give us enough info to identify what's the issue there.

@ynakao
Copy link
Author

ynakao commented Jan 29, 2023

It seems error occurs when uploading mp4 files. I guess this is because video support in GoToSocial will be landed on v0.7.0 and I'm running the latest release v0.6.0. Are you running GoToSocial latest git commit version as you can post videos in your test?

Except for this, all operation worked properly in my case.

Error log 
(pleroma-bot-v1.2.1-rc18) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc18]% ./bin/pleroma-bot --archive twitter-archive-2022-12-27.zip
...
⚠ 2023-01-29 09:59:48,337 - pleroma_bot - WARNING - Raising max_tweets to the maximum allowed value (_utils.py:612)
ℹ 2023-01-29 09:59:54,767 - pleroma_bot - INFO - tweets gathered:        23497
Processing tweets... : 100%|█████████████████████████████████████████████████████████████| 23497/23497 [04:37<00:00, 84.69it/s]
ℹ 2023-01-29 10:04:32,289 - pleroma_bot - INFO - tweets to post:         23497
Posting tweets... :  82%|██████████████████████████████████████████████████▋           | 19205/23497 [2:54:55<39:33,  1.81it/s]✖ 2023-01-29 12:59:28,928 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       773124013677654016
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/773124013677654016/773124013677654016-PTZmtVxlXHRDGMcg.mp4
Size:   0.09MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 12:59:28,929 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  82%|██████████████████████████████████████████████████▋           | 19209/23497 [2:54:58<41:13,  1.73it/s]✖ 2023-01-29 12:59:31,233 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       773142132597125120
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/773142132597125120/773142132597125120-CrrAqk5VYAAWl88.mp4
Size:   0.1MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 12:59:31,234 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  84%|███████████████████████████████████████████████████▉          | 19699/23497 [2:59:30<36:35,  1.73it/s]✖ 2023-01-29 13:04:03,915 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       811535032577732609
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/811535032577732609/811535032577732609-5xBU-2VYTfLvI_FV.mp4
Size:   0.54MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:04:03,921 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  84%|███████████████████████████████████████████████████▉          | 19701/23497 [2:59:31<36:10,  1.75it/s]✖ 2023-01-29 13:04:05,060 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       811538743119052800
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/811538743119052800/811538743119052800-C0MqLTtVQAAwNfx.mp4
Size:   0.22MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:04:05,061 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  94%|██████████████████████████████████████████████████████████▍   | 22140/23497 [3:22:16<14:28,  1.56it/s]✖ 2023-01-29 13:26:49,244 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       1114169013477629952
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/1114169013477629952/1114169013477629952-D3ZSzfbU0AAxZg3.mp4
Size:   0.12MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:26:49,245 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... :  99%|█████████████████████████████████████████████████████████████▎| 23253/23497 [3:32:43<02:33,  1.59it/s]✖ 2023-01-29 13:37:16,749 - pleroma_bot - ERROR - Error uploading media:
Status code:    200
Tweet ID:       1505537146530430977
Filename:       /home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/tweets/1505537146530430977/1505537146530430977-FOS96kZaMAEFNmy.mp4
Size:   0.11MB
Response:       null (_pleroma.py:317)
✖ 2023-01-29 13:37:16,749 - pleroma_bot - ERROR - 'NoneType' object is not subscriptable (_pleroma.py:333)
Traceback (most recent call last):
  File "/home/hetzarch/test/pleroma-bot/pleroma-bot-v1.2.1-rc18/lib/python3.10/site-packages/pleroma_bot/_pleroma.py", line 309, in post_pleroma
    media_id = json.loads(response.text)["id"]
TypeError: 'NoneType' object is not subscriptable
Posting tweets... : 100%|██████████████████████████████████████████████████████████████| 23497/23497 [3:35:02<00:00,  1.82it/s]
ℹ 2023-01-29 13:39:34,789 - pleroma_bot - INFO - Updating profile:       <Response [200]>
(pleroma-bot-v1.2.1-rc18) [hetzarch@hetzarch][~/test/pleroma-bot/pleroma-bot-v1.2.1-rc18]%

@robertoszek
Copy link
Owner

Are you running GoToSocial latest git commit version as you can post videos in your test?

Yep, I just pulled the latest snapshot:
https://hub.docker.com/r/superseriousbusiness/gotosocial/tags

https://docs.gotosocial.org/en/latest/installation_guide/docker/#version

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@robertoszek robertoszek

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@robertoszek @ynakao