Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add config to manage how long logs are stored #3257

Merged
merged 9 commits into from Nov 19, 2023
Merged

Add config to manage how long logs are stored #3257

merged 9 commits into from Nov 19, 2023

Conversation

ghost
Copy link

@ghost ghost commented Mar 13, 2023

Closes #2942

This pr adds a config option called log_expiration which can be configured in the same way as thread_cooldown etc.
According to the closed_at field this config option will delete logs which have passed the time specified in log_expiration.

This will work retrospectively.

@ghost
Copy link
Author

ghost commented Mar 13, 2023

I have tested this extensively but it would be better if someone could review this to make sure its efficient. Also since setting the config option will immediately start deleting old logs not sure if we should have some kind of safeguard?

@StephenDaDev
Copy link
Member

I think a safeguard is probably a good measure just to make sure nobody does anything they'd rather not do

@ghost
Copy link
Author

ghost commented Mar 14, 2023

I think a safeguard is probably a good measure just to make sure nobody does anything they'd rather not do

Actually if you look at it, the config option is set after the bot is running, they will need to restart the bot for it to come into effect. In this time they can remove the config option to prevent logs from being deleted. However vise versa if the logs are being deleted and they remove the config option, the deletion will stop without needing a restart.

Maybe this is enough?

@StephenDaDev
Copy link
Member

@lorenzo132 What do you think?

@lorenzo132
Copy link
Member

@lorenzo132 What do you think?

I'm a fan of the logs expiration config, specialy on a company wise level this is a very nice feature to have.

@ghost
Copy link
Author

ghost commented Mar 14, 2023

Used delete_many instead of looping delete_one based on recommendation by @khakers

khakers
khakers reviewed Mar 14, 2023
View reviewed changes
core/config_help.json Outdated Show resolved Hide resolved
bot.py Show resolved Hide resolved
Cordila and others added 2 commits March 14, 2023 15:18
Taaku18
Taaku18 requested changes Jul 14, 2023
View reviewed changes
bot.py Show resolved Hide resolved
bot.py Outdated Show resolved Hide resolved
bot.py Outdated Show resolved Hide resolved
@ghost
Copy link
Author

ghost commented Sep 8, 2023

All requested changes have been done and tested.

Jerrie-Aries
Jerrie-Aries reviewed Sep 8, 2023
View reviewed changes
Copy link
Contributor

@Jerrie-Aries Jerrie-Aries left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There were copy and paste errors in config_help.json, I believe.

Also a suggestion about not stopping the loop if the config for it is not set.

core/config_help.json Outdated Show resolved Hide resolved
bot.py Show resolved Hide resolved
@Jerrie-Aries
Copy link
Contributor

I did a couple of quick tests. Seems working fine for me.

Taaku18
Taaku18 approved these changes Nov 19, 2023
View reviewed changes
Copy link
Collaborator

@Taaku18 Taaku18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for making the requested changes! Looks good.

@Taaku18 Taaku18 added staged Staged for next version and removed in progress labels Nov 19, 2023
@Taaku18 Taaku18 merged commit 54f7b1c into modmail-dev:development Nov 19, 2023
2 checks passed
@Taaku18 Taaku18 mentioned this pull request Nov 20, 2023
Merged
raidensakura added a commit to raidensakura/modmail that referenced this pull request Dec 26, 2023
@raidensakura
Add config to manage how long logs are stored (modmail-dev#3257)
b1e4e6c
raidensakura added a commit to raidensakura/modmail that referenced this pull request Dec 27, 2023
@raidensakura @khakers @Taaku18
Merge develop into stable for v4.2.0 (#23)
aba4834 
* A few optimizations

- Reduce Docker image size
- Migrate requirements.txt to Pipfile
- Remove unused dependencies
- Add a few unnecessary files in .dockerignore

* Pull modmail-dev#3281

* Pull modmail-dev#3279

* Pull modmail-dev#3278

* Pull modmail-dev#3277 & proper indentation

* Squashed commit of the following:

commit 2e32d6e
Merge: 51a023c b21037c
Author: Martin <[email protected]>
Date:   Sun Apr 9 21:22:08 2023 +0200

    Merge branch 'modmail-dev:master' into master

commit 51a023c
Author: Martin <[email protected]>
Date:   Mon Apr 3 13:35:56 2023 +0200

    Changelog+Small fixes

commit f5f25b5
Author: Martin <[email protected]>
Date:   Mon Apr 3 13:10:35 2023 +0200

    confirm_thread_creation Buttons instead of reactions

commit 11e094c
Merge: ebd38d3 e919304
Author: Taku <[email protected]>
Date:   Tue Mar 14 05:19:54 2023 -0700

    Merge branch 'master' into master

commit ebd38d3
Author: Taku <[email protected]>
Date:   Tue Mar 14 05:18:17 2023 -0700

    Fix indentations

commit b77e435
Author: Martin <[email protected]>
Date:   Sun Mar 12 17:22:42 2023 +0100

    Reminder Plugin

    Created a reminder plugin

* Pull modmail-dev/modmail#modmail-dev#3271

* Squashed commit of the following:

commit 13437cd
Author: baptiste0928 <[email protected]>
Date:   Tue Mar 21 22:29:24 2023 +0100

    Format with black

commit 8a6ecd6
Merge: d9e364e 02250e8
Author: Taku <[email protected]>
Date:   Tue Mar 21 13:30:40 2023 -0700

    Merge branch 'development' into feat/guild-icon-size

    Signed-off-by: Taku <[email protected]>

commit d9e364e
Author: baptiste0928 <[email protected]>
Date:   Sun Mar 19 20:17:35 2023 +0100

    Update changelog

commit d0bfeb4
Author: baptiste0928 <[email protected]>
Date:   Sun Mar 19 20:06:20 2023 +0100

    Set size for guild icons on embeds

* Format with black

* Pull modmail-dev/modmail#modmail-dev#3233

* Squashed commit of the following:

commit cf3444a
Author: Khakers <[email protected]>
Date:   Tue Jan 24 14:13:15 2023 -0800

    Add dm_channel_id field to log entry

* Update changelog, workflow & remove bandit

* Pull modmail-dev#3272 + lock dependencies

* Add docs boilerplate + bumppython version

* Dpy bump, add versions for ?about

* Format with black

* Update README.md

* Fix permission error when installing plugin

* Add new mention_message config, black + ruff

* Switch from root to app user for Docker

* Add support for trailing space in prefix

* Add logviewer as built-in plugin

* logviewer plugin enhancements

* Minor logviewer changes

* Update changelog + logviewer pomelo change

* add pomelo for logbase

* Change logviewer favicon

* Refactor loglist object with proper serialization

* Remove favicon to untrack later

* fix log entry key error

* Replace local favicon with remote avatar

* Add markdown header support

* Switch repo link and branch

* Change more repo links

* Handle properly when loglist is the index page

* toggle off internal message checkbox on load

* Remove trailing slash, fix broken log url on index

* Fix broken default avatar url

* Fix pathing issue on displaying log list in index

* Add placeholder for not logged in user

* Fix blockquote parsing emote ending

* Whoops, remove comment

* Fix blockquote parsing + oauth navbar indicator

* Update docs

* config and docs change

* Update FUNDING.yml

* Update docs

* Update ignores + workflow

* Tweak loadcheck

* tweak loadcheck (2)

* blegh

* ✨

* Attempt to fix invalid guild_id on test

* Docs update

* Terminate process more gracefully

* Update dependencies for Python 3.11

* Refactor logviewer pomelo changes

* Minify docker image size

* Remove unused files

* bump black to fix checks

* Update dev requirements

* Update workflow

* More pomelo changes

* Replace avatar with local asset

* Fix broken author avatar on `logs` command

* Implement pomelo on channel name

* Format with black

* Add pre-commit hook

* Update requirements

* Overhaul block functionality

* Add plugin uninstall alias

* Update python badge

* Update README.md

* Update README.md

* Fix block datetime not being timezone aware

* Remove cairosvg

* Fix copy paste mistake

* Another copy paste issue sigh

* Fix NoneType on parse

* Improve block check and add message cooldown

* bump dpy

* Update internal dpy version

* Update ?info command for pomelo

* Bump version

* Fix missing package error

* Fix index compile error

* Fix space in filepath failing to load local plugin

* Escape codeblock language in class name

* Fix genesis message retrieval failure

* Update docs

* Don't build image on PR

* Blocklist rewrite (#11)

* Add blocklist

blocklist.py contains the initial classes and code for the new blocklist system.
The intention of this system is to unify blocking to one new mongodb collection and vastly simplify bot code required to manage it. Additionally, most simply methods should be side effect free.

* Add migrations.py

migrations.py contains functions to migrate legacy blocklists to the new blocklist. contains handling for both legacy modmail string and the newer dict storage

* Add blocklist bot.py integration

* rename BlockedUser to BlocklistItem & add more blocklist methods

Adds add_block, unblock_id, and is_id_blocked to blocklist class

* migrate block and unblock commands to new blocklist

* fix overzealous ide rename

* add get_all_blocks method and documentation + cleanup to blocklist.py

* refactor blocked command to use new block system

* cleanup block command code

now with more DRY

* migrate bot.py is_blocked method to new blocklist system

* migrate whitelist command and refactor is_id_blocked to return tuple

is_id_blocked now returns a tuple[bool, Optional[BlocklistItem]] so the calling code can get information about the block easily

* rename BlocklistItem to BlocklistEntry

* Add deprecation warning logs to legacy block code and docstrings to bot.py block related code

* clean up migrations.py code

* Add [p]migration command

* add final deprecation docstrings and refactor contact command to use blocklist.is_user_blocked() instead of core is_blocked()

* remove blocklist print statements

* Format with black

---------

Co-authored-by: Raiden Sakura <[email protected]>

* fix note code

upgrade signature of note code state store_user method to match expected discord.py method

use a typed checked dict for author instead of making a declaring a class in a for loop that replicates a different type. hopefully a type checker would raise warnings the next time one of these changes

* Fix failing tests and docker build

* fix blocklist bugs (#13)

* fix missing await statements in new block method usage

* remove Self type importing and usage

* rename placeholder variable name

* StrEnum is only in 3.11...

---------

Co-authored-by: Raiden Sakura <[email protected]>

* oops

* Fix object is not subscriptable for 3.8

* Pull 2 changes from upstream (#14)

* Fixed unset registry_plugins_only causing non-registry plugins to fail to install

* To avoid confusion, config.get() only accepts one pos argument now

---------

Co-authored-by: Taku <[email protected]>

* Format with black

* More blocklist fixes (#15)

* update requirements.txt

* fix blocklist migration detection

* fix migrations.py type hints for python 3.8

* Add index for channel_id

Channel_id is the main query field for nearly every log related database operation and for some reason it isn't indexed. This should speed up all log related database reliant operations especially on large collections

* Update CHANGELOG.md

Update changelog with my changes

* Remove CairoSVG dependencies from Dockerfile

* Improve is_image detection to actually detect images (excluding videos) and add content_type value

* Fix logviewer positional arg error

* partially reverts 2f76202 which added lottie sticker support and updates dependency lock (#20)

* partially reverts 2f76202 which added lottie sticker support and updates dependency lock

* Lint

---------

Co-authored-by: Raiden Sakura <[email protected]>

* Synchronize thread NSFW status and title with the database (#18)

* Changing the thread title or nsfw status now immediately updates in the database

!Changes signature of thread.set_title()

adds update_nsfw and update_title methods to the mongodb client

* Lint

---------

Co-authored-by: Raiden Sakura <[email protected]>

* Refactor docker workflow to multiplatform build (#21)

Upgrades out of date action & adds dependency actions (QEMU & BuildX) for multiplatform build. Sets platforms to Linux/amd64 and Linux/arm64

* Update funding

* Remove internal logviewer plugin

* Add .gitignore for local plugin

* Migrate to poetry

* Update workflow for poetry

* Update workflow

* Oops, forgot to install deps

* Update launch command

* Attempt to solve image not building arm64

* Add missing libffi

* Refactor Dockerfile

* Resolve deprecated pkg_resources warning

* Update changelog

* Bump workflow versions

* oops

* Add [speed] extra for dpy

* Bump dpy, don't exit when dpy version mismatch

* Fix sticker replies not sending

* Update changelog

* Improved join/leave message for multiple servers.

From modmail-dev@0086e04

* Fix code scanning alert #5

* Fixed compat with MissingRequiredArgument requiring additional param

modmail-dev@9464c5d

* Fix plugin update bug on broken plugin

https://github.com/modmail-dev/Modmail/pull/3295/files

* Update CHANGELOG.md

* Fix rate limit issue on raw reaction add/remove events

https://github.com/modmail-dev/Modmail/pull/3306/files

* Remove Heroku Reference from Debug Command

modmail-dev@d891ad1

* Update CHANGELOG.md

* Remove unused files

* Update pip

* Bump aiohttp and use [speedups]

* Update .dockerignore

* Rename project

* Add shell script for pm2

* Update docs

* Update env example

* Remove app.json

* Update dependencies

* Update readme

* Add config to manage how long logs are stored (modmail-dev#3257)

* Add presence intent into .env.example

* Formating

* Fix missing guild icon url

* Revert gyazo image embedding

* Fix overly permissive regex

* Specify fork on metadata

* Update changelog

* Bump version

---------

Co-authored-by: khakers <[email protected]>
Co-authored-by: Taku <[email protected]>
raidensakura pushed a commit to raidensakura/modmail that referenced this pull request Apr 10, 2024
@khakers @Taaku18
@Jerrie-Aries @raidensakura
Add config to manage how long logs are stored (modmail-dev#3257)
614d37c 
* Add config to change how long logs are stored

* Optimize deleting logs

* Update core/config_help.json

Co-authored-by: khakers <[email protected]>
Signed-off-by: Cordila <[email protected]>

* Testing changes

* Update core/config_help.json (Fix copy paste)

Co-authored-by: Jerrie <[email protected]>
Signed-off-by: Cordila <[email protected]>

---------

Signed-off-by: Cordila <[email protected]>
Co-authored-by: Cordila <[email protected]>
Co-authored-by: khakers <[email protected]>
Co-authored-by: Taku <[email protected]>
Co-authored-by: Jerrie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
staged Staged for next version
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@StephenDaDev @lorenzo132 @Jerrie-Aries @khakers @Taaku18