Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: [WIP] Moving off and off-pro to off2 #243

Merged
merged 70 commits into from
Nov 23, 2023
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
44a8562
chore: wip in moving off and off-pro to off2
alexgarel Jul 19, 2023
dcdc446
docs: wip on off reinstall
alexgarel Jul 25, 2023
de1b1e2
docs: wip on off reinstall
alexgarel Jul 25, 2023
43e6e39
chore: wip on off install on off2 - lxc confs
Jul 26, 2023
ec1f5ee
feat: more sanoid syncs on off2
alexgarel Jul 27, 2023
fcf049d
chore: wip on off reinstall on off2
alexgarel Jul 27, 2023
0fa984e
chore: wip
alexgarel Jul 29, 2023
cf6f1b0
fix: enable large file upload for product opener instances
alexgarel Aug 9, 2023
6e2dbc2
fix: share data folder between off and off-pro
alexgarel Aug 9, 2023
94f2b59
docs: wip on off2 migration
alexgarel Aug 10, 2023
6df4f5b
chore: export_files is shared between off and off-pro
alexgarel Aug 11, 2023
52e9040
chore: wip on reinstall on off2
alexgarel Aug 11, 2023
d26424b
chore: wip on reinstall on off2
alexgarel Aug 14, 2023
1d04b45
chore: wip on off reinstall on off2
alexgarel Aug 17, 2023
8e6bc90
chore: add memcached configuration
alexgarel Aug 22, 2023
8abb35b
docs: explain how to use munin
alexgarel Aug 22, 2023
d7b3c41
chore: wip on off reinstall on off2
alexgarel Aug 22, 2023
2a27b37
Merge branch 'develop' into off2-off-reinstall
alexgarel Aug 22, 2023
4236f08
chore: small progresses on off2 install
alexgarel Aug 25, 2023
02c685d
chore: nginx proxy conf for off and off-pro
alexgarel Aug 29, 2023
3cd9fba
fix: change snapshot-purge to target right ZFS pool
alexgarel Aug 31, 2023
5989800
feat: off2 munin configuration
alexgarel Aug 31, 2023
a274262
docs: wip on off reinstall on off2
alexgarel Aug 31, 2023
ca7c34f
feat: changed lxc config for id mapping
Sep 1, 2023
c0fd13c
wip on off reinstall on off2
alexgarel Sep 1, 2023
4bf4499
feat: add logs mound for off-pro
alexgarel Sep 5, 2023
535cd44
fix: pro instance must have it's own ssl certificate
alexgarel Sep 5, 2023
0995b8b
feat: sshd config for sftp
alexgarel Sep 5, 2023
dc12c9c
wip on off reinstall on off2
alexgarel Sep 5, 2023
5ad4079
chore: madenear.me config on reveres proxy
Sep 5, 2023
4966e49
wip on off2 migration:
alexgarel Sep 5, 2023
12044f4
feat: confs to get fail2ban working with nftables
alexgarel Sep 8, 2023
e3b8355
fix: fix syncoid.service
alexgarel Sep 11, 2023
4981fe8
fix: remove obsolete users sync
alexgarel Sep 11, 2023
5c0991d
chore: add ovh3 rsync_off1 script
alexgarel Sep 11, 2023
0c4e697
feat: howmuchsugar reverse proxy
alexgarel Sep 11, 2023
ea966c7
docs: using fail2ban to ban bots
alexgarel Sep 8, 2023
f1405fd
docs: wip on off reinstall on off2
alexgarel Sep 8, 2023
7d32fd2
docs: migration for howmuchsugar
alexgarel Sep 11, 2023
e6a2eb8
docs: zfs munin
alexgarel Sep 12, 2023
8f5cbe7
Merge branch 'develop' into off2-off-reinstall
alexgarel Sep 12, 2023
67956e8
docs: more munin stuff
alexgarel Sep 13, 2023
37093b7
feat: nginx images serving on off2
alexgarel Sep 13, 2023
600d847
feat: dry_run req limiting of off2 images serving
alexgarel Sep 13, 2023
6876c4d
docs: wip on off reinstall on off2
alexgarel Sep 14, 2023
713a950
fix: use a specific jail for manual ban
alexgarel Sep 15, 2023
f7c292a
fix: use a specific jail for manual ban
alexgarel Sep 15, 2023
3a73c21
chore: wip in migration (scripts)
alexgarel Sep 20, 2023
881c463
fix: fix on off2 (images) nginx config
alexgarel Oct 11, 2023
01390ab
fix: add off-pro images access to off container
alexgarel Oct 11, 2023
7fddb61
fix: typo in off container config
alexgarel Oct 11, 2023
e1203fe
doc: wip on off migration to off2
alexgarel Oct 11, 2023
f3fb1cc
chore: wip on off migration to off2
alexgarel Oct 17, 2023
67b10aa
chore: wip on off migration to off2
alexgarel Oct 19, 2023
c5389ff
docs: adding some general doc about the install
alexgarel Oct 27, 2023
80ef2b9
feat: add a script to snapshots promox subvolumes for a container
alexgarel Nov 9, 2023
e19911b
fix: add buffering on reverse nginx on access logs
Nov 10, 2023
232d202
feat: better clones update script
alexgarel Nov 15, 2023
8cb49e6
Merge branch 'develop' into off2-off-reinstall
alexgarel Nov 20, 2023
4b2d091
feat: reverse proxy mails on systemd errors
Nov 21, 2023
9109c82
feat: snapshot and sync pve volumes from off2 to ovh3
Nov 21, 2023
a14e384
feat: handling off2 pve volumes snapshots on ovh3
alexgarel Nov 21, 2023
7df8d2f
feat: maintenance page for off2 reverse proxy on openfoodfacts
Nov 22, 2023
c327355
chore: all your bases are belong to sanoid
alexgarel Nov 22, 2023
b9623f8
chore: certificates for howmuchsugare and madenear.me
alexgarel Nov 22, 2023
bcd75e9
chore: just before migration
alexgarel Nov 22, 2023
a1f311b
docs: fix during migration
alexgarel Nov 22, 2023
cb72ad1
chore: off switch on off2
alexgarel Nov 22, 2023
87a53fb
fix: mount off products in off-pro (to have internal_code.sto)
alexgarel Nov 22, 2023
77a6c6b
fix: handle internal_code.sto on off-pro
alexgarel Nov 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions confs/off-memcached/memcached.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# memcached default config file
# 2003 - Jay Bonci <[email protected]>
# This configuration file is read by the start-memcached script provided as
# part of the Debian GNU/Linux distribution.

# Run memcached as a daemon. This command is implied, and is not needed for the
# daemon to run. See the README.Debian that comes with this package for more
# information.
-d

# Log memcached's output to /var/log/memcached
logfile /var/log/memcached.log

# Be verbose
# -v

# Be even more verbose (print client commands as well)
# -vv

# Start with a cap of 64 megs of memory. It's reasonable, and the daemon default
# Note that the daemon will grow to this size, but does not start out holding this much
# memory
# 3,5 G
-m 3584

# Default connection port is 11211
-p 11211

# Run the daemon as root. The start-memcached will default to running as root if no
# -u command is present in this config file
-u memcache

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 0.0.0.0
alexgarel marked this conversation as resolved.
Show resolved Hide resolved

# Limit the number of simultaneous incoming connections. The daemon default is 1024
# -c 1024

# Lock down all paged memory. Consult with the README and homepage before you do this
# -k

# Return error when memory is exhausted (rather than removing items)
# -M

# Maximize core file limit
# -r
67 changes: 67 additions & 0 deletions confs/off2/munin/munin-node.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
#
# Example config-file for munin-node
#

log_level 4
log_file /var/log/munin/munin-node.log
pid_file /var/run/munin/munin-node.pid

background 1
setsid 1

user root
group root

# This is the timeout for the whole transaction.
# Units are in sec. Default is 15 min
#
# global_timeout 900

# This is the timeout for each plugin.
# Units are in sec. Default is 1 min
#
# timeout 60

# Regexps for files to ignore
ignore_file [\#~]$
ignore_file DEADJOE$
ignore_file \.bak$
ignore_file %$
ignore_file \.dpkg-(tmp|new|old|dist)$
ignore_file \.rpm(save|new)$
ignore_file \.pod$

# Set this if the client doesn't report the correct hostname when
# telnetting to localhost, port 4949
#
#host_name localhost.localdomain

# A list of addresses that are allowed to connect. This must be a
# regular expression, since Net::Server does not understand CIDR-style
# network notation unless the perl module Net::CIDR is installed. You
# may repeat the allow line as many times as you'd like

allow ^127\.0\.0\.1$
allow ^::1$
allow ^10\.0\.0

# If you have installed the Net::CIDR perl module, you can use one or more
# cidr_allow and cidr_deny address/mask patterns. A connecting client must
# match any cidr_allow, and not match any cidr_deny. Note that a netmask
# *must* be provided, even if it's /32
#
# Example:
#
# cidr_allow 127.0.0.1/32
# cidr_allow 192.0.2.0/24
# cidr_deny 192.0.2.42/32

cidr_allow 82.64.249.221/32 # cquest freebox
cidr_allow 212.129.55.232/32 # cquest gateway

# Which address to bind to;
host *
# host 127.0.0.1

# And which port
port 4949
17 changes: 17 additions & 0 deletions confs/off2/pve/lxc_101.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
arch: amd64
cores: 2
hostname: proxy
memory: 2048
mp0: /zfs-hdd/off-pro/sftp,mp=/mnt/off-pro/sftp
net0: name=eth0,bridge=vmbr1,firewall=1,hwaddr=82:B8:56:FD:98:6E,ip=10.1.0.101/24,type=veth
net1: name=net1,bridge=vmbr0,firewall=1,gw=213.36.253.222,hwaddr=8A:54:38:D6:95:DB,ip=213.36.253.214/27,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: zfs-hdd:subvol-101-disk-0,size=32G
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 999
lxc.idmap: g 0 100000 999
lxc.idmap: u 1000 1000 64536
lxc.idmap: g 1000 1000 64536
7 changes: 3 additions & 4 deletions confs/off2/pve/lxc_110.conf
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,11 @@ mp9: /zfs-hdd/opf/images,mp=/mnt/opf/images
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.2,hwaddr=AA:ED:55:47:6B:EF,ip=10.1.0.110/24,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: zfs-hdd:subvol-110-disk-0,size=30G
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 999
lxc.idmap: g 0 100000 999
lxc.idmap: u 1000 1000 10
lxc.idmap: g 1000 1000 10
lxc.idmap: u 1011 101011 64525
lxc.idmap: g 1011 101011 64525
lxc.idmap: u 1000 1000 64536
lxc.idmap: g 1000 1000 64536
8 changes: 4 additions & 4 deletions confs/off2/pve/lxc_111.conf
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ mp7: /zfs-hdd/off/images,mp=/mnt/off/images
mp8: /zfs-hdd/opff/products,mp=/mnt/opff/products
mp9: /zfs-hdd/opff/images,mp=/mnt/opff/images
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.2,hwaddr=E2:FC:A7:2D:02:A9,ip=10.1.0.111/24,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: zfs-hdd:subvol-111-disk-0,size=30G
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 999
lxc.idmap: g 0 100000 999
lxc.idmap: u 1000 1000 10
lxc.idmap: g 1000 1000 10
lxc.idmap: u 1011 101011 64525
lxc.idmap: g 1011 101011 64525
lxc.idmap: u 1000 1000 64536
lxc.idmap: g 1000 1000 64536
8 changes: 4 additions & 4 deletions confs/off2/pve/lxc_112.conf
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ mp7: /zfs-hdd/off/images,mp=/mnt/off/images
mp8: /zfs-hdd/opff/products,mp=/mnt/opff/products
mp9: /zfs-hdd/opff/images,mp=/mnt/opff/images
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.2,hwaddr=EE:91:F9:92:F2:89,ip=10.1.0.112/24,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: zfs-hdd:subvol-112-disk-0,size=30G
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 999
lxc.idmap: g 0 100000 999
lxc.idmap: u 1000 1000 10
lxc.idmap: g 1000 1000 10
lxc.idmap: u 1011 101011 64525
lxc.idmap: g 1011 101011 64525
lxc.idmap: u 1000 1000 64536
lxc.idmap: g 1000 1000 64536
29 changes: 29 additions & 0 deletions confs/off2/pve/lxc_113.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
arch: amd64
cores: 8
features: nesting=1
hostname: off
memory: 40960
mp0: /zfs-hdd/off,mp=/mnt/off
mp1: /zfs-nvme/off/products,mp=/mnt/off/products
mp10: /zfs-hdd/opff/images,mp=/mnt/opff/images
mp11: /zfs-hdd/opf/products,mp=/mnt/opf/products
mp12: /zfs-hdd/opf/images,mp=/mnt/opf/images
mp13: /zfs-hdd/off/logs,mp=/mnt/off/logs
mp14: /zfs-hdd/off-pro/cache/export_files,mp=/mnt/off-pro/cache/export_files
mp2: /zfs-hdd/off/users,mp=/mnt/off/users
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be useful to put users and orgs on the nvme disk, as they are quite small and accessed often

mp3: /zfs-hdd/off/orgs,mp=/mnt/off/orgs
mp4: /zfs-hdd/off/images,mp=/mnt/off/images
mp5: /zfs-hdd/off/html_data,mp=/mnt/off/html_data
mp6: /zfs-hdd/off/cache,mp=/mnt/off/cache
mp7: /zfs-hdd/obf/products,mp=/mnt/obf/products
mp8: /zfs-hdd/obf/images,mp=/mnt/obf/images
mp9: /zfs-hdd/opff/products,mp=/mnt/opff/products
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.2,hwaddr=62:79:6B:52:14:A3,ip=10.1.0.113/24,type=veth
ostype: debian
rootfs: zfs-hdd:subvol-113-disk-0,size=30G
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 999
lxc.idmap: g 0 100000 999
lxc.idmap: u 1000 1000 64536
lxc.idmap: g 1000 1000 64536
25 changes: 25 additions & 0 deletions confs/off2/pve/lxc_114.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
arch: amd64
cores: 4
features: nesting=1
hostname: off-pro
memory: 6144
mp0: /zfs-hdd/off-pro,mp=/mnt/off-pro
mp1: /zfs-nvme/off-pro/products,mp=/mnt/off-pro/products
mp2: /zfs-hdd/off/users,mp=/mnt/off-pro/users
mp3: /zfs-hdd/off/orgs,mp=/mnt/off-pro/orgs
mp4: /zfs-hdd/off-pro/images,mp=/mnt/off-pro/images
mp5: /zfs-hdd/off-pro/html_data,mp=/mnt/off-pro/html_data
mp6: /zfs-hdd/off-pro/cache,mp=/mnt/off-pro/cache
mp7: /zfs-hdd/off/data,mp=/mnt/off-pro/data
mp8: /zfs-hdd/off-pro/sftp,mp=/mnt/off-pro/sftp
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.2,hwaddr=6E:F7:BA:06:2D:90,ip=10.1.0.114/24,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: zfs-hdd:subvol-114-disk-0,size=30G
swap: 0
unprivileged: 1
lxc.idmap: u 0 100000 999
lxc.idmap: g 0 100000 999
lxc.idmap: u 1000 1000 64536
lxc.idmap: g 1000 1000 64536
14 changes: 14 additions & 0 deletions confs/off2/pve/lxc_120.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#Host postgresql for off minions.
arch: amd64
cores: 2
features: nesting=1
hostname: off-postgres
memory: 2048
mp0: zfs-nvme:subvol-120-disk-0,mp=/var/lib/postgresql/,backup=1,mountoptions=noatime,size=5G
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.2,hwaddr=82:79:7C:ED:7A:FF,ip=10.1.0.120/24,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: zfs-hdd:subvol-120-disk-0,size=20G
swap: 0
unprivileged: 1
13 changes: 13 additions & 0 deletions confs/off2/pve/lxc_121.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#This machine only host a simple memcached.
arch: amd64
cores: 2
features: nesting=1
hostname: off-memcached
memory: 4096
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.2,hwaddr=D6:C7:BB:09:0D:F1,ip=10.1.0.121/24,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: zfs-hdd:subvol-121-disk-0,size=15G
swap: 0
unprivileged: 1
52 changes: 52 additions & 0 deletions confs/off2/sanoid/sanoid.conf
Original file line number Diff line number Diff line change
Expand Up @@ -66,14 +66,66 @@

# off

[zfs-hdd/off]
use_template=prod_data
recursive=no

[zfs-hdd/off/cache]
use_template=prod_data
recursive=no

[zfs-hdd/off/html_data]
use_template=prod_data
recursive=no

[zfs-hdd/off/images]
use_template=prod_data
recursive=no

[zfs-hdd/off/logs]
use_template=prod_data
recursive=no

[zfs-hdd/off/orgs]
use_template=prod_data
recursive=no

# not yet !
# [zfs-nvme/off/products]
# use_template=prod_data
# recursive=no

[zfs-hdd/off/users]
use_template=synced_data
recursive=no

# off-pro

[zfs-hdd/off-pro]
use_template=prod_data
recursive=no

[zfs-hdd/off-pro/cache]
use_template=prod_data
recursive=no

[zfs-hdd/off-pro/html_data]
use_template=prod_data
recursive=no

[zfs-hdd/off-pro/images]
use_template=prod_data
recursive=no

[zfs-hdd/off-pro/logs]
use_template=prod_data
recursive=no

# not yet !
# [zfs-nvme/off-pro/products]
# use_template=prod_data
# recursive=no


[template_prod_data]
# How often snapshots should be taken under an hour
Expand Down
12 changes: 12 additions & 0 deletions confs/off2/sanoid/syncoid-args.conf
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,16 @@
--no-sync-snap zfs-hdd/opff/images [email protected]:rpool/opff/images
--no-sync-snap zfs-hdd/opff/products [email protected]:rpool/opff/products
# off
--no-sync-snap zfs-hdd/off/cache [email protected]:rpool/off/cache
--no-sync-snap zfs-hdd/off/html_data [email protected]:rpool/off/html_data
--no-sync-snap zfs-hdd/off/logs [email protected]:rpool/off/logs
--no-sync-snap zfs-hdd/off/images [email protected]:rpool/off/images
# not yet --no-sync-snap zfs-nvme/off/products [email protected]:rpool/off/products
--no-sync-snap zfs-hdd/off/orgs [email protected]:rpool/off/orgs
--no-sync-snap zfs-hdd/off/users [email protected]:rpool/off/users
# off-pro
--no-sync-snap zfs-hdd/off-pro/cache [email protected]:rpool/off-pro/cache
--no-sync-snap zfs-hdd/off-pro/html_data [email protected]:rpool/off-pro/html_data
--no-sync-snap zfs-hdd/off-pro/logs [email protected]:rpool/off-pro/logs
--no-sync-snap zfs-hdd/off-pro/images [email protected]:rpool/off-pro/images
# not yet --no-sync-snap zfs-nvme/off-pro/products [email protected]:rpool/off-pro/products
7 changes: 7 additions & 0 deletions confs/off2/subgid
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
root:1000:64536
root:100000:65536
alex:165536:65536
stephane:231072:65536
teolemon:296608:65536
raphael:362144:65536
CharlesNepote:427680:65536
7 changes: 7 additions & 0 deletions confs/off2/subuid
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
root:1000:64536
root:100000:65536
alex:165536:65536
stephane:231072:65536
teolemon:296608:65536
raphael:362144:65536
CharlesNepote:427680:65536
3 changes: 3 additions & 0 deletions confs/proxy-off/nginx/openbeautyfacts.org
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,10 @@ server {
add_header X-Content-Type-Options nosniff;
add_header X-Frame-Options DENY;

# enable large uploads
client_max_body_size 20M;

# logs location
access_log /var/log/nginx/openbeautyfacts.org.log main;
error_log /var/log/nginx/openbeautyfacts.org.errors.log;

Expand Down
Loading
Loading