Crawl Minecraft Overviewer-based maps and save updated map tiles, in hope to view them later [to do] 抓取 Overviewer 渲染的 Minecraft 地图并差分入库,以期日后可以查看Overviewer历史地图[待做]
WARNING: project in early development, unstable, and no backward compatibility by design. If you happen to have used it and generated some json files, try to comprehend tools/json-to-sqlite that I use to convert data file. 警告,本项目尚处于早期开发阶段,无力保证向下兼容(兼容以前的数据文件/配置文件格式)。 tools/json-to-sqlite 目录提供了作者自用的数据转换脚本。
Python 3.7+
pip3
pytz <unify timezone|统一时区>
python-telegram-bot <Send status message to Telegram | 向TG发送运行状态/进度消息>
aiohttp <single-threaded concurrency|单线程并发抓取>
boto3 <S3 support|S3 对象存储支持库>
Modify configs/config.py (general settings) and configs/crawl_list.py (map settings), instructions are in those files.
修改上述文件(抓取器设置,地图设置),备注见文件内注释。
cd
to directory that have crawl.py
,run python3 ./crawl.py
. Look at its console log output, use ctrl+c
to terminate when error happens.
cd
到 crawl.py
所在目录,运行 python3 ./crawl.py
,观察终端输出,有异常时ctrl+c
关停程序
在配置文件中配置S3Config()
相关信息,将config.py的storage_type改为S3
运行 python3 syncer.py [1] [2]/data DB --having sqlite3
上传抓取记录数据库,已上传的数据库文件不会从本地删除。
运行 python3 syncer.py [1] [2]/images img
上传已抓取图片,已成功上传的图片会从本地文件系统删除
(Optional, unmature | 可选不成熟功能) 运行 python3 syncer.py [1] [2]/log log --having .log
上传运行日志,已上传的日志不会从本地删除。
其中 [1] 为 config.py/CrawlerConfig.project_root, 默认值 ..
; [2] 为 config.py/CrawlerConfig.data_foldersW, 默认值 data-production
Under default configurations, crawl.py will create data folder at ..
, assume crawl.py in .
;
data folder have 3 sub-folders:
- data (crawl records)
- images (saved images)
- log (log)
此处放置了曾经已实现的 joiner(拼图器)所生产的图片,笔者希望藉此展示自己想做的内容。
diff出的图片(5MB) (Diff:ed world map, 5MB)
仅进行拼接的v2全图 警告,非常大80MB (World map, joined only, WARNING: very large, 80MB)