QGIS3 QGIS Server code sprint Notes
14 developpers and contributors to QGIS and QGIS servers met in city of Lyon in France the 12 th and 13th december 2016
Just after coffee, a common brainstorming to share the global vision over QGIS server :
QEP 74 from Martin Dobias over Qgis layer registry refactoring will bring multiple project support. Oslandia would like to work on a prototype
- get rid of the old map renderer (https://github.com/qgis/QGIS/pull/3809 by Paul Blottiere)
- avoid too much gui dependencies
- prototype
- finish pull request
- work on SLD FE
- Functionality Matrix
-
architecture: NGINX + QGIS server cluster
- deploy cluster of fcgi apps sucks
- hacking around plugins to get syslog logging
- Server cache invalidation not working over NTFS (inotify problem).
-
RABBITMQ + qgis server cluster
- fcgi not applicable
- Embedding is very basic (no control over I/O)
-
Qgis server Services as plugins like providers
-
python loader
-
response object
- should be python compliant
- enable streaming
- do not presume HTTP being the protocol response format
- Provide I/O falicities
-
implemented: abstract interfaces / service registry / native loader
-
Work in Progress
- python loader
- Tests
- Write Services
See https://github.com/dmarteau/QGIS
boundless Alessandro Passotti request for an external cache to allow scaling / load balancing system each qgis instance should be able to centralize cache (qgis server node farm use case) to avoid cache duplication
Discussion,"do we need a project cache? " why not just export a binary blob when saving qgs project? No more xml parsing then.
- Performance test for QGIS 2 and QGIS 3
- Add environment variables to manage configuration of test (server, warm up, users, etc.)
- Profiling of the server: Callgrind
- Slowness in initializing the layers at each request: will be solved by the new QProject
- Big perf problems saving the PNG in GetMap. Will be investigated.
-
https://github.com/qgis/QGIS/pull/3809 - replace QgsMapRenderer has been merged. During this work Paul noticed that multithread rendering on server side worked, but that all tests failed. One hypothesis is that tests are not compatible with multithread rendering. If that's verified, we could expect huge performance improvements on rendering! A good candidate for day-2 work.
-
Performance benchmark framework improvements
Second day started with following points for each:
- work on backport recent compliance server OGC CITE improvements to 2.18 and 2.14 (original work from Marco)
- work on implementing modular approach in server
- testing multiprocessing rendering branch
- architecture refactor
- play CITE compliance tests and check New documentation
- Server Doc
Still heavy discussion about caching of projects. Is a cache really mandatory (again)?
- Performance tests:
- Test QGIS 3 and QGIS 2 (results)
- Prepare Multithreading settings in QGIS3 config
- QGIS Server Optimization from performance test results
- Diagram legends improvements
- Processing: QEP#84
-> add issues to the QEP:
- get rid of the server project parser, replace it by new QGsProject non sigleton (by january 2017)
- Cache mecanism to upper lever, simplify cache and invalidation policy
- OGC compliancy (CITE). Scripts couldn't yet be tested during the sprint (nginx problems). Plans are to run them and integrate them in a Travis test suite.
- Integration of the new module architecture by 3Liz. Work in progress
- Multithreading branch tested. Works but still work needs to be done to check if threading really happens.
- Stretching bug found in the server, 3liz did fix it
- Performance profiling by C2C. Saving png seems time consuming. Could by related to a fancy Qt filter applied by default. Describing the method for profiling could be shared, docker images are already in C2c Github https://github.com/camptocamp/ms_perfs
- 3liz tested compiling expression to postgis, but there are still issues.
- Another need raised about testing security issues like SQL injections