This repository has been archived by the owner on Mar 3, 2020. It is now read-only.
Live Sync API #459
Merged
Live Sync API #459
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@justinwray I think it would be better to have example curl command for developer. wp-api's api reference is easy to read and understand. Example reference: http://v2.wp-api.org/reference/posts/ If Live Sync API reference is like this, I think it will be grate... |
|
@kamikaze123 Absolutely. I think the long-term plan would be to include actual developer documentation (in the Wiki, most likely). We simply included some of the information within the commit log so that the PR was easier to review. |
|
Can you 1) rebase so the existing tests pass and 2) add some tests for the new code? |
|
Rebased and Unit Tests added. |
* Introducing the Live Sync API.
* The Live Sync API allows administrators to import game activity in near-real-time. Users can link their accounts on one or more FBCTF platform instances and their scores will be automatically imported into the systems that have been linked.
* The primary use-case revolves around event aggregation across multiple FBCTF instances. Event organizers can now separate FBCTF instances and combine scores into one global scoreboard.
* The Live Sync API will import Levels, Categories, Scoring Events, and Hint Usage. Scores are automatically calculated, and bonuses are updated to ensure accurate scoring across linked FBCTF instances.
* Administrators determine which systems, if any, are linked.
* Users must link their account in order for their activity to be synced.
* The UI/UX of FBCTF has been updated to include a mechanism for users to configure their Live Sync credentials.
* Users cannot obtain hints or capture levels on the importing system.
* The API is JSON based and the schema is generalized so that it can leveraged by other platforms or external processes. So data can be synced from non-FBCTF platforms.
* The importing script will automatically handle country conflicts (if two systems have the same country selected for a level).
* USER GUIDE (Documentation):
* Users must first have an account on all FBCTF instances they wish to link.
* The user must then login and access the game board.
* From the navigation menu, the user should select "Account."
* The user must then set a Live Sync username and password. The Live Sync username and password must not be their login credentials. In fact, users are prohibited from using their account password as their Live Sync password.
* The user would repeat the above steps for each FBCTF instance they wish to link. The Live Sync credentials must be the same on each FBCTF instance or their accounts will not be linked.
* ADMIN GUIDE (Documentation):
* The admin is free to sync as many platforms as their desire. Additionally the admin may import from as many API sources as their desire.
* The admin will need to launch the "live import" script, on any importing systems, from the command-line:
* `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php <Space Delimited URLs to API Endpoints> <Sleep Between Cycles> <Disable SSL Certificate Verification> <Show Debug Messages>`
* Disabling of the SSL Verification and Debugging are both optional. The URL(s) and Sleep time are required arguments.
* EXAMPLE: `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php "https://10.10.10.101/data/livesync.php https://10.10.10.102/data/livesync.php https://10.10.10.103/other/platform/api" 300 true true`
* API SCHEMA (JSON):
* JSON:
[{"active":true,"type":"flag","title":"Example Level 1","description":"This is the first example level.","entity_iso_code":"TJ","category":"None","points":100,"bonus":30,"bonus_dec":10,"penalty":10,"teams":{"fbctf:user1:$2y$12$a1T4KyqqxADi3YIJ7M2sf.VoSHz6qMBx.zrxAIvZnD8de95EsLeny":{"timestamp":"2017-02-17 02:20:22","capture":true,"hint":false}}}]
* Explained (Formatted output for readability - the actually data must be in valid JSON format):
[0] => Array
(
[active] => 1 // Level Status (Enabled/Disabled)
[type] => flag // Level Type (Flag or Quiz)
[title] => Example Level 1 // Level Name
[description] => This is the first example level. // Level Description
[entity_iso_code] => US // Country Code (Mapping)
[category] => None // Level Category
[points] => 100 // Points
[bonus] => 30 // Bonus Points
[bonus_dec] => 10 // Bonus Point Decrement
[penalty] => 0 // Hint Cost
[teams] => Array
(
[fbctf:user3:$2y$12$GIR7V0Q2OMDv8cTTOnzKVpGYgR4.pWTsPRHtZ3yenKZ9JxOabx4m2] => Array // Live Sync Type, Live Sync Username, Live Sync Key (Hash)
(
[timestamp] => 2017-02-17 01:09:24 // Activity Timestamp
[capture] => 1 // Capture Status
[hint] => // Hint Used
)
)
)
* Example (Formatted output for readability - the actually data must be in valid JSON format):
Array
(
[0] => Array
(
[active] => 1
[type] => flag
[title] => Example Level 1
[description] => This is the first example level.
[entity_iso_code] => US
[category] => None
[points] => 100
[bonus] => 30
[bonus_dec] => 10
[penalty] => 0
[teams] => Array
(
)
)
[1] => Array
(
[active] => 1
[type] => flag
[title] => Example Level 2
[description] => This is the second example level.
[entity_iso_code] => OM
[category] => None
[points] => 100
[bonus] => 30
[bonus_dec] => 10
[penalty] => 0
[teams] => Array
(
[fbctf:user1:$2y$12$n.VmlNNwxmZ/OkGGuhVhFeX0VExAgjoaYzyetLCIemSXN/yxWXLyO] => Array
(
[timestamp] => 2017-02-17 01:01:49
[capture] => 1
[hint] => 1
)
[fbctf:user2:$2y$12$GIDv8cR7V0nzKVpQ2OMTTOGYgR4.pWTxOPRH9abtsJZ3yenKZx4m2] => Array
(
[timestamp] => 2017-02-17 01:21:13
[capture] => 1
[hint] => 1
)
)
)
[2] => Array
(
[active] => 1
[type] => flag
[title] => Example Level 3
[description] => This is the third example level.
[entity_iso_code] => MA
[category] => None
[points] => 100
[bonus] => 30
[bonus_dec] => 10
[penalty] => 0
[teams] => Array
(
[fbctf:user2:$2y$12$GIDv8cR7VpQ2OM0nzKVTTOGYgR4.pWTxOabtsPRH9JZ3yenKZx4m2] => Array
(
[timestamp] => 2017-02-17 01:18:45
[capture] => 1
[hint] =>
)
[fbctf:user1:$2y$12$n.VmlNNwxmZ/OkGGuhVhFeXYzExAg0VoajyetLCIemSXN/yxWXLyO] => Array
(
[timestamp] => 2017-02-17 01:01:41
[capture] => 1
[hint] =>
)
)
)
[3] => Array
(
[active] => 1
[type] => flag
[title] => Example Level 4
[description] => This is the second example level.
[entity_iso_code] => RO
[category] => None
[points] => 100
[bonus] => 30
[bonus_dec] => 10
[penalty] => 0
[teams] => Array
(
[fbctf:user3:$2y$12$GIDv8cR7V02OnzKVpQMTTOGYgR4.pWTsPOabtZRH9Jx3yenKZx4m2] => Array
(
[timestamp] => 2017-02-17 01:09:24
[capture] => 1
[hint] =>
)
)
)
)
* TO DO (Enhancements):
* Implemented alternative Live Sync key/authentication mechanisms, such as: Facebook Login, OAuth, etc.
* Improve the processing of Bases/Progressive scoring.
* Integrate password strength enforcement for the Live Sync credentials.
* Google OAuth can now be used with the Live Sync when the exporting system provides the "google_oauth" type and provides the email address of the user in base64 encoded form. * Added Google OAuth UI/UX. If enabled, this allows a user to link their Google account to their FBCTF account using Google OAuth. The user simply navigates to the account page and clicks the "Link Your Google Account" button and completes the sign-in/authorization process. * Administrators must enable Google OAuth. When disabled the option does not appear for the users. To enable Google OAuth the administrator must first create a Google API account and then place the API secrets file on the system (in a non-web directory). The administrator would then set the full path to the API secrets file in the settings.ini file, within the GOOGLE_OAUTH_FILE field. * The Live Sync API has been updated to handle the "google_oauth" type case. * The liveimport.php script has been updated to set default values for some of the API fields. The following fields are mandatory: * title * description * points * teams * The live import code has also been updated to ensure duplicate levels, when using a combination of non-defined and defined countries, are not generated. * The project now requires google/apiclient ^2.0 from composer. Updated composer.json and composer.lock to define the new dependencies. * Minor formatting updates.
|
Added Google OAuth to Live Sync API and Rebased. |
* The live sync import script (livesync.php) now utilizes `getopts()` to provide more user-friendly option input to the command-line script. The script will provide a help message upon usage without the required field(s). Here is the help message text:
```
Usage:
hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php
--url <Sync URL> [Switched allowed multiple times. Optionally provide custom HTTP headers after URL, pipe delimited]
--sleep <Time to Sleep Between Cycles> [Default: 300]
--disable-ssl-verification [Optional: Disables SSL Certification Verification]
--debug [Optional: Enables Debug Output]
```
* Custom HTTP Headers are now supported via the `url` CLI argument, using a pipe (`|`) delimiter. Multiple headers may be provided by the user.
* Users of the live sync import script (livesync.php) must provide at minimum the `url` argument. Multiple URLs are allowed in order to import from more than one host. Additionally, users may specify custom HTTP headers as necessary per-URL.
* The `sleep`, `disable-ssl-verification`, and `debug` arguments are optional. By default the script will sleep for 300 seconds between imports; this may be changed with the `sleep` option. By default the script will enforce SSL security and verification, however there are times when this may need to be disabled. SSL Verification can be disabled with the `disable-ssl-verification` option. The `debug` option provides a detailed output of the scripts activity; unless `debug` is used, the script will provide no output unless an error is encountered.
* Here are some example usages of the liveimport.php script:
* Single URL, 30 second cycles:
`hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --url "https://10.10.10.101/data/livesync.php" --sleep 30`
* Multiple URLs, 90 second cycles, Disable SSL Verification:
`hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --disable-ssl-verification --url "https://10.10.10.101/data/livesync.php" --url "https://10.10.10.103/data/livesync.php" --sleep 90`
* Multiple URLs, One Custom HTTP Header, Debug Enabled:
`hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --debug --url "https://10.10.10.101/data/livesync.php" --url "https://10.10.10.102/data/livesync.php|X-API-KEY: f4f0d2154f338fd8edb38fc3839f22dd"`
* The live sync import script (livesync.php) now performs XSSI filtering on the returned JSON and logs an error message if the JSON data is missing or malformed.
* Inverse conditions within the live sync import script (livesync.php) have been reformatted to provide clearer readability.
* The Google OAuth Live Sync key has been switched from the user's email address to their Google Profile ID. This value does not need to be encoded.
* Added a specific redirect URL for the Google OAuth process. This will ensure the user is redirected back to the same (sub)domain they came from. This is primarily useful if the FBCTF instance is accessible from multiple domains or subdomains (like www). The redirect URL must still be authorized in the Google API settings/console. * Added a date/time output for the debug mode of the live import script. * Set the modal title to a static value.
|
Has anyone used Live Sync API in the new commit? |
|
Also where is the liveimport.php file? hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php |
|
ok I found the liveimport.php file but do you move the livesync.php file to the directory from the /fbctf/src/data location? Also where is the /other/platform/api? Maybe just update the admin instructions above. hvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php "https://10.10.10.101/data/livesync.php https://10.10.10.102/data/livesync.php https://10.10.10.103/other/platform/api" 300 true true |
|
I moved the livesync.php file only to get this error when running the following command of course replacing the IP with the actual server url I am using. hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/data/liveimport.php "https://10.10.10.101/data/livesync.php https://10.10.10.103/other/platform/api" 300 false true Error: |
|
For the Live Sync API, there are two relevant files that serve two distinct purposes: The The To successfully sync data from one host to another you would run the To run the the Here are some example usages of the liveimport.php script:
Note: We are working on complete API documentation that will provide the information needed to publish an API endpoint for Live Sync as well as usage information for the process of importing data. You should not move any of the files within the project, as they will be in the correct location by default. The For clarification, the Additionally, in the future, it would be best to address these types of issues in an issue so that other users in the future may be able to find resolutions to a similar issue or concern. |
gsingh93
pushed a commit
that referenced
this pull request
Jun 5, 2017
* add hindi translation * added hindi translation * Update lang_hi.php * Error Checking During Build Tests (#452) * Error Checking During Build Tests * Execute hh_client during build tests. * Currently the PHP built-in getimagesizefromstring function is not in the HHVM upstream hhi, and therefore generates an error. In the future, once getimagesizefromstring is added upstream, we should use the hh_client exit status. * * Readded execute permissions to the script. * HHVM/Hack Typing Error Fixes (#450) * HHVM/Hack Typing Error Fixes * Fixed a few HHVM/Hack typing and strict compliance issues. * This is necessary before hh_client can run and be enforced during the build process. (See comments on #435) * * Updated formatting. * Require bxslider version 4.2.6 (Fixes #455) (#458) * This resolves a current build error #455. * bxslider was updated from 4.2.6 to 4.2.7 on February 14th. Previously FBCTF allowed for a near match to 4.2.6. However, FBCTF fails to build with 4.2.7. During the installation, process grunt failed to build the browserify javascript. * Fixed Syntax Errors in Hindi Language (Fixes Build Errors) (#460) * Fixed minor syntax error due to character encoding. * This will ensure the project builds (no Hack errors). * Automated Game Start and Stop (#449) * Automated Game Start and Stop * Games will automatically start and stop at their scheduled times. Administrators can still manually start or stop a game regardless of the configured schedule. * Both Control::genAutoBegin() and Control::genAutoEnd() were added to check the current time against the scheduled start or stop time and perform the relevant action (Control::genBegin or Control::getEnd). * Control::genAutoRun() checks the current game status and determine if the game should be starting or ending, calling the appropriate function (Control::genAutoBegin or Control::getAutoEnd) and is exclusively used in the new autorun.php script. * Control::genRunAutoRunScript() runs the new autorun.php script, ensuring the script is not already running before starting a new copy. * The Router class was updated to include a call to Control::genRunAutoRunScript(), this ensures the script is always running. This script status check, and execution when needed, only takes place on a full page load. * The autorun.php script runs Control::genAutoRun() and sleeps up to 30 seconds. * If the game is scheduled to start or stop within 30 seconds, the script will sleep for the necessary amount of time. * Games will always start with at most a 29-second difference from the scheduled time. This discrepancy can only take place if the schedule is changed within 30 seconds of the previously scheduled time. Otherwise, the execution will happen at the scheduled time. * This automation is self-contained and requires no additional dependencies or external services (like cron, etc.). * * Allow administrators to define the cycle time (in seconds) for the autorun process. This time will be used for the sliding sleep. * * Added sanitization to the autorun script path/file. * Attachments and Links Import/Export, Database Restore, and Control Cleanup (#451) * Attachments and Links Import/Export, Database Restore, and Control Cleanup * Attachments can now be exported and imported. On export, attachments are downloaded into a Tar Gzip and securely extracted on import. * Links and Attachments data is now provided within the Levels export. Users must import both the Level data and the Attachment files to restore the levels with attachments. * A database restore option has been added which utilizes the backed up database content. This overwrites all data in the database. * The Control page has been reorganized to align the various functionality better. * Memcached flushing has been added to all relevant data imports. * Error handling has been added to the various import functions. * * Removed getter function for the Attachment constant. * Switched double quotes with single quotes. * Update README.md * Update README.md * Update README.md * Update README.md * Live Sync API (#459) * Live Sync API * Introducing the Live Sync API. * The Live Sync API allows administrators to import game activity in near-real-time. Users can link their accounts on one or more FBCTF platform instances and their scores will be automatically imported into the systems that have been linked. * The primary use-case revolves around event aggregation across multiple FBCTF instances. Event organizers can now separate FBCTF instances and combine scores into one global scoreboard. * The Live Sync API will import Levels, Categories, Scoring Events, and Hint Usage. Scores are automatically calculated, and bonuses are updated to ensure accurate scoring across linked FBCTF instances. * Administrators determine which systems, if any, are linked. * Users must link their account in order for their activity to be synced. * The UI/UX of FBCTF has been updated to include a mechanism for users to configure their Live Sync credentials. * Users cannot obtain hints or capture levels on the importing system. * The API is JSON based and the schema is generalized so that it can leveraged by other platforms or external processes. So data can be synced from non-FBCTF platforms. * The importing script will automatically handle country conflicts (if two systems have the same country selected for a level). * USER GUIDE (Documentation): * Users must first have an account on all FBCTF instances they wish to link. * The user must then login and access the game board. * From the navigation menu, the user should select "Account." * The user must then set a Live Sync username and password. The Live Sync username and password must not be their login credentials. In fact, users are prohibited from using their account password as their Live Sync password. * The user would repeat the above steps for each FBCTF instance they wish to link. The Live Sync credentials must be the same on each FBCTF instance or their accounts will not be linked. * ADMIN GUIDE (Documentation): * The admin is free to sync as many platforms as their desire. Additionally the admin may import from as many API sources as their desire. * The admin will need to launch the "live import" script, on any importing systems, from the command-line: * `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php <Space Delimited URLs to API Endpoints> <Sleep Between Cycles> <Disable SSL Certificate Verification> <Show Debug Messages>` * Disabling of the SSL Verification and Debugging are both optional. The URL(s) and Sleep time are required arguments. * EXAMPLE: `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php "https://10.10.10.101/data/livesync.php https://10.10.10.102/data/livesync.php https://10.10.10.103/other/platform/api" 300 true true` * API SCHEMA (JSON): * JSON: [{"active":true,"type":"flag","title":"Example Level 1","description":"This is the first example level.","entity_iso_code":"TJ","category":"None","points":100,"bonus":30,"bonus_dec":10,"penalty":10,"teams":{"fbctf:user1:$2y$12$a1T4KyqqxADi3YIJ7M2sf.VoSHz6qMBx.zrxAIvZnD8de95EsLeny":{"timestamp":"2017-02-17 02:20:22","capture":true,"hint":false}}}] * Explained (Formatted output for readability - the actually data must be in valid JSON format): [0] => Array ( [active] => 1 // Level Status (Enabled/Disabled) [type] => flag // Level Type (Flag or Quiz) [title] => Example Level 1 // Level Name [description] => This is the first example level. // Level Description [entity_iso_code] => US // Country Code (Mapping) [category] => None // Level Category [points] => 100 // Points [bonus] => 30 // Bonus Points [bonus_dec] => 10 // Bonus Point Decrement [penalty] => 0 // Hint Cost [teams] => Array ( [fbctf:user3:$2y$12$GIR7V0Q2OMDv8cTTOnzKVpGYgR4.pWTsPRHtZ3yenKZ9JxOabx4m2] => Array // Live Sync Type, Live Sync Username, Live Sync Key (Hash) ( [timestamp] => 2017-02-17 01:09:24 // Activity Timestamp [capture] => 1 // Capture Status [hint] => // Hint Used ) ) ) * Example (Formatted output for readability - the actually data must be in valid JSON format): Array ( [0] => Array ( [active] => 1 [type] => flag [title] => Example Level 1 [description] => This is the first example level. [entity_iso_code] => US [category] => None [points] => 100 [bonus] => 30 [bonus_dec] => 10 [penalty] => 0 [teams] => Array ( ) ) [1] => Array ( [active] => 1 [type] => flag [title] => Example Level 2 [description] => This is the second example level. [entity_iso_code] => OM [category] => None [points] => 100 [bonus] => 30 [bonus_dec] => 10 [penalty] => 0 [teams] => Array ( [fbctf:user1:$2y$12$n.VmlNNwxmZ/OkGGuhVhFeX0VExAgjoaYzyetLCIemSXN/yxWXLyO] => Array ( [timestamp] => 2017-02-17 01:01:49 [capture] => 1 [hint] => 1 ) [fbctf:user2:$2y$12$GIDv8cR7V0nzKVpQ2OMTTOGYgR4.pWTxOPRH9abtsJZ3yenKZx4m2] => Array ( [timestamp] => 2017-02-17 01:21:13 [capture] => 1 [hint] => 1 ) ) ) [2] => Array ( [active] => 1 [type] => flag [title] => Example Level 3 [description] => This is the third example level. [entity_iso_code] => MA [category] => None [points] => 100 [bonus] => 30 [bonus_dec] => 10 [penalty] => 0 [teams] => Array ( [fbctf:user2:$2y$12$GIDv8cR7VpQ2OM0nzKVTTOGYgR4.pWTxOabtsPRH9JZ3yenKZx4m2] => Array ( [timestamp] => 2017-02-17 01:18:45 [capture] => 1 [hint] => ) [fbctf:user1:$2y$12$n.VmlNNwxmZ/OkGGuhVhFeXYzExAg0VoajyetLCIemSXN/yxWXLyO] => Array ( [timestamp] => 2017-02-17 01:01:41 [capture] => 1 [hint] => ) ) ) [3] => Array ( [active] => 1 [type] => flag [title] => Example Level 4 [description] => This is the second example level. [entity_iso_code] => RO [category] => None [points] => 100 [bonus] => 30 [bonus_dec] => 10 [penalty] => 0 [teams] => Array ( [fbctf:user3:$2y$12$GIDv8cR7V02OnzKVpQMTTOGYgR4.pWTsPOabtZRH9Jx3yenKZx4m2] => Array ( [timestamp] => 2017-02-17 01:09:24 [capture] => 1 [hint] => ) ) ) ) * TO DO (Enhancements): * Implemented alternative Live Sync key/authentication mechanisms, such as: Facebook Login, OAuth, etc. * Improve the processing of Bases/Progressive scoring. * Integrate password strength enforcement for the Live Sync credentials. * * Added unit tests for Live Sync to TeamTest * * Updated unit tests for the Live Sync API. * Added Google OAuth to Live Sync API * Google OAuth can now be used with the Live Sync when the exporting system provides the "google_oauth" type and provides the email address of the user in base64 encoded form. * Added Google OAuth UI/UX. If enabled, this allows a user to link their Google account to their FBCTF account using Google OAuth. The user simply navigates to the account page and clicks the "Link Your Google Account" button and completes the sign-in/authorization process. * Administrators must enable Google OAuth. When disabled the option does not appear for the users. To enable Google OAuth the administrator must first create a Google API account and then place the API secrets file on the system (in a non-web directory). The administrator would then set the full path to the API secrets file in the settings.ini file, within the GOOGLE_OAUTH_FILE field. * The Live Sync API has been updated to handle the "google_oauth" type case. * The liveimport.php script has been updated to set default values for some of the API fields. The following fields are mandatory: * title * description * points * teams * The live import code has also been updated to ensure duplicate levels, when using a combination of non-defined and defined countries, are not generated. * The project now requires google/apiclient ^2.0 from composer. Updated composer.json and composer.lock to define the new dependencies. * Minor formatting updates. * * Ensure mandatory fields are set, gracefully skip when they are not. * Refined Live Import CLI Options and Updated Google OAuth Data * The live sync import script (livesync.php) now utilizes `getopts()` to provide more user-friendly option input to the command-line script. The script will provide a help message upon usage without the required field(s). Here is the help message text: ``` Usage: hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --url <Sync URL> [Switched allowed multiple times. Optionally provide custom HTTP headers after URL, pipe delimited] --sleep <Time to Sleep Between Cycles> [Default: 300] --disable-ssl-verification [Optional: Disables SSL Certification Verification] --debug [Optional: Enables Debug Output] ``` * Custom HTTP Headers are now supported via the `url` CLI argument, using a pipe (`|`) delimiter. Multiple headers may be provided by the user. * Users of the live sync import script (livesync.php) must provide at minimum the `url` argument. Multiple URLs are allowed in order to import from more than one host. Additionally, users may specify custom HTTP headers as necessary per-URL. * The `sleep`, `disable-ssl-verification`, and `debug` arguments are optional. By default the script will sleep for 300 seconds between imports; this may be changed with the `sleep` option. By default the script will enforce SSL security and verification, however there are times when this may need to be disabled. SSL Verification can be disabled with the `disable-ssl-verification` option. The `debug` option provides a detailed output of the scripts activity; unless `debug` is used, the script will provide no output unless an error is encountered. * Here are some example usages of the liveimport.php script: * Single URL, 30 second cycles: `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --url "https://10.10.10.101/data/livesync.php" --sleep 30` * Multiple URLs, 90 second cycles, Disable SSL Verification: `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --disable-ssl-verification --url "https://10.10.10.101/data/livesync.php" --url "https://10.10.10.103/data/livesync.php" --sleep 90` * Multiple URLs, One Custom HTTP Header, Debug Enabled: `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --debug --url "https://10.10.10.101/data/livesync.php" --url "https://10.10.10.102/data/livesync.php|X-API-KEY: f4f0d2154f338fd8edb38fc3839f22dd"` * The live sync import script (livesync.php) now performs XSSI filtering on the returned JSON and logs an error message if the JSON data is missing or malformed. * Inverse conditions within the live sync import script (livesync.php) have been reformatted to provide clearer readability. * The Google OAuth Live Sync key has been switched from the user's email address to their Google Profile ID. This value does not need to be encoded. * Minor Updates * Added a specific redirect URL for the Google OAuth process. This will ensure the user is redirected back to the same (sub)domain they came from. This is primarily useful if the FBCTF instance is accessible from multiple domains or subdomains (like www). The redirect URL must still be authorized in the Google API settings/console. * Added a date/time output for the debug mode of the live import script. * Set the modal title to a static value. * I think its bad (#446) * Fix fail error * Order by name in category list The category filter in gameboard aren't ordered alphabetically * Autofocus in team name * Fail * Fix Fail autofocus * Fix error with autofocus * Not very good For security reasons, I think it's not very good insert the admin id by default at session table * Update SessionTest.php * Updated LiveSync Security (#494) * Updated LiveSync Security * Live Sync API is now disabled by default. * Admins can enable or disable the Live Sync API from the Administration Configuration page. * Live Sync API now has an optional 'Auth Key.' When the auth key is set, anyone attempting to pull from the API must supply the auth key value in their request. Without the auth key, no data is provided by the Live Sync API endpoint. * When using the Auth Key, it must be added as a parameter to the URL value in the `liveimport` script: ```?auth=XXXXX_``` * Example (with an auth key of `1234567890`: * `hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php --url 'https://10.10.10.101/data/livesync.php?auth=1234567890'` * Note: When using the Auth Key you should use a secure key. * The `livesync` API endpoint will provide error messages if the API is disabled, the key is missing or invalid, or if any general error is encountered. * The `liveimport` script will check for errors and display those in the output if any are encountered. * Updated LiveSync Security * Combined Awaitables throughout LiveSync endpoint. * Used hash_equals() for API key verification, mitigating timing attacks on the key.# Please enter the commit message for your changes. Lines starting * Temp fix for issue 499 & 500. Forcing Grunt to continue as it is not correctly detecting node_modules in the folder (#502) * Merge Deconflict of /dev and /master (#503) * Registration enforcing strong passwords (#442) * Password types in admin * Fully functional password complexity enforcement for registration * lowercase word in text * Adding test for password types regex and fixing all errors for hh_client * Updating outdated schema for tests * Custom branding for icon and text (#448) * Custom branding for icon and text * Replace async calls branding xhp by attributes * Use genRenderBranding in genRenderMobilePage and combine awaitables
Merged
justinwray
added a commit
that referenced
this pull request
Aug 5, 2017
Merged `master` onto `dev` Commits: * add hindi translation (f9f69a7) * added hindi translation (c5a3eb2) * Update lang_hi.php (6bf8317) * Merge pull request #454 from Akhil09/master (f19b5d5) * Error Checking During Build Tests (#452) (de72b28) * HHVM/Hack Typing Error Fixes (#450) (8e4151e) * Require bxslider version 4.2.6 (Fixes #455) (#458) (c2957d7) * Fixed Syntax Errors in Hindi Language (Fixes Build Errors) (#460) (899ab2b) * Automated Game Start and Stop (#449) (ca40091) * Attachments and Links Import/Export, Database Restore, and Control Cleanup (#451)(be35c6b) * Update README.md (8d1db8c) * Update README.md (8f16141) * Update README.md (53a6fa7) * Update README.md (300091c) * Live Sync API (#459) (0b7ca09) * I think its bad (#446) (d326564) * Updated LiveSync Security (#494) (e880251) * Temp fix for issue 499 & 500. Forcing Grunt to continue as it is not correctly detecting node_modules in the folder (#502) (dd3c874) * Merge Deconflict of /dev and /master (#503) (51e06a7) * Added Slack registration link to README (b11322e) * Merge branch 'dev' into WraySec/fbctf/merge@7f8c281 * Fixed conflict in ScoreLog (e181da2) * Fixed conflict in ScoreLog (dd9de6e) * Fixed conflict in Control (f7c6439) * Merge /dev/ to /master (#542) (3047351)
justinwray
added a commit
that referenced
this pull request
Nov 17, 2017
* Major Performance Enhancements and Bug Fixes * **NOTICE**: _This PR is extremely large. Due to the interdependencies, these code changes are being included as a single PR._ * **Local Caching of Database and Memcached Results** * Replaced PR #574 * Results from the database (MySQL) are stored and queried from the Cache service (Memcached). Results from Memcached are now stored locally within HHVM memory as well. * New class `Cache` has been added, with four methods: * `setCache()` * `getCache()` * `deleteCache()` * `flushCache()` * The new `Cache` class object is included as a static property of the `Model` class and all children. * Through the new `Cache` object, all `Model` sub-classes will automatically utilize the temporary HHVM-memory cache for results that have already been retrieved. * The `Cache` object is created and used throughout a single HHVM execution thread (user request). * This change results in a massive reduction in Memcached requests (over 99% - at scale), providing a significant improvement in scalability and performance. * The local cache can be deleted or flushed from the `Model` class or any child class with the usage of `deleteLocalCache()`. The `deleteLocalCache()` method works like `invalidateMCRecords()`. When called from a child class, a `$key` can be passed in, matching the MC key identifier used elsewhere, or if no `$key` is specified all of the local caches for that class will be deleted. If the calling class is `Model` then the entire local cache is flushed. * Some non-HTTP code has local cache values explicitly deleted, or the local cache completely flushed, as the execution thread is continuous: * Prevent `autorun.php` from storing timestamps in the local cache, forever (the script runs continuously). * Flush the local cache before the next cycle of `bases.php` to ensure the game is still running and the configuration of the bases has not changed (the script runs continuously). * Flush the local cache before the next import cycle of `liveimport.php` to ensure we get the up-to-date team and level data (the script runs continuously). * The `Cache` class is specifically separate from `Model` (as an independent class) so that other code may instantiate and utilize a temporary (request-exclusive) local-memory-based caching solution, with a common interface. The usage provides local caching without storing the data in MySQL, Memcached, or exposing it to other areas of the application. (For example, this is being utilized in some `Integration` code already.) * Implemented CR from PR #574. * Relevant: Issue #456 and Comment #456 (comment) * **Blocking AJAX Requests** * Replaced PR #575 * Expansion and Bug Fixes of PR #565 * AJAX requests for the gameboard are now individually blocking. A new request will not be dispatched until the previous request has completed. * AJAX requests will individually stop subsequent requests on a hard-error. * The blocking of continuous AJAX requests, when the previous has not yet returned, or on a hard error, provides a modest performance benefit by not compounding the issue with more unfulfillable requests. * Forceful refreshes are still dispatched every 60 seconds, regardless of the blocking state on those requests. * Relevant: Issue #456 and Comment #456 (comment) * **AJAX Endpoint Optimization** * Removed nested loops within multiple AJAX endpoints: * `map-data.php` * `country-data.php` * ` leaderboard.php` * All Attachments, including Link and Filename, are now cached and obtained through: `Attachment::genAllAttachmentsFileNamesLinks()`. * All Team names of those who have completed a level, are now cached and obtained through `MultiTeam::genCompletedLevelTeamNames()`. * All Levels and Country for map displays are cached and obtained through `Level::genAllLevelsCountryMap()` and `Country::genAllEnabledCountriesForMap()`. * Relevant: Issue #456 * **Memcached Cluster Support** * The platform now supports a cluster of Memcached nodes. * Configuration for the `MC_HOST` within the `settings.ini` file is now an array, instead of a single value: * `MC_HOST[] = 127.0.0.1` * Multiple Memcached servers can be configured by providing additional `MC_HOST` lines: * `MC_HOST[] = 1.2.3.4` * `MC_HOST[] = 5.6.7.8` * The platform uses a Write-Many Read-Once approach to the Memcached Cluster. Specifically, data is written to all of the configured Memcached nodes and then read from a single node at random. This approach ensures that all of the nodes stay in sync and up-to-date while providing a vital performance benefit to the more expensive and frequent operation of reading. * The existing `Model` methods (`setMCRecords()` and `invalidateMCRecords()`) all call and utilize the new cluster methods: * `writeMCCluster()` * `invalidateMCCluster()` * The flushing of Memcached has also been updated to support the multi-cluster approach: `flushMCCluster()`. * Note that the usage of a Memcached Cluster is seamless for administrators and users, and works in conjunction with the Local Cache. Also note, the platform works identically, for administrators and users, for both single-node and multi-node Memcached configurations. * The default configuration remains a single-node configuration. The utilization of a Memcached Cluster requires the following: * The configuration and deployment of multiple Memcached nodes (the `quick_setup install_multi_cache` or Memcached specific provision, will work). * The modification of `settings.ini` to include all of the desired Memcached hosts. * All Memcached hosts must be identically configured. * Usage of a Memcached Cluster is only recommended in the Multi-Server deployment modes. * Relevant: Issue #456 * **Load Balancing of Application Servers (HHVM)** * The platform now supports the ability to load balance multiple HHVM servers. * To facilitate the load balancing of the HHVM servers, the following changes were made: * Scripts (`autorun`, `progressive`, etc.) are now tracked on a per-server level, preventing multiple copies of the scripts from being executed on the HHVM servers. * Additional database verification on scoring events to prevent multiple captures. * Load Balancing of HHVM is only recommended in the Multi-Server deployment modes. * Relevant: Issue #456 * **Leaderboard Limits** * `MultiTeam::genLeaderboard()` now limits the total number of teams returned based on a configurable setting. * A new argument has been added to `MultiTeam::genLeaderboard()`: `limit`. This value, either `true` or `false`, indicates where the limit should be enforced, and defaults to `true`. * When the data is not cached, `MultiTeam::genLeaderboard()` will only build, cache, and return the number of teams needed to meet the limit. * When the data is already cached, `MultiTeam::genLeaderboard()` will ensure the limit value has not changed and returned the cached results. If the configured limit value has been changed, `MultiTeam::genLeaderboard()` will build, cache, and return the number based on the new limit. * The "Scoreboard" modal (found from the main gameboard) is a special case where all teams should be displayed. As such, the Scoreboard modal sets the `limit` value to `false` retuning all teams. This full leaderboard will be cached, but all other display limits are still enforced based on the configured limit. Once invalidated, the cached data will return to the limited subset. * Because a full leaderboard is not always cached, this does result in the first hit to the Scoreboard modal requiring a database hit. * A user, whose rank is above the limit, will have their rank shown to them as `$limit+`. For example, if the limit is set to `50` and the user's rank is above `50`, they would see: `51+` as their rank. * Overall, the caching of the Leaderboard, one of the more resource-intensive and frequent queries, resulted in significant performance gains. * The Leaderboard limit is configurable by administrators within the administrative interface. The default value is `50`. * Relevant: Issue #456 * **Activity Log Limits** * The Activity Log is now limited to the most recent `100` log entries. The value is not configurable. * The activity log is continually queried and contains a large amount of data, as such, it is a very resource-intensive request. * The limit on the results built, cached, and returned for the activity log provides a notable improvement in performance. * Relevant: Issue #456 * **Database Optimization** * Expansion of PR #564 * Added additional indexing of the database tables in the schema. * The additional indexing provides further performance improvements to the platform queries, especially those found in `MultiTeam` and those queries continually utilized as a result of the AJAX calls. * Relevant: Issue #456 and Comment #456 (comment) * **Team and MultiTeam Performance Improvements** * Updated numerous `Team::genTeam()` calls to used the cached version: `MultiTeam::genTeam()`. * Optimized the database query within `MultiTeam::genFirstCapture()` to return the `team_id` and build the `Team` from the cache. * Optimized the database query within `MultiTeam::genCompletedLevel()` to return the `team_id` and build the `Team` from the cache. * Optimized the database query within `MultiTeam::genAllCompletedLevels()` to return the `team_id` and build the `Team` from the cache. * A full invalidation of the `MultiTeam` cache is no longer executed when a new team is created. Newly created teams will not have any valid scoring activity. Delaying the rebuild of the scoring related cache provides a modest performance improvement. The new team will not show up in certain areas (namely the full scoreboard) until they or someone else perform a scoring action. To ensure the team is properly functioning, following cache is specifically invalided on a new team creation: * `ALL_TEAMS` * `ALL_ACTIVE_TEAMS` * `ALL_VISIBLE_TEAMS` * `TEAMS_BY_LOGO` * Fixed an extremely rare race condition within `MultiTeam::genFirstCapture()`. * Relevant: Issue #456 * **Combined Awaitables** * Combined Awaitables which were not in nested loops. * Combined Awaitables found in some nested loops, where existing code provided a streamlined approach. * Given the lack of support for concurrent queries to a single database connection, some queries were combined via `multiQuery()` (in the case where the queries were modifying data within the database). TODO: Build and utilize additional `AsyncMysqlConnection` within the pool for suitable concurrent queries. * Annotated Awaitables within a nested loop for future optimization. * Relevant: Issue #577 * **Facebook and Google Login Integration** * Replaced PR #573 * The platform now supports Login via OAuth2 for Facebook and Google. When configured and enabled, users will have the option to link and login to their existing account with a Facebook or Google account. * Automated registration through Facebook or Google OAuth2 is now supported. When configured and enabled, users will have the option to register an account by using and linking an existing account with Facebook or Google. * New `configuration` options added to the database schema: * Added `facebook_login`. This configuration option is a toggleable setting to enable or disable login via Facebook. * Added `google_login`. This configuration option is a toggleable setting to enable or disable login via Google. * Added `facebook_registration`. This configuration option is a toggleable setting to enable or disable registration via Facebook. * Added `google_registration`. This configuration option is a toggleable setting to enable or disable registration via Google. * Added `registration_prefix`. This configuration option is a string that sets the prefix for the randomly generated username/team name for teams registered via (Facebook or Google) OAuth. * New Integration section within the Administrative interface allows for control over the Facebook and Google Login, Registration, and the automatic team name prefix option. * Overhauled the Login page to support the new Login buttons. Login page now displays appropriate messages based on the configuration of login. * Login form is dynamically generated, based on the configuration options and settings. * Overhauled the Registration page to support the new Registration buttons. The registration page now displays appropriate messages based on the configuration of registration. * The registration form is dynamically generated, based on the configuration options and settings. * Account Linking for Facebook sets both the Login OAuth values and the LiveSync values (single step for both). * Account Linking for Google sets both the Login OAuth values and the LiveSync values (single step for both). * Facebook Account linkage option has been added to the Account modal. * The Account modal now shows which accounts are already linked. * The Account modal will color-code the buttons on an error (red) and success (green). * New table "teams_oauth" has been added to handle the OAuth data for Facebook and Google account linkage. * New class `Integration` handles the linkage of Facebook or Google accounts with an FBCTF account (both Login OAuth values and the LiveSync values). The Integration class also includes the underlying methods for authentication in both the linkage and login routines and the OAuth registration process. * New URL endpoints have been created and simplified for the `Integration` actions: * New data endpoint `data/integration_login.php`. This endpoint accepts a type argument, currently supporting types of `facebook` and `google`. Through this endpoint, the login process is handled in conjunction with the Integration class. * The new callback URL for Facebook Login: `/data/integration_login.php?type=facebook` * The new callback URL for Google Login: `/data/integration_login.php?type=google` * New data endpoint data/integration_oauth.php. This endpoint accepts a type argument, currently supporting types of `facebook'`and `google`. Through this endpoint, the OAuth account linkage is handled in conjunction with the Integration class. * The new callback URL for Facebook linkage: /data/integration_login.php?type=facebook * The new callback URL for Google linkage: /data/integration_login.php?type=google * Old Google-specific endpoint (data/google_oauth.php) has been removed. * New Team class methods: `genAuthTokenExists()`, `genTeamFromOAuthToken()`, `genSetOAuthToken()`. * `Team::genAuthTokenExists()` allows an OAuth token to be verified. * `Team::genTeamFromOAuthToken()` returns a Team object based on the OAuth token supplied. * `Team::genSetOAuthToken()` sets the OAuth token for a team. * The `settings.ini` (including the packaged example file) and `Configuration` have methods to verify and return Facebook and Google API settings. * `Configuration::getFacebookOAuthSettingsExists()` verifies the Facebook API _App ID_ and _APP Secret_ are set in the `settings.ini` file. * `Configuration::getFacebookOAuthSettingsAppId()` returns the Facebook API _App ID_. * `Configuration::getFacebookOAuthSettingsAppSecret()` returns the Facebook API _App Secret_. * `Configuration::getGoogleOAuthFileExists()` verifies the Google API JSON file is set and exists in the `settings.ini` file. * `Configuration::getGoogleOAuthFile()` returns the filename for the Google API JSON file. * All of Facebook and Google API configuration values are cached (in Memcached) to prevent the repeated loading, reading, and parsing of the `settings.ini` file. * To use the new Facebook or Google integration the following must be completed: * A Facebook and/or Google Application must be created, and OAuth2 API keys must be obtained. * The API keys must be provided in the Settings.ini file. * Desired settings must be configured from within the administrative interface (Configuration) - the default has all integration turned off. * The Facebook OAuth code provides CSRF protection through the Graph SDK. * The Google OAuth code provides CSRF protection through the usage of the `integration_csrf_token` cookie and API state value. * Note: Facebook Login/Integration will not work in development mode - this is due to a pending issue in the Facebook Graph SDK (facebookarchive/php-graph-sdk#853) utilization of the pending PR (facebookarchive/php-graph-sdk#854) resolves this issue. Alternatively, the Integration with Facebook will work in production mode, the recommended mode for a live game. * Implemented CR from PR #573. * Relevant: PR #591 and PR #459. * **LiveSync API and LiveImport Script Update** * LiveSync has been updated to support and supply Facebook and Google OAuth output. All of the users LiveSync integrations (FBCTF, Facebook, and Google) are now provided through the API. As a result, so long as one of the three LiveSync methods are configured by the user (which happens automatically when linking an account to Facebook or Google) the data will become available through the LiveSync API. * LiveSync now includes a "general" type. The new `general` type output includes the scoring information using the local team name on the FBCTF instance. This new type is not for importation on another FBCTF instance but does provide the opportunity for third-parties to use the data for score tracking, metric collections, and displays. As such, this new LiveSync data allows the scoring data for a single FBCTF instance to be tracked. * The `liveimport.sh` script, used to import LiveSync API data, will ignore the new `general` LiveSync type. * Updated `Team::genLiveSyncKeyExists()` and `Team::genTeamFromLiveSyncKey()` to use the new Integration class methods: `Integration::genFacebookThirdPartyExists)` and `Integration::genFacebookThirdPartyEmail()`. * Within the `liveimport.sh` script: when the type is `facebook_oauth`, `Team::genLiveSyncKeyExists()` and `Team::genTeamFromLiveSyncKey()` properly use the Facebook `third_party_id`. * `Integration::genFacebookThirdPartyExists()` and `Integration::genFacebookThirdPartyEmail()` query the Facebook API for the coorosponding user, storing the results in a temporary HHVM-memory cache, via the `Cache` class. * Given that `liveimport.sh` now needs to query the Facebook API for any `facebook_oauth` typed items, the script will utilize the HHVM-memory cache of `Integration` to limit the number of hits to the Facebook API. * The `liveimport.sh` script now includes the `Cache` class and the Facebook Graph SDK. * Relevant: PR #459. * **Error and Exception Handling** * All Exceptions, including Redirect Exceptions, are now caught. * The NGINX configuration has been updated to catch errors from HHVM (FastCGI) and return `error.php`. * The `error.php` page has been updated with a themed error page. * The `error.php` page will redirect to `index.php?page=error` so long as `index.php?page=error` is not generating any HTTP errors. If an error is detected on `index.php?page=error` then no redirect will occur. The verification of the HTTP status ensures no redirect loops occur. * The `DataController` class now includes a `sendData()` method to catch errors and exceptions. `DataController` children classes now utilize `sendData()` instead of outputing their results directly. * On Exception within an AJAX request, an empty JSON array is returned. This empty array prevents client-side errors. * The `ModuleController` class now includes a `sendRender()` method to catch errors and exceptions. `ModuleController` children classes now utilize `sendRender()` instead of outputing their results directly. * On Exception within a Module request, an empty string is returned. This empty string prevents client-side and front-end errors. * A new AJAX endpoint has been added: `/data/session.php`. The response of the endpoint is used to determine if the user's session is still active. If a user's session is no longer active, they will be redirected from the gameboard to the login page. This redirection ensures that they do not continually perform AJAX requests. * Custom HTTP headers are used to monitor AJAX responses: * The Login page now includes a custom HTTP header: `Login-Page`. * The Error page now includes a custom HTTP header: `Error-Page`. * The custom HTTP headers are used client-side (JS) to determine if a request or page rendered an error or requires authentication. * Exception log outputs now include additional information on which Exception was thrown. * Users should no longer directly receive an HTTP 500. * These Exception changes prevent the error logs from being filled with unauthenticated requests. The changes also provide a user-friendly experience when things malfunction or a user needs to reauthenticate. * Relevant: #563 * **Team Account Modal Update** * Users can now change their team name from within the Account modal. * The account Modal now contains the following options: * Team Name * Facebook Account Linkage * Google Account Linkage * FBCTF LiveSync Authentication * Relevant: PR #459. * **Non-Visible/Inactive Team Update** * Ensure that non-visible or inactive team do not show up for any other users. * Non-Visible/Inactive teams are not awarded as the "first capture." * Non-Visible/Inactive teams do not show in the "captured by" list. * Countries will not show as captured (for other teams) if only captured by a Non-Visible/Inactive team. * Activity Log entries for a Non-Visible or Inactive team are not included in the activity log for other users. * Updated `ScoreLog::genAllPreviousScore()` and `ScoreLog::genPreviousScore()` to only include Visible and Active teams, or the user's own team. * Teams who are Non-Visible or Inactive will have a rank of "N/A." * Relevant: PR #513 * **Mobile Page Update** * The mobile page is shown when a user's window has a resolution under `960px`. While this is geared towards mobile users, it can happen when the window size on a non-mobile device is too small. * The mobile page now includes a "Refresh" button, which will reload the page. * The mobile page will refresh, attempting to re-render correctly, after `2` seconds. * If a user resizes their window to a larger size, they should reload into a properly displayed screen, and not the mobile warning. * **Login and Registration JS Fixes** * Consistently corrected the usage of `teamname` and `team_name` across PHP and JS code. * Ensured that all JavaScript is using `team_name`. * Ensured that all PHP is using `team_name` when interacting with JS. * Updated the input filters within PHP when retrieving input for the team name, using `team_name`. * Updated Login errors to highlight the username and password fields. * Relevant: Issue #571, Issue #558, Issue #521, PR #592, and PR #523 * **System Statistics JSON Endpoint** * A new administrative-only JSON endpoint has been added that provides statistical data about the platform and game. * The endpoint is found at `/data/stata.php`. Access to the endpoint requires an authenticated administrative session. * The endpoint provides the following information: * Number of Teams (`teams`) * Number of Sessions (`sessions`) * Total Number of Levels (`levels`) * Number of Active Levels (`active_levels`) * Number of Hints (`hints`) * Number of Captures (`captures`) * `AsyncMysqlConnectionPool` Statistics (`database`) * Created Connections (`created_pool_connections`) * Destroyed Connections (`destroyed_pool_connections`) * Connection Requests (`connections_requested`) * Pool Hits (`pool_hits`) * Pool Misses (`pool_misses`) * `Memcached` Statistics (`memcached`) * Node Address * Node Address:Port * Process ID (`pid`) * Uptime (`uptime`) * Threads (`threads`) * Timestamp (`time`) * Size of Pointer (`pointer_size`) * Total User Time for Memcached Process (`rusage_user_seconds`) * Total User Time for Memcached Process (`rusage_user_microseconds`) * Total System Time for Memcached Process (`rusage_system_seconds`) * Total System Time for Memcached Process (`rusage_system_microseconds`) * Current Items in Cache (`curr_items`) * Total Items in Cache (`total_items`) * Max Bytes Limit (`limit_maxbytes`) * Number of Current Connections (`curr_connections`) * Number of Total Connections (`total_connections`) * Number of Current Connection Structures Allocated (`connection_structures`) * Number of Bytes Used (`bytes`) * Total Number of Cache Get Requests (`cmd_get`) * Total Number of Cache Set Requests (`cmd_set`) * Total Number Successful of Cache Retrievals (`get_hits`) * Total Number Unsuccessful of Cache Retrievals (`get_ misses`) * Total Number of Cache Evictions (`evictions`) * Total Number of Bytes Read (`bytes_read`) * Total Number of Bytes Written (`bytes_writter`) * Memcached Version (`version`) * System Load Statistics (`load`) * One Minute Average (`0`) * Five Minute Average (`1`) * Fifteen Minute Average (`2`) * System CPU Utilization (`load`) * Userspace Utilization Percentage (`user`) * Nice Utilization Percentage (`nice`) * System Utilization Percentage (`sys`) * Idle Percentage (`idle`) * The endpoint provides current data and can be pooled/ingested for historical data reporting. * For more information on the `AsyncMysqlConnectionPool` statistics, please see: https://docs.hhvm.com/hack/reference/class/AsyncMysqlConnection/ and https://docs.hhvm.com/hack/reference/class/AsyncMysqlConnectionPool/getPoolStats/ * For more information on the `Memcached` statistics, please see: https://github.com/memcached/memcached/blob/master/doc/protocol.txt and https://secure.php.net/manual/en/memcached.getstats.php * **Miscellaneous Changes** * Added `Announcement` and `ActivityLog` to `autorun.php`. * Added `Announcement` and `ActivityLog` to `bases.php`. * Added/Updated UTF-8 encoding on various user-controlled values, such as team name. * Changed the "Sign Up" link to a button on the login page. * Allow any Logo to be re-used once all logos are in use. * Invalidate Scores and Hint cache when a Team is deleted. * Reverify the game status (running or stopped) before the next cycle of base scoring in `bases.php`. * Allow the game to stop even if some scripts (`autorun`, `bases`, `progressive`, etc.) are not running. * Fixed a bug where teams with a custom logo could not be edited by an administrator. * Added "Reset Schedule" button to administrative interface to completely remove a previously set game schedule. The game schedule can only be reset if the game is not running. Otherwise, the existing schedule must be modified. * Moved "Begin Game," "Pause Game," and "End Game" outside of the scrollable admin list into a new fixed pane below the navigation list. * Formatted all code files as part of this PR. * Updated ActivityLog to delete entries when deleting a team. * Updated PHPUnit tests based on the new changes.
iliushin-a
added a commit
to iliushin-a/fbctf
that referenced
this pull request
May 16, 2018
* Major Performance Enhancements and Bug Fixes * **NOTICE**: _This PR is extremely large. Due to the interdependencies, these code changes are being included as a single PR._ * **Local Caching of Database and Memcached Results** * Replaced PR facebookarchive#574 * Results from the database (MySQL) are stored and queried from the Cache service (Memcached). Results from Memcached are now stored locally within HHVM memory as well. * New class `Cache` has been added, with four methods: * `setCache()` * `getCache()` * `deleteCache()` * `flushCache()` * The new `Cache` class object is included as a static property of the `Model` class and all children. * Through the new `Cache` object, all `Model` sub-classes will automatically utilize the temporary HHVM-memory cache for results that have already been retrieved. * The `Cache` object is created and used throughout a single HHVM execution thread (user request). * This change results in a massive reduction in Memcached requests (over 99% - at scale), providing a significant improvement in scalability and performance. * The local cache can be deleted or flushed from the `Model` class or any child class with the usage of `deleteLocalCache()`. The `deleteLocalCache()` method works like `invalidateMCRecords()`. When called from a child class, a `$key` can be passed in, matching the MC key identifier used elsewhere, or if no `$key` is specified all of the local caches for that class will be deleted. If the calling class is `Model` then the entire local cache is flushed. * Some non-HTTP code has local cache values explicitly deleted, or the local cache completely flushed, as the execution thread is continuous: * Prevent `autorun.php` from storing timestamps in the local cache, forever (the script runs continuously). * Flush the local cache before the next cycle of `bases.php` to ensure the game is still running and the configuration of the bases has not changed (the script runs continuously). * Flush the local cache before the next import cycle of `liveimport.php` to ensure we get the up-to-date team and level data (the script runs continuously). * The `Cache` class is specifically separate from `Model` (as an independent class) so that other code may instantiate and utilize a temporary (request-exclusive) local-memory-based caching solution, with a common interface. The usage provides local caching without storing the data in MySQL, Memcached, or exposing it to other areas of the application. (For example, this is being utilized in some `Integration` code already.) * Implemented CR from PR facebookarchive#574. * Relevant: Issue facebookarchive#456 and Comment facebookarchive#456 (comment) * **Blocking AJAX Requests** * Replaced PR facebookarchive#575 * Expansion and Bug Fixes of PR facebookarchive#565 * AJAX requests for the gameboard are now individually blocking. A new request will not be dispatched until the previous request has completed. * AJAX requests will individually stop subsequent requests on a hard-error. * The blocking of continuous AJAX requests, when the previous has not yet returned, or on a hard error, provides a modest performance benefit by not compounding the issue with more unfulfillable requests. * Forceful refreshes are still dispatched every 60 seconds, regardless of the blocking state on those requests. * Relevant: Issue facebookarchive#456 and Comment facebookarchive#456 (comment) * **AJAX Endpoint Optimization** * Removed nested loops within multiple AJAX endpoints: * `map-data.php` * `country-data.php` * ` leaderboard.php` * All Attachments, including Link and Filename, are now cached and obtained through: `Attachment::genAllAttachmentsFileNamesLinks()`. * All Team names of those who have completed a level, are now cached and obtained through `MultiTeam::genCompletedLevelTeamNames()`. * All Levels and Country for map displays are cached and obtained through `Level::genAllLevelsCountryMap()` and `Country::genAllEnabledCountriesForMap()`. * Relevant: Issue facebookarchive#456 * **Memcached Cluster Support** * The platform now supports a cluster of Memcached nodes. * Configuration for the `MC_HOST` within the `settings.ini` file is now an array, instead of a single value: * `MC_HOST[] = 127.0.0.1` * Multiple Memcached servers can be configured by providing additional `MC_HOST` lines: * `MC_HOST[] = 1.2.3.4` * `MC_HOST[] = 5.6.7.8` * The platform uses a Write-Many Read-Once approach to the Memcached Cluster. Specifically, data is written to all of the configured Memcached nodes and then read from a single node at random. This approach ensures that all of the nodes stay in sync and up-to-date while providing a vital performance benefit to the more expensive and frequent operation of reading. * The existing `Model` methods (`setMCRecords()` and `invalidateMCRecords()`) all call and utilize the new cluster methods: * `writeMCCluster()` * `invalidateMCCluster()` * The flushing of Memcached has also been updated to support the multi-cluster approach: `flushMCCluster()`. * Note that the usage of a Memcached Cluster is seamless for administrators and users, and works in conjunction with the Local Cache. Also note, the platform works identically, for administrators and users, for both single-node and multi-node Memcached configurations. * The default configuration remains a single-node configuration. The utilization of a Memcached Cluster requires the following: * The configuration and deployment of multiple Memcached nodes (the `quick_setup install_multi_cache` or Memcached specific provision, will work). * The modification of `settings.ini` to include all of the desired Memcached hosts. * All Memcached hosts must be identically configured. * Usage of a Memcached Cluster is only recommended in the Multi-Server deployment modes. * Relevant: Issue facebookarchive#456 * **Load Balancing of Application Servers (HHVM)** * The platform now supports the ability to load balance multiple HHVM servers. * To facilitate the load balancing of the HHVM servers, the following changes were made: * Scripts (`autorun`, `progressive`, etc.) are now tracked on a per-server level, preventing multiple copies of the scripts from being executed on the HHVM servers. * Additional database verification on scoring events to prevent multiple captures. * Load Balancing of HHVM is only recommended in the Multi-Server deployment modes. * Relevant: Issue facebookarchive#456 * **Leaderboard Limits** * `MultiTeam::genLeaderboard()` now limits the total number of teams returned based on a configurable setting. * A new argument has been added to `MultiTeam::genLeaderboard()`: `limit`. This value, either `true` or `false`, indicates where the limit should be enforced, and defaults to `true`. * When the data is not cached, `MultiTeam::genLeaderboard()` will only build, cache, and return the number of teams needed to meet the limit. * When the data is already cached, `MultiTeam::genLeaderboard()` will ensure the limit value has not changed and returned the cached results. If the configured limit value has been changed, `MultiTeam::genLeaderboard()` will build, cache, and return the number based on the new limit. * The "Scoreboard" modal (found from the main gameboard) is a special case where all teams should be displayed. As such, the Scoreboard modal sets the `limit` value to `false` retuning all teams. This full leaderboard will be cached, but all other display limits are still enforced based on the configured limit. Once invalidated, the cached data will return to the limited subset. * Because a full leaderboard is not always cached, this does result in the first hit to the Scoreboard modal requiring a database hit. * A user, whose rank is above the limit, will have their rank shown to them as `$limit+`. For example, if the limit is set to `50` and the user's rank is above `50`, they would see: `51+` as their rank. * Overall, the caching of the Leaderboard, one of the more resource-intensive and frequent queries, resulted in significant performance gains. * The Leaderboard limit is configurable by administrators within the administrative interface. The default value is `50`. * Relevant: Issue facebookarchive#456 * **Activity Log Limits** * The Activity Log is now limited to the most recent `100` log entries. The value is not configurable. * The activity log is continually queried and contains a large amount of data, as such, it is a very resource-intensive request. * The limit on the results built, cached, and returned for the activity log provides a notable improvement in performance. * Relevant: Issue facebookarchive#456 * **Database Optimization** * Expansion of PR facebookarchive#564 * Added additional indexing of the database tables in the schema. * The additional indexing provides further performance improvements to the platform queries, especially those found in `MultiTeam` and those queries continually utilized as a result of the AJAX calls. * Relevant: Issue facebookarchive#456 and Comment facebookarchive#456 (comment) * **Team and MultiTeam Performance Improvements** * Updated numerous `Team::genTeam()` calls to used the cached version: `MultiTeam::genTeam()`. * Optimized the database query within `MultiTeam::genFirstCapture()` to return the `team_id` and build the `Team` from the cache. * Optimized the database query within `MultiTeam::genCompletedLevel()` to return the `team_id` and build the `Team` from the cache. * Optimized the database query within `MultiTeam::genAllCompletedLevels()` to return the `team_id` and build the `Team` from the cache. * A full invalidation of the `MultiTeam` cache is no longer executed when a new team is created. Newly created teams will not have any valid scoring activity. Delaying the rebuild of the scoring related cache provides a modest performance improvement. The new team will not show up in certain areas (namely the full scoreboard) until they or someone else perform a scoring action. To ensure the team is properly functioning, following cache is specifically invalided on a new team creation: * `ALL_TEAMS` * `ALL_ACTIVE_TEAMS` * `ALL_VISIBLE_TEAMS` * `TEAMS_BY_LOGO` * Fixed an extremely rare race condition within `MultiTeam::genFirstCapture()`. * Relevant: Issue facebookarchive#456 * **Combined Awaitables** * Combined Awaitables which were not in nested loops. * Combined Awaitables found in some nested loops, where existing code provided a streamlined approach. * Given the lack of support for concurrent queries to a single database connection, some queries were combined via `multiQuery()` (in the case where the queries were modifying data within the database). TODO: Build and utilize additional `AsyncMysqlConnection` within the pool for suitable concurrent queries. * Annotated Awaitables within a nested loop for future optimization. * Relevant: Issue facebookarchive#577 * **Facebook and Google Login Integration** * Replaced PR facebookarchive#573 * The platform now supports Login via OAuth2 for Facebook and Google. When configured and enabled, users will have the option to link and login to their existing account with a Facebook or Google account. * Automated registration through Facebook or Google OAuth2 is now supported. When configured and enabled, users will have the option to register an account by using and linking an existing account with Facebook or Google. * New `configuration` options added to the database schema: * Added `facebook_login`. This configuration option is a toggleable setting to enable or disable login via Facebook. * Added `google_login`. This configuration option is a toggleable setting to enable or disable login via Google. * Added `facebook_registration`. This configuration option is a toggleable setting to enable or disable registration via Facebook. * Added `google_registration`. This configuration option is a toggleable setting to enable or disable registration via Google. * Added `registration_prefix`. This configuration option is a string that sets the prefix for the randomly generated username/team name for teams registered via (Facebook or Google) OAuth. * New Integration section within the Administrative interface allows for control over the Facebook and Google Login, Registration, and the automatic team name prefix option. * Overhauled the Login page to support the new Login buttons. Login page now displays appropriate messages based on the configuration of login. * Login form is dynamically generated, based on the configuration options and settings. * Overhauled the Registration page to support the new Registration buttons. The registration page now displays appropriate messages based on the configuration of registration. * The registration form is dynamically generated, based on the configuration options and settings. * Account Linking for Facebook sets both the Login OAuth values and the LiveSync values (single step for both). * Account Linking for Google sets both the Login OAuth values and the LiveSync values (single step for both). * Facebook Account linkage option has been added to the Account modal. * The Account modal now shows which accounts are already linked. * The Account modal will color-code the buttons on an error (red) and success (green). * New table "teams_oauth" has been added to handle the OAuth data for Facebook and Google account linkage. * New class `Integration` handles the linkage of Facebook or Google accounts with an FBCTF account (both Login OAuth values and the LiveSync values). The Integration class also includes the underlying methods for authentication in both the linkage and login routines and the OAuth registration process. * New URL endpoints have been created and simplified for the `Integration` actions: * New data endpoint `data/integration_login.php`. This endpoint accepts a type argument, currently supporting types of `facebook` and `google`. Through this endpoint, the login process is handled in conjunction with the Integration class. * The new callback URL for Facebook Login: `/data/integration_login.php?type=facebook` * The new callback URL for Google Login: `/data/integration_login.php?type=google` * New data endpoint data/integration_oauth.php. This endpoint accepts a type argument, currently supporting types of `facebook'`and `google`. Through this endpoint, the OAuth account linkage is handled in conjunction with the Integration class. * The new callback URL for Facebook linkage: /data/integration_login.php?type=facebook * The new callback URL for Google linkage: /data/integration_login.php?type=google * Old Google-specific endpoint (data/google_oauth.php) has been removed. * New Team class methods: `genAuthTokenExists()`, `genTeamFromOAuthToken()`, `genSetOAuthToken()`. * `Team::genAuthTokenExists()` allows an OAuth token to be verified. * `Team::genTeamFromOAuthToken()` returns a Team object based on the OAuth token supplied. * `Team::genSetOAuthToken()` sets the OAuth token for a team. * The `settings.ini` (including the packaged example file) and `Configuration` have methods to verify and return Facebook and Google API settings. * `Configuration::getFacebookOAuthSettingsExists()` verifies the Facebook API _App ID_ and _APP Secret_ are set in the `settings.ini` file. * `Configuration::getFacebookOAuthSettingsAppId()` returns the Facebook API _App ID_. * `Configuration::getFacebookOAuthSettingsAppSecret()` returns the Facebook API _App Secret_. * `Configuration::getGoogleOAuthFileExists()` verifies the Google API JSON file is set and exists in the `settings.ini` file. * `Configuration::getGoogleOAuthFile()` returns the filename for the Google API JSON file. * All of Facebook and Google API configuration values are cached (in Memcached) to prevent the repeated loading, reading, and parsing of the `settings.ini` file. * To use the new Facebook or Google integration the following must be completed: * A Facebook and/or Google Application must be created, and OAuth2 API keys must be obtained. * The API keys must be provided in the Settings.ini file. * Desired settings must be configured from within the administrative interface (Configuration) - the default has all integration turned off. * The Facebook OAuth code provides CSRF protection through the Graph SDK. * The Google OAuth code provides CSRF protection through the usage of the `integration_csrf_token` cookie and API state value. * Note: Facebook Login/Integration will not work in development mode - this is due to a pending issue in the Facebook Graph SDK (facebookarchive/php-graph-sdk#853) utilization of the pending PR (facebookarchive/php-graph-sdk#854) resolves this issue. Alternatively, the Integration with Facebook will work in production mode, the recommended mode for a live game. * Implemented CR from PR facebookarchive#573. * Relevant: PR facebookarchive#591 and PR facebookarchive#459. * **LiveSync API and LiveImport Script Update** * LiveSync has been updated to support and supply Facebook and Google OAuth output. All of the users LiveSync integrations (FBCTF, Facebook, and Google) are now provided through the API. As a result, so long as one of the three LiveSync methods are configured by the user (which happens automatically when linking an account to Facebook or Google) the data will become available through the LiveSync API. * LiveSync now includes a "general" type. The new `general` type output includes the scoring information using the local team name on the FBCTF instance. This new type is not for importation on another FBCTF instance but does provide the opportunity for third-parties to use the data for score tracking, metric collections, and displays. As such, this new LiveSync data allows the scoring data for a single FBCTF instance to be tracked. * The `liveimport.sh` script, used to import LiveSync API data, will ignore the new `general` LiveSync type. * Updated `Team::genLiveSyncKeyExists()` and `Team::genTeamFromLiveSyncKey()` to use the new Integration class methods: `Integration::genFacebookThirdPartyExists)` and `Integration::genFacebookThirdPartyEmail()`. * Within the `liveimport.sh` script: when the type is `facebook_oauth`, `Team::genLiveSyncKeyExists()` and `Team::genTeamFromLiveSyncKey()` properly use the Facebook `third_party_id`. * `Integration::genFacebookThirdPartyExists()` and `Integration::genFacebookThirdPartyEmail()` query the Facebook API for the coorosponding user, storing the results in a temporary HHVM-memory cache, via the `Cache` class. * Given that `liveimport.sh` now needs to query the Facebook API for any `facebook_oauth` typed items, the script will utilize the HHVM-memory cache of `Integration` to limit the number of hits to the Facebook API. * The `liveimport.sh` script now includes the `Cache` class and the Facebook Graph SDK. * Relevant: PR facebookarchive#459. * **Error and Exception Handling** * All Exceptions, including Redirect Exceptions, are now caught. * The NGINX configuration has been updated to catch errors from HHVM (FastCGI) and return `error.php`. * The `error.php` page has been updated with a themed error page. * The `error.php` page will redirect to `index.php?page=error` so long as `index.php?page=error` is not generating any HTTP errors. If an error is detected on `index.php?page=error` then no redirect will occur. The verification of the HTTP status ensures no redirect loops occur. * The `DataController` class now includes a `sendData()` method to catch errors and exceptions. `DataController` children classes now utilize `sendData()` instead of outputing their results directly. * On Exception within an AJAX request, an empty JSON array is returned. This empty array prevents client-side errors. * The `ModuleController` class now includes a `sendRender()` method to catch errors and exceptions. `ModuleController` children classes now utilize `sendRender()` instead of outputing their results directly. * On Exception within a Module request, an empty string is returned. This empty string prevents client-side and front-end errors. * A new AJAX endpoint has been added: `/data/session.php`. The response of the endpoint is used to determine if the user's session is still active. If a user's session is no longer active, they will be redirected from the gameboard to the login page. This redirection ensures that they do not continually perform AJAX requests. * Custom HTTP headers are used to monitor AJAX responses: * The Login page now includes a custom HTTP header: `Login-Page`. * The Error page now includes a custom HTTP header: `Error-Page`. * The custom HTTP headers are used client-side (JS) to determine if a request or page rendered an error or requires authentication. * Exception log outputs now include additional information on which Exception was thrown. * Users should no longer directly receive an HTTP 500. * These Exception changes prevent the error logs from being filled with unauthenticated requests. The changes also provide a user-friendly experience when things malfunction or a user needs to reauthenticate. * Relevant: facebookarchive#563 * **Team Account Modal Update** * Users can now change their team name from within the Account modal. * The account Modal now contains the following options: * Team Name * Facebook Account Linkage * Google Account Linkage * FBCTF LiveSync Authentication * Relevant: PR facebookarchive#459. * **Non-Visible/Inactive Team Update** * Ensure that non-visible or inactive team do not show up for any other users. * Non-Visible/Inactive teams are not awarded as the "first capture." * Non-Visible/Inactive teams do not show in the "captured by" list. * Countries will not show as captured (for other teams) if only captured by a Non-Visible/Inactive team. * Activity Log entries for a Non-Visible or Inactive team are not included in the activity log for other users. * Updated `ScoreLog::genAllPreviousScore()` and `ScoreLog::genPreviousScore()` to only include Visible and Active teams, or the user's own team. * Teams who are Non-Visible or Inactive will have a rank of "N/A." * Relevant: PR facebookarchive#513 * **Mobile Page Update** * The mobile page is shown when a user's window has a resolution under `960px`. While this is geared towards mobile users, it can happen when the window size on a non-mobile device is too small. * The mobile page now includes a "Refresh" button, which will reload the page. * The mobile page will refresh, attempting to re-render correctly, after `2` seconds. * If a user resizes their window to a larger size, they should reload into a properly displayed screen, and not the mobile warning. * **Login and Registration JS Fixes** * Consistently corrected the usage of `teamname` and `team_name` across PHP and JS code. * Ensured that all JavaScript is using `team_name`. * Ensured that all PHP is using `team_name` when interacting with JS. * Updated the input filters within PHP when retrieving input for the team name, using `team_name`. * Updated Login errors to highlight the username and password fields. * Relevant: Issue facebookarchive#571, Issue facebookarchive#558, Issue facebookarchive#521, PR facebookarchive#592, and PR facebookarchive#523 * **System Statistics JSON Endpoint** * A new administrative-only JSON endpoint has been added that provides statistical data about the platform and game. * The endpoint is found at `/data/stata.php`. Access to the endpoint requires an authenticated administrative session. * The endpoint provides the following information: * Number of Teams (`teams`) * Number of Sessions (`sessions`) * Total Number of Levels (`levels`) * Number of Active Levels (`active_levels`) * Number of Hints (`hints`) * Number of Captures (`captures`) * `AsyncMysqlConnectionPool` Statistics (`database`) * Created Connections (`created_pool_connections`) * Destroyed Connections (`destroyed_pool_connections`) * Connection Requests (`connections_requested`) * Pool Hits (`pool_hits`) * Pool Misses (`pool_misses`) * `Memcached` Statistics (`memcached`) * Node Address * Node Address:Port * Process ID (`pid`) * Uptime (`uptime`) * Threads (`threads`) * Timestamp (`time`) * Size of Pointer (`pointer_size`) * Total User Time for Memcached Process (`rusage_user_seconds`) * Total User Time for Memcached Process (`rusage_user_microseconds`) * Total System Time for Memcached Process (`rusage_system_seconds`) * Total System Time for Memcached Process (`rusage_system_microseconds`) * Current Items in Cache (`curr_items`) * Total Items in Cache (`total_items`) * Max Bytes Limit (`limit_maxbytes`) * Number of Current Connections (`curr_connections`) * Number of Total Connections (`total_connections`) * Number of Current Connection Structures Allocated (`connection_structures`) * Number of Bytes Used (`bytes`) * Total Number of Cache Get Requests (`cmd_get`) * Total Number of Cache Set Requests (`cmd_set`) * Total Number Successful of Cache Retrievals (`get_hits`) * Total Number Unsuccessful of Cache Retrievals (`get_ misses`) * Total Number of Cache Evictions (`evictions`) * Total Number of Bytes Read (`bytes_read`) * Total Number of Bytes Written (`bytes_writter`) * Memcached Version (`version`) * System Load Statistics (`load`) * One Minute Average (`0`) * Five Minute Average (`1`) * Fifteen Minute Average (`2`) * System CPU Utilization (`load`) * Userspace Utilization Percentage (`user`) * Nice Utilization Percentage (`nice`) * System Utilization Percentage (`sys`) * Idle Percentage (`idle`) * The endpoint provides current data and can be pooled/ingested for historical data reporting. * For more information on the `AsyncMysqlConnectionPool` statistics, please see: https://docs.hhvm.com/hack/reference/class/AsyncMysqlConnection/ and https://docs.hhvm.com/hack/reference/class/AsyncMysqlConnectionPool/getPoolStats/ * For more information on the `Memcached` statistics, please see: https://github.com/memcached/memcached/blob/master/doc/protocol.txt and https://secure.php.net/manual/en/memcached.getstats.php * **Miscellaneous Changes** * Added `Announcement` and `ActivityLog` to `autorun.php`. * Added `Announcement` and `ActivityLog` to `bases.php`. * Added/Updated UTF-8 encoding on various user-controlled values, such as team name. * Changed the "Sign Up" link to a button on the login page. * Allow any Logo to be re-used once all logos are in use. * Invalidate Scores and Hint cache when a Team is deleted. * Reverify the game status (running or stopped) before the next cycle of base scoring in `bases.php`. * Allow the game to stop even if some scripts (`autorun`, `bases`, `progressive`, etc.) are not running. * Fixed a bug where teams with a custom logo could not be edited by an administrator. * Added "Reset Schedule" button to administrative interface to completely remove a previously set game schedule. The game schedule can only be reset if the game is not running. Otherwise, the existing schedule must be modified. * Moved "Begin Game," "Pause Game," and "End Game" outside of the scrollable admin list into a new fixed pane below the navigation list. * Formatted all code files as part of this PR. * Updated ActivityLog to delete entries when deleting a team. * Updated PHPUnit tests based on the new changes.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introducing the Live Sync API.
The Live Sync API allows administrators to import game activity in near-real-time. Users can link their accounts on one or more FBCTF platform instances and their scores will be automatically imported into the systems that have been linked.
The primary use-case revolves around event aggregation across multiple FBCTF instances. Event organizers can now separate FBCTF instances and combine scores into one global scoreboard.
The Live Sync API will import Levels, Categories, Scoring Events, and Hint Usage. Scores are automatically calculated, and bonuses are updated to ensure accurate scoring across linked FBCTF instances.
Administrators determine which systems, if any, are linked.
Users must link their account in order for their activity to be synced.
The UI/UX of FBCTF has been updated to include a mechanism for users to configure their Live Sync credentials.
Users cannot obtain hints or capture levels on the importing system.
The API is JSON based and the schema is generalized so that it can be leveraged by other platforms or external processes. So data can be synced from non-FBCTF platforms.
The importing script will automatically handle country conflicts (if two systems have the same country selected for a level).
USER GUIDE (Documentation):
Users must first have an account on all FBCTF instances they wish to link.
The user must then login and access the game board.
From the navigation menu, the user should select "Account."
The user must then set a Live Sync username and password. The Live Sync username and password must not be their login credentials. In fact, users are prohibited from using their account password as their Live Sync password.
The user would repeat the above steps for each FBCTF instance they wish to link. The Live Sync credentials must be the same on each FBCTF instance or their accounts will not be linked.
ADMIN GUIDE (Documentation):
The admin is free to sync as many platforms as their desire. Additionally, the admin may import from as many API sources as their desire.
The admin will need to launch the "live import" script, on any importing systems, from the command-line:
hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php <Space Delimited URLs to API Endpoints> <Sleep Between Cycles> <Disable SSL Certificate Verification> <Show Debug Messages>Disabling of the SSL Verification and the Debugging arguments are both optional. The URL(s) and Sleep time are required arguments.
EXAMPLE:
hhvm -vRepo.Central.Path=/var/run/hhvm/.hhvm.hhbc_liveimport /var/www/fbctf/src/scripts/liveimport.php "https://10.10.10.101/data/livesync.php https://10.10.10.102/data/livesync.php https://10.10.10.103/other/platform/api" 300 true trueAPI SCHEMA (JSON):
[{"active":true,"type":"flag","title":"Example Level 1","description":"This is the first example level.","entity_iso_code":"TJ","category":"None","points":100,"bonus":30,"bonus_dec":10,"penalty":10,"teams":{"fbctf:user1:$2y$12$a1T4KyqqxADi3YIJ7M2sf.VoSHz6qMBx.zrxAIvZnD8de95EsLeny":{"timestamp":"2017-02-17 02:20:22","capture":true,"hint":false}}}]TO DO (Enhancements):
Implemented alternative Live Sync key/authentication mechanisms, such as: Facebook Login, OAuth, etc.
Improve the processing of Bases/Progressive scoring.
Integrate password strength enforcement for the Live Sync credentials.