EQL
EQL is a language that can match events, generate sequences, stack data, build aggregations, and perform analysis. EQL is schemaless and supports multiple database backends. It supports field lookups, boolean logic, comparisons, wildcard matching, and function calls.
-- EndGame
Within DeTT&CT, EQL provides you powerful options to exclude or include particular objects (detections, visibility or data sources) from your techniques and data sources YAML administration files. You can find more information on how to write EQL queries and its syntax here: EndGame's - Query Guide
Content:
When including/excluding certain YAML objects, the result should match the schema. If not, DeTT&CT throws an error. Function call like | count will therefore not work. However, it does execute and shows you the result of the query.
When you want to visualise how detection/visibility scores looked like in a certain period (e.g. to show how you have improved) using visuals created in the ATT&CK navigator. Be sure to use the option --all-scores. Otherwise, you will include only the most recent scores in the EQL query. So, looking back in time use the argument --all-scores.
We have no support, yet, for performing EQL queries on Group YAML files.
When writing a query, and you are not sure how to build your query. It can help to create an invalid query on purpose. This will print out the schema. For example, see below the schema for the detection object:
python dettect.py d -ft sample-data/techniques-administration-endpoints.yaml --search-detection "schema"
Error at line:1,column:1
Field not recognised
schema
^^^^^^
Take into account the following schema:
{'techniques': {'detection': {'applicable_to': ['string'],
'comment': 'string',
'location': ['string'],
'score_logbook': {'comment': 'string',
'date': 'mixed',
'score': 'number'}},
'event_type': 'string',
'technique_id': 'string',
'technique_name': 'string'}}
Below you will find a few examples on how to get started using EQL in DeTT&CT.
Filter on the key-value pair applicable_to within the techniques administration YAML file. For example, to only include 'Windows workstations':
python dettect.py d -ft sample-data/techniques-administration-endpoints.yaml --layer --search-detection "techniques where arrayContains(detection.applicable_to, 'Windows workstations')"
Multiple applicable_to values can be provided by adding a comma after 'Windows workstations' and a new value.
The EQL query for including only detections implemented at a specific location can be achieved with a very similar query as used within filtering on applicable_to. Again we use the EQL function arrayContains:
--search-detection "techniques where arrayContains(detection.location, 'EDR')"
You may have a use case in which you want to filter out detection with a low score such as one and focus on the higher scores:
--search-detection "techniques where detection.score_logbook.score > 1"
When generating a rough visibility score based on the number of data sources you have available you can choose, in multiple ways, to modify which data sources are included in the calculations.
For example:
- Only include data sources that can be used in data analytics:
python dettect.py ds -fd sample-data/data-sources-endpoints.yaml -l --search "data_sources where available_for_data_analytics = true"
- Only include data source for which the data quality dimension device completeness, data field completeness and retention have a score higher or equal to 3:
python dettect.py ds -fd sample-data/data-sources-endpoints.yaml -l --search "data_sources where data_quality.device_completeness >= 3 and data_quality.data_field_completeness >= 3 and data_quality.retention >= 3"
Once you have built-up history on when detection/visibility scores have changed (within the score_logbook), you can visualise this change within an ATT&CK Navigator layer file using an EQL query. Of course, also without much history, this can be done purely based on when you have added new detections or visibility.
First, we create a layer showing only the detections with a date before 2021-10-01. Be aware that in these cases you have to include the option --all-scores. Otherwise, the EQL query will only include the most recent detection score objects (from within a score_logbook) and thereby creating a false representation of your detection coverage. (For example, the detection for T1569.002 (as present in the sample technique administration) was improved after 2021-10-01 with a score of level 4. In this case, the whole detection for T1569.002 would not part of the Navigator layer file, as only the most recent score objects were included in the search).
Detection coverage before 2021-10-01:
python dettect.py d -ft sample-data/techniques-administration-endpoints.yaml -l --search-detection "techniques where detection.score_logbook.date < '2021-10-01'" --all-scores
And then we create a Navigator layer showing the current detection coverage:
python dettect.py d -ft sample-data/techniques-administration-endpoints.yaml -l
{'data_sources': {'applicable_to': 'string',
'available_for_data_analytics': 'boolean',
'comment': 'string',
'data_quality': {'consistency': 'number',
'data_field_completeness': 'number',
'device_completeness': 'number',
'retention': 'number',
'timeliness': 'number'},
'data_source_name': 'string',
'date_connected': 'null',
'date_registered': 'null',
'products': ['string']}}{'techniques': {'technique_id': 'string',
'technique_name': 'string',
'visibility': {'applicable_to': ['string'],
'location': ['string'],
'comment': 'string',
'score_logbook': {'date': 'mixed',
'score': 'number',
'comment': 'string'}}}}{'techniques': {'technique_id': 'string',
'technique_name': 'string',
'detection': {'applicable_to': ['string'],
'comment': 'string',
'score_logbook': {'date': 'mixed',
'score': 'number',
'comment': 'string'}}}}