Velociraptor & OS Query Data providers

[ianhelle]

[ianhelle]

@juju4 has created a couple of PoC notebooks importing and manipulating Velociraptor and OSQuery data.
Three main components:

1. Data ingestion
2. Data wrangling to get data to work with MSTICPy components
3. Common analysis techniques for the data.

My thoughts on 1 and 2:

• Create a QueryProvider for each of OSQuery and VR - supply an input path for the data
  • Based (loosely) on the LocalData provider
  • Implement queries for the most common data formats
  • Add pandas processing instructions to the query that tells pandas how to treat the data (e.g. which fields are dates)
  • Add a facility to specify a data transform function for more complex manipulation (if needed)
  • Create generic queries for less common data sets

qp = QueryProvider("OSQuery", source_path="~/host-inv-123")
proc_df = qp.list_processes()
proc_df.mp_plot.process_tree()
proc_df.mp_plot.timeline()

Analysis functions

Some of these seem pretty generically useful - e.g. processes masquerading as system processes.
Not sure what the best place to put these is:

1. Could be a notebook that uses the data providers
2. Could create analysis modules in msticpy.analysis (some of these would need to be partly data/config-driven - e.g. list of system procs that were spoofing targets). This list might evolve over time and users might want to add their own custom items
3. Create notebooklets that implement this kind of analysis (and notebooks that use them)
   (we could also do 2 and then migrate to 3)

My worry with 2 is that we are treading into a detection area that we might not have the capability to support properly. This seems more like Sigma territory. Long-term it would be better if we could consume this type of analysis from somewhere else - where it can be better maintained by the community.

Open to discussion on this.

[ianhelle]

Notes from 2023-01-26 discussion:

• Velociraptor and OSQuery data can be stored in a number of DB backends
• Common use case is to take a snapshot of data sets saved as CSVs in a zip file or folder
• We might be able to cover both cases by implementing an ODBC helper class (maybe mixin)
  • Usage might be something like:

  qry_prov = QueryProvider("OSQuery", path="/foo")
  qry_prov = QueryProvider("OSQuery", db="sqllite_dsn")   # sqllite_dsn is ODBC connection defn in msticpyconfig
  qyr_prov = QueryProvider("OSQuery", db="the_full_dsn")  # the connection string info

• The DSN reference could be in DataProviders section and should be able to use KeyVault/Keyring to obtain secrets
• We might want to use something like DuckDB to create a SQL interface over the data read into pandas

Implementation notes/thoughts

• Rather than implementing specific driver classes for VR, OSQ, Velocity3, etc. we can use a generic class with
  a specific ID that reflects the data source.
• When reading in queries the driver instance will look for queries with matching data source ID.
• The implementation can be partly based on the localdata_driver - which is just a convenience wrapper for importing CSV or pickled DF files.
• Queries definitions - are essentially just mappings to the different data sets - e.g. processes, network_connections, etc. and (initially)
  don't any any filtering, joins, etc.

Maybe a good class structure would be:

• Generic data driver
• Mixins for file, and ODBC sources
• When instantiating a driver for the QueryProvider, user-supplied parameters would select the data type ID (defining which queries should be loaded) and access method (file, DB) which controls how the data is going to be pulled in.

[juju4]

Great notes!
As discussed, will likely start with Local provider and queries definition which would help both as demo and value case.
With various backends, more on the logging side (sentinel, elastic and so on), there will be variation on the schema that needs to be handled either with column renaming/mapping, either with a normalized schema (OSSEM or other)

Available data will also depend on tool configuration

• Velociraptor: using offline collection data. Example Windows data set from BlueTeamVillage project obsidian
• Osquery: depends on query packs used, to start with default ones (https://github.com/osquery/osquery/tree/master/packs) with maybe a bit of extra for important columns like https://github.com/juju4/ansible-osquery/blob/master/templates/osquery-custom-pack.conf.j2

LocalData provider reference
https://github.com/microsoft/msticpy/blob/main/docs/source/data_acquisition/DataProv-LocalData.rst
https://github.com/microsoft/msticpy/blob/main/msticpy/data/drivers/local_data_driver.py

[juju4]

I submitted few PR

• one to enrich local_data_driver with json files - this helps cover velociraptor offline collection which output is either json, either csv - local_data_driver: add file_extensions option and json files processing #622 add localdata queries file and notebook examples to process velociraptor offline collection #623
• one for local_osquery_driver based on local data one. I felt that it was necessary as an osquery log file contains output for multiple queries. Not very optimized but working - add LocalOsquery driver based on LocalData one #624
  In both cases, also including a base queries yaml file.

To review separately for the normalization and analysis function.

[juju4]

Velociraptor offline collector local json outputs handled by fantastic PR from @ianhelle: #668