AIL Framework version 2.0 released including a user management functionality, many bugs fixed and improvements

v2.0 (2019-07-05)

Changes

• [helper] dump crawler history by daterange. [Terrtia]
• [UI submit items] bootstrap 4 migration. [Terrtia]
• [Flask login] add brute force protection + log login errors. [Terrtia]
• [helper] generate self signed certificates. [Terrtia]
• [Flask server] https support + create self signed certificate.
  [Terrtia]
• [user_management 2.0] add update scripts + fix create_default_user.
  [Terrtia]
• [UI user_management] user_role acl: hide admin panel. [Terrtia]
• [UI user_management] incorrect passwords: display errors. [Terrtia]
• [user_management endpoint] check user roles + add 503 template.
  [Terrtia]
• [UI dashboard + search] bootstrap 4 migration: dashbaord + fix search
  input. [Terrtia]
• [UI crawler, show_domain] domain history: remove target blank.
  [Terrtia]
• [UI crawler, show_domain] add domain history list + navigation.
  [Terrtia]
• [user_managemant] clean code + check password and email length.
  [Terrtia]
• [user_management UI] add admin section: edit + create users. [Terrtia]
• [user_management UI] edit my_profile + renew api tokens. [Terrtia]
• [user_management] create default admin user (temp passwd save in
  AIL_HOME) + change password UI + logout UI + create random password.
  [Terrtia]
• [restapi] add rest api authentification + create default user.
  [Terrtia]
• [user_management] add user role_management. [Terrtia]
• [user_management] create + check user password. [Terrtia]
• [UI] add basic user management. [Terrtia]

Fix

• [items submit UI] fix tags dropdown. [Terrtia]
• [helper dump_crawler] fix files not found. [Terrtia]
• [helper dump_crawler] fix empty dict. [Terrtia]
• [MISP export] fix event creation. [Terrtia]
• [UI items_submit] add active tag + fix template name. [Terrtia]
• [UI login/change_password] add missing ail-logo. [Terrtia]
• [Update] clean output. [Terrtia]
• [Update] add default update script. [Terrtia]
• [install_dep] create update current_version. [Terrtia]
• [UI settings] fix toggle_sidebar. [Terrtia]
• [install_dep] create default user. [Terrtia]
• [user_management] fix tokens duplicate + check user_acl_integrity +
  add login errors messages. [Terrtia]
• [server endpoint] unknow users: avoid endpoint enumeration. [Terrtia]
• [TheHive feeder] create_alert: push all items tags TODO check items
  status + add more item metadata. [Terrtia]
• [domain history] fix domain status. [Terrtia]

Other

• Merge pull request #359 from CIRCL/user_management. [Alexandre
  Dulaunoy]

  User/role management

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]

• Update README.md. [Thirion Aurélien]

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'user_management' of https://github.com/CIRCL/AIL-
  framework into user_management. [Terrtia]

• Update README.md. [Thirion Aurélien]

  Update install instructions

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' into user_management. [Terrtia]

AIL Framework version 1.8 released including many bug fixes, improved PGP handling, ability to export crawled resource.

v1.8 (2019-06-12)

Changes

• [UI crawled domains] Download all domain content (HTML + HAR +
  screenshot) [Terrtia]
• [backend crawler] domains: download 1 archive by crawled (most recent)
  [Terrtia]
• [paste_submit UI] add debug. [Terrtia]
• [template] the "item" project. [Alexandre Dulaunoy]

Fix

• [PgpDump] catch UnicodeDecodeError error. [Terrtia]
• [backend crawler] rename downloaded archive. [Terrtia]
• [paste_submit UI] filter empty file field. [Terrtia]
• [PgpDump] process large pgp blocks. [Terrtia]
• [paste_submit UI] filter empty file field. [Terrtia]
• [UI crawler endpoints] display crawler status + fix #353. [Terrtia]
• [update v1.7] add bs4 requirement. [Terrtia]

Example

Finding relationships between two hidden services via the PGP key published on the website.

AIL Framework version 1.7 released including new correlation features between PGP UIDs and cryptocurrency addresses

Changes

• [correlation] clean files. [Terrtia]
• [update v1.7] update thirdparty. [Terrtia]
• [correlation] add cryptocurrency + refractor correlation. [Terrtia]
• [Bitcoin] map cryptocurrency: bitcoin (DB pivot) [Terrtia]
• [update v1.7] add update scripts. [Terrtia]
• [pgpdump] reprocess tagged items + fix pgpdump. [Terrtia]
• [Update] force update order. [Terrtia]
• [PgpDump] fix graph + add new tags: pgp-signature pgp-public-key-block
  • avoid keys injection in pgp user_id. [Terrtia]
• [decoded UI] add PgpDump UI + fix hashdecoded js. [Terrtia]
• [decoded items] bootstrap 4 migration. [Terrtia]
• [PgpDump] add PgpDump backend TODO: UI. [Terrtia]
• [crawler] manual/auto crawler: always save screenshots. [Terrtia]
• [crawler] manual/auto crawler: always save screenshots. [Terrtia]

Fix

• [correlation] fix endpoint. [Terrtia]

Other

• Update README.md. [Thirion Aurélien]

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]

• Merge pull request #349 from kovacsbalu/fix-paste-encoding. [Thirion
  Aurélien]

  Fix #314

• Use default encoding error from redis. [kovacsbalu]

• Fix #314 Replace char on redis encoding error. Try to use local file
  on other error. [kovacsbalu]

• Merge pull request #350 from kovacsbalu/fix-crawler-rotation. [Thirion
  Aurélien]

  fix: [crawler] rotation

• Hopp, single quote :) [kovacsbalu]

• Fix crawler rotation. [kovacsbalu]

  Before this, crawler processed prioritized onions and after all starts prioritized regular.

AIL Framework version 1.6 released with many bugs fixed and testing improvement

Changes

• [travis] Travis has his own venv where it installs "stuff". Now we
  detect and us it in the launcher. [Steve Clement]
• [travis] Require Python 3.6 to make build faster. [Steve Clement]
• [doc] Some stats on build status/gitter etc. [Steve Clement]
• [hashDecoded] cleanup for the VT message + PEP. [Alexandre Dulaunoy]

Fix

• [faup] fix new return types (bytes to str) [Terrtia]
• [Crawler] force domains/subdomains lower case (rfc4343) [Terrtia]
• [showpaste] fix: #346, avoid None screenshots. [Terrtia]
• [python requirements] rename file. [Terrtia]
• [crawler] typo: domains down. [Terrtia]
• [travis] LAUNCH.sh needs -l to launch... [Steve Clement]
• [travis] Next round of travis fixes. LAUNCH.sh is the only launch
  script needed. chg: [installer] Be way more quiet, watching unzips is
  only fun during development. chg: [installer] Make the arch one +x.
  [Steve Clement]
• [installer] Avoid doing funky sudo pip install moves, it breaks
  python on package managed python installs, if ever, use the --user
  flag. [Steve Clement]
• [travis] Try and require xenial (16.04) and see if it works better.
  [Steve Clement]
• [travis ] There are issues on the 14.04 build system of Travis. This
  fixes it temporarily. [Steve Clement]
• [Onion] typo. [Terrtia]

Other

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
• Chg; [doc crawler] use the torproject torrc. [Terrtia]

AIL Framework version 1.5 released including major improvements in crawler, server management, bootstrap 4 support and many more

AIL Framework version 1.5 released including major improvements in crawler, server management, bootstrap 4 support and many more.

• [UI] Server management. Check for new updates/versions and show background update progress
• [update] Background update process introduced
• [UI] Bootstrap 4 migration started (crawler and tags view)
• Crawler includes new functionalities
  • Port can now be configured
  • Configurable crawling including one-time crawler, regular crawling and type of crawling (e.g. including HAR, screenshots, blacklist management)
• All Items (pastes) are now tagged by date-range
• Many bugs were fixed
• And significant performance improvement in the back-end

AIL Framework version 1.4 released including Tor hidden services crawler and monitoring

AIL Framework version 1.4 released including Tor hidden services crawler and monitoring.

Major new feature:

• Tor hidden service crawler. AIL now includes the ability to crawl and parse output crawled from Tor hidden services.
• Tor onion availability is monitored to detect up and down of hidden services.
• Screenshots are captured and integrated in the analysed output.
• Blurred interface functionality has been added to avoid "burning the eyes" of the security analyst with specific content.
• As the collected information is part of the standard framework, all the AIL modules are available to the crawled hidden services.

New features:

• New export modules for statistics including credentials, phones, banking and TLDs.

Many bugs were fixed.

AIL Framework version 1.3 released including automatic decoding of files from unstructured data

AIL Framework version 1.3 released including automatic decoding of files from unstructured data

New features:

• Detection of IBAN bank accounts are now included
• A cleaner module for decoding files (Base64, hex encoded) from unstructured data
• A new UI for browsing decoded files, their types and relationship

Many bugs fixed and small improvements.

AIL Framework version 1.2 released including TheHive and MISP integration

AIL Framework version 1.2 has been released including TheHive and MISP integration.

• AIL Framework is now capable of auto-publishing (based on tags) event in MISP or alerts in TheHive. Events or cases can be created on request when browsing pastes.
• A new submit interface to ease the submission and processing of data in AIL Framework.
• Introduction of a false-positive or true-positive on tag to allow the classification of information. This will be used later to improve automatic classification of information processed.
• GDPR document released about the use of AIL in the scope of processing personal information.

AIL Framework v1.1 released

A new release of the AIL Framework includes new functionalities such as tagging and classification relying on MISP taxonomies and galaxy. For more information, have a look at the AIL wiki.

• Tagging can now be used in the UI and also the modules are automatically tagging based on detection/analysis.
• Tags are now displayed while browsing important pastes and in search results.
• A bug in duplicate modules detection has been fixed to avoid empty hash creation.
• Duplicate hashes are now persistent and stored on disk.

AIL Framework version 1.0 released

AIL Framework version 1.0 released including a migration from Python 2 to Python 3.

• Redis level-db has been moved to ARDB
• UI improvement including the ability to get paste in raw format
• New Base64 detection module to find and decode Base64 data (and save extracted files including metadata)
• New API key detection module such as Google, AWS or alike
• New bitcoin address detection and validation module
• Improved X.509 certificate detection (Keys module)
• Many bug fixes