Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AsyncIO rewrite #126

Merged
merged 90 commits into from
Sep 3, 2024
Merged

AsyncIO rewrite #126

merged 90 commits into from
Sep 3, 2024

Commits on Aug 22, 2024

  1. Update git ignore to include test files

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    cb21714 View commit details
    Browse the repository at this point in the history

  2. Update the parsing logic to include 0-length entries.

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    813cd42 View commit details
    Browse the repository at this point in the history

  3. Add constants to common imports

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ec1eb54 View commit details
    Browse the repository at this point in the history

  4. Add a "visited" flag to keep better track of which records we've iter… 

    …ated to.
    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    4bb33e2 View commit details
    Browse the repository at this point in the history

  5. Update bodyfile and l2t file creation to properly output the records … 

    …we parse.
    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    a763ca2 View commit details
    Browse the repository at this point in the history

  6. Bump up to 64-bit integer buffers for all time values

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    25b4132 View commit details
    Browse the repository at this point in the history

  7. Rewrite WindowsTime using the Python NTP library for inspiration on h… 

    …anding time values.
    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    1597241 View commit details
    Browse the repository at this point in the history

  8. Amend csv_writer to use our revised WindowsTime class

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    080f399 View commit details
    Browse the repository at this point in the history

  9. Update time checks in mft_parser

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    73ae6e2 View commit details
    Browse the repository at this point in the history

  10. Add WindowsTime checks to attribute_parser and mft_record for early w… 

    …arning of invalid or incorrect time information.
    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d0aa920 View commit details
    Browse the repository at this point in the history

  11. Add logging features to report WindowsTIme class failures, modify log… 

    …ger.verbose to logger.info in the main function.
    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    be0ea47 View commit details
    Browse the repository at this point in the history

  12. Fixed a couple missed logger.verbose to logger.infos

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    402584e View commit details
    Browse the repository at this point in the history

  13. Added a few logging levels (last commit) and updated with this commit… 

    … mft_parser verbose to info log messages.
    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f4463c0 View commit details
    Browse the repository at this point in the history

  14. Standardize common_imports and add it to every file.

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    86fb9cc View commit details
    Browse the repository at this point in the history

  15. Update common_imports with an __all__ feature.

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    05d415d View commit details
    Browse the repository at this point in the history

  16. Remove common imports and restore file based importing

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    70ed15a View commit details
    Browse the repository at this point in the history

  17. Fix threading

    @rowingdude
    rowingdude committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    50929b4 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Add record_number and filename @properties

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    fee3082 View commit details
    Browse the repository at this point in the history

  2. Cleaned up the threading for parse_mft_records with ProcessPoolExec, … 

    …it's now a single while loop
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    8964682 View commit details
    Browse the repository at this point in the history

  3. Update analyzemft.py with safety checks.

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    25523ac View commit details
    Browse the repository at this point in the history

  4. Update options_parser menus

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    690ce92 View commit details
    Browse the repository at this point in the history

  5. Add self functions to file handler to have more granular control over… 

    … program operation.
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    d05eb16 View commit details
    Browse the repository at this point in the history

  6. Update Options_Parser menu functions to have more granular control ov… 

    …er how we interact with files through the menu.
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    1688077 View commit details
    Browse the repository at this point in the history

  7. Fix modified logger strings away from calling options

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    f228e3b View commit details
    Browse the repository at this point in the history

  8. Write out the object_id checks, update attribute parser

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    13c1a1d View commit details
    Browse the repository at this point in the history

  9. Add a logging function for parsing records.

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    b7f95ac View commit details
    Browse the repository at this point in the history

  10. Clean up WindowsTime, preparing for attribute_parser further modulari… 

    …zaton
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    0bf3698 View commit details
    Browse the repository at this point in the history

  11. Add a method for directly invoking datetime, add stylistic date/time … 

    …output.
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    ac9cad1 View commit details
    Browse the repository at this point in the history

  12. Forgot to add Windows_time typing import

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    a558659 View commit details
    Browse the repository at this point in the history

  13. Replace valueError with self.logger

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    1f14831 View commit details
    Browse the repository at this point in the history

  14. On a test file, we got that timestamp isn't defined, so we're adding … 

    …a bit of control code to limit it.
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    efa2865 View commit details
    Browse the repository at this point in the history

  15. Fix up code styling

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    027cf5e View commit details
    Browse the repository at this point in the history

  16. Add safety checks to mft_parser

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    7b44423 View commit details
    Browse the repository at this point in the history

  17. Add a dataclass to mftParser, update functions to async

    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    3710951 View commit details
    Browse the repository at this point in the history

  18. Rewrite logger with a dataclass method and add typehints to each func… 

    …tion.
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    0e6b6a2 View commit details
    Browse the repository at this point in the history

  19. Added functionality to filehandler - a dataclass, estimated_records, … 

    …and tyhpehints.
    @rowingdude
    rowingdude committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    a8e5163 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Updates to type

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    5c49a70 View commit details
    Browse the repository at this point in the history

  2. Final directory layout

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    35fe353 View commit details
    Browse the repository at this point in the history

  3. Add init.py to each new directory

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    94e9428 View commit details
    Browse the repository at this point in the history

  4. Fix import pointers to new folder locations

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    8dddb7b View commit details
    Browse the repository at this point in the history

  5. Fix analyze_mft/init.py to new locations

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    821d391 View commit details
    Browse the repository at this point in the history

  6. Create error_handler.py and add appropately sectioned code.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    569def2 View commit details
    Browse the repository at this point in the history

  7. Moved error_handler @wraps out of main AnalyzeMFT script, deleted fro… 

    …m main script.
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    92437ad View commit details
    Browse the repository at this point in the history

  8. Convert CSV generation process to ASYNCio

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    0a98370 View commit details
    Browse the repository at this point in the history

  9. Use aioFiles for async file operations

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    911cc91 View commit details
    Browse the repository at this point in the history

  10. Add more output for troubleshooting any process breakage.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3af7b40 View commit details
    Browse the repository at this point in the history

  11. Add CSV creation debugging

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    372c8af View commit details
    Browse the repository at this point in the history

  12. Adding more file handling output

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    4a8e2e6 View commit details
    Browse the repository at this point in the history

  13. Add csvtimeline and body file

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3723d48 View commit details
    Browse the repository at this point in the history

  14. Add body and timeline outputs

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    f70d2d8 View commit details
    Browse the repository at this point in the history

  15. Update constants.py with static data.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3448bec View commit details
    Browse the repository at this point in the history

  16. Move revise attribute parsing logic after constants.py update

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    dc21300 View commit details
    Browse the repository at this point in the history

  17. Revise index_allocation, ea functions

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    c5b69cd View commit details
    Browse the repository at this point in the history

  18. Simplify MFTParser with constants.py and attribute_parser corrected r… 

    …eferences.
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    1b87ac8 View commit details
    Browse the repository at this point in the history

  19. Simplify Analyze.MFT by moving functionality to module files

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    824e129 View commit details
    Browse the repository at this point in the history

  20. Update FileHandler to correctly reference aioFiles

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    4b00fbe View commit details
    Browse the repository at this point in the history

  21. Add log file option to output

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    85ef334 View commit details
    Browse the repository at this point in the history

  22. Update parsing time handling and revise resident data check

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    e34d128 View commit details
    Browse the repository at this point in the history

  23. Trial a tqdm status bar

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    7a00fb4 View commit details
    Browse the repository at this point in the history

  24. Migrate bodyfile writer, l2twriter to their own class module files, u… 

    …pdate references to them.
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    84d789b View commit details
    Browse the repository at this point in the history

  25. Update file references

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    1554d4c View commit details
    Browse the repository at this point in the history

  26. update initialize_components in analyzeMFT.py

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    488c089 View commit details
    Browse the repository at this point in the history

  27. Update nain function, remove some of the extra debug clutter as it'll… 

    … be in the error handler wrapper.
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    bfa801e View commit details
    Browse the repository at this point in the history

  28. Place if name is main in a try block

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    0d9df53 View commit details
    Browse the repository at this point in the history

  29. Update logger with debug and verbose output sub functions.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    a177b6c View commit details
    Browse the repository at this point in the history

  30. Make new sub modules for options with options validation checks.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    860b881 View commit details
    Browse the repository at this point in the history

  31. Simplify WindowsTime with condensed checks and more descriptive errors.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    167f305 View commit details
    Browse the repository at this point in the history

  32. Add async for filepaths and folderpaths. Use asyncio.sleep for syncin… 

    …g threads.
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    3c7c019 View commit details
    Browse the repository at this point in the history

  33. Update all methods to async, return specific types to help with debug… 

    …ging errors and validation, return None when there is insufficient data.
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    9197c48 View commit details
    Browse the repository at this point in the history

  34. Update csv_writer with typehints and full async methods.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    1ef3916 View commit details
    Browse the repository at this point in the history

  35. json_writer - add async to each function

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    7a9aea3 View commit details
    Browse the repository at this point in the history

  36. body_writer - add async to each function, add string format validater.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    7281bd7 View commit details
    Browse the repository at this point in the history

  37. csv_timeline - finish prepare_csv_timeline_record and convert all fun… 

    …ctions to async
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    59f2044 View commit details
    Browse the repository at this point in the history

  38. Update json output reference in mft_parser.py

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    d865893 View commit details
    Browse the repository at this point in the history

  39. update mft_record to point at revised attribute_parser class sub-func… 

    …tions. Added two @properties - record_number && filename
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    51c8c06 View commit details
    Browse the repository at this point in the history

  40. Update attribute_parser imports to typecheck a list.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    8688d66 View commit details
    Browse the repository at this point in the history

  41. Add body file, csv_timeline to main script (they were left out due to… 

    … being in development). Add keyboard interrupt handler for graceful ctrl-c exits
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    8f20c43 View commit details
    Browse the repository at this point in the history

  42. Remove thread_manager and all references to it.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    54d9270 View commit details
    Browse the repository at this point in the history

  43. Update references to the logger class, and imports.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    a31a02e View commit details
    Browse the repository at this point in the history

  44. Change mft_record to explicit imports from constants.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    6fcaf4d View commit details
    Browse the repository at this point in the history

  45. Update usage.md with statusbar info

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    7590363 View commit details
    Browse the repository at this point in the history

  46. Update logging references for error, info, verbose, and warning

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    92d9a53 View commit details
    Browse the repository at this point in the history

  47. Update dict to a listed dictionary to use append and list functions.

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    2fc60cd View commit details
    Browse the repository at this point in the history

  48. Update parsing logic to try to catch malformed data

    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    4a36657 View commit details
    Browse the repository at this point in the history

  49. Start work on an attributes folder where I will define each attribute… 

    … as a class and provide individual parsing logic.
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    8da4784 View commit details
    Browse the repository at this point in the history

  50. modify methods to use from_bytes to have more control over data valid… 

    …ation and flow
    @rowingdude
    rowingdude committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    e46adc7 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. aSyncIO rewrite - stripped to the basics mft in, csv out.

    @rowingdude
    rowingdude authored Sep 3, 2024
    Configuration menu
    Copy the full SHA
    d16106c View commit details
    Browse the repository at this point in the history

  2. Create pyproject.toml and populate the constants.py file.

    @rowingdude
    rowingdude authored Sep 3, 2024
    Configuration menu
    Copy the full SHA
    037abd6 View commit details
    Browse the repository at this point in the history

  3. Rewrite main program to point into src/analyzeMFT

    @rowingdude
    rowingdude authored Sep 3, 2024
    Configuration menu
    Copy the full SHA
    117a1c4 View commit details
    Browse the repository at this point in the history

  4. Update Changes.md

    @rowingdude
    rowingdude authored Sep 3, 2024
    Configuration menu
    Copy the full SHA
    314cd7b View commit details
    Browse the repository at this point in the history