Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Refactor pstd/env with std::filesystem #1470

Merged
merged 1 commit into from
Jun 6, 2023

Conversation

4kangjc
Copy link
Contributor

@4kangjc 4kangjc commented May 9, 2023

No description provided.

@4kangjc 4kangjc force-pushed the filesystem branch 3 times, most recently from 6df4992 to 8cf9c16 Compare May 10, 2023 09:08
@AlexStocks
Copy link
Collaborator

@4kangjc pls fix the file conflict and continue to finish this pr.

@@ -209,8 +208,7 @@ Status StableLog::PurgeFileAfter(uint32_t filenum) {
for (auto& it : binlogs) {
if (it.first > filenum) {
// Do delete
Status s = pstd::DeleteFile(log_path_ + it.second);
if (!s.ok()) {
if (pstd::DeleteFile(log_path_ + it.second)) {
return s;
}
LOG(WARNING) << "Delete file " << log_path_ + it.second;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch seems to be trying to delete some log files, and there are no obvious bugs in the code. However, there are some suggestions for improvement:

  1. Error Handling: It might be useful to add an error handling mechanism to handle cases where pstd::DeleteFile return an error status.

  2. Logging: The logging statement inside the loop might cause a lot of unnecessary repetition if there are a large number of files being deleted. Consider moving the message outside the loop or adding a counter to reduce the frequency of logs.

  3. Magic Number: The values 111 and 209 used in the code patch are magic numbers, and it's not clear from the code snippet what they represent. Consider using named constants instead.

  4. Naming: The functions and variables used in the patch are not named consistently in terms of capitalization. Consider following a consistent naming convention.

  5. Code Clarity: The code patch could be made more readable by adding comments explaining what each section does.

Overall, the code patch seems safe, but I recommend reviewing it in the context of the overall codebase to ensure that it integrates properly with other parts of the code.

c++fs
)
endif()

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch seems to be adding support for linking the C++ filesystem library for GNU and Clang compilers based on their IDs. Here are some suggestions:

  1. It is a good practice to add comments in code that explain why certain changes are made. This can help other developers understand the reasoning behind the changes easily.

  2. The code patch only checks if the compiler is GNU or Clang, but it does not verify if the compiler supports the C++ filesystem library. You may want to consider adding an additional check for this using the check_cxx_compiler_flag function and setting a flag variable accordingly.

  3. When linking libraries, it is recommended to use target-based linkage instead of specifying libraries directly. This way, dependencies between targets can be automatically resolved by CMake. You can create a target with the add_library command and specify its dependencies using target_link_libraries. Then, link the target to your main target using target_link_libraries.

  4. It is better to use a switch statement instead of multiple if-elseif blocks when checking for different compiler IDs. This can make the code more concise and easier to read.

  5. Finally, you may want to consider using consistent casing for variable names and formatting the code consistently throughout the project.

int GetChildren(const std::string& dir, std::vector<std::string>& result);
bool GetDescendant(const std::string& dir, std::vector<std::string>& result);
void GetDescendant(const std::string& dir, std::vector<std::string>& result);

uint64_t NowMicros();
void SleepForMicroseconds(int micros);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the provided diff, here are a few observations:

  • The return type of DeleteFile is changed from Status to bool. This may introduce some semantic changes and can affect error handling depending on how the API is used.
  • LockFile and UnlockFile functions are removed from the header file, so they are no longer part of the public API. This may be intentional or accidental, but it's worth checking if any downstream code depends on these functions.
  • The return type of GetDescendant is changed from bool to void, implying that the function no longer returns success/failure status. It's unclear if this was intentional.
  • Overall, it's hard to say anything more definitive without knowing what the implementation of these functions looks like, their purpose, and how they're used in the code.

}
LOG(WARNING) << "Delete file " << log_path_ + it.second;
LOG(WARNING) << "Delete file " << filename;
}
}
return Status::OK();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch looks fine.
There is an improvement suggestion that can be made, where instead of concatenating strings in the DeleteFile function call, it will be better to pass it as a separate parameter.

So instead of:

pstd::DeleteFile(log_path_ + it->second)

It should be:

pstd::DeleteFile(log_path_, it->second)

This can help make the code more understandable and avoid any string concatenation issues.

endif()
endif()

target_link_libraries(pstd PUBLIC ${PSTD_DEP_LIBS})
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code patch modifies the CMake project file for a target named "pstd". The patch adds a condition to link the standard C++ file system library "stdc++fs" on GNU compilers versions less than 9, and "c++fs" on Clang compilers versions less than 9. This is done by appending the required library to a list of dependent libraries called PSTD_DEP_LIBS. The patch also replaces the direct linking of dependent libraries with the target_link_libraries command, making it more concise.

A possible improvement to this patch is to use the CMAKE_CXX_STANDARD_LIBRARY_LIB variable instead of manually appending the required library for each compiler and version. The CMAKE_CXX_STANDARD_LIBRARY_LIB variable contains the name of the C++ standard library implementation (e.g., "stdc++", "c++") and can be used to link the corresponding file system library.

Another potential improvement is to add a comment explaining the reason for linking the file system library only on specific compiler versions. Without proper documentation, future maintainers might not understand why the library is linked conditionally, leading to confusion and potential bugs.

int GetChildren(const std::string& dir, std::vector<std::string>& result);
bool GetDescendant(const std::string& dir, std::vector<std::string>& result);
void GetDescendant(const std::string& dir, std::vector<std::string>& result);

uint64_t NowMicros();
void SleepForMicroseconds(int micros);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch seems to change the return type of the DeleteFile function from Status to bool, and remove the declarations for LockFile and UnlockFile. It also changes the return type of GetDescendant from bool to void.

Without seeing the implementation of these functions or the context in which they are used, it is hard to determine whether these changes introduce any bugs or potential improvements. However, changing the return types could break compatibility with existing code that uses these functions, so it is important to make sure any other code using these functions is updated accordingly.

@4kangjc 4kangjc changed the title [WIP]refactor: Refactor pstd/env with std::filesystem refactor: Refactor pstd/env with std::filesystem Jun 4, 2023
}
LOG(WARNING) << "Delete file " << log_path_ + it.second;
LOG(WARNING) << "Delete file " << filename;
}
}
return Status::OK();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch seems to be a review of a piece of code that deals with purging files. Here are a few points about the code:

  • There are no obvious bug risks in the code patch.
  • The improvement that could be suggested in this code patch is to log the actual reason for failure when deleting a file fails instead of using a generic error message. This can give more information on what went wrong during the delete operation and help debug the issue.
  • In the function PurgeFileAfter, it would be better to concatenate the filename outside of the pstd::DeleteFile call and store it in a variable to avoid code duplication, as done in the PurgeFiles function.
  • It may be good to consider adding more checks before deleting files, such as verifying file ownership (if applicable) and checking whether the file being deleted is actually the one intended to be deleted.

endif()
endif()

target_link_libraries(pstd PUBLIC ${PSTD_DEP_LIBS})
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch seems to be modifying the CMake build script for the pstd target, specifically the linking of dependent libraries.

The changes appear to add conditional linker flags/arguments to link against the stdc++fs or c++fs library depending on the version of the GCC or Clang compiler.

There are no obvious bug risks in this code patch, but it is always good to test the changes and ensure they work as intended. As for improvement suggestions, it is not clear from this code patch if there are any other areas that could be improved without more context and knowledge about the project.

int GetChildren(const std::string& dir, std::vector<std::string>& result);
bool GetDescendant(const std::string& dir, std::vector<std::string>& result);
void GetDescendant(const std::string& dir, std::vector<std::string>& result);

uint64_t NowMicros();
void SleepForMicroseconds(int micros);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch looks like a simple header file with function signatures and declarations, so it's hard to provide any concrete feedback without seeing the implementation details. However, some general suggestions include:

  • Use consistent naming conventions for functions, such as using either camelCase or snake_case, and stick to them throughout the codebase.
  • Use descriptive function names that make clear what the function does, especially for public-facing functions.
  • Consider using std::filesystem instead of rolling your own file manipulation functions, as it provides a more robust and standardized API for dealing with files and directories.
  • The return type of the DeleteFile() function has been changed from Status to bool. Make sure this is intentional and doesn't introduce bugs or unexpected behavior elsewhere in the code.
  • Consider adding documentation comments (///) to the function signatures, describing their purpose and expected behavior.

}
LOG(WARNING) << "Delete file " << log_path_ + it.second;
LOG(WARNING) << "Delete file " << filename;
}
}
return Status::OK();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, the code patch looks fine. However, there are a few improvements that could be made:

  1. In both functions, it would be better to use a range-based for loop instead of an iterator.

  2. In the PurgeFiles function, the if (s.ok()) check is unnecessary. The return value of pstd::DeleteFile is already a boolean indicating success or failure.

  3. In both functions, it would be better to store the full path of the file to be deleted in a separate variable before calling pstd::DeleteFile. This makes it easier to log in case of errors.

  4. In the error message of the PurgeFiles function, it would be better to provide more information about why the delete failed instead of just saying "delete file failed".

  5. In the PurgeFileAfter function, the misspelling of "failed" in the error message should be corrected.

  6. In both functions, it might be a good idea to add a check that the file exists before attempting to delete it, to avoid deleting non-existent files.

  7. Finally, it would be nice if there were comments explaining what the functions do and how they work.

endif()
endif()

target_link_libraries(pstd PUBLIC ${PSTD_DEP_LIBS})
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code patch seems to update the target_link_libraries command for the pstd target in a CMake project.

The changes look reasonable, as it defines an intermediate PSTD_DEP_LIBS variable to hold the dependent libraries and then links them with the pstd target using target_link_libraries.

The additional code checks for the compiler type and version. If the compiler is GCC or Clang and its version is less than 9, it appends the stdc++fs or c++fs library (depending on the compiler) to the PSTD_DEP_LIBS variable. This is because these libraries were introduced in GCC and Clang 9 respectively.

One potential improvement could be to use target_compile_features instead of checking the compiler versions manually. For example, you could use target_compile_features(pstd PUBLIC cxx_std_17) to enforce that the code is compiled with C++17 or later features, which should automatically include the filesystem library for GCC 9+ and Clang 9+.

int GetChildren(const std::string& dir, std::vector<std::string>& result);
bool GetDescendant(const std::string& dir, std::vector<std::string>& result);
void GetDescendant(const std::string& dir, std::vector<std::string>& result);

uint64_t NowMicros();
void SleepForMicroseconds(int micros);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch seems to be a header file defining various file-related functions and classes. Here are some observations:

  1. It would be better to return Status instead of bool in the DeleteFile function, as it allows conveying specific error messages to the caller.

  2. Similarly, LockFile and UnlockFile should also return Status instead of bool.

  3. The visibility of the GetDescendant function is unclear - it is not clear whether it is supposed to be used outside the header file or just within its implementation. If it is meant to be used outside, then it should have a status return value to indicate any errors.

  4. It would be good to add documentation (comments or doxygen-style) to provide context and usage examples for all these functions.

  5. It's hard to give specific improvement suggestions without knowing the intended use of these functions. For example, it might make sense to add overloads for DeleteFile and LockFile that take std::filesystem::path objects instead of a string.

@wanghenshui wanghenshui merged commit c128813 into OpenAtomFoundation:unstable Jun 6, 2023
lqxhub pushed a commit to lqxhub/pika that referenced this pull request Jun 18, 2023
AlexStocks added a commit that referenced this pull request Jul 2, 2023
* feat: supported 'ptype' command

* Simplified command `type` and `ptype` code

* Remove the capacity judgment

* fix slavaof serialize response bug (#1583)

Co-authored-by: liuyuecai <[email protected]>

* fix (#1587) (#1588)

* fix unit test:type/set (#1577)

* Fix memory leaks in HandleBGWorkerWriteDB and HandleMetaSyncResponse (#1590)

* fix memory leaks in PikaReplBgWorker::HandleBGWorkerWriteDB(void* arg) and PikaReplClientConn::HandleMetaSyncResponse(void* arg).

* add address/thread sanitizer to CMakeLists

---------

Co-authored-by: cjh <[email protected]>

* refactor: replace pstd/env with std::filesystem (#1470)

* fix bug issue 1554: using unique_ptr.reset to fix the SIGABRT error (#1595)

using unique_ptr.reset to fix the SIGABRT error && using time-wait lock to notify quickly

* fix: incorrect manner of terminating the process (#1596)

* fix issue#1597: add rocksdb dependency to pstd (#1599)

* fix g++15 compile failure

* add rocksdb dependency to pstd

---------

Co-authored-by: J1senn <[email protected]>

* fix_info_command

* fix: gcc13 compile failed (#1601)

* ci: add unit test to github action (#1604)

* ci: add unit test in github workflow

* chore:change `PLATFORM` field logic (#1615)

* fix issue 1517: scan 命令不支持 type 参数 (#1582)

* fix: the unit test of type/set (#1617)

* test: optimize the return results of srandmember to avoid approximate results

* fix: use last_seed for random engine

* [fix issue1621] fix deadlock (#1620)

* [fix] fix deadlock

* [fix] fix

* command `type` and `ptype` add unit test

* fix member variable initialization

* Update issue templates

* feat: supported 'ptype' command

* fix unit test:type/set (#1577)

* fix issue#1597: add rocksdb dependency to pstd (#1599)

* fix g++15 compile failure

* add rocksdb dependency to pstd

---------

Co-authored-by: J1senn <[email protected]>

* fix: gcc13 compile failed (#1601)

* fix: the unit test of type/set (#1617)

* test: optimize the return results of srandmember to avoid approximate results

* fix: use last_seed for random engine

* Modify other modules to use the new `type` function

* fix function repeat

---------

Co-authored-by: Yuecai Liu <[email protected]>
Co-authored-by: liuyuecai <[email protected]>
Co-authored-by: Peter Chan <[email protected]>
Co-authored-by: chenbt <[email protected]>
Co-authored-by: Junhua Chen <[email protected]>
Co-authored-by: cjh <[email protected]>
Co-authored-by: kang jinci <[email protected]>
Co-authored-by: Xin.Zh <[email protected]>
Co-authored-by: machinly <[email protected]>
Co-authored-by: A2ureStone <[email protected]>
Co-authored-by: J1senn <[email protected]>
Co-authored-by: chejinge <[email protected]>
Co-authored-by: baerwang <[email protected]>
Co-authored-by: ptbxzrt <[email protected]>
Co-authored-by: 你不要过来啊 <[email protected]>
}
result.emplace_back(entry->d_name);
for (auto& de : filesystem::directory_iterator(dir)) {
result.emplace_back(de.path());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
result.emplace_back(de.path());
result.emplace_back(de.path().filename());

@yaoyinnan yaoyinnan changed the title refactor: Refactor pstd/env with std::filesystem feat: Refactor pstd/env with std::filesystem Jul 25, 2023
bigdaronlee163 pushed a commit to bigdaronlee163/pika that referenced this pull request Jun 8, 2024
bigdaronlee163 pushed a commit to bigdaronlee163/pika that referenced this pull request Jun 8, 2024
* feat: supported 'ptype' command

* Simplified command `type` and `ptype` code

* Remove the capacity judgment

* fix slavaof serialize response bug (OpenAtomFoundation#1583)

Co-authored-by: liuyuecai <[email protected]>

* fix (OpenAtomFoundation#1587) (OpenAtomFoundation#1588)

* fix unit test:type/set (OpenAtomFoundation#1577)

* Fix memory leaks in HandleBGWorkerWriteDB and HandleMetaSyncResponse (OpenAtomFoundation#1590)

* fix memory leaks in PikaReplBgWorker::HandleBGWorkerWriteDB(void* arg) and PikaReplClientConn::HandleMetaSyncResponse(void* arg).

* add address/thread sanitizer to CMakeLists

---------

Co-authored-by: cjh <[email protected]>

* refactor: replace pstd/env with std::filesystem (OpenAtomFoundation#1470)

* fix bug issue 1554: using unique_ptr.reset to fix the SIGABRT error (OpenAtomFoundation#1595)

using unique_ptr.reset to fix the SIGABRT error && using time-wait lock to notify quickly

* fix: incorrect manner of terminating the process (OpenAtomFoundation#1596)

* fix issue#1597: add rocksdb dependency to pstd (OpenAtomFoundation#1599)

* fix g++15 compile failure

* add rocksdb dependency to pstd

---------

Co-authored-by: J1senn <[email protected]>

* fix_info_command

* fix: gcc13 compile failed (OpenAtomFoundation#1601)

* ci: add unit test to github action (OpenAtomFoundation#1604)

* ci: add unit test in github workflow

* chore:change `PLATFORM` field logic (OpenAtomFoundation#1615)

* fix issue 1517: scan 命令不支持 type 参数 (OpenAtomFoundation#1582)

* fix: the unit test of type/set (OpenAtomFoundation#1617)

* test: optimize the return results of srandmember to avoid approximate results

* fix: use last_seed for random engine

* [fix issue1621] fix deadlock (OpenAtomFoundation#1620)

* [fix] fix deadlock

* [fix] fix

* command `type` and `ptype` add unit test

* fix member variable initialization

* Update issue templates

* feat: supported 'ptype' command

* fix unit test:type/set (OpenAtomFoundation#1577)

* fix issue#1597: add rocksdb dependency to pstd (OpenAtomFoundation#1599)

* fix g++15 compile failure

* add rocksdb dependency to pstd

---------

Co-authored-by: J1senn <[email protected]>

* fix: gcc13 compile failed (OpenAtomFoundation#1601)

* fix: the unit test of type/set (OpenAtomFoundation#1617)

* test: optimize the return results of srandmember to avoid approximate results

* fix: use last_seed for random engine

* Modify other modules to use the new `type` function

* fix function repeat

---------

Co-authored-by: Yuecai Liu <[email protected]>
Co-authored-by: liuyuecai <[email protected]>
Co-authored-by: Peter Chan <[email protected]>
Co-authored-by: chenbt <[email protected]>
Co-authored-by: Junhua Chen <[email protected]>
Co-authored-by: cjh <[email protected]>
Co-authored-by: kang jinci <[email protected]>
Co-authored-by: Xin.Zh <[email protected]>
Co-authored-by: machinly <[email protected]>
Co-authored-by: A2ureStone <[email protected]>
Co-authored-by: J1senn <[email protected]>
Co-authored-by: chejinge <[email protected]>
Co-authored-by: baerwang <[email protected]>
Co-authored-by: ptbxzrt <[email protected]>
Co-authored-by: 你不要过来啊 <[email protected]>
cheniujh pushed a commit to cheniujh/pika that referenced this pull request Sep 24, 2024
cheniujh added a commit to cheniujh/pika that referenced this pull request Sep 24, 2024
* feat: supported 'ptype' command

* Simplified command `type` and `ptype` code

* Remove the capacity judgment

* fix slavaof serialize response bug (OpenAtomFoundation#1583)

Co-authored-by: liuyuecai <[email protected]>

* fix (OpenAtomFoundation#1587) (OpenAtomFoundation#1588)

* fix unit test:type/set (OpenAtomFoundation#1577)

* Fix memory leaks in HandleBGWorkerWriteDB and HandleMetaSyncResponse (OpenAtomFoundation#1590)

* fix memory leaks in PikaReplBgWorker::HandleBGWorkerWriteDB(void* arg) and PikaReplClientConn::HandleMetaSyncResponse(void* arg).

* add address/thread sanitizer to CMakeLists

---------

Co-authored-by: cjh <[email protected]>

* refactor: replace pstd/env with std::filesystem (OpenAtomFoundation#1470)

* fix bug issue 1554: using unique_ptr.reset to fix the SIGABRT error (OpenAtomFoundation#1595)

using unique_ptr.reset to fix the SIGABRT error && using time-wait lock to notify quickly

* fix: incorrect manner of terminating the process (OpenAtomFoundation#1596)

* fix issue#1597: add rocksdb dependency to pstd (OpenAtomFoundation#1599)

* fix g++15 compile failure

* add rocksdb dependency to pstd

---------

Co-authored-by: J1senn <[email protected]>

* fix_info_command

* fix: gcc13 compile failed (OpenAtomFoundation#1601)

* ci: add unit test to github action (OpenAtomFoundation#1604)

* ci: add unit test in github workflow

* chore:change `PLATFORM` field logic (OpenAtomFoundation#1615)

* fix issue 1517: scan 命令不支持 type 参数 (OpenAtomFoundation#1582)

* fix: the unit test of type/set (OpenAtomFoundation#1617)

* test: optimize the return results of srandmember to avoid approximate results

* fix: use last_seed for random engine

* [fix issue1621] fix deadlock (OpenAtomFoundation#1620)

* [fix] fix deadlock

* [fix] fix

* command `type` and `ptype` add unit test

* fix member variable initialization

* Update issue templates

* feat: supported 'ptype' command

* fix unit test:type/set (OpenAtomFoundation#1577)

* fix issue#1597: add rocksdb dependency to pstd (OpenAtomFoundation#1599)

* fix g++15 compile failure

* add rocksdb dependency to pstd

---------

Co-authored-by: J1senn <[email protected]>

* fix: gcc13 compile failed (OpenAtomFoundation#1601)

* fix: the unit test of type/set (OpenAtomFoundation#1617)

* test: optimize the return results of srandmember to avoid approximate results

* fix: use last_seed for random engine

* Modify other modules to use the new `type` function

* fix function repeat

---------

Co-authored-by: Yuecai Liu <[email protected]>
Co-authored-by: liuyuecai <[email protected]>
Co-authored-by: Peter Chan <[email protected]>
Co-authored-by: chenbt <[email protected]>
Co-authored-by: Junhua Chen <[email protected]>
Co-authored-by: cjh <[email protected]>
Co-authored-by: kang jinci <[email protected]>
Co-authored-by: Xin.Zh <[email protected]>
Co-authored-by: machinly <[email protected]>
Co-authored-by: A2ureStone <[email protected]>
Co-authored-by: J1senn <[email protected]>
Co-authored-by: chejinge <[email protected]>
Co-authored-by: baerwang <[email protected]>
Co-authored-by: ptbxzrt <[email protected]>
Co-authored-by: 你不要过来啊 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants