Improved IO Bytes Size Hint

[Xavientois]

After trying to implement better size_hint() return values for File in this PR and changing to implementing it for BufReader in this PR, I have arrived at this implementation that provides tighter bounds for the Bytes iterator of various readers including BufReader, Empty, and Chain.

Unfortunately, for BufReader, the size_hint only improves after calling fill_buffer due to it using the contents of the buffer for the hint. Nevertheless, the the tighter bounds should result in better pre-allocation of space to handle the contents of the Bytes iterator.

Closes #81052

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @cramertj (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[bors]

☔ The latest upstream changes (presumably #81578) made this pull request unmergeable. Please resolve the merge conflicts.

[rust-log-analyzer]

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

configure: rust.channel         := nightly
configure: rust.debug-assertions := True
configure: llvm.assertions      := True
configure: dist.missing-tools   := True
configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
configure: writing `config.toml` in current directory
configure: 
configure: run `python /checkout/x.py --help`
configure: 
---
skip untracked path cpu-usage.csv during rustfmt invocations
skip untracked path src/doc/book/ during rustfmt invocations
skip untracked path src/doc/rust-by-example/ during rustfmt invocations
skip untracked path src/llvm-project/ during rustfmt invocations
Diff in /checkout/library/std/src/io/util.rs at line 4:
Running `"/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustfmt" "--config-path" "/checkout" "--edition" "2018" "--unstable-features" "--skip-children" "--check" "/checkout/library/std/src/io/util.rs"` failed.
If you're running `tidy`, try again with `--bless`. Or, if you just want to format code, run `./x.py fmt` instead.
 mod tests;
 use crate::fmt;
 use crate::fmt;
-use crate::io::{self, BufRead, Initializer, IoSlice, IoSliceMut, Read, Seek, SeekFrom, SizeHint, Write};
+use crate::io::{
+    self, BufRead, Initializer, IoSlice, IoSliceMut, Read, Seek, SeekFrom, SizeHint, Write,
 
 
 /// A reader which is always at EOF.
failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --stage 2 src/tools/tidy
Build completed unsuccessfully in 0:00:16

[Dylan-DPC-zz]

@cramertj this is ready for review

[chadbrewbaker]

Sorry if I am blind, but I didn't see an implementation. A benchmark would be nice. BufReader is a pain point for me right now in several ways.

Chiefly, read_until doesn't support stream processing. You should be able to pass in a function that does a left fold over fixed buffer reads until it hits the end of line or whatnot. I'm agnostic if the buffer stays as Vector that doesn't grow or is a u8 array.

fn read_until_streaming(&mut self, byte: u8, buf: &mut Vec<u8>, left_fold_func, init_state ) ->
 (Result<usize>, final_state) {
    }

A "hello world" might preform an XOR over the bytes of each line.

struct XorState {
    sum: u8 =0;
}

fun xor_fold( &mut state: XorState, buf: &mut Vec<u8>, byte: u8){
   // while (buf.peek() != byte)
   // state.sum = xor(state.sum, buf.get() );
}

[Xavientois]

@chadbrewbaker I'm not sure I understand your question. If you're wondering where the updated size hint is implemented for BufReader it is on line 441 of library/std/src/io/buffered/bufreader.rs.

Are you asking benchmarks to show the performance boost for this improved size hint? I am not sure what I would be checking since the goal of this PR is to provide tighter bounds from the size hint when possible, which is shown by the unit tests I included.

[chadbrewbaker]

Sorry, missed the unit test. Ok.

[cramertj]

@bors r+

[bors]

📌 Commit 7674ae1 has been approved by cramertj

[bors]

⌛ Testing commit 7674ae1 with merge b347d73d836ef2d36e2e8160b567c5b3edeee259...

[rust-log-analyzer]

The job mingw-check failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

Removing intermediate container 93ce468f24a5
 ---> 7ae884a133af
Step 5/10 : RUN npm install es-check -g
 ---> Running in 440ac6ccb27e
/node-v14.4.0-linux-x64/bin/es-check -> /node-v14.4.0-linux-x64/lib/node_modules/es-check/index.js

> [email protected] postinstall /node-v14.4.0-linux-x64/lib/node_modules/es-check/node_modules/spawn-sync
> node postinstall
+ [email protected]
added 95 packages from 44 contributors in 3.766s
Removing intermediate container 440ac6ccb27e
 ---> b4d6f0a1ef82
---
Cloning into 'rust-toolstate'...
<Nothing changed>
+ es-check es5 ../src/librustdoc/html/static/main.js ../src/librustdoc/html/static/settings.js ../src/librustdoc/html/static/source-script.js ../src/librustdoc/html/static/storage.js

Cannot read property 'includes' of undefined

[bors]

💔 Test failed - checks-actions

[JohnTitor]

Unrelated CI failure, @bors retry