Replaced linear token counting macros with optimized implementation #59600

tobia · 2019-04-01T09:23:23Z

There are currently two distinct token-counting macros in the source. Both implement the trivial algorithm, with linear complexity. They may or may not be adequate for their use case, but considering that other people are probably going to copy and paste them whenever they need a token-counting macro, I replaced them with an optimized implementation with logarithmic complexity.

rust-highfive · 2019-04-01T09:23:26Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @KodrAus (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

rust-highfive · 2019-04-01T09:30:22Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:02f2ff0d:start=1554110687908288597,finish=1554110688767636698,duration=859348101
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
$ export GCP_CACHE_BUCKET=rust-lang-ci-cache
Setting environment variables from .travis.yml
---
[00:05:14]    Compiling rustc_apfloat v0.0.0 (/checkout/src/librustc_apfloat)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:735:32
[00:05:14]     |
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14]     | |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14]     | |_- in this expansion of `count!`
[00:05:14] 738 | 
[00:05:14] 739 | / macro_rules! tuple {
[00:05:14] 740 | |     () => ();
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | |_- in this expansion of `tuple!`
[00:05:14]     | |_- in this expansion of `tuple!`
[00:05:14] 771 | 
[00:05:14] 772 |   tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     |   ------------------------------------------------------------ in this macro invocation
[00:05:14]     = note: `-D unused-parens` implied by `-D warnings`
[00:05:14] 
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:736:32
[00:05:14]    --> src/libserialize/serialize.rs:736:32
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:735:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14]     | |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:736:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:735:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14]     | |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:736:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:735:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14]     | |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:736:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:735:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14]     | |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:736:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:14] error: unnecessary parentheses around assigned value
[00:05:14]    --> src/libserialize/serialize.rs:735:32
[00:05:14]     |
[00:05:14]     |
[00:05:14] 722 | / macro_rules! peel {
[00:05:14] 723 | |     ($name:ident, $($other:ident,)*) => (tuple! { $($other,)* })
[00:05:14] 724 | | }
[00:05:14] 724 | | }
[00:05:14]     | |_- in this expansion of `peel!` (#2)
[00:05:14] 732 | / macro_rules! count {
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 733 | |     ()                     => (0usize);
[00:05:14] 734 | |     ($one:tt)              => (1usize);
[00:05:14] 735 | |     ($($pairs:tt $_p:tt)*) => ((count!($($pairs)*) << 1usize));
[00:05:14]     | |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these parentheses
[00:05:14] 736 | |     ($odd:tt $($rest:tt)*) => ((count!($($rest)*) | 1usize));
[00:05:14] 737 | | }
[00:05:14]     | |_- in this expansion of `count!` (#4)
[00:05:14] 739 |   macro_rules! tuple {
[00:05:14]     |  _-
[00:05:14]     | |_|
[00:05:14]     | |
[00:05:14]     | |
[00:05:14] 740 | |     () => ();
[00:05:14] 741 | |     ( $($name:ident,)+ ) => (
[00:05:14] 742 | |         impl<$($name:Decodable),*> Decodable for ($($name,)*) {
[00:05:14] ...   |
[00:05:14] 745 | |                 let len: usize = count!($($name)*);
[00:05:14] ...   |
[00:05:14] ...   |
[00:05:14] 768 | |         peel! { $($name,)* }
[00:05:14] 769 | |     )
[00:05:14] 770 | | }
[00:05:14]     | | -
[00:05:14]     | |_|
[00:05:14]     | |_|
[00:05:14]     | |_in this expansion of `tuple!` (#1)
[00:05:14]     |   in this expansion of `tuple!` (#3)
[00:05:14] 771 | 
[00:05:14] 772 | | tuple! { T0, T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, }
[00:05:14]     | | ------------------------------------------------------------ in this macro invocation (#1)
[00:05:16] error: aborting due to 11 previous errors
[00:05:16] 
[00:05:16] error: Could not compile `serialize`.
[00:05:16] warning: build failed, waiting for other jobs to finish...

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Centril · 2019-04-01T09:38:44Z

Should we deduplicate one of these macros?

pnkfelix · 2019-04-04T08:56:55Z

I don't know if the size of the final generated code here matters in practice. The previous code being generated looks quite trivial to optimize; but the logarithmic variant probably also optimizes well.

Out of curiosity, did you attempt to measure the expansion time itself of the old and new versions on large inputs?

Update: I don't want to give the wrong impression: I'm not objecting to this PR. I just was a little surprised to see it.

pnkfelix · 2019-04-12T11:01:34Z

(also, in case you hadn't noticed, travis is complaining about unneeded parentheses in the macro definitions here...)

tobia · 2019-04-12T17:50:26Z

Hello. I've removed the extra parentheses, let's see if the warnings are gone.

To me it seemed obvious that a linear counter was a weak solution, that could cause trouble sooner or later. Be it for the time taken to count things, for the memory used by macro recursion, or for Rust's default recursion limit. Considering that a token counter could conceivably be used to implement initializers like vec![...] (even though the standard ones don't use it) having someone apply such an initializer to a large number of elements, only to get a macro expansion error, is in the realm of possibility.

I agree that it should be deduplicated in a standard place. But I don't know Rust's sources well enough to suggest the correct place. Also, there may be other instances of that linear counting algorithm elsewhere in the source tree, that my grep-fu didn't turn up.

Anyways, I did an accurate measurement, using this source code. I did the various measurements by hand, changing the number of tokens in the macro call, commenting / decommenting the two macros, and running each test several times in a Bash loop, with rustc test_count.rs && ./test_count, in order to get some precision when I computed the average. For example, the smaller tests would return either 0 or 1 ticks, so I ran 100 of them and computed the average. I also did a measurement with no count!() macro call and subtracted that baseline from all other results. Ticks per second can be obtained with the getconf CLK_TCK command. Results:

Tokens	New macro	Old macro
10	0.1ms	0.1ms
100	0.2ms	(*) 3.1ms
1,000	2.9ms	(*) 232.6ms
10,000	8.9ms	ERROR
100,000	90.3ms	ERROR

(*) Rust's default recursion limit does not allow the old linear macro to run on more than 61 tokens, so for the test's sake I artificially increased it with ![recursion_limit="..."]. Still, beyond 1,000 tokens even that was not enough: the compiler crashed with Fatal runtime error: stack overflow.

Also, in case anybody is curious, this is the expansion for 10,000 tokens:

(((((1 << 1 << 1 << 1 | 1) << 1 | 1) << 1 | 1) << 1 << 1 << 1 << 1 | 1) << 1 << 1 << 1 << 1)

Compare that to summing 1 to itself 10,000 times, in a non-tail recursive call, needing 10,000 stack frames.

Centril · 2019-04-28T12:00:27Z

r? @pnkfelix

panaman67 · 2019-05-18T21:46:20Z

Whats the status on this?

Centril · 2019-06-10T01:24:06Z

Ping from triage @pnkfelix

pnkfelix · 2019-06-10T08:53:04Z

@bors r+ rollup

bors · 2019-06-10T08:53:05Z

📌 Commit a4a07e0 has been approved by pnkfelix

tesuji · 2019-06-10T08:55:07Z

This PR may affect parsing performance. Do it need to be rollup?

pnkfelix · 2019-06-10T09:04:14Z

@bors r+ rollup

bors · 2019-06-10T09:04:15Z

💡 This pull request was already approved, no need to approve it again.

There's another pull request that is currently being tested, blocking this pull request: Changed usages of mir in librustc::mir and librustc_mir to body #61506

bors · 2019-06-10T09:04:16Z

📌 Commit a4a07e0 has been approved by pnkfelix

pnkfelix · 2019-06-10T09:07:09Z

I'm having some issues interacting with github at the moment (thus the duplicate approvals). I don't care one way or another whether this is rolled up. But I don't expect there to be much impact on performance from it one way or another (apart from the case of microbenchmarks like the ones given above).

Replaced linear token counting macros with optimized implementation There are currently two distinct token-counting macros in the source. Both implement the trivial algorithm, with linear complexity. They may or may not be adequate for their use case, but considering that other people are probably going to copy and paste them whenever they need a token-counting macro, I replaced them with an optimized implementation with logarithmic complexity.

@ghost

Rollup of 5 pull requests Successful merges: - #59600 (Replaced linear token counting macros with optimized implementation) - #61501 (get rid of real_intrinsics module) - #61570 (Fix issues with const argument inference) - #61683 (Haiku: the maximum stack size is 16 MB) - #61697 (submodules: update clippy from 71be6f6 to c0dbd34) Failed merges: r? @ghost

rust-highfive assigned KodrAus Apr 1, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 1, 2019

pnkfelix added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 12, 2019

Replaced linear token counting macros with optimized implementation

a4a07e0

tobia force-pushed the master branch from 559d330 to a4a07e0 Compare April 12, 2019 16:24

rust-highfive assigned pnkfelix and unassigned KodrAus Apr 28, 2019

Centril added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 10, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 10, 2019

Centril mentioned this pull request Jun 10, 2019

Rollup of 6 pull requests #61713

Closed

Centril mentioned this pull request Jun 10, 2019

Rollup of 5 pull requests #61714

Closed

Centril mentioned this pull request Jun 10, 2019

Rollup of 5 pull requests #61716

Merged

bors merged commit a4a07e0 into rust-lang:master Jun 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replaced linear token counting macros with optimized implementation #59600

Replaced linear token counting macros with optimized implementation #59600

tobia commented Apr 1, 2019

rust-highfive commented Apr 1, 2019

rust-highfive commented Apr 1, 2019

Centril commented Apr 1, 2019

pnkfelix commented Apr 4, 2019 •

edited

Loading

pnkfelix commented Apr 12, 2019

tobia commented Apr 12, 2019 •

edited

Loading

Centril commented Apr 28, 2019

panaman67 commented May 18, 2019

Centril commented Jun 10, 2019

pnkfelix commented Jun 10, 2019

bors commented Jun 10, 2019

tesuji commented Jun 10, 2019

pnkfelix commented Jun 10, 2019

bors commented Jun 10, 2019

bors commented Jun 10, 2019

pnkfelix commented Jun 10, 2019

Replaced linear token counting macros with optimized implementation #59600

Replaced linear token counting macros with optimized implementation #59600

Conversation

tobia commented Apr 1, 2019

rust-highfive commented Apr 1, 2019

rust-highfive commented Apr 1, 2019

Centril commented Apr 1, 2019

pnkfelix commented Apr 4, 2019 • edited Loading

pnkfelix commented Apr 12, 2019

tobia commented Apr 12, 2019 • edited Loading

Centril commented Apr 28, 2019

panaman67 commented May 18, 2019

Centril commented Jun 10, 2019

pnkfelix commented Jun 10, 2019

bors commented Jun 10, 2019

tesuji commented Jun 10, 2019

pnkfelix commented Jun 10, 2019

bors commented Jun 10, 2019

bors commented Jun 10, 2019

pnkfelix commented Jun 10, 2019

pnkfelix commented Apr 4, 2019 •

edited

Loading

tobia commented Apr 12, 2019 •

edited

Loading