Introduce a shared extractor library #12546
Conversation
hmac
force-pushed
the
extractor-shared-library
branch
2 times, most recently
from
efc8b34
to
3829454
Compare
This makes it possible for different languages to share this extractor.
This mirrors the structure we have in the Ruby extractor, and will allow us to share more code.
Ensure that builds via cargo-cross, which are executed in a docker container, can see the shared library.
hmac
force-pushed
the
extractor-shared-library
branch
from
3829454
to
856132b
Compare
This include support for mounting external path dependencies as volumes.
hmac
force-pushed
the
extractor-shared-library
branch
9 times, most recently
from
0fc65dc
to
24baf64
Compare
hmac
force-pushed
the
extractor-shared-library
branch
from
24baf64
to
30eacd0
Compare
.github/workflows/ruby-build.yml
Outdated
| @@ -60,8 +60,10 @@ jobs: | |||
| path: | | |||
| ruby/extractor/target/release/autobuilder | |||
| ruby/extractor/target/release/autobuilder.exe | |||
| ruby/extractor/target/x86_64-unknown-linux-gnu/release/autobuilder | |||
There was a problem hiding this comment.
Is this still needed? It looks like you are mv-ing those paths when building the extractor.
There was a problem hiding this comment.
Good spot. I've removed these paths.
| run: cd extractor && cross build --release | ||
| run: | | ||
| cd extractor | ||
| cross build --release |
There was a problem hiding this comment.
Perhaps use --target-dir instead of using mv afterwards?
There was a problem hiding this comment.
I don't think that will work. The default target-dir is ./target. Inside this dir, cross creates the x86_64-unknown-linux-gnu directory.
shared/extractor/src/options.rs
Outdated
| /// assert_eq!(parse_codeql_threads("-5", 4), Some(1)); | ||
| /// assert_eq!(parse_codeql_threads("nope", 4), None); | ||
| /// ``` | ||
| pub fn parse_codeql_threads(threads_str: &str, num_cpus: usize) -> Option<usize> { |
There was a problem hiding this comment.
| pub fn parse_codeql_threads(threads_str: &str, num_cpus: usize) -> Option<usize> { | |
| fn parse_codeql_threads(threads_str: &str, num_cpus: usize) -> Option<usize> { |
There was a problem hiding this comment.
I'm afraid we need the pub for the doc tests to work
There was a problem hiding this comment.
That's just annoying. I quite like the doc tests. Let's leave it.
| @@ -14,13 +14,21 @@ else | |||
| fi | |||
|
|
|||
| (cd extractor && "$CARGO" build --release) | |||
| extractor/target/release/generator --dbscheme ql/lib/ruby.dbscheme --library ql/lib/codeql/ruby/ast/internal/TreeSitter.qll | |||
|
|
|||
| # If building via cross, the binaries will be in extractor/target/<triple>/release | |||
There was a problem hiding this comment.
Shall we try to use the --target-dir flag to make cross output the files to the same directory as cargo would?
There was a problem hiding this comment.
Good idea - I'll give that a try
There was a problem hiding this comment.
Unfortunately this doesn't work - see my other comment for more info.
It is now called `tree-sitter-extractor`, to make it clearer that it builds on tree-sitter grammars.
Move most of the common code between the Ruby and QL extractors into a library at
shared/extractor.