DevUtils: inplace builds, caching of libraries, hot reloading #529
Conversation
taliesinb
changed the title
taliesinb
force-pushed
the
InplaceBuilds
branch
4 times, most recently
from
b7ff088
to
86a7b8c
Compare
|
@maxitg CircleCI is failing because it the images don't have GitLink on them apparently. Should we add it to the images? |
taliesinb
force-pushed
the
InplaceBuilds
branch
from
86a7b8c
to
31c413c
Compare
|
I added an explicit This is good news! It indicates that is the only remaining issue. I think we should keep the |
taliesinb
force-pushed
the
InplaceBuilds
branch
6 times, most recently
from
707a9bf
to
59abded
Compare
taliesinb
force-pushed
the
InplaceBuilds
branch
from
c9990bd
to
7f560a0
Compare
pretty natural place for it. Also call InstallGitLink in the pack.wls script.
taliesinb
force-pushed
the
InplaceBuilds
branch
from
7f560a0
to
e10300e
Compare
There was a problem hiding this comment.
This looks to be a fantastic workflow! One extra thing I like about it is that we can eventually separate DevUtils into a separate repository, and use it in multiple packages which would save us from lots of duplicated code.
Reviewed 24 of 28 files at r1, 2 of 2 files at r2, 2 of 2 files at r3.
Reviewable status: all files reviewed, 51 unresolved discussions (waiting on @aokellermann, @daneelsan, @maxitg, and @taliesinb)
dev_build.wls, line 1 at r1 (raw file):
#!/usr/bin/env wolframscript
Looks like this one should not be here.
install.wls, line 8 at r1 (raw file):
Print["Message occurred during loading of DevUtils. Build failed."]; Exit[1]; ];
I think we should wrap everything in Check because messages can occur in InstallGitLink[], PacletFind[...], etc., and it's much easier to deal with a single check capturing everything than individual ones that can be missed.
The same applies to the other scripts.
README.md, line 107 at r1 (raw file):
## Build Instructions For users who wish to make use of SetReplace functionality, and not modify the source code itself, we recommend simply building and installing the paclet.
Use italic SetReplace as is a convention throughout the documentation.
README.md, line 118 at r1 (raw file):
Please note that if you do not have GitLink installed, it will be installed for you. For more info about doing development on the SetReplace codebase and the associated workflows, see the Contributing guide.
Use italic SetReplace.
README.md, line 118 at r1 (raw file):
Please note that if you do not have GitLink installed, it will be installed for you. For more info about doing development on the SetReplace codebase and the associated workflows, see the Contributing guide.
"Contributing guide" should be a link.
README.md, line 120 at r1 (raw file):
3. Run `./install.wls` to install the paclet into your Wolfram system. 4. Evaluate `PacletDataRebuild[]` in all running Wolfram kernels. 5. Evaluate ``<< SetReplace` `` every time before using the package.
Don't remove this line because some users might not know how packages work.
.github/CONTRIBUTING.md, line 83 at r2 (raw file):
### Building in-place The main workflow we recommend is what we are calling an "in-place build". This largely happens automatically when you load the SetReplace package by calling `Get["~/git/SetReplace/Kernel/init.m"]` or equivalent. If the C++ library has not already been built, it will automatically be built for you, and the resulting libraries placed in the `LibraryResources` subdirectory of the repository root.
Use italics for SetReplace.
.github/CONTRIBUTING.md, line 85 at r2 (raw file):
The main workflow we recommend is what we are calling an "in-place build". This largely happens automatically when you load the SetReplace package by calling `Get["~/git/SetReplace/Kernel/init.m"]` or equivalent. If the C++ library has not already been built, it will automatically be built for you, and the resulting libraries placed in the `LibraryResources` subdirectory of the repository root. If you later modify the C++ code and call `Get[...]` again, the library will be automatically rebuilt, and hot-loaded into your current Mathematica session (you do not need to Quit). Moreover, builds of the library will be cached based on the hash of the C++ code, making it easy to switch quickly between several library versions (say, in different Git branches).
Quit should be formatted as code.
.github/CONTRIBUTING.md, line 105 at r2 (raw file):
### Build Paclets You may occasionally want to build and install a paclet from the current state of the repository. This will package together all the Wolfram Language source code, along with the library, and various metadata, into a single ".paclet" file, which has a automatically computed version number associated with it.
"... has a automatically ..." -> "... has an automatically ..."
.github/CONTRIBUTING.md, line 123 at r2 (raw file):
The paclet will be installed in your system, replacing any existing version of the paclet you may have. This will allow you to load the paclet in future by running simply
Get["SetReplace`"].
in the future
.github/CONTRIBUTING.md, line 125 at r2 (raw file):
The paclet will be installed in your system, replacing any existing version of the paclet you may have. This will allow you to load the paclet in future by running simply ``Get["SetReplace`"] ``. A less frequently updated version is available through the Wolfram public paclet server and can be installed with `PacletInstall["SetReplace"]`.
If we are going to mention this, it should be in the README since it's not very useful for developers usually being at least a week old.
.github/CONTRIBUTING.md, line 153 at r2 (raw file):
### Automated tests To run the tests, `cd` to the repository root, and run `./test.wls` from the command line. Note that `./test.wls` will automatically rebuild libSetReplace for you before running the tests if your libSetReplace is out of date (for example, if you changed some C++ files but you did not either `Get` SetReplace or run `./build.wls`). If everything is ok, you will see `[ok]` next to each group of tests, and "Tests passed." message at the end. Otherwise, you will see error messages telling you which test inputs failed and for what reason.
Similarly, libSetReplace and SetReplace should be italic.
.github/CONTRIBUTING.md, line 394 at r2 (raw file):
### Scripts The three main scripts of *SetReplace* are [build.wls](/build.wls), [install.wls](/install.wls) and [test.wls](/test.wls). The build script is the most complex of the three, and it uses additional definitions in [buildInit.wl](/scripts/buildInit.wl). In addition to building the C++ code and packing the paclet, it also auto-generates the paclet version number based on the number of commits to master from the checkpoint defined in [version.wl](/scripts/version.wl). Some of the code in the [scripts](/scripts) folder is only used for building *SetReplace* on the internal Wolfram Research systems and should not be modified by external developers as CI has no way of testing it.
This needs to be updated as well to explain how DevUtils work.
DevUtils/BuildLibrary.m, line 4 at r2 (raw file):
PackageImport["GeneralUtilities`"]
Nit: Don't leave more than one empty line between code lines. The same applies elsewhere.
DevUtils/BuildLibrary.m, line 7 at r2 (raw file):
PackageExport["BuildLibSetReplace"]
Wouldn't it be better to rename this file BuildLibSetReplace.m to be consistent with the function name?
DevUtils/BuildLibrary.m, line 81 at r2 (raw file):
"LibraryPath" -> libraryPath, "LibraryFileName" -> libraryFileName, "LibraryBuildTime" -> DateList[FileDate[libraryPath], TimeZone -> "UTC"],
I think we should round the seconds. Since precision does not go further than seconds anyway, I don't think it makes any sense to have a weird float in the JSON file.
DevUtils/BuildLibrary.m, line 82 at r2 (raw file):
"LibraryFileName" -> libraryFileName, "LibraryBuildTime" -> DateList[FileDate[libraryPath], TimeZone -> "UTC"], "LibrarySourceHash" -> Hash[sourceHashes]
Shouldn't it be finalHash? It's a bit weird that we are including DevUtils sources but not options. And it's quite counterintuitive that this hash is not the same as is used for caching.
DevUtils/BuildLibrary.m, line 173 at r2 (raw file):
flushLibrariesIfFull[libraryDirectory_] := Scope[ files = FileNames["lib*", libraryDirectory]; If[Length[files] > 128,
Shouldn't it be 127 if this is called before the build and we claim in the usage that 128 is the maximum?
DevUtils/BuildLibrary.m, line 174 at r2 (raw file):
files = FileNames["lib*", libraryDirectory]; If[Length[files] > 128, oldestFile = MinimalBy[files, FileDate, 8];
Why 8 instead of Length[files] - 127?
DevUtils/BuildLibrary.m, line 50 at r3 (raw file):
BuildLibSetReplace[OptionsPattern[]] := ModuleScope[
Remove the empty line.
DevUtils/BuildPaclet.m, line 16 at r2 (raw file):
CreateSetReplacePaclet::buildfailed = "Could not build paclet from `` into ``.";
buildfailed is confusing because it's not actually building it. Why don't we call it packfailed and change the message accordingly?
DevUtils/BuildPaclet.m, line 17 at r2 (raw file):
CreateSetReplacePaclet::buildfailed = "Could not build paclet from `` into ``."; CreateSetReplacePaclet::nogitlink = "GitLink is not installed, so the built paclet version cannot be correctly calculated. Proceed with caution, and consider installing GitLink by running InstallGitLink[]."
Nit: Split into two lines under 120 chars.
DevUtils/BuildPaclet.m, line 26 at r2 (raw file):
* The source for the PacletObject is given by the contents of the 'Kernel' and 'LibraryResources' directories \ of the current repo, but can be overridden with the 'RootDirectory' option. * The minor version is derived from the number of commits since the last checkpoint, which is obtained from \
This sounds like the checkpoint is obtained from the master branch. I think we need to say more clearly that it's obtained from the number of commits to master since the last checkpoint, and the checkpoint is at scripts/version.wl.
DevUtils/BuildPaclet.m, line 30 at r2 (raw file):
" $gitLinkDownloadCode = "PacletInstall[\"https://www.wolframcloud.com/obj/maxp1/GitLink-2019.11.26.01.paclet\"]";
This is no longer used anywhere.
DevUtils/BuildPaclet.m, line 32 at r2 (raw file):
$gitLinkDownloadCode = "PacletInstall[\"https://www.wolframcloud.com/obj/maxp1/GitLink-2019.11.26.01.paclet\"]"; CreateSetReplacePaclet[OptionsPattern[]] := ModuleScope[
Shouldn't it build first as well to be consistent with the scripts and avoid potential mistakes?
DevUtils/BuildPaclet.m, line 47 at r2 (raw file):
buildInfo = <|"GitSHA" -> gitSHA, "BuildTime" -> Round[DateList[TimeZone -> "UTC"]]|>; tempBuildInfoFile = FileNameJoin[{$TemporaryDirectory, "PacletBuildInfo.json"}];
I think we should put it into $TemporaryDirectory/SetReplace/PacletBuildInfo.json to avoid potential conflicts with other packages.
DevUtils/BuildPaclet.m, line 50 at r2 (raw file):
Developer`WriteRawJSONFile[tempBuildInfoFile, buildInfo]; fileTree = Flatten @ List[
Why use a dangerous Flatten here if you can just put them all in a single list?
DevUtils/BuildPaclet.m, line 66 at r2 (raw file):
pacletInfo = Association @@ Import[pacletInfoFilename]; versionString = pacletInfo[Version] <> "." <> ToString[minorVersionNumber]; tempFilename = FileNameJoin[{$TemporaryDirectory, "PacletInfo.m"}];
Similarly, I think we should put it to $TemporaryDirectory/SetReplace/PacletInfo.m.
DevUtils/BuildPaclet.m, line 7 at r3 (raw file):
PackageExport["CreateSetReplacePaclet"]
Same here, why not name this file CreateSetReplacePaclet.m for consistency?
DevUtils/Console.m, line 6 at r3 (raw file):
PackageExport["ConsoleBuildLibSetReplace"]
Shouldn't we have a usage message for these two as well since they are public?
DevUtils/Console.m, line 9 at r3 (raw file):
ConsoleBuildLibSetReplace[opts___] := ModuleScope @ Check[
Remove the empty line.
DevUtils/Console.m, line 20 at r3 (raw file):
If[result["FromCache"], Print["Using cached build"];
The periods at the end of printed lines should be consistent.
DevUtils/Console.m, line 36 at r3 (raw file):
ConsoleCreateSetReplacePaclet[opts___] := ModuleScope @ Check[
Remove the empty line.
DevUtils/Console.m, line 77 at r3 (raw file):
SetUsage @ " ConsoleTryEnvironment[var$, default$] will look up the value of the environment variable var$, but use default$ if it is not availabe.
This line is over 120 chars.
DevUtils/GitLink.m, line 46 at r3 (raw file):
PackageExport["CalculateMinorVersionNumber"]
Usage?
DevUtils/GitLink.m, line 55 at r3 (raw file):
gitRepo, Except[versionInformation["Checkpoint"]], GitLink`GitMergeBase[gitRepo, "HEAD", masterBranch]] - 1]];
The -1 should be after the next bracket, the version it currently generates is off by one.
Kernel/buildData.m, line 21 at r3 (raw file):
(* forwarders for the functions we want from DevUtils. This is done so we don't create the SetReplaceDevUtils context for ordinary users (when DevUtils *isn't* available) *) $buildLibSetReplace = Symbol["SetReplaceDevUtils`BuildLibSetReplace"];
Why $? It's a function, not a constant.
Kernel/buildData.m, line 27 at r3 (raw file):
(* if there is a frontend, then give a temporary progress panel, otherwise just Print *) If[TrueQ @ $Notebooks, Internal`WithLocalSettings[
Can you add a comment explaining what WithLocalSettings is? It does not have a usage message.
Kernel/buildData.m, line 79 at r3 (raw file):
SetUsage @ " $SetReplaceBuildTime gives the time at which this SetReplace paclet was built. * When evaluated for an in-place build, this time is Now.
It's not actually Now, it's the time it was loaded.
Kernel/setSubstitutionSystem$cpp.m, line 5 at r3 (raw file):
PackageImport["GeneralUtilities`"]
Why so many empty lines? Some people have a limited monitor height.
Kernel/setSubstitutionSystem$cpp.m, line 13 at r3 (raw file):
(* Interface to the C++ implementation of setSubstitutionSystem. *) (* this function is defined now, but only run the *next* time Kernel/init.m is called, before all symbols are cleared. *)
This line goes over the 120 chars limit.
Kernel/setSubstitutionSystem$cpp.m, line 93 at r3 (raw file):
$Failed]; $libraryFunctions = {
Why don't we define it like this:
$libraryFunctions = {
$cpp$setCreate = ...,
$cpp$setDelete = ...
};so that we don't have a list of functions in two different places?
scripts/re_build_library.wls, line 17 at r3 (raw file):
$systemID = AntProperty["system_id"]; $targetDirectory = FileNameJoin[{AntProperty["files_directory"], AntProperty["component"], "LibraryResources", $systemID}];
This line is over 120 chars.
scripts/re_build_paclet.wls, line 1 at r3 (raw file):
#!/usr/bin/env wolframscript
I think re_pack_paclet.wls would be a better name.
scripts/re_build_SetReplace.xml, line 45 at r3 (raw file):
</target> <target name='NativeLibrary.SetReplace.MacOSX-x86-64.postbuild' extensionOf='NativeLibrary.postbuild' if='is.linux'>
The target name should be NativeLibrary.SetReplace.Linux-x86-64.postbuild.
scripts/test_re_scripts.wls, line 1 at r3 (raw file):
#!/usr/bin/env wolframscript
I think we should add a comment saying it does not test the RE scripts entirely, and even if this test passes, the RE build might still fail.
scripts/test_re_scripts.wls, line 1 at r3 (raw file):
#!/usr/bin/env wolframscript
I'm currently running the RE build, but there is a 40 mins congestion. I'll comment here if the build fails.
scripts/test_re_scripts.wls, line 16 at r3 (raw file):
(* we'll disable Exit[0], since we want don't want our *own* script to end *) Exit[0] /; $wrap := Print[$greenColor, "Exit[0]", $endColor];
RE script ends due to this as well. We need to remove Exit[0] from the RE scripts.
Tests/buildData.wlt, line 12 at r3 (raw file):
"$SetReplaceGitSHA" -> <| "tests" -> { (* This tests will fail if the paclet was built from a dirty repo, i.e. there were uncommitted changes.
It doesn't anymore.
Tests/meta.wlt, line 17 at r3 (raw file):
Not @ StringQ @ MessageName[symbol, "usage"], HoldFirst]; VerificationTest[ Select[exports, hasNoSymbolUsageQ], HoldComplete[]
Put HoldCompete[] on the next line to make it obvious that VerificationTest is comparing the two. Same for the cases below.
Tests/meta.wlt, line 31 at r3 (raw file):
(* Test coverage: all public symbols appear in unit tests *) allSource = StringJoin[FileString /@ FileNames["*.wlt", $testsDirectory]]; doesNotAppearInSourceQ = Function[symbol,
I think this should be called doesNotAppearInTestsQ. Otherwise, the error message is confusing.
There was a problem hiding this comment.
Agree -- it would be nice to share these utilities.
Reviewable status: 7 of 26 files reviewed, 49 unresolved discussions (waiting on @aokellermann, @daneelsan, @maxitg, and @taliesinb)
dev_build.wls, line 1 at r1 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Looks like this one should not be here.
Done.
install.wls, line 8 at r1 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I think we should wrap everything in
Checkbecause messages can occur inInstallGitLink[],PacletFind[...], etc., and it's much easier to deal with a single check capturing everything than individual ones that can be missed.The same applies to the other scripts.
Done.
README.md, line 107 at r1 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Use italic SetReplace as is a convention throughout the documentation.
Done.
README.md, line 118 at r1 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Use italic SetReplace.
Done.
README.md, line 118 at r1 (raw file):
Previously, maxitg (Max Piskunov) wrote…
"Contributing guide" should be a link.
Done.
README.md, line 120 at r1 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Don't remove this line because some users might not know how packages work.
Done.
.github/CONTRIBUTING.md, line 83 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Use italics for SetReplace.
Done.
.github/CONTRIBUTING.md, line 85 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Quitshould be formatted as code.
Done.
.github/CONTRIBUTING.md, line 105 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
"... has a automatically ..." -> "... has an automatically ..."
Done.
.github/CONTRIBUTING.md, line 123 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
in the future
Done.
.github/CONTRIBUTING.md, line 125 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
If we are going to mention this, it should be in the README since it's not very useful for developers usually being at least a week old.
I'll just remove it.
.github/CONTRIBUTING.md, line 153 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Similarly, libSetReplace and SetReplace should be italic.
Done.
.github/CONTRIBUTING.md, line 394 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
This needs to be updated as well to explain how
DevUtilswork.
Done.
DevUtils/Console.m, line 6 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Shouldn't we have a usage message for these two as well since they are public?
Done.
DevUtils/Console.m, line 9 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Remove the empty line.
Done.
DevUtils/Console.m, line 20 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
The periods at the end of printed lines should be consistent.
Done.
DevUtils/Console.m, line 36 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Remove the empty line.
Done.
DevUtils/Console.m, line 77 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
This line is over 120 chars.
Done.
DevUtils/GitLink.m, line 46 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Usage?
Done.
DevUtils/GitLink.m, line 55 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
The
-1should be after the next bracket, the version it currently generates is off by one.
Done.
Kernel/buildData.m, line 21 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Why
$? It's a function, not a constant.
Done.
Kernel/buildData.m, line 27 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Can you add a comment explaining what
WithLocalSettingsis? It does not have a usage message.
Done.
Kernel/buildData.m, line 79 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
It's not actually
Now, it's the time it was loaded.
Done.
Kernel/setSubstitutionSystem$cpp.m, line 5 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Why so many empty lines? Some people have a limited monitor height.
Done.
Kernel/setSubstitutionSystem$cpp.m, line 13 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
This line goes over the 120 chars limit.
Done.
Kernel/setSubstitutionSystem$cpp.m, line 93 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Why don't we define it like this:
$libraryFunctions = { $cpp$setCreate = ..., $cpp$setDelete = ... };so that we don't have a list of functions in two different places?
Done.
scripts/re_build_library.wls, line 17 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
This line is over 120 chars.
Done.
scripts/re_build_SetReplace.xml, line 45 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
The target name should be
NativeLibrary.SetReplace.Linux-x86-64.postbuild.
Done.
Tests/buildData.wlt, line 12 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
It doesn't anymore.
Done.
Tests/meta.wlt, line 17 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Put
HoldCompete[]on the next line to make it obvious thatVerificationTestis comparing the two. Same for the cases below.
Done.
Tests/meta.wlt, line 31 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I think this should be called
doesNotAppearInTestsQ. Otherwise, the error message is confusing.
Done.
DevUtils/BuildLibrary.m, line 7 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Wouldn't it be better to rename this file
BuildLibSetReplace.mto be consistent with the function name?
Done.
DevUtils/BuildLibrary.m, line 81 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I think we should round the seconds. Since precision does not go further than seconds anyway, I don't think it makes any sense to have a weird float in the JSON file.
Done.
DevUtils/BuildLibrary.m, line 82 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Shouldn't it be
finalHash? It's a bit weird that we are includingDevUtilssources but not options. And it's quite counterintuitive that this hash is not the same as is used for caching.
Good catch. This was actually correct, when we were doing "consistency checking" -- where you wouldn't know the compile options at load time, only what the source hashes were. But now this isn't relevant anymore since we attempt a rebuild every time you load SetReplace.
DevUtils/BuildLibrary.m, line 174 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Why
8instead ofLength[files] - 127?
It's an amortization to avoid having to reread all the file dates every single time we do a build if the cache is exactly full.
DevUtils/BuildLibrary.m, line 50 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Remove the empty line.
Done.
DevUtils/BuildPaclet.m, line 16 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
buildfailedis confusing because it's not actually building it. Why don't we call itpackfailedand change the message accordingly?
Done.
DevUtils/BuildPaclet.m, line 26 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
This sounds like the checkpoint is obtained from the
masterbranch. I think we need to say more clearly that it's obtained from the number of commits tomastersince the last checkpoint, and the checkpoint is atscripts/version.wl.
Done.
DevUtils/BuildPaclet.m, line 30 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
This is no longer used anywhere.
Done.
DevUtils/BuildPaclet.m, line 32 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Shouldn't it build first as well to be consistent with the scripts and avoid potential mistakes?
I don't agree. BuildLibSetReplace requires options like Compiler etc, which would have to be forwarded by CreateSetReplacePaclet, which is inelegant (and not useful to anyone currently, since the console scripts need to carefully check the output of the CreateSetReplacePaclet in order to print things nicely, etc). Keeping them orthogonal is clean and simple.
Also, if we did this, the error handling in CreateSetReplacePaclet for the case where BuildLibSetReplace fails would then redundantly replicate the same error handling done by the console versions of the scripts.
But I understand it might be confusing to someone who decides to use these functions directly (which we do not advertise or document in the MD files), so I'll add a line to the usage that warns that this does not build the library first.
DevUtils/BuildPaclet.m, line 47 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I think we should put it into
$TemporaryDirectory/SetReplace/PacletBuildInfo.jsonto avoid potential conflicts with other packages.
Done.
DevUtils/BuildPaclet.m, line 50 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Why use a dangerous
Flattenhere if you can just put them all in a single list?
Done.
DevUtils/BuildPaclet.m, line 66 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Similarly, I think we should put it to
$TemporaryDirectory/SetReplace/PacletInfo.m.
Done.
DevUtils/BuildPaclet.m, line 7 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Same here, why not name this file
CreateSetReplacePaclet.mfor consistency?
Done.
scripts/re_build_paclet.wls, line 1 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I think
re_pack_paclet.wlswould be a better name.
Done.
scripts/test_re_scripts.wls, line 1 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I think we should add a comment saying it does not test the RE scripts entirely, and even if this test passes, the RE build might still fail.
I'll just rename this file to run_re_scripts.wls.
scripts/test_re_scripts.wls, line 1 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I'm currently running the RE build, but there is a 40 mins congestion. I'll comment here if the build fails.
Great.
scripts/test_re_scripts.wls, line 16 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
RE script ends due to this as well. We need to remove
Exit[0]from the RE scripts.
Not exactly sure what you're saying, but I've removed Exit[0] from re_build_library.wls and re_pack_paclet.wls as requested.
There was a problem hiding this comment.
Reviewable status: 7 of 26 files reviewed, 49 unresolved discussions (waiting on @aokellermann, @daneelsan, and @maxitg)
DevUtils/BuildLibrary.m, line 173 at r2 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Shouldn't it be
127if this is called before the build and we claim in the usage that128is the maximum?
Done.
taliesinb
force-pushed
the
InplaceBuilds
branch
from
3d01a42
to
dfbbdbb
Compare
There was a problem hiding this comment.
Reviewable status: 7 of 26 files reviewed, 52 unresolved discussions (waiting on @aokellermann, @daneelsan, @maxitg, and @taliesinb)
DevUtils/GitLink.m, line 10 at r3 (raw file):
(* unfortunately, owing to a bug in GitLink, GitLink *needs* to be on the $ContextPath or GitRepo objects end up in the wrong context, since they are generated in a loopback link unqualified *) $GitLinkAvailableQ := $GitLinkAvailableQ = !FailureQ[Quiet @ Check[Needs["GitLink`"], $Failed]];
I think the caching here won't work if $GitLinkAvailableQ is called first, then GitLink is installed, and then $GitLinkAvailableQ is called again.
DevUtils/GitLink.m, line 38 at r3 (raw file):
" InstallGitLink[] := If[PacletFind["GitLink", "Internal" -> All] === {},
Prepend PacletManager` context to PacletFind and PacletInstall. The RE build does not work otherwise.
scripts/re_pack_paclet.wls, line 20 at r4 (raw file):
$opts = { "RootDirectory" -> $buildDir,
This root directory does not have .git and version.wl, so some of the git-related functions fail (see the comment in the xml file).
taliesinb
force-pushed
the
InplaceBuilds
branch
from
7a82b0b
to
cc35578
Compare
There was a problem hiding this comment.
Reviewable status: 7 of 26 files reviewed, 52 unresolved discussions (waiting on @aokellermann, @daneelsan, and @maxitg)
DevUtils/GitLink.m, line 10 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
I think the caching here won't work if
$GitLinkAvailableQis called first, thenGitLinkis installed, and then$GitLinkAvailableQis called again.
I've turned off caching.
DevUtils/GitLink.m, line 38 at r3 (raw file):
Previously, maxitg (Max Piskunov) wrote…
Prepend
PacletManager`context toPacletFindandPacletInstall. The RE build does not work otherwise.
That won't work under 12.1, unfortunately, since those symbols were moved. I'll just PackageImport["PacletManager`"], which won't have any real effect >= 12.1
scripts/re_pack_paclet.wls, line 20 at r4 (raw file):
Previously, maxitg (Max Piskunov) wrote…
This root directory does not have
.gitandversion.wl, so some of the git-related functions fail (see the comment in thexmlfile).
Fixed by introducing the new RepositoryDirectory option.
depending on its role. Also introduce LibraryDirectory which is where the built libraries are assumed to live.
taliesinb
force-pushed
the
InplaceBuilds
branch
from
e1d71df
to
78f822c
Compare
instead. Also make sure that the Compiler environment variable has an effect for both pack and install. More error checking during packing to make sure the input files are present. Also rename function to PackSetReplace.
taliesinb
force-pushed
the
InplaceBuilds
branch
from
78f822c
to
e0ae9b8
Compare
There was a problem hiding this comment.
The RE builds still fail, but as discussed, we can merge now and fix it on Monday (since they only affect me anyway). There is a small typo in the new text, you can open a new PR to fix it. Other than that, everything looks ok.
Reviewed 10 of 20 files at r4, 1 of 4 files at r5, 10 of 10 files at r6.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @aokellermann, @daneelsan, and @taliesinb)
README.md, line 118 at r6 (raw file):
Please note that if you do not have GitLink installed, it will be installed for you. Now that you have installed the *SetReplace* paclet, you should evaluate ``<< SetReplace` `` every time you start a fresh Mathematica session. This will load the paclet and bring the various functions into scope, so that you can call thaem.
Typo: thaem -> them.
scripts/re_build_SetReplace.xml, line 15 at r1 (raw file):
Component (platform-independent) --> <target name='Component.SetReplace.execute' extensionOf='Component.execute'>
This did not work. Here is the log:
Build 'Internal / SetReplace / Component / Build' #168, default branch '<default>'
Triggered 2020-11-25 20:05:59 by 'Snapshot dependency; Max Piskunov (maxp); Internal / SetReplace / Build, build #194'
Started 2020-11-25 20:14:28 on agent 'build34'
Finished 2020-11-25 20:14:38 with status FAILURE 'Exit code 1 (Step: NativeLibrary.build (Ant)) (new)'
VCS revisions: 'ReScripts' (Git, instance id 11676): 'b4dc2a7f6e34b97dc847462aac30d235b342514d' (branch: 'refs/heads/master', checkout rules: '+:. => re-scripts')
'Internal_SetReplace_SetReplace' (Git, instance id 13859): 'e0ae9b89cba3aba2f86ad53d25a44278916c9170' (branch: 'refs/heads/internalBuild', checkout rules: '+:. => SetReplace')
TeamCity URL https://teamcity.wolfram.com/viewLog.html?buildId=1797868&buildTypeId=Internal_SetReplace_Component_Build
TeamCity server version is 2020.1.4 (build 78906), server timezone: CST (UTC-06:00)
[20:05:59]E: bt15934 (8m:38s)
[20:05:59]i: TeamCity server version is 2020.1.4 (build 78906)
[20:06:00] : Collecting changes in 3 VCS roots (5s)
[20:06:00] : [Collecting changes in 3 VCS roots] VCS Root details
[20:06:00] : [VCS Root details] "SetReplace" {instance id=13859, parent internal id=1366, parent id=Internal_SetReplace_SetReplace, description: "ssh://[email protected]:7999/pac/setreplace.git#refs/heads/internalBuild"}
[20:06:00] : [VCS Root details] "RETC/Internal" {instance id=12572, parent internal id=1325, parent id=Internal, description: "ssh://[email protected]:7999/retc/internal.git#refs/heads/master"}
[20:06:00] : [VCS Root details] "re-scripts" {instance id=11676, parent internal id=1300, parent id=ReScripts, description: "ssh://[email protected]:7999/re/re-scripts.git#refs/heads/master"}
[20:06:00]i: [Collecting changes in 3 VCS roots] Detecting changes in VCS root 'RETC/Internal' (used in 'Build', 'Build' and 18 other configurations)
[20:06:00]i: [Collecting changes in 3 VCS roots] Will collect changes for 'RETC/Internal' starting from revision c07d1689672df05b6d23e3c3cf7a409db9c37140
[20:06:00]i: [Collecting changes in 3 VCS roots] Detecting changes in VCS root 'SetReplace' (used in 'Build', 'Build' and 4 other configurations)
[20:06:00]i: [Collecting changes in 3 VCS roots] Will collect changes for 'SetReplace' starting from revision 71791b24cc36e29bb159249f44d594641d898135
[20:06:00]i: [Collecting changes in 3 VCS roots] VCS revisions for 'SetReplace' - 71791b24cc36e29bb159249f44d594641d898135..e0ae9b89cba3aba2f86ad53d25a44278916c9170
[20:06:00]i: [Collecting changes in 3 VCS roots] Processing combined checkout rule for 'SetReplace'
[20:06:00]i: [Collecting changes in 3 VCS roots] Detecting changes in VCS root 're-scripts' (used in 'AlphaDocs (.html)', 'AlphaDocs (.nb)' and 393 other configurations)
[20:06:00]i: [Collecting changes in 3 VCS roots] Will collect changes for 're-scripts' starting from revision b4dc2a7f6e34b97dc847462aac30d235b342514d
[20:06:01]i: [Collecting changes in 3 VCS roots] VCS root 'SetReplace': git fetch ssh://[email protected]:7999/pac/setreplace.git
[20:06:04]i: [Collecting changes in 3 VCS roots] VCS root 'SetReplace': update ref remote name: refs/heads/internalBuild, local name: refs/heads/internalBuild, old object id: 71791b24cc36e29bb159249f44d594641d898135, new object id: e0ae9b89cba3aba2f86ad53d25a44278916c9170, result: FORCED
[20:06:04]i: [Collecting changes in 3 VCS roots] VCS root 'SetReplace': Remote process messages:
Total 54 (delta 42), reused 0 (delta 0)
[20:06:04]i: [Collecting changes in 3 VCS roots] VCS root 'SetReplace': git fetch ssh://[email protected]:7999/pac/setreplace.git finished
[20:06:05]i: [Collecting changes in 3 VCS roots] Done collecting changes for 'SetReplace': 4 changes collected 4 changes persisted, total time: 4s,632ms, persisting time: 290ms
[20:06:05] : [Collecting changes in 3 VCS roots] Compute revision for 'SetReplace'
[20:06:05] : [Compute revision for 'SetReplace'] Upper limit revision: e0ae9b89cba3aba2f86ad53d25a44278916c9170
[20:06:05]i: [Compute revision for 'SetReplace'] MaxModId = 2054591
[20:06:05] : [Compute revision for 'SetReplace'] Latest commit attached to build configuration (with id <= 2054591): e0ae9b89cba3aba2f86ad53d25a44278916c9170
[20:06:05] : [Compute revision for 'SetReplace'] Computed revision: e0ae9b89cba3aba2f86ad53d25a44278916c9170
[20:06:05] : [Collecting changes in 3 VCS roots] Compute revision for 're-scripts'
[20:06:05] : [Compute revision for 're-scripts'] Upper limit revision: b4dc2a7f6e34b97dc847462aac30d235b342514d
[20:06:05]i: [Compute revision for 're-scripts'] MaxModId = 2054591
[20:06:05] : [Compute revision for 're-scripts'] Latest commit attached to build configuration (with id <= 2054591): b4dc2a7f6e34b97dc847462aac30d235b342514d
[20:06:05] : [Compute revision for 're-scripts'] Computed revision: b4dc2a7f6e34b97dc847462aac30d235b342514d
[20:06:05] : [Collecting changes in 3 VCS roots] Compute revision for 'RETC/Internal'
[20:06:05] : [Compute revision for 'RETC/Internal'] Upper limit revision: c07d1689672df05b6d23e3c3cf7a409db9c37140
[20:06:05]i: [Compute revision for 'RETC/Internal'] MaxModId = 2054591
[20:06:05] : [Compute revision for 'RETC/Internal'] There is no first revision stored for the branch refs/heads/master
[20:06:05] : [Compute revision for 'RETC/Internal'] Latest commit attached to build configuration (with id <= 2054591): 73541372e375892429826ca0e2ba3be8ccc89758
[20:06:05] : [Compute revision for 'RETC/Internal'] Computed revision: 73541372e375892429826ca0e2ba3be8ccc89758
[20:14:28] : The build is removed from the queue to be prepared for the start
[20:14:28] : Starting the build on the agent "build34"
[20:14:28]i: Agent time zone: US/Central
[20:14:28]i: Agent is running under JRE: 1.8.0_222-b10
[20:14:28] : Updating tools for build
[20:14:28] : [Updating tools for build] Found 1 tool used by the build: ant-1.10.3
[20:14:28] : [Updating tools for build] All used tools are up-to-date
[20:14:28] : Clearing temporary directory: /Developer/teamcity/temp/buildTmp
[20:14:28] : Publishing internal artifacts (4s)
[20:14:32] : [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[20:14:32] : [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[20:14:28] : Clean build enabled: removing old files from /Developer/teamcity/work/95e96ae48e9bf59e
[20:14:28] : Checkout directory: /Developer/teamcity/work/95e96ae48e9bf59e
[20:14:28] : Updating sources: auto checkout (on agent) (3s)
[20:14:28] : [Updating sources] Will use agent side checkout
[20:14:28] : [Updating sources] Full checkout enforced. Reason: ["Delete all files before the build" turned on]
[20:14:28] : [Updating sources] VCS Root: SetReplace (2s)
[20:14:28] : [VCS Root: SetReplace] checkout rules: =>SetReplace; revision: e0ae9b89cba3aba2f86ad53d25a44278916c9170
[20:14:28] : [VCS Root: SetReplace] Git version: 2.22.0.0
[20:14:28] : [VCS Root: SetReplace] Update git mirror (/Developer/teamcity/system/git/git-A106202C.git) (1s)
[20:14:28] : [Update git mirror (/Developer/teamcity/system/git/git-A106202C.git)] /usr/bin/git config http.sslCAInfo
[20:14:28] : [Update git mirror (/Developer/teamcity/system/git/git-A106202C.git)] /usr/bin/git show-ref
[20:14:28] : [Update git mirror (/Developer/teamcity/system/git/git-A106202C.git)] /usr/bin/git -c credential.helper= ls-remote origin
[20:14:30] : [Update git mirror (/Developer/teamcity/system/git/git-A106202C.git)] /usr/bin/git show-ref refs/remotes/origin/internalBuild
[20:14:30] : [Update git mirror (/Developer/teamcity/system/git/git-A106202C.git)] /usr/bin/git show-ref refs/heads/internalBuild
[20:14:30] : [Update git mirror (/Developer/teamcity/system/git/git-A106202C.git)] /usr/bin/git pack-refs --all
[20:14:30] : [VCS Root: SetReplace] Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] The .git directory is missing in '/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace'. Running 'git init'...
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git init
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git config lfs.storage /Developer/teamcity/system/git/git-A106202C.git/lfs
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git config core.sparseCheckout true
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git config http.sslCAInfo
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git show-ref
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git show-ref refs/remotes/origin/internalBuild
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git log -n1 --pretty=format:%H%x20%s e0ae9b89cba3aba2f86ad53d25a44278916c9170 --
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git branch
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git update-ref refs/heads/internalBuild e0ae9b89cba3aba2f86ad53d25a44278916c9170
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git -c credential.helper= checkout -q -f internalBuild
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git branch --set-upstream-to=refs/remotes/origin/internalBuild
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] Cleaning SetReplace in /Developer/teamcity/work/95e96ae48e9bf59e/SetReplace the file set ALL_UNTRACKED
[20:14:30] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/SetReplace)] /usr/bin/git clean -f -d -x
[20:14:30] : [Updating sources] VCS Root: re-scripts (1s)
[20:14:30] : [VCS Root: re-scripts] checkout rules: =>re-scripts; revision: b4dc2a7f6e34b97dc847462aac30d235b342514d
[20:14:30] : [VCS Root: re-scripts] Git version: 2.22.0.0
[20:14:30] : [VCS Root: re-scripts] Update git mirror (/Developer/teamcity/system/git/git-B4574875.git) (1s)
[20:14:30] : [Update git mirror (/Developer/teamcity/system/git/git-B4574875.git)] /usr/bin/git config http.sslCAInfo
[20:14:30] : [Update git mirror (/Developer/teamcity/system/git/git-B4574875.git)] /usr/bin/git show-ref
[20:14:30] : [Update git mirror (/Developer/teamcity/system/git/git-B4574875.git)] /usr/bin/git -c credential.helper= ls-remote origin
[20:14:32] : [Update git mirror (/Developer/teamcity/system/git/git-B4574875.git)] /usr/bin/git show-ref refs/remotes/origin/master
[20:14:32] : [Update git mirror (/Developer/teamcity/system/git/git-B4574875.git)] /usr/bin/git show-ref refs/heads/master
[20:14:32] : [Update git mirror (/Developer/teamcity/system/git/git-B4574875.git)] /usr/bin/git pack-refs --all
[20:14:32] : [VCS Root: re-scripts] Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] The .git directory is missing in '/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts'. Running 'git init'...
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git init
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git config lfs.storage /Developer/teamcity/system/git/git-B4574875.git/lfs
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git config core.sparseCheckout true
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git config http.sslCAInfo
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git show-ref
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git show-ref refs/remotes/origin/master
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git log -n1 --pretty=format:%H%x20%s b4dc2a7f6e34b97dc847462aac30d235b342514d --
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git branch
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git -c credential.helper= reset --hard b4dc2a7f6e34b97dc847462aac30d235b342514d
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git branch --set-upstream-to=refs/remotes/origin/master
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] Cleaning re-scripts in /Developer/teamcity/work/95e96ae48e9bf59e/re-scripts the file set ALL_UNTRACKED
[20:14:32] : [Update checkout directory (/Developer/teamcity/work/95e96ae48e9bf59e/re-scripts)] /usr/bin/git clean -f -d -x
[20:14:32] : Build preparation done
[20:14:32] : Step 1/2: Mathematica Install (Mathematica CustomTool Installer)
[20:14:32] : [Step 1/2] Step 1/1: Mathematica CustomTool Installer (Python)
[20:14:32] : [Step 1/1] Starting: python .script.py
[20:14:32] : [Step 1/1] in directory: /Developer/teamcity/work/95e96ae48e9bf59e
[20:14:32] : [Step 1/1] [20:14:32] === Installing LINUX 12.0 ===
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] Checking for obsolete artifacts
[20:14:32] : [Step 1/1] in /re/download/M-LINUX-Internal_12.0
[20:14:32] : [Step 1/1] None found.
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] [20:14:32] Downloading artifact MD5
[20:14:32] : [Step 1/1] [20:14:32] "Layout.M-LINUX64-Internal.12.0_Files.tar.gz.MD5"
[20:14:32] : [Step 1/1] [20:14:32] to "/re/download/M-LINUX-Internal_12.0"
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] [20:14:32] MD5s match.
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] [20:14:32] Downloading artifact
[20:14:32] : [Step 1/1] [20:14:32] "Layout.M-LINUX64-Internal.12.0_Files.tar.gz"
[20:14:32] : [Step 1/1] [20:14:32] to "/re/download/M-LINUX-Internal_12.0"
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] [20:14:32] Artifact already present, skipping download.
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] [20:14:32] Installing
[20:14:32] : [Step 1/1] [20:14:32] "/re/download/M-LINUX-Internal_12.0/Layout.M-LINUX64-Internal.12.0_Files.tar.gz"
[20:14:32] : [Step 1/1] [20:14:32] to "/re/tools/M-LINUX-Internal_12.0" ...
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] [20:14:32] Already present, skipping install.
[20:14:32] : [Step 1/1]
[20:14:32] : [Step 1/1] [20:14:32] === Done (0.058 seconds) ===
[20:14:32]i: [Step 1/1] ##teamcity[setParameter name='re.tools.mathematica.directory' value='/re/tools/M-LINUX-Internal_12.0%re.tools.mathematica.bundle_subdirectory%']
[20:14:32] : [Step 1/1] Process exited with code 0
[20:14:32]E: Step 2/2: NativeLibrary.build (Ant) (3s)
[20:14:33] : [Step 2/2] Starting: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/bin/java -Dagent.home.dir=/Developer/teamcity -Dagent.name=build34 -Dagent.ownPort=9090 -Dagent.work.dir=/Developer/teamcity/work -Dant.home=/Developer/teamcity/tools/ant-1.10.3 -Dbuild.number=168 -Dbuild.vcs.number.Internal_SetReplace_SetReplace=e0ae9b89cba3aba2f86ad53d25a44278916c9170 -Dbuild.vcs.number.ReScripts=b4dc2a7f6e34b97dc847462aac30d235b342514d -Dcomponent=SetReplace -Djava.io.tmpdir=/Developer/teamcity/temp/buildTmp -Dre.tools.mathematica=Y -Dteamcity.agent.cpuBenchmark=404 -Dteamcity.agent.dotnet.agent_url=http://localhost:9090/RPC2 -Dteamcity.agent.dotnet.build_id=1797868 -Dteamcity.agent.ensure.free.space=50gb -Dteamcity.auth.password=******* -Dteamcity.auth.userId=TeamCityBuildId=1797868 -Dteamcity.build.changedFiles.file=/Developer/teamcity/temp/buildTmp/changedFiles1162204349901525149.txt -Dteamcity.build.checkoutDir=/Developer/teamcity/work/95e96ae48e9bf59e -Dteamcity.build.id=1797868 -Dteamcity.build.properties.file=/Developer/teamcity/temp/buildTmp/teamcity.build626878933372352302.properties -Dteamcity.build.tempDir=/Developer/teamcity/temp/buildTmp -Dteamcity.build.workingDir=/Developer/teamcity/work/95e96ae48e9bf59e -Dteamcity.buildConfName=Build -Dteamcity.buildType.id=Internal_SetReplace_Component_Build -Dteamcity.configuration.properties.file=/Developer/teamcity/temp/buildTmp/teamcity.config1023016500270962465.properties -Dteamcity.projectName=Component -Dteamcity.runner.properties.file=/Developer/teamcity/temp/buildTmp/teamcity.runner4017526455061717591.properties -Dteamcity.runtime.props.file=/Developer/teamcity/temp/agentTmp/ant80688828464479924runtime -Dteamcity.tests.recentlyFailedTests.file=/Developer/teamcity/temp/buildTmp/testsToRunFirst8114612047522303683.txt "-Dteamcity.version=2020.1.4 (build 78906)" -classpath /Developer/teamcity/tools/ant-1.10.3/lib/ant-launcher.jar org.apache.tools.ant.launch.Launcher -lib /Developer/teamcity/plugins/antPlugin/ant-runtime.jar:/Developer/teamcity/lib/runtime-util.jar:/Developer/teamcity/lib/serviceMessages.jar -listener jetbrains.buildServer.agent.ant.TeamCityAntBuildListener -buildfile /Developer/teamcity/work/95e96ae48e9bf59e/SetReplace/scripts/re_build_SetReplace.xml Component.build
[20:14:33] : [Step 2/2] in directory: /Developer/teamcity/work/95e96ae48e9bf59e
[20:14:34] : [Step 2/2] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [Step 2/2] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [Step 2/2] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:34] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [include] include
[20:14:35] : [Step 2/2] include
[20:14:35] : [Step 2/2] include
[20:14:35]W: [Step 2/2] Component.config.check.java
[20:14:35]W: [Component.config.check.java] echo
[20:14:35]W: [echo] Java 1.8.0_222
[20:14:35]W: [Component.config.check.java] echo
[20:14:35]W: [echo] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre
[20:14:35]W: [Step 2/2] Component.config.check.ant
[20:14:35]W: [Component.config.check.ant] echo
[20:14:35]W: [echo] Ant Apache Ant(TM) version 1.10.3 compiled on March 24 2018
[20:14:35]W: [Component.config.check.ant] echo
[20:14:35]W: [echo] /Developer/teamcity/tools/ant-1.10.3
[20:14:35] : [Step 2/2] Component.config.check.platform
[20:14:35] : [Step 2/2] Component.config.check.init
[20:14:35] : [Step 2/2] Component.config.Jenkins.check
[20:14:35] : [Step 2/2] Component.config.Jenkins.init
[20:14:35] : [Step 2/2] Component.config.TeamCity.check
[20:14:35]W: [Step 2/2] Component.config.TeamCity.init
[20:14:35]W: [Component.config.TeamCity.init] echo
[20:14:35]W: [echo] TeamCity 2020.1.4 (build 78906)
[20:14:35]W: [Component.config.TeamCity.init] echo
[20:14:35]W: [echo] Project == Component
[20:14:35]W: [Component.config.TeamCity.init] echo
[20:14:35]W: [echo] Build == 168
[20:14:35] : [Step 2/2] Component.config.Legacy.check
[20:14:35] : [Step 2/2] Component.config.Legacy.init
[20:14:35] : [Step 2/2] Component.config.executive
[20:14:35]W: [Step 2/2] Component.config.ant-contrib.load
[20:14:35]W: [Component.config.ant-contrib.load] echo
[20:14:35]W: [echo] /re/tools/antlib/config/ant-contrib-1.0b3.jar loaded.
[20:14:35] : [Step 2/2] Component.config.ant-contrib.init
[20:14:35] : [Step 2/2] Component.config.plugins
[20:14:35] : [Step 2/2] Component.config.MathematicaTask.check
[20:14:35]W: [Step 2/2] Component.config.MathematicaTask.init
[20:14:35]W: [Component.config.MathematicaTask.init] echo
[20:14:35]W: [echo] Mathematica Directory == "/re/tools/M-LINUX-Internal_12.0"
[20:14:36] : [Step 2/2] Component.config.SignTask.check
[20:14:36] : [Step 2/2] Component.config.SignTask.init
[20:14:36] : [Step 2/2] Component.config.VerboseLoggingTask.init
[20:14:36] : [Step 2/2] Component.config.tasks
[20:14:36] : [Step 2/2] Component.config.CMake.CMake-3.0
[20:14:36] : [Step 2/2] Component.config.CMake.CMake
[20:14:36] : [Step 2/2] Component.config.CMake.init
[20:14:36] : [Step 2/2] Component.config.GradleHome.check
[20:14:36] : [Step 2/2] Component.config.GradleHome.init
[20:14:36] : [Step 2/2] Component.config.Maven.check
[20:14:36] : [Step 2/2] Component.config.Maven.init
[20:14:36] : [Step 2/2] Component.config.Python.osx
[20:14:36] : [Step 2/2] Component.config.Python.unix
[20:14:36] : [Step 2/2] Component.config.Python.windows
[20:14:36]W: [Step 2/2] Component.config.Python.init
[20:14:36]W: [Component.config.Python.init] echo
[20:14:36]W: [echo] python.exe == "/usr/bin/python2.7"
[20:14:36] : [Component.config.Python.init] exec
[20:14:36] : [exec] Python 2.7.15
[20:14:36] : [Step 2/2] Component.config.tools
[20:14:36]W: [Step 2/2] Component.config.CreationID.init
[20:14:36]W: [Component.config.CreationID.init] echo
[20:14:36]W: [echo] Job Name == "Internal_SetReplace_Component_Build"
[20:14:36]W: [Component.config.CreationID.init] echo
[20:14:36]W: [echo] Build Number == "168"
[20:14:36]W: [Component.config.CreationID.init] echo
[20:14:36]W: [echo] Build URL == "https://teamcity.wolfram.com/viewType.html?buildTypeId=Internal_SetReplace_Component_Build&buildId=1797868"
[20:14:36]W: [Component.config.CreationID.init] echo
[20:14:36]W: [echo] CreationID == "2020.11.25.168" / "20201125168"
[20:14:36] : [Step 2/2] Component.config.Parse.cd_code
[20:14:36] : [Step 2/2] Component.config.Parse.branding-cd_code
[20:14:36] : [Step 2/2] Component.config.Parse.version
[20:14:36] : [Step 2/2] Component.config.Parse.branding-version
[20:14:36] : [Step 2/2] Component.config.Parse.language-old-check
[20:14:36] : [Step 2/2] Component.config.Parse.language-old
[20:14:36] : [Step 2/2] Component.config.Parse.language-new-check
[20:14:36] : [Step 2/2] Component.config.Parse.language-new
[20:14:36] : [Step 2/2] Component.config.Parse.language
[20:14:36] : [Step 2/2] Component.config.Parse.layout-os
[20:14:36] : [Step 2/2] Component.config.Parse.layout-gui
[20:14:36] : [Step 2/2] Component.config.Parse.init
[20:14:36] : [Step 2/2] Component.config.info
[20:14:36]W: [Step 2/2] Component.config.init
[20:14:36]W: [Component.config.init] echo
[20:14:36]W: [echo] checkout_directory == /Developer/teamcity/work/95e96ae48e9bf59e
[20:14:36]W: [echo] output_directory == /Developer/teamcity/work/95e96ae48e9bf59e/output
[20:14:36]W: [echo] scratch_directory == /Developer/teamcity/work/95e96ae48e9bf59e/scratch
[20:14:36]W: [echo] tmp_directory == /Developer/teamcity/work/95e96ae48e9bf59e/tmp
[20:14:36]W: [echo]
[20:14:36]W: [echo] files_directory == /Developer/teamcity/work/95e96ae48e9bf59e/output/Files
[20:14:36]W: [echo] debugfiles_directory == /Developer/teamcity/work/95e96ae48e9bf59e/DebugFiles
[20:14:36]W: [Step 2/2] Component.init
[20:14:36]W: [Component.init] echo
[20:14:36]W: [echo] === Component SetReplace ===
[20:14:36] : [Step 2/2] Component.init.custom
[20:14:36] : [Step 2/2] Component.BuildMonitor.get-build-code-check
[20:14:36] : [Step 2/2] Component.BuildMonitor.get-build-code
[20:14:36] : [Step 2/2] Component.BuildMonitor.init.custom
[20:14:36] : [Step 2/2] Component.config.rmdir.files_directory
[20:14:36] : [Step 2/2] Component.config.clean.files_directory
[20:14:36] : [Component.config.clean.files_directory] mkdir
[20:14:36] : [mkdir] Created dir: /Developer/teamcity/work/95e96ae48e9bf59e/output/Files
[20:14:36] : [Step 2/2] Component.config.rmdir.scratch_directory
[20:14:36] : [Step 2/2] Component.config.clean.scratch_directory
[20:14:36] : [Component.config.clean.scratch_directory] mkdir
[20:14:36] : [mkdir] Created dir: /Developer/teamcity/work/95e96ae48e9bf59e/scratch
[20:14:36] : [Step 2/2] Component.config.rmdir.tmp_directory
[20:14:36] : [Step 2/2] Component.config.clean.tmp_directory
[20:14:36] : [Component.config.clean.tmp_directory] mkdir
[20:14:36] : [mkdir] Created dir: /Developer/teamcity/work/95e96ae48e9bf59e/tmp
[20:14:36] : [Step 2/2] Component.config.clean.kernel_processes-check
[20:14:36] : [Step 2/2] Component.config.clean.kernel_processes
[20:14:36] : [Step 2/2] Component.config.clean
[20:14:36] : [Step 2/2] Component.clean
[20:14:36] : [Step 2/2] Component.BuildMonitor.collect-metadata-pre-check
[20:14:36] : [Step 2/2] Component.BuildMonitor.collect-metadata-pre
[20:14:36] : [Step 2/2] Component.BuildMonitor.unarchive-artifacts-check
[20:14:36] : [Step 2/2] Component.BuildMonitor.unarchive-artifacts
[20:14:36] : [Step 2/2] Component.BuildMonitor.prebuild
[20:14:36] : [Step 2/2] Component.prebuild
[20:14:36] : [Step 2/2] Component.execute
[20:14:36] : [Step 2/2] Component.postbuild
[20:14:36] : [Step 2/2] Component.artifacts.check-custom
[20:14:36] : [Step 2/2] Component.artifacts.custom
[20:14:36] : [Step 2/2] Component.Artifacts.postbuild
[20:14:36] : [Step 2/2] Component.Artifacts.init
[20:14:36] : [Step 2/2] Component.Artifacts.check-tar
[20:14:36]E: [Step 2/2] Component.Artifacts.tar
[20:14:36]W: [Component.Artifacts.tar] echo
[20:14:36]W: [echo] Creating Internal_SetReplace_Component_Build_Files.tar.gz...
[20:14:36]E: [Component.Artifacts.tar] fail
[20:14:36]E: [fail] /Developer/teamcity/work/95e96ae48e9bf59e/output/Files is empty. (How did *that* happen?)
[20:14:36]W: [Step 2/2] Process exited with code 1
[20:14:36]W: [Step 2/2] Ant output
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.BuildMonitor.unarchive-artifacts-check:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.BuildMonitor.unarchive-artifacts:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.BuildMonitor.prebuild:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.prebuild:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.execute:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.postbuild:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.artifacts.check-custom:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.artifacts.custom:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.Artifacts.postbuild:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.Artifacts.init:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.Artifacts.check-tar:
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Component.Artifacts.tar:
[20:14:36]W: [Ant output] [echo] Creating Internal_SetReplace_Component_Build_Files.tar.gz...
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] BUILD FAILED
[20:14:36]W: [Ant output] /re/tools/antlib/artifacts/Artifacts.xml:75: /Developer/teamcity/work/95e96ae48e9bf59e/output/Files is empty. (How did *that* happen?)
[20:14:36]W: [Ant output]
[20:14:36]W: [Ant output] Total time: 2 seconds
[20:14:36]E: [Step 2/2] Process exited with code 1 (Step: NativeLibrary.build (Ant))
[20:14:36]E: [Step 2/2] Step NativeLibrary.build (Ant) failed
[20:14:36] : Publishing internal artifacts (1s)
[20:14:37] : [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[20:14:37] : [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[20:14:36]W: Publishing artifacts (1s)
[20:14:36] : [Publishing artifacts] Collecting files to publish: [+:output/**, -:output/Files]
[20:14:36]W: [Publishing artifacts] Artifacts path 'output/**' not found
[20:14:36]W: [Publishing artifacts] Artifacts path 'output/Files' not found
[20:14:38] : Build finished
|
@maxitg The RE failure you pasted is in all likelihood because of the deletion of the XML snippet that copied the Kernel/ files into the build directory (which we believed to b e pointless!). It may actually still be pointless, and RE needs to be informed that they can remove the corresponding component so that it doesn't fail. Also, I've opened a PR to fix the typo. |
## Changes * `./performanceTest.wls` got broken after the introduction of in-place builds #529. This PR fixes it. * As a result, `./performanceTest.wls` no longer installs any paclets, and tests everything in-place. * Also added `./performanceTest.wls master @Head 2` to CI to prevent the script from breaking in the future, and to have an easy access to performance results. ## Comments * Having it in CI causes a ~30 seconds slowdown. * I'm only running 2 tests instead of the default 5 to minimize that slowdown. * CI won't fail if there is a slowdown. The purpose of this PR is to test the script itself, not its output. ## Examples * Check Performance Test in CI: https://app.circleci.com/pipelines/github/maxitg/SetReplace/1367/workflows/a36a3d08-0121-4b76-822b-457b8b680c1d/jobs/1825.
This PR changes the way we build the library (and indeed the entire paclet) to an "inplace model". This means things happen within the git tree only, and we don't copy source code around, build libraries into temporary directories, etc. The benefit is there is much less "state" to keep track of.
Some highlights of this PR are:
Get["~/git/SetReplace/Kernel.init.m"]will (when DevUtils is available) cause the C++ library to be rebuilt if necessary.pacletfiles (or install them), since it doesn't really have any benefits. You can just run Get, as mentioned above. You don't even need to./build.wlsany more, since simply getting the package will do a rebuild when required. Paclet files can still be produced by running./pack.wls, and installed by running./install.wls.BuildLibSetReplace[]andCreateSetReplacePaclet[]This is ready for code review.
Note this incorporates the changes from #526
This change is