Julia script creation system custom images

[amartinhuertas]

Hi @fverdugo, @santiagobadia ... I think the script for compiling Gridap sysimages is ready to be merged into master. Note that there is a README.md file with up-to-date usage instructions, caveats, etc.

[codecov-io]

Codecov Report

Merging #432 into master will increase coverage by 1.23%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #432      +/-   ##
==========================================
+ Coverage   87.34%   88.57%   +1.23%     
==========================================
  Files         158      158              
  Lines       11170    10216     -954     
==========================================
- Hits         9756     9049     -707     
+ Misses       1414     1167     -247     

Impacted Files	Coverage Δ
src/FESpaces/FECellBases.jl	33.33% <0.00%> (-23.81%)	⬇️
src/Arrays/SubVectors.jl	80.00% <0.00%> (-20.00%)	⬇️
src/Integration/Quadratures.jl	70.00% <0.00%> (-11.82%)	⬇️
src/Arrays/ArrayPairs.jl	81.81% <0.00%> (-11.04%)	⬇️
src/TensorValues/SymFourthOrderTensorValueTypes.jl	64.44% <0.00%> (-10.56%)	⬇️
src/TensorValues/TensorValueTypes.jl	66.66% <0.00%> (-10.15%)	⬇️
src/FESpaces/DirichletFESpaces.jl	90.90% <0.00%> (-9.10%)	⬇️
src/TensorValues/SymTensorValueTypes.jl	56.71% <0.00%> (-8.57%)	⬇️
src/CellData/QPointCellFields.jl	80.85% <0.00%> (-8.44%)	⬇️
src/FESpaces/SingleFieldFEFunctions.jl	92.30% <0.00%> (-7.70%)	⬇️
... and 121 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0975a1a...ba1faa0. Read the comment docs.

[amartinhuertas]

Do we have to update NEWS.md for this addition?

[fverdugo]

Do we have to update NEWS.md for this addition?

it would be nice

[amartinhuertas]

Done!

[amartinhuertas]

Hi @fverdugo ... I guess that we can merge this right? or are we waiting for something I am not aware of?

[fverdugo]

@amartinhuertas perhaps it is a good idea to release 0.14.2. Can you prepare the release?

https://github.com/gridap/Gridap.jl/wiki/How-to-register-a-julia-package#specific-steps-to-registering-gridap

[amartinhuertas]

@amartinhuertas perhaps it is a good idea to release 0.14.2. Can you prepare the release?

Yes, I can do it ... but now I am starting to become concerned about the fact that there are no actually tests for the Julia script I have added ... perhaps I can modify travis.yml such that the image is created and then the subpackage GridapTests is run with the image ... what do u think? Any better idea ?

[fverdugo]

Testing is a good idea!

Now we have 2 stages:

• test (with 3 jobs)
• Documentation (with 1 job)

You want to create and use the system image in the 3rd job in the test stage, right?

There is something that is problematic though. The default sys image creation depends on Tutorials and Tutorials depends on Gridap... It will cause some trouble when developing braking releases... Perhaps in travis, we want to generate the sys image by snooping the tests in GridapTests instead of cloning Tutorials.

@amartinhuertas What do you think?

[amartinhuertas]

Now we have 2 stages:
test (with 3 jobs)
Documentation (with 1 job)

I am aware of the two stages. But not actually of the three jobs (I know almost nothing about travis and associated nomenclature). By jobs, do you mean the three Unix shell commands below the script: section?

    - stage: "test"
      julia: 1.5
      os: linux
      script:
        - echo $PWD
        - julia --project=. --color=yes -e 'using Pkg; Pkg.instantiate()'
        - julia --project=. --color=yes --check-bounds=yes -e '(1,) .== 1; include("test/GridapTests/runtests.jl")'
      after_success: skip

You want to create and use the system image in the 3rd job in the test stage, right?

I want to modify the script: section such that the image is created before the 3rd command, and then passed to the third command.

There is something that is problematic though. The default sys image creation depends on Tutorials and Tutorials depends on > Gridap... It will cause some trouble when developing braking releases... Perhaps in travis, we want to generate the sys image > by snooping the tests in GridapTests instead of cloning Tutorials.
@amartinhuertas What do you think?

I think you are right as usual. Yes, I will do that, I have to modify the create_gridap_image.jl a little bit, but not such a big deal. I will do it!

[fverdugo]

@amartinhuertas Another minor remark. I would not use --check-bounds=no by default in the creation of the sys image. Even in the ideal case of a bug-free Gridap, the user can use the library in a "wrong" way which can lead to seg faults. Instead, I would not pass this flag to julia by default. what do you think?

[fverdugo]

I want to modify the script: section such that the image is created before the 3rd command, and then passed to the third command.

yes this, section defines the 3rd job (or whatever is called) in the test stage. Note that that jobs 1 and 2 on this stage are automatically created by the julia pluggin in travis.

[amartinhuertas]

@amartinhuertas Another minor remark. I would not use --check-bounds=no by default in the creation of the sys image. Even in the ideal case of a bug-free Gridap, the user can use the library in a "wrong" way which can lead to seg faults. Instead, I would not pass this flag to julia by default. what do you think?

I am not sure if I understand this comment.

In particular, where am I using --check-bounds=no by default in the creation of the sys image? Do u actually mean in the README.md documentation? Because AFAIK, I only use that flag in the documentation of the script, and not in the script itself.

I agree that in .travis.yml is not convenient at all to use --check-bounds=no.

[amartinhuertas]

BTW ... not related to this PR, but, when I go to travis in order to observe the progress of the Travis CI jobs, I get the following warning message on top of the page:

Builds have been temporarily disabled for public repositories due to a negative credit balance. Please go to the Plan page to replenish your credit balance or alter your Consume paid credits for OSS setting.

Do u observe the same as well ?

[fverdugo]

Builds have been temporarily disabled for public repositories due to a negative credit balance. Please go to the Plan page to replenish your credit balance or alter your Consume paid credits for OSS setting.

Really ?? 😱😱😱 My last push has passed the CI. I hope it was a temporary bug in travis. 🤞

[amartinhuertas]

Really ?? screamscreamscream My last push has passed the CI. I hope it was a temporary bug in travis. crossed_fingers

I keep getting the abovementioned error message in GridapDistributed.jl's Travis CI page. There have been recent changes into Travis pricing:

https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

In particular, the text under the header "Building on a public repositories only" precisely documents the behaviour that I am currently observing.

[fverdugo]

Yes you are right! I also have seen the same problem. What these 1000 credits mean? We are already out of budget before finishing the first week of the month?

Perhaps it has arrived the time to move to github CI.

[amartinhuertas]

Yes you are right! I also have seen the same problem. What these 1000 credits mean? We are already out of budget before finishing the first week of the month?

As far as I understand from that page, we were automatically put into the free trial plan right at the start of the week. This means that we had 10K credits (approx 1000 CPU minutes). Once these are consumed, we cannot run anymore on Travis if we do not hire one of the paid plans. There is the exception below, though.

• When your credit allotment runs out - we’d love for you to consider which of our plans will meet your needs.
• We will be offering an allotment of OSS minutes that will be reviewed and allocated on a case by case basis. Should you want to apply for these credits please open a request with Travis CI support stating that you’d like to be considered for the OSS allotment. Please include:
  • Your account name and VCS provider (like travis-ci.com/github/[your account name] )
  • How many credits (build minutes) you’d like to request (should your run out of credits again you can repeat the process to request more or discuss a renewable amount)
• Usage will be tracked under your account information so that you can better understand how many credits/minutes are being used

Just guessing, we quickly ran out of these 1000 minutes because they are shared by all projects within the Gridap org. But not totally sure about this, as I was not able to figure out, using the TRAVIS-CI web app interface, how many credits we have already consumed.

Perhaps it has arrived the time to move to github CI.

You mean Github actions, right? There are GitHub-hosted runners where you can run your jobs in. By now, it seems to be free for open source projects (https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration), but this may change in the future. There is also the concept of self-hosted runners (we have two machines in CIMNE Castelldefels that we could use for this purpose), although its use is highly discouraged for public repos, because of the following computer security reasons: https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

I think I am going to migrate GridapDistributed.jl to github actions, and then share with you what I have learnt, ok?