-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can we upgrade to pymc (v4) and astrea (previously theano-pymc)? #153
Comments
I remember the pymc3 system we have set up is very fragile, I distinctly recall that we had an issue using virtual environments with pymc3 due it its theano requirements which the hub did not let be installed. That means that using virtual environments will likely not be feasible and this replacement will likely override any pymc3 work we are pursuing. We should test this on our current system before implementing fully. |
This needs exploration |
The image is now built (in 20 min) and testable in a new option when spawning a server. To test against it using a more powerful server one can also visit https://hub.jupytearth.org/services/configurator/ and specify it there temporarily just before starting the server.
|
So far, this didn't pan out @abbyazari so its not worth testing on your end yet. I'm trying three more builds and hoping to have a version relevant for you to test soon. (The previous everything is working image b4aa089)
|
Thank you @consideRatio for pursuing this! I am using pymc3 in the Hub and even when there is a new version, I think it will be wise to keep the original pymc3 version for now. If possible, I can make a test and run the code I have with the new pymc4 and see that there are no major changes. Would it be possible to keep pymc3 for now but have a new environment with pymc4 to test our code? Thank you! |
@facusapienza21 @abbyazari can you clarify how you are using pymc3 currently? Are you always activating the conda environment shared/envs/pymc3 ( The key point here in my mind is that if you are not using pymc3 from the base image, where both When I trial this code from #104 (comment), it doesn't work. But, if I either import numpy as np
import pymc3 as pm
# True parameter values
alpha, sigma = 1, 1
beta = [1, 2.5]
# Size of dataset
size = 100
# Predictor variable
X1 = np.random.randn(size)
X2 = np.random.randn(size) * 0.2
# Simulate outcome variable
Y = alpha + beta[0] * X1 + beta[1] * X2 + np.random.randn(size) * sigma
basic_model = pm.Model()
with basic_model:
# Priors for unknown model parameters
alpha = pm.Normal("alpha", mu=0, sigma=10)
beta = pm.Normal("beta", mu=0, sigma=10, shape=2)
sigma = pm.HalfNormal("sigma", sigma=1)
# Expected value of outcome
mu = alpha + beta[0] * X1 + beta[1] * X2
# Likelihood (sampling distribution) of observations
Y_obs = pm.Normal("Y_obs", mu=mu, sigma=sigma, observed=Y) @facusapienza21 if you want to trial something when |
To clarify, I'm not in any way or form suggesting that we modify If you don't use it, I figure we could even remove it from there. I think that installation of pymc3 is malfunctioning based on testing with the example above. |
@consideRatio we are using the pymc3 kernel that we set up a few months back as a solution to the pymc3 issues, I do not believe we ever got pymc3 to work in any other way except the kernel. I am unsure if this has something to do with the image... |
@abbyazari okay thanks for the clarification which I read as that you are using the python kernel within the pymc3 conda environment! Since I keep failing to get pymc to work directly as part of the base image, I'll trial removing pymc3 from the base image and see if the pymc3 conda environment keeps working as before. |
We are using the python kernel called pymc3, more details are on the slack message (including a screenshot). Let me know when you want me to test out anything in our workflow! I can hop on the hub and see if our code still runs etc. |
@abbyazari thanks for the clarification! Okay I'll go ahead and then cleanup pymc that is installed and has never worked in the base conda environment and docker image and validate the dedicated conda environment in shared/envs/pymc3 still work. Given that, you are in full control of if you want to transition to pymc (v4) from pymc3 etc and it won't impact the maintenance of the base image if you do or don't. Thank you for your help understanding the situation Abby and Facu! |
@facusapienza21 @abbyazari I have concluded that a reason for our image size has grown immensively (now 21GB!), is partially the pymc3 installation as it forces a downgrade of numpy and scipy. If we would use the new version of pymc called just pymc (version 4+), we won't run into that downgrade issue.
My question becomes, could we drop pymc3 entirely and go for pymc (version 4) instead? And, can we replace theano-pymc with aesara that seem to be the stand in replacement? See the notes here: https://github.com/pymc-devs/pymc/blob/main/RELEASE-NOTES.md.
The text was updated successfully, but these errors were encountered: