Allow environment configuration override via Java System Properties or Environment Variables

[hbobenicio]

I'm not sure if I got it right but... the only way to configure migrations environments is via a Properties File (for example: ./enviroment/someEnv.properties). Am I correct? (at least I didn't find more on that on the docs page )

If so, it would be also interesting to configure some of the environment properties (or override them) via CLI arguments, Java System Properties or Environment Variables.

for exemple:

Today:
migrate --path=path/to/repo --env=myenv -> requires full configuration on path/to/repo/repositories/myenv.properties

Maybe a more flexible alternative:

(path/to/repo/repositories/myenv.properties without url, username and password properties)

migrate --path=path/to/repo --env=myenv --url=$MYENV_URL --username=$MYENV_USERNAME --password=$MYENV_PASSWORD

or

migrate --path=path/to/repo --env=myenv -Durl=$MYENV_URL -Dusername=$MYENV_USERNAME -Dpassword=$MYENV_PASSWORD

or

env URL=$MYENV_URL USERNAME=$MYENV_USERNAME PASSWORD=$MYENV_PASSWORD migrate --path=path/to/repo --env=myenv

Motivation:
In some deployment environments, a safe approach of storing sensitive data (like db username/password) is via protected environment variables. This way, one would let url, username and password properties out of the environment property file and define it as OS Environment Variable to allow migrations to execute on the CI pipeline.

Sorry for the long description. Any thoughts about it? Is this already possible to be accomplished somehow?

[harawata]

Hi @hbobenicio ,
Thanks for the request!

@h3adache ,
I'm not sure if I understand your comment on #10 correctly, but is this already possible with Maven or Gradle plugin?

[h3adache]

Yes @harawata my development.properties looks something like:

driver=${db.driver}
url=${db.url}
username=${db.username}
password=${db.password}
delimiter=;
changelog=Changelog
auto_commit=false
send_full_script=false

In gradle I use:

processResources {
        filesMatching(['**/*.properties']) {
            expand(project.properties)
        }
    }

Which allows me to replace those values using properties defined in my gradle file (I use a separate environments.gradle file) and also allows me to override using command line (system properties) to the gradle build.

Maven supports something very similar. I switched away from maven a while ago so I don't have the examples but they look pretty much the same on the properties file.

[harawata]

Thank you for the explanation, @h3adache !
So, the substitution occurs when building the project, right?
If the goal is to avoid committing password etc. to VCS, it may be an acceptable solution indeed.

@hbobenicio ,
Does @h3adache 's method meet your requirement?

p.s.
Just curious, are those 'protected environment variables' different from normal environment variables in any way?

[hbobenicio]

In my case, I'm using the migrations-maven-plugin.
I tried many ways to define -Durl, -Dusername and -Dpassword properties in my maven project (inside pom.xml, as command line argument, as the migrations-maven-plugin configuration) without any success. Maybe I'm missing something. Maybe there is a similar way to expand properties with maven like the way h3adache showed with gradle (I'm not a maven expert, as you can see ^^).

Anyway... just thought it would be nice to have support for another direct way to define those properties

BTW, I got it working in my case with a workaround on my CI pipeline like:

echo "url=$URL" >> my-mybatis-environment.properties
echo "username=$USERNAME" >> my-mybatis-environment.properties
echo "password=$PASSWORD" >> my-mybatis-environment.properties
mvn migration:up -Dmigration.env=my-mybatis-environment

@harawata ,
those are just regular environment variables, but with a security layer on branches
https://docs.gitlab.com/ce/ci/variables/README.html#secret-variables
https://docs.gitlab.com/ce/ci/variables/README.html#protected-secret-variables

[harawata]

Migrations or migrations-maven-plugin itself does not provide filtering (i.e. variable substitution).
But Maven's resource plugin does.
https://maven.apache.org/plugins/maven-resources-plugin/examples/filter.html
This should be slightly easier than your current method, I think.
If you had any difficulty, please let me know and I'll try it myself.

And thanks for the links.
I've never had a chance to use GitLab, but it seems like a neat feature.
Hopefully, it will work nicely with the filtering.

[hbobenicio]

thanks for the tip @harawata !
I really appreciate that!
I didn't know about this plugin. It really seems like a better approach, for sure.
Gonna try that this week.

Well then... you guys can tell if this may be a nice desirable feature or if this runs out of the scope of the project. In case of the later, I'm ok about closing this issue.

Once again, thanks for all the tips!

[harawata]

You're welcome!
And we thank you for taking the time to submit the request.

Regarding the enhancement idea, I'm not opposed to it.
It's just that doing it with build tools (i.e. Maven or Gradle) seems to be sufficient, so far.
And it also is a little bit unclear to me how it affects Maven/Gradle plugins (I'm not an expert either).

[h3adache]

Hi @hbobenicio and @harawata! Sorry for the late response. Yes the filtering is exactly what I use.
You can use the filtering along with env/system/even settings.xml values using https://maven.apache.org/pom.html#Properties
If you set up env or system properties in your gitlab ci pipeline (as properties of your runners) then it will automagically use those.
Hope that helps.
The downside of this approach is that your build artifacts will contain those values.
I can see that as a use case for reading those straight from environment properties.

[stephen-bealine]

Hi @h3adache, any chance you could expand on how you use gradle to replace the tokens in your environment file. I am trying to do the same (or at least similar) and having no luck.

I have a ci.properties which contains the following
url=jdbc:postgresql://localhost:5432/${db_name}

Am just wanting to use a dynamic DB name within our CI environment.

Ideally I'd like to use a custom task, which populates the above token with a command line property and then run the migrateUp task. But no matter what I do, the token in not replaced and hence connection fails. This is probably down to my lack of gradle knowledge, so any help would be appreciated.

Think the difference is that you are using environment variables