Recommended patterns for custom k8s interaction

[benjaminleighton]

Hi I'd like to use qhub commands to init render and deploy qhub with terraform customisations but running these commands overwrites any modifications I make to terraform configuration. Applying back customisations from git branches might work but is this the best way?

Related, I'd like to run lower level terraform commands to deploy or otherwise interrogate the infrastructure created by qhub. I'm able to guess some of this and running terraform commands in various directories works but would it be worthwhile documenting the full sequence in which qhub itself runs these commands?

Thanks

[pierrotsmnrd]

Hi @benjaminleighton

Which kind of terraform customisations are you trying to do ?
I'm interested by your initial need, your use-case may be relevant for other people.

I'm pinging @costrouc and @tylerpotts, two of our QHub engineers, who may provide an answer to your technical question.

[benjaminleighton]

One of the things I was trying to do was set a --verbose flage on the conda-store.py command line. This is related to my other question #936 (comment) but another thing I'd like to do is assign an s3 read role to the eks-node-group-role that qhub creates. I've been able to do that manually but it would be great to have it as infrastructure as code.

I imagine there are many use cases for installing various additional kubernetes datascience related components https://artifacthub.io/packages/search?ts_query_web=&ts_query=machine-learning&sort=relevance&page=1 or perhaps other workflow engines as an alternative to prefect.

[benjaminleighton]

I've a prototype approach to this which is to store the qhub terraform configuration in git and maintain a branch that minimally adds additional terraform outputs, kind of like hooks. Then, in another git repo I have a seperate terraform configuration that loads the provisioned qhub terraform state from my remote s3 backend and using the defined outputs modifies the state.

So far I've tested the by exposing the cluster iam role like:

output "cluster_aws_iam_role" {   
  description = "aws iam role"
  value          = module.kubernetes.cluster_aws_iam_role
}

and then in my other terraform configuration referencing it like

data "terraform_remote_state" "common" {
  backend = "s3"
  config = {
      bucket         = "my-qhub-terraform-state"
      key            = "terraform/my-qhub.tfstate"
      region         = "ap-southeast-2"
      encrypt        = true
      dynamodb_table = "my-qhub-terraform-state-lock"
  }
}
resource "aws_iam_role_policy_attachment" "cluster-policy" {
  count = length(local.cluster_policies)
  policy_arn = local.cluster_policies[count.index]
  role       =  data.terraform_remote_state.common.outputs.cluster_aws_iam_role 
}

I'm hoping this approach will be pretty robust to changes in qhub terraform structure and that worse case it will be a matter of rewiring my branch to work against a new master. Best case I'll qhub deploy destroying my existing infrastructure and qhub terraform config and regenerating a clean version. Then I'll sync that to my git master and remerge my branch to reenable hooks. Then I'll reapply my other terraform customisations using remote state.

I also noticed I can do qhub deploy --disable-render which I think answers the question of how to replicate the qhub terraform commands without deleting the existing terraform configuration, I don't think it will help propogate changes to qhub-config.yaml. I think I'm safest to treat the qhub terraform configuration as emphemeral there being no built in mechanism to rerender but maintain direct, aka merge, custom changes to terraform config.

[Adam-D-Lewis]

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.