Function to return the unix timestamp form of a given timestamp

[rwblokzijl]

Terraform Version

Terraform v1.2.8

Use Cases

Certain data sources require a timestamp as input. For example: google_service_account_jwt optionally takes an exp claim in its json payload.

Attempted Solutions

Currently this is the only way i manage to do it:

data "external" "time" {
  program = ["bash", "-c", "jq -n --arg data \"$(date +%s)\" '{\"data\":$data}'"]
}

locals {
  now = tonumber(data.external.time.result["data"])
}

The problem is that is only works on linux and depends on jq being installed.

Proposal

Add the option for the formatdate function to convert to unix time.

References

No response

[apparentlymart]

Hi @rwblokzijl! Thanks for sharing this feature request.

The implementation of formatdate actually lives in a third-party library, but conveniently the third party is actually just me wearing a different hat, so I can temporarily put my other hat on for the sake of this response.

Someone recently requested the same thing with the intention of using it in Packer, over in zclconf/go-cty#130. As you can see in that issue, my decision was that Unix timestamps are outside of formatdate's scope because unix timestamps are not typically used as part of a human-readable timestamp, but rather are used instead of a human-readable timestamp.

In that issue I proposed that downstream applications which need Unix timestamps should offer separate functions tailored to that purpose, separately from formatdate. I don't personally have interest in maintaining such a function in upstream go-cty, but there's no reason why Packer and Terraform could not offer such a function. Indeed, HashiCorp has a shared library of functions that both Terraform and Packer can use, separately from the ones I maintain upstream in go-cty.

---

Now putting my Terraform maintainer hat 🎩 back on: I think the Terraform team here must decide first if it agrees with my upstream decision; the Terraform team could potentially fork formatdate if we collectively disagree with the upstream me. But if not, I think this amounts to a request for a new function in Terraform which returns a Unix timestamp typed as a number.

As some background context to help inform that decision, we originally selected an RFC3339-based syntax as Terraform's canonical convention for representing timestamps because it seemed to strike a good balance between being compact and being human-readable for inclusion in plan output. Unix timestamps are more compact but most humans cannot easily understand what wallclock time a particular timestamp represents without the help of tooling.

Our position so far has therefore been that providers designed for use with Terraform should be built to accept and produce RFC3339, even if the upstream system they represent uses a different format. This then allows Terraform providers and built-in functions to compose together well, without explicit format conversions.

However, any convention like this runs into the same challenge: not everything a Terraform configuration is integrating with has a first-class Terraform provider, and in such cases the module itself must implement the translation between Terraform's conventions and the external system's conventions. This seems like a variation of that situation: there is a Terraform provider here that could in principle adapt the timestamp format, but the provider here is just passing through a JSON payload verbatim without actually interpreting it, and the recipient of that JSON payload isn't part of Terraform and has its own conventions. It does therefore seem necessary to generate a Unix timestamp here despite our usual compromise/convention of using RFC3339 for readability.

When considering this we should also take into account whether this function seems like a good candidate for being a provider-contributed function, once there is a mechanism for providers to bring their own functions (#31225). That could either mean a function specifically for generating Unix timestamps as described above, or it could mean something more specialized for google_service_account_jwt which lives inside the Google Cloud Platform provider.

(The Packer team has a similar feature request at hashicorp/packer#11945. We should communicate with them to try to make the same decision in both products unless we have a strong reason for them to differ.)

[rwblokzijl]

Thank you @apparentlymart for taking the time to provide such a detailed answer. I appreciate the context and different perspectives.

Wearing the only hat I have, my user cap 🧢, I would like to share my perspective:

While I do think the google_service_account_jwt should support the timestamp as part of the resource, I would personally also advocate for a rich set of tools within terraform. I have only been working with Terraform for 1.5 years, but during this time I have noticed that the bulk of my time is not spent coding and designing infra, but rather dealing with the bugs and shortcomings of Providers, the SDKs they rely on, the underlying APIs, and the architecture and feature sets of the cloud services.

Terraform is the main tool we have for dealing with these issues. It is the glue that connects a wide range of APIs and providers of varying design philosophies and quality. When terraform does not provide proper tools to deal with an issue, we're stuck with hacky solutions, waiting for github issues at providers, or just not being able to solve the problem at all.

If I allow myself to dream, I imagine terraform with the possibility to write and import user defined functions. I think a functional extention to terraform would work very well with its declarative nature. If neccessary seperating pure and impure functions (though i think that should be the users responsibility).

Overall I love terraform. I think its declarative nature is a beautiful abstraction over REST apis. I can't wait to see how it develops into the future.

[ag-TJNII]

I also need a unix stamp. I need a timestamp that I can do some math on in a locals {} block to generate a resource input structure. time_static has a unix output, but for this case I actually want the current time as I plan to use this for automatic credential rotation. Unix stamps are very common, while I appreciate the human-friendly default of RFC3339 the lack of a unix stamp is a gap when we need a machine-friendly timestamp we can manipulate in the DSL. Some functions to go from a RFC3339 timestamp to a unix timestamp and vice-versa would be handy.

[mikew3432]

Having 'rfc3339_to_unix()' and 'unix_to_rfc3339()' functions would enable use of arithmetic operators and comparison operations on dates and times for scheduling puposes. E.g. calculate desired start and end windows, based on open ended repeating intervals from a base date time reference. Especially when crossing year boundaries.

One of those math operators is of course the % operator (modulo arithmetic).

Having come up against difficulties and limitations with "time_rotating" resource, timeadd() function, regex, formatdate() - it shouldn't be this hard. Rfc3339 exists purely for readability and we shouldn't have to resort to arcane code to do basic maths.

[apparentlymart]

Hi @mikewoodd3432,

While it is true that you can potentially treat a Unix timestamp as a number and do arithmetic to it, it has been my experience that systems which do that inevitably have weird edge cases due to their unawareness of calendar and of details like leap seconds. I'm not meaning to suggest that this is true in all cases or true in your case, but if more complex date and time manipulation were the use-case I expect we'd aim to meet it by providing time and calendar aware functionally specific to timestamps, rather than encouraging the naive use of integer arithmetic on timestamps.

I expect that once we can support functions contributed by providers the hashicorp/time provider would be a natural home for those functions, because that provider already has awareness of calendar concerns built into it. (Calendar-aware calculations are out of the scope of the core language today, which is why timeadd only works in units that don't require calendar awareness. External providers are not subject to that scope, though.)

If we were to add a function for generating Unix timestamps then I expect it would be only to solve the original stated problem of integrating with a system that can only accept that as its input format, with the assumption that the resulting timestamp will just be passed verbatim and not used as part of other calculations. At the moment though it seems like this is niche enough that we'll probably delegate it also to a provider (perhaps hashicorp/time again) rather than incorporating it into the main language as a built-in.

(There are many other requests for specialized functions like this and for our planning and prioritization purposes we are considering them all together to be indirect justification for #31225, which would make provider-contributed functions possible. In the meantime providers can achieve a similar effect with a considerably more awkward syntax using data sources that perform only local calculations.)

[mikew3432]

I expect that once we can support functions contributed by providers the hashicorp/time provider would be a natural home for those functions, because that provider already has awareness of calendar concerns built into it. (Calendar-aware calculations are out of the scope of the core language today, which is why timeadd only works in units that don't require calendar awareness. External providers are not subject to that scope, though.)

Thanks @apparentlymart ok I admit you're making some sense, but in reply I would say, terraform appears to have gotten providers to standardise (nicely) on RFC3339 as the standard date-time data type, and could do with providing a bit more filler for the current gaps? Someone might debate that hashicorp/time is just another consumer of RFC3339. And although it emits unix time which is at odds with "the standard", I feel unix time is still valid and rfc3339_to_unix and unix_to_rfc3339 are good functions for cross-platform and "completeness" reasons (is completeness a taboo in software development these days??). But if you are aiming to future proof by avoiding unix time altogether, fair play for thinking ahead...

In summary I do feel strongly the "Data and Time functions" category is rather wanting. Lack of calendar aware "dateadd" or equivalent enhancement to "timeadd" is glaring (understandable but glaring). Calendar aware functions would be my first wish. Unix time would be nice.

[apparentlymart]

In case it's interesting to anyone, support for provider-contributed functions is planned for the forthcoming Terraform v1.8, and so once the plugin framework is updated to help providers expose such functions it would become technically possible to offer a variety of time and date related functions in the hashicorp/time provider.

I don't belong to the team that is ultimately responsible for that provider and so I cannot decide on their behalf that such things are in scope there, but it might be worth starting a discussion in an issue over there about it, if anyone who was following this issue is still interested.

[bflad]

The team maintaining the hashicorp/time provider intends to release a rfc3339_parse() function that will work with Terraform 1.8 and later. That function will return an object with one of the attributes being unix (representing the number of seconds elapsed since January 1, 1970 UTC). The draft implementation can be found at hashicorp/terraform-provider-time#280 and its release will occur before Terraform 1.8 GA is released.

If you have any other RFC3339 or standards-based time functionality requests, please feel free to create a feature request in that provider's issue tracker: https://github.com/hashicorp/terraform-provider-time/issues

[crw]

Thank you for your continued interest in this issue.

Terraform version 1.8 launches with support of provider-defined functions. It is now possible to implement your own functions! We would love to see this implemented as a provider-defined function.

Please see the provider-defined functions documentation to learn how to implement functions in your providers. If you are new to provider development, learn how to create a new provider with the Terraform Plugin Framework. If you have any questions, please visit the Terraform Plugin Development category in our official forum.

We hope this feature unblocks future function development and provides more flexibility for the Terraform community. Thank you for your continued support of Terraform!

[austinvalle]

Hey everyone 👋🏻 ,

Following up on @bflad's #31751 (comment), the hashicorp/time provider has just released this new rfc3339_parse function in v0.11.

You can test this out now with Terraform v1.8.0-beta1 or wait for the upcoming v1.8.0 release.

# Configuration using provider functions must include required_providers configuration.
terraform {
  required_providers {
    time = {
      source  = "hashicorp/time"
      version = "0.11.1"
    }
  }
  # Provider functions require Terraform 1.8 and later.
  required_version = ">= 1.8.0"
}

locals {
  plan_time = provider::time::rfc3339_parse(plantimestamp())
}

output "unix" {
  value = local.plan_time.unix
}

$ terraform plan

Changes to Outputs:
  + unix = 1710196634

If you run into any problems with the new function, please create an issue over in the time provider issue tracker, thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.