Builtin SHA256 hashing #6977

MatthewJohnHeath · 2024-08-09T17:43:50Z

Expose the functionality of Zig's std.crypto.hash.sha2.Sha256 through a pure-functional interface to Roc.

lukewilliamboswell · 2024-08-14T01:25:53Z

@MatthewJohnHeath I tried to push to your branch but I got permission denied. Made a PR in case that helps.

MatthewJohnHeath · 2024-08-20T09:27:28Z

Hi Luke, Sorry I'm slow to respond. I've been travelling. There are definitely some problems on the branch; it builds but then in the REPL the functions seem to be missing (although triggering a different error to *actually* non-existent function names). I created the initial PR mostly in order to get feedback on what might be wring (I was also expecting to be able to get more time online these couple of weeks). I think I am going to need some help to get it working.

…

On Wed, 14 Aug 2024 at 02:26, Luke Boswell ***@***.***> wrote: @MatthewJohnHeath <https://github.com/MatthewJohnHeath> I tried to push to your branch but I got permission denied. Made a PR in case that helps. — Reply to this email directly, view it on GitHub <#6977 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEZKVQQ7LL7GIFEDIJSQHBTZRKW3PAVCNFSM6AAAAABMI4AGLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBXGY2TMNZVHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Matthew Heath <http://mattheath.wordpress.com>

smores56 · 2024-08-30T19:05:53Z

The shape of this PR is a good start, but I'm seeing a potential issue if we add other cryptographic code to this module, which would be quite unsurprising. You have emptySha256, addBytes, and digest. The first is named based on the hash, but the second and third are nominally agnostic to the hash, but only work for SHA-256. Could you rename them to addBytesSha256 or something similar?

smores56 · 2024-08-30T19:06:12Z

Otherwise, I'm attempting to get things running

smores56 · 2024-08-30T20:00:53Z

Okay, I'm almost done getting this working, just need to coordinate the Zig and Roc builtin types.

smores56 · 2024-08-30T20:53:33Z

The REPL worked for me! We'll still need some tests, but it should be good to go, above comments notwithstanding.

smores56 · 2024-08-30T21:01:41Z

Also, there's no code currently to extract state from the new Digest256 opaque type. Could you add a function or something to do that? Maybe Crypt.digest256ToBytes : Digest256 -> List U8?

MatthewJohnHeath · 2024-08-31T15:03:19Z

Also, there's no code currently to extract state from the new Digest256 opaque type. Could you add a function or something to do that? Maybe Crypt.digest256ToBytes : Digest256 -> List U8?

I was thinking of (only) having the type implement Eq. Can you think of use cases where the individual bytes would be needed?

smores56 · 2024-09-02T09:42:09Z

I was thinking of (only) having the type implement Eq. Can you think of use cases where the individual bytes would be needed?

If I want to store the hash for future comparison. Not that using SHA-256 would be recommended over Argon2, but the idea holds. There are a few algorithms out there that use SHA-256 digest bytes for computation (Wikipedia), at least UUID v5 uses SHA-1 in that way.

smores56 · 2024-09-09T17:18:58Z

Tests are currently failing. @MatthewJohnHeath do you have time to implement the above fixes any time soon? If not, I can help, but it's your PR, so I'll defer to you.

MatthewJohnHeath · 2024-09-09T18:08:05Z

Tests are currently failing. @MatthewJohnHeath do you have time to implement the above fixes any time soon? If not, I can help, but it's your PR, so I'll defer to you.

@smores56 It's not really clear to me why the build is hanging (which seems to be what is happening). I will try a few things, but might need to ask for help

MatthewJohnHeath · 2024-09-09T21:13:50Z

@smores56 It's building now. Hopefully I can do those last 2 renamings of functions you asked for tomorrow.

MatthewJohnHeath · 2024-09-11T17:32:40Z

If this gets through the tests, would it be to merge and then do docs and tests? Or do those need to be there first?

smores56 · 2024-09-11T18:37:32Z

It's always preferable to do at least testing, but ideally also docs, in the same PR. We'd usually avoid them if they're going to make the PR too big, or if they're blocking us merging an important change/bug fix. Since it's neither of those, I'd prefer if we can get them in this change. I'm happy to help with either the tests or docs!

smores56 · 2024-09-12T20:38:14Z

crates/compiler/builtins/roc/Crypt.roc

+Sha256 := { location : U64 }
+
+## Represents the digest of soem data produced by the SHA256 cryptographic hashing function as an opaque type.
+## `Digest256`implements the `Eq` ability.


nitpick: Digest256(space)implements ...

also, consider adding an empty doc comment between these two doc comments, that'll follow the "one-line summary on top, context beneath" structure for docs.

smores56 · 2024-09-12T21:25:20Z

crates/compiler/builtins/roc/Crypt.roc

+Digest256 := { firstHalf : U128, secondHalf : U128 } implements [Eq]
+
+## Returns a `Sha256` to which no data have been added.
+emptySha256 : {} -> Sha256


I'd recommend "An empty SHA-256 digest."

And it's "to which no data has been added." https://www.thesaurus.com/e/grammar/data-is-or-data-are/

The word shouldn't be "digest", I think. We should save that for the Digest256 returned after "finalize" has been called on the zig object.
I am struggling to find a good term to describe what the Sha256 object does. My mental model of it is as the state of the algorithm, but that doesn't seem useful wording here. Something like "An empty SHA-256 hasher", maybe?

I agree that data (EDIT)is in this context; I've had too much time in settings where I get corrected the other way.

smores56 · 2024-09-13T08:43:58Z

crates/compiler/builtins/roc/Crypt.roc

+import Result
+import Str
+
+## Represents, as an opaque type, the state of a SHA256 cryptographic hashing function, after some (or no) data have been added to the hash.


nitpick: it being an opaque type is implied, I don't think we need to explicitly mention that here.

This reverts commit 77b9223.

smores56 · 2024-10-08T08:34:48Z

So I made progress on running things. Tests are running for the builtins, and there is some LLVM issue I'll have to investigate tomorrow:

smores@smortress:~/dev/roc$ cargo run --release --bin roc -- test crates/compiler/builtins/roc/Crypt.roc
    Finished release [optimized] target(s) in 0.16s
     Running `target/release/roc test crates/compiler/builtins/roc/Crypt.roc`
Call parameter type does not match function signature!
%list.RocList %"#arg2"
 ptr  %call_builtin = call i64 @roc_builtins.crypt.sha256AddBytes(i64 %"#arg1", %list.RocList nocapture nonnull readonly byval(%list.RocList) %"#arg2"), !dbg !1635


Function "Crypt_sha256AddBytes_ade2dc9e385e74c61c8d36210907131d7823a7fe8d06c7bd978c1f46a9d3830" failed LLVM verification in NON-OPTIMIZED build. Its content was:

define internal fastcc i64 @Crypt_sha256AddBytes_ade2dc9e385e74c61c8d36210907131d7823a7fe8d06c7bd978c1f46a9d3830(i64 %"#arg1", %list.RocList %"#arg2") !dbg !1634 {
entry:
  %call_builtin = call i64 @roc_builtins.crypt.sha256AddBytes(i64 %"#arg1", %list.RocList nocapture nonnull readonly byval(%list.RocList) %"#arg2"), !dbg !1635
  ret i64 %call_builtin, !dbg !1635
}
thread 'main' panicked at crates/compiler/gen_llvm/src/llvm/build.rs:5810:21:
😱 LLVM errors when defining function "Crypt_sha256AddBytes_ade2dc9e385e74c61c8d36210907131d7823a7fe8d06c7bd978c1f46a9d3830"; I wrote the full LLVM IR to "/tmp/nix-shell.7F740v/test.ll"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

smores56 · 2024-10-21T21:00:28Z

I've been distracted, but I'll be looking at this again today. I might have to ask for help from @bhansconnect since I'm not much of an LLVM guy, which seems to be the last issue here. Needless to say, @MatthewJohnHeath there's nothing left that you need to do on this PR. If you have any changes you want to make, feel free, but otherwise I'm planning on merging as soon as I fix the LLVM issue.

Signed-off-by: Sam Mohr <[email protected]>

bhansconnect · 2024-11-18T03:44:21Z

crates/compiler/builtins/bitcode/src/crypt.zig

+};
+
+pub fn emptySha256() callconv(.C) Sha256 {
+    const allocation = utils.allocateWithRefcount(@sizeOf(sha2.Sha256), @alignOf(sha2.Sha256), false);


Why is this allocated with a refcount?

bhansconnect · 2024-11-18T03:45:08Z

crates/compiler/builtins/bitcode/src/crypt.zig

+const utils = @import("utils.zig");
+
+const Sha256 = extern struct {
+    location: [*]u8,


Why the extra pointer indirection. I think we should just expose the zig sha256 directly. Cast it to an array probably so it can potentially be inline instead of requiring an allocation and extra indirection.

On the roc side can just make it a tuple of integers that end up being sized the same as the zig type.

Just add a comptime assert on the size here. That way on zig update we can correct. I think it is currently 112 bytes before padding. So a tuple of 7 u128s should work to represent it on the roc size. On this zig side, it should be a [7]u128 align(16)

bhansconnect · 2024-11-18T03:47:26Z

crates/compiler/gen_llvm/src/llvm/lowlevel.rs

+        }
+        CryptSha256Digest => {
+            arguments!(sha);
+            call_bitcode_fn(env, &[sha], bitcode::CRYPT_SHA256_DIGEST)


I think you need to use call_list_bitcode_fn and put the list as the first argument to the function. I think that should fix the current bug.

MatthewJohnHeath force-pushed the main branch 3 times, most recently from 344ac11 to 4da18e6 Compare August 30, 2024 14:49

MatthewJohnHeath force-pushed the main branch 8 times, most recently from f7fc408 to c9ab4e5 Compare September 4, 2024 20:55

MatthewJohnHeath force-pushed the main branch 2 times, most recently from b2f8ca2 to fcee861 Compare September 11, 2024 17:25

MatthewJohnHeath force-pushed the main branch from 29db7e0 to aada47b Compare September 12, 2024 21:08

smores56 reviewed Sep 13, 2024

View reviewed changes

smores56 previously approved these changes Sep 13, 2024

View reviewed changes

MatthewJohnHeath and others added 21 commits October 2, 2024 19:39

WIP filling in missing match cases

622e1e0

Filling in missing matches before rebase

03fd10d

Fixed rebase

63228ea

Adds case staements for lowlevel to compile

1c4c465

switch out bad pointer rep

ae43b93

Fix Crypt builtin

aef515c

Fix formatting

e76f880

Adding digest inspectors

2123b2c

Revert "Adding digest inspectors"

5128ce3

This reverts commit 77b9223.

Adds functions to access digest

8aaa397

add missing module import

15af72b

Fixes structured binding in digest256ToBytes

f4f61f9

Name changes

c391566

Tidy formatting

7699493

Attempt to fix formatting

a00ab4a

Ran fomatter

bab7c9d

tests

ff0703b

Docs for exposed functions and types

5054c8b

Fix spelling and zig fmt

1ea8bf1

Response to review on comment

38f0c63

Fixes typo in test

e092345

MatthewJohnHeath force-pushed the main branch from 208fc98 to e092345 Compare October 2, 2024 18:40

smores56 added 2 commits October 7, 2024 22:37

update mono tests

9176bcb

Merge branch 'main' into MatthewJohnHeath/main

3dc1b65

Merge branch 'main' into main

5879947

Signed-off-by: Sam Mohr <[email protected]>

smores56 enabled auto-merge November 5, 2024 06:04

Fix broken test

1610708

bhansconnect reviewed Nov 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Builtin SHA256 hashing #6977

Builtin SHA256 hashing #6977

MatthewJohnHeath commented Aug 9, 2024 •

edited

Loading

lukewilliamboswell commented Aug 14, 2024

MatthewJohnHeath commented Aug 20, 2024 via email

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

MatthewJohnHeath commented Aug 31, 2024

smores56 commented Sep 2, 2024

smores56 commented Sep 9, 2024

MatthewJohnHeath commented Sep 9, 2024

MatthewJohnHeath commented Sep 9, 2024

MatthewJohnHeath commented Sep 11, 2024

smores56 commented Sep 11, 2024

smores56 Sep 12, 2024

smores56 Sep 12, 2024

MatthewJohnHeath Sep 13, 2024 •

edited

Loading

smores56 Sep 13, 2024

smores56 commented Oct 8, 2024

smores56 commented Oct 21, 2024

bhansconnect Nov 18, 2024

bhansconnect Nov 18, 2024

bhansconnect Nov 18, 2024

bhansconnect Nov 18, 2024

Builtin SHA256 hashing #6977

Are you sure you want to change the base?

Builtin SHA256 hashing #6977

Conversation

MatthewJohnHeath commented Aug 9, 2024 • edited Loading

lukewilliamboswell commented Aug 14, 2024

MatthewJohnHeath commented Aug 20, 2024 via email

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

smores56 commented Aug 30, 2024

MatthewJohnHeath commented Aug 31, 2024

smores56 commented Sep 2, 2024

smores56 commented Sep 9, 2024

MatthewJohnHeath commented Sep 9, 2024

MatthewJohnHeath commented Sep 9, 2024

MatthewJohnHeath commented Sep 11, 2024

smores56 commented Sep 11, 2024

smores56 Sep 12, 2024

Choose a reason for hiding this comment

smores56 Sep 12, 2024

Choose a reason for hiding this comment

MatthewJohnHeath Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

smores56 Sep 13, 2024

Choose a reason for hiding this comment

smores56 commented Oct 8, 2024

smores56 commented Oct 21, 2024

bhansconnect Nov 18, 2024

Choose a reason for hiding this comment

bhansconnect Nov 18, 2024

Choose a reason for hiding this comment

bhansconnect Nov 18, 2024

Choose a reason for hiding this comment

bhansconnect Nov 18, 2024

Choose a reason for hiding this comment

MatthewJohnHeath commented Aug 9, 2024 •

edited

Loading

MatthewJohnHeath Sep 13, 2024 •

edited

Loading