API: MaxTealSourceBytes to 1M #6031

[scholtz]

MaxTealSourceBytes to 1M

Summary

Solves issue #6031

Test Plan

Test plan not needed. Just increases one constant to little bigger number so that the teal compilation proceed.

[jannotti]

The issue you link to says:

Please increase post body size to the limit where any future build of the contract would rather fail on "teal code too big" instead of the input body is too big.

I don't see any place in go-algorand that such an error ("teal code too big") is returned. Can you point it out? Ideally, there should indeed be a test that shows that such an error is generated.

[scholtz]

@jannotti Just read the #6031 thread pls. You can reproduce the issue when you create teal code longer then 200kb .. Feel free to write test for it.

You can reproduce the issue when you clone https://github.com/scholtz/BiatecCLAMM/tree/feat/update-tealscript and run npm i and npm run test

[jannotti]

@jannotti Just read the #6031 thread pls. You can reproduce the issue when you create teal code longer then 200kb .. Feel free to write test for it.

You can reproduce the issue when you clone https://github.com/scholtz/BiatecCLAMM/tree/feat/update-tealscript and run npm i and npm run test

Since I quoted from that thread in my message, and I was the first commenter on that thread, you may safely assume that I have read it and understand it. Nowhere does that thread explain where the error "teal code to big" might come from, since that text does not appear in go-algorand.

I suspect that the error message in question is somewhere in algokit, which would also explain why you are so reluctant to provide a test case. It is impossible to provide a test case in go-algorand because you are describing a poor interaction between algokit and go-algorand, not a bug in go-algorand. It appears that algokit does not understand that when it receives a "request body too large" response, it should report the error to the user as "teal code to big". That is something that should be changed in algokit.

I do not object to raising the size of programs that algod can assemble, but if the fundamental problem is that algokit does not understand the error in question, we have simply masked the problem until a larger a program is sent. If algokit has a built-in constant that limits the size of teal code to 1M, that chould also be changed in algokit. It would be better for algokit to simply submit what it's given, understand the error "request body too large" and report the size limitation when the error is received.

At any rate, there's work to be done in algokit, and possible also a raise of this constant if we actually think it's reasonable to send such large programs to algod. As is, this change masks the real problem.

[jannotti]

If the previous hand-wavy argument that 200_000 was reasonable has caused problems, then there should be a number that is correct in the sense that it can't cause those problems. But I don't see what those problems were, exactly, nor why 1MB is the number that is correct to solve them.

[scholtz]

my current teal code for clamm when i remove the comments from the code is 190kb .. when i keep the comments its 210kb.. i prefer not to code a feature to trim the comments from the higher level language because i like when the code is commented, and it is better for auditing process when teal code is commented

i believe 1MB is the better limit because this way devs will hit different limits and not the length of teal code program limit

[jannotti]

Could you tell me how many bytes your teal compiles to? Since we are considering raising the potential size of a logicsig to 16kb, I'd like to have enough headroom to allow for program that compiles to something that big, along with a reasonable amount of comments.

[scholtz]

hi @jannotti I believe the error is thrown here in algod:

go-algorand/daemon/algod/api/server/v2/test/handlers_test.go

Line 1560 in 48a539f

tealDisassembleTest(t, largeProgram, 400, "http: request body too large", true)

when building the contract with bigger size then allowed.. Feel free to rename the text "http: request body too large" to "We allow maximum 1MB of teal code to be compiled" .. but my issue is not to get proper error text, but not to get the error when compiling.. i believe 1MB should be sustainable and usable long term for any optional extensions.. it does not effect any of other compilation limitations at the moment like opcode budget checking...

I believe this teal code size limit is not something that should limit devs from building, but rather the opcode budget should be proper limit.

Please merge it in pls.

[pbennett]

I've got a contract that compiles to 6320 bytes or so (and will grow) and the teal for it is 142K. 200K seems a little on the edge in what's allowed to even be compiled at all, so raising the limit seems warranted. Obviously there's no way 1M teal would be runnable with current limits but going from 142K to 200K seems very easy to hit and still possibly have a 'runnable' contract <=8K.

[joe-p]

I imagine a significant factor here is bytes pseudo-op arguments. For example, TEALScript uses bytes 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF for uint256 encoding. During assembly all of these values get reduced due to it being put in the constant block, but that doesn't help if the bottleneck is sending the TEAL to the node. For this specific scenario a solution is to optimize byte constants prior to sending to algod.

[algorandskiy]

I guess it is safe to raise since it is developer API (normally restricted)

[jannotti]

Can I get this answer? I'm totally ok with using a constant that is reasonably proportional to how many bytes you need to end up with a program that's about as big as allowed:

Could you tell me how many bytes your teal compiles to? Since we are considering raising the potential size of a logicsig to 16kb, I'd like to have enough headroom to allow for program that compiles to something that big, along with a reasonable amount of comments.

[jannotti]

I've got a contract that compiles to 6320 bytes or so (and will grow) and the teal for it is 142K.

If we assume this is a reasonable ratio of source TEAL bytes to bytecode (22.4) then a maximum sized (current) app of 8k would need 8096*22.4 = 182kb source program. Since we're contemplating 16000 byte logicsigs, we might want to compile programs about 16000*22.4 = 358kb.

Allow a little slack, and I'd be in favor of a half mb limit.

I won't fight that hard against 1 mb if others prefer it. Please speak up.

[jannotti]

Suggested change

	// in the source TEAL, so we allow up to 1MB
	const MaxTealSourceBytes = 1_000_000
	// in the source TEAL. We have some indication that real TEAL programs with comments are about 20 times bigger than the bytecode they produce, and we may soon allow 16,000 byte logicsigs, implying a maximum of 320kb. Let's call it half a meg for a little room to spare.
	const MaxTealSourceBytes = 512*1024

[gmalouf]

I'd like to see the above change committed, then happy to approve.

[jannotti]

With my suggested change, I'm happy. With the 1MB limit, I'm basically ok too. I'll approve and let another approver weigh in.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.13%. Comparing base (1493410) to head (f2995f7).
Report is 53 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6068      +/-   ##
==========================================
+ Coverage   55.77%   56.13%   +0.36%     
==========================================
  Files         488      488              
  Lines       69434    69434              
==========================================
+ Hits        38730    38980     +250     
+ Misses      28017    27808     -209     
+ Partials     2687     2646      -41     

Flag	Coverage Δ
	56.13% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.