Memory usage/Performance?

[jabinb]

I'm currently loading around 350~ CUE files that on disk are a few kb each (1.5mb total). They're organised into packages and then those packages are included by a root cue file.

The structure looks like this:

├── Creatures
│   ├── Animal
│   ├── Animated
│   ├── Beholder
│   ├── Daemon
│   ├── Dragonkin
│   ├── Elemental
│   ├── Familiars
│   ├── Gargoyle
│   ├── Giantkin
│   ├── Human
│   ├── Misc
│   ├── Ophidian
│   ├── Orc
│   ├── Plant
│   ├── Ratkin
│   ├── Slime
│   ├── Terathan
│   ├── Troll
│   └── Undead
└── Zuluhotel.cue

However it takes 15-20s to run cue export and uses a fair amount of CPU/RAM:

Command being timed: "cue export Zuluhotel.cue"
User time (seconds): 39.33
System time (seconds): 0.69
Percent of CPU this job got: 257%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.54
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 848912
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 233937
Voluntary context switches: 24721
Involuntary context switches: 3965
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

I have no idea of what to expect of CUE's performance when loading many small files, is this normal? Or could I be doing something wrong?

[verdverm]

Performance can be quite varied based on how you write your CUE. There are small test cases which are known to have overly long runtimes. Performance has improved, but there is more to do. You can filter the issues by the performance label to find a lot of related content. The main upcoming enhancement is called "Structural Sharing" though I am not sure which release that is scheduled for.

In the meantime, there are things that can be done to improve performance. I have found reducing choices for the evaluator (like conjunctions and disjunctions) and "unrolling" various constructs can make large differences, depending on the CUE.

Do you have a link to the code you can share? It can be a lot easier to point these out potential changes if so.

[jabinb]

Thanks for the help, you can find my code at https://github.com/zuluhotelaustralia/zuluhotel/tree/master/Distribution/Configuration

I've narrowed it down to a particular field comprehension I'm doing. Basically I'm constructing a type that has all the fields of X but with a default from Y:

https://github.com/zuluhotelaustralia/zuluhotel/blob/f4614b361e2ebed33e94fb3a12e35bb1f062273a/Distribution/Configuration/Types/Types.cue#L3-L13

If I remove this and replace it with a static type, e.g.

#NpcEntry: {
	Kind:                    "NpcEntry"
	Name:                    string
	CorpseNameOverride:      *#NpcDefaults.CorpseNameOverride | #CreatureProperties.CorpseNameOverride
	BaseType:                *#NpcDefaults.BaseType | #CreatureProperties.BaseType
	Str:                     *#NpcDefaults.Str | #CreatureProperties.Str
	Int:                     *#NpcDefaults.Int | #CreatureProperties.Int
	Dex:                     *#NpcDefaults.Dex | #CreatureProperties.Dex
// rest of the fields

The export goes from 15s to 4s

[verdverm]

Nice, I was thinking that field comprehension (for {...}) might help out. It acts as a sort of copy in the evaluator and removes "links" that cause some excessive comparisons. (using non-technical terms here)

4s seems on par for this many lines of code at the moment. This should get better with the proposed enhancements.