It's slower than package.el

[raxod502]

My startup time was 0.5s before using straight.el; now it's 0.9s.

[raxod502]

I think it got slower after the recent refactor. :'(

[dieggsy]

Can confirm - it's around 1-2 seconds slower for me. (My init time went from 2.5-ish seconds to 4.2-ish)

...would this have anything to do with the fact that emacs has to follow symlinks?

EDIT: Scratch that, seems like my config has gotten slower for other reasons besides this, it's only really like .1-.3 seconds slower

[raxod502]

It's great to hear that it's not actually that much slower. My config has also gotten slower for other reasons that I haven't bothered to fix yet, so I haven't done profiling on straight.el specifically since I first added the build caching system.

See also #41, which should eliminate even the 100–300ms slowdown.

By the way, how many packages do you have? (You can get that number by evaluating (length (hash-table-keys straight--success-cache)).)

[raxod502]

I'm pretty sure the symlinks are negligible. The slowdown should be almost entirely due to the fact that straight.el uses find(1) to check all of your packages for modifications on init. (With that in mind, you may be impressed that it is so fast!)

[dieggsy]

I have 173 packages, apparently. (Huh, I should probably revise this and get rid of the cruft) I :defer t most of that, I think, with probably around 10-ish packages actually loaded on init, 20-ish deferred with some time.

[raxod502]

I believe that you are not in fact imagining things, and straight.el really is 1–2 seconds slower. I think the reason you are having trouble attributing it to straight.el is that the package modifications check (the part we expect to be slow, and the part that happens all at once) is about 300ms, but every other straight-use-package invocation is ever-so-slightly slower than it should be, which adds up to 1–2 seconds.

I'm finally going to get my Emacs config back to a <0.5s init, so I'm investigating this issue now.

[raxod502]

Some profiling.

(benchmark 10 '(straight-use-package 'cider))

"Elapsed time: 4.143111s (0.600769s in 13 GCs)"

(benchmark 10000000 '(straight--convert-recipe 'cider))

"Elapsed time: 1.680602s"

(let ((recipe (straight--convert-recipe 'cider)))
  (benchmark 100000 '(straight--register-recipe recipe)))

"Elapsed time: 2.212280s (0.308168s in 8 GCs)"

(let ((recipe (straight--convert-recipe 'cider)))
  (benchmark 10000 '(straight--repository-is-available-p recipe)))

"Elapsed time: 0.904636s (0.244791s in 6 GCs)"

(benchmark 10000 '(file-exists-p "~/.emacs.d/straight/repos/cider/"))

"Elapsed time: 0.692318s (0.191917s in 5 GCs)"

(let ((recipe (straight--convert-recipe 'cider)))
  (benchmark 100000 '(straight--add-package-to-load-path recipe)))

"Elapsed time: 6.580693s (2.727027s in 71 GCs)"

(benchmark 1000 '(straight--load-build-cache))

"Elapsed time: 2.250950s (1.583563s in 44 GCs)"

(benchmark 100 '(straight--save-build-cache))

"Elapsed time: 4.071628s (0.470273s in 13 GCs)"

(let ((recipe (straight--convert-recipe 'cider)))
  (benchmark 1000 '(straight--package-might-be-modified-p recipe)))

"Elapsed time: 5.798578s (0.103037s in 2 GCs)"

(benchmark 10000000 '(straight--get-dependencies 'cider))

"Elapsed time: 1.157992s"

(let ((recipe (straight--convert-recipe 'cider)))
  (benchmark 1000 '(straight--activate-package-autoloads recipe)))

"Elapsed time: 2.689443s (0.944570s in 24 GCs)"

;; So it looks like the major bottlenecks are, in decreasing order of
;; severity: saving the build cache, loading the build cache, checking
;; for package modifications, and generating autoloads. The problem is
;; made significantly worse by repeated loading of dependencies, so we
;; can probably see the biggest immediate improvement by making sure
;; to only process a given package once within either a single
;; `straight-use-package' invocation or a single init-file load.

;; After replacing `pp' with `insert' when saving the build cache:

(benchmark 100 '(straight--save-build-cache))

;; was: "Elapsed time: 4.071628s (0.470273s in 13 GCs)"
"Elapsed time: 0.223231s (0.096819s in 2 GCs)"

;; So that's much better. Now we don't use pretty-printing anywhere. I
;; could probably optimize this further by hardcoding the printing
;; algorithm, but that shouldn't be too important since we already
;; optimize to do this once per init.

[vyp]

So is pretty printing necessary? Or what's the advantage of it? Nice work btw 🎉

[raxod502]

Pretty-printing is nice since (1) you can read the build cache and (2) Emacs can edit it without choking on it being a single long line.

But the delay is too high a cost to pay, so pretty-printing is disabled now (once I push my performance-improving commits).

[vyp]

Ah it's a shame emacs chokes on it, I was thinking if anyone needs to read it, they can just pretty print it interactively from emacs. I'm not sure what the solution to that is then. 😕

[vyp]

Maybe there could be a configuration option for it.

[raxod502]

Emacs can perfectly well read and write it. It's just that if you try to edit the file interactively, there will be a fair bit of lag. And you're right that inspection of the value can also happen from within Emacs (C-h v). I don't really see any use case for having it pretty-printed.

[raxod502]

Alright. I've eliminated all the duplicate work being done (e.g #117), and improved performance in a number of other situations (e.g. evaluating a single file of straight-use-package declarations, init time when only one package has been modified) by introducing a new transaction-based API (see the referenced commit and its children).

However, performance improvements are not as significant as I would like. Further profiling reveals the following remaining bottlenecks on my config of ~108 packages:

• 550ms for the bulk find(1) operation at straight.el load time. Can be eliminated by Add option to only rebuild packages when changed in Emacs #41.
• 300ms total for activating autoloads. Can possibly be improved by concatenating autoloads.
• 220ms for saving the build cache, even though we no longer pretty-print it. Can most likely be improved by either hardcoding the printing algorithm or implementing some kind of compression scheme.

[tummychow]

Just ported to straight.el and wanted to chime in on this issue - my config lives on spinning rust, so the first time I open emacs after booting, it takes 50 (fifty, five zero) seconds of continuous disk io to spawn a window. Subsequent loads are snappy (sub-1s) because my config is relatively small, but that first invocation of find(1) appears to be staggeringly expensive.

[raxod502]

@tummychow How many packages do you have?

(length (hash-table-keys straight--profile-cache))

I have 126, and my M-x emacs-init-time is 2.6 seconds. I'm surprised there's such a difference.

[tummychow]

Here's my lockfile. Running that snippet of elisp shows 26 packages. This is Arch Linux, GNU findutils 4.6.0, emacs 25.3.1. The config lives on a WDC WD10SPCX-60K (5400rpm laptop-size spinning rust). Any debugging you'd like me to try?

[raxod502]

@tummychow Try evaluating the following:

(benchmark 1 '(unwind-protect (straight--cache-package-modifications) (straight--uncache-package-modifications)))

That should give you just the time of the find(1) command, so we can tell if it's the disk or your configuration.

What I just noticed is that your init-file is loaded by org-tangle. This might mean that your configuration code is loaded after after-init-hook is run, which means you must load straight.el in a special way. I'm guessing that this is why it is so slow.

It might be a good idea to add a warning in the case that straight.el detects the loading of a large number of packages outside of a transaction block.

[tummychow]

My init is in orgmode, but I tangle the file every time I save, not when emacs starts up. My init.el is exactly the contents of the org-src blocks.

That snippet executes in <0.1s, but the disk is already warmed up at this point. Similarly, if I kill-emacs and launch it again, the startup time is a much more reasonable 0.8s instead of 50+. Is there a way to test it on the very first load?

[raxod502]

The easiest way would probably just be to edit straight--cache-package-modifications to print the find(1) command instead of running it. Then restart and run the command in your shell, and see how long it takes.

[tummychow]

Bingo:

[~] $  time ./straightfind

real    0m49.936s
user    0m0.420s
sys     0m1.112s
[~] $  time ./straightfind

real    0m0.092s
user    0m0.076s
sys     0m0.016s

First run was immediately after booting, second run was immediately after that. Full contents of the find invocation here.

[raxod502]

That's impressive. I had no idea how pronounced the difference was for a non-SSD. In any case, I think the only solution is #41.

[Compro-Prasad]

IMHO package.el does none in comparison to straight.el. So, why bother about it?

[raxod502]

@Compro-Prasad Because I would like to present straight.el as superior to package.el in all respects, not only some. Inertia is a powerful force in how people make decisions like which software to use, and I'd like to counteract it by not leaving people any reason at all except inertia.

More generally, straight.el should be faster than all other package managers, not just package.el.

[Compro-Prasad]

@raxod502 What about using seq library instead of mapcar and similar functions?

[raxod502]

Are you suggesting that this would improve performance? If so, why do you think so?

[Compro-Prasad]

I don't know. Not an elisp expert. Just saw that seq was the new thing. If I remember correctly it was added in Emacs 25.

[raxod502]

seq.el provides a higher-level, generic API for sequences, so it would be slower, not faster.

[OldhamMade]

Hey, not sure if this is useful at all, but I'm moving to straight.el and I'm using esup to track where time is spent during my init, and the slowest part of my setup is the following:

bootstrap.el:83  1.405sec   61%
(straight-use-recipes '(melpa :type git :host github
:repo "melpa/melpa"
:no-build t))

I'm on macOS Mojave with a 3.1GHz i7, 16GB ram, and an SSD.

[raxod502]

@OldhamMade If you're using the default modification checking, then you'll see a large amount of time spent on whatever straight-use-package call gets executed first, since that's when straight.el performs its bulk find(1) command to check for modifications to all known packages. Combining the commands makes performance profiling more difficult, but nevertheless improves the performance by several times.

[sergeyklay]

Just ported to straight.el and run esup.

packaging.el:23  1.069sec   62%
(load-file (expand-file-name "straight.el" user-site-lisp-dir))

62% of time Emacs spent for loading straight.el. I have 119 package, and my M-x emacs-init-time is 2.48 seconds (with package.el is 1.2 second).

1.069sec for loading straight.el using NVMe M.2 SSD

[raxod502]

@sergeyklay Something seems fishy there. Loading straight.el itself should take hardly any time, especially on SSD, as it is under 6,000 lines of code and it does not do any actual package management during loading. For example, I benchmark a load time of 87ms. (This is actually long enough that I might consider splitting up straight.el to optimize startup, but it's nowhere near what you are suggesting.) But I am a little confused about the line of code you are pointing to. Why are you loading straight.el directly instead of doing it the standard way, which is by loading bootstrap.el? Or is straight.el in this case actually a file that you wrote, which includes a number of straight-use-package forms?

You should also check the value of straight-check-for-modifications and see if it is as you desire.

[sergeyklay]

Actually my straight.el file is:

;;; straight.el --- straight.el installer. -*- lexical-binding: t; -*-

;;; Commentary:

;; This library simply installs straight.el.
;; No further configuration is done.

;;; Code:

(unless (featurep 'straight)
  (defvar bootstrap-version)

  (let ((bootstrap-file (concat user-emacs-directory
                                "straight/repos/straight.el/bootstrap.el"))
        (bootstrap-version 5))
    (unless (file-exists-p bootstrap-file)
      (with-current-buffer
          (url-retrieve-synchronously
           "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
           'silent 'inhibit-cookies)
        (goto-char (point-max))
        (eval-print-last-sexp)))
    (load bootstrap-file nil 'nomessage)))

;; Local Variables:
;; flycheck-disabled-checkers: (emacs-lisp-checkdoc)
;; End:

;;; straight.el ends here

[wedens]

Counterpoint: I have 157 packages (most of them are lazy loaded) and emacs takes 0.35s to start with scratch buffer.

esup shows this code as most time consuming in straight.el:

bootstrap.el:81  0.011sec   5%
(straight-use-recipes '(org-elpa :local-repo nil))

I use emacs 27 and everything is on SSD.

[sergeyklay]

@wedens How much does Emacs spend loading a bootstrap file?

[wedens]

@sergeyklay code you mentioned above takes 0.03sec (with already downloaded bootstrap.el)

[sergeyklay]

@wedens What exactly do you mean saying "most of them are lazy loaded". Could you please provide some typical examples?

[wedens]

@sergeyklay I use use-package with (setq use-package-always-defer t) which defers loading packages until necessary (via autoloads, for example). For some (minority of) packages it's not enough and more tricks are required.

Another popular startup time optimization is to increase gc-cons-threshold and set file-name-handler-alist to nil early in config and reset them to default values in emacs-startup-hook.

You may want to try (or at least looks at its code) doom-emacs which uses straight.el and does a lot optimizations (including mentioned above) to achieve fast startup.

But all this is not really specific to straight.el.

[sergeyklay]

@wedens Yes, you're right all of this is not really specific to straight.el (even though it is extremely interesting to me now). However, what I'm trying to understand is why straight.el so slow (2x) compared to package.el with almost the same Emacs configuration. Below are the changes I made to migrate to straight.el:

• Removed all package.el related code, e.g. package-initialize, package-refresh-contents an so on
• Added a bootstrap file called straight.el as I provided above
  
• Replaced :ensure nil to :straight (:type built-in) wherever I used this for use-package
• Added the following bootstrap code:

(setq-default straight-vc-git-default-clone-depth 1)
(load-file (expand-file-name "straight.el" user-site-lisp-dir))

;; Install use-package using straight.el
(straight-use-package 'use-package)

;; Use straight.el by default in use-package directives
(setq straight-use-package-by-default t)

that's all

[wedens]

@sergeyklay What emacs version do you use? emacs 27 starts noticeable faster.

You can try installing watchexec and setting (setq straight-check-for-modifications '(watch-files find-when-checking)). By default straight.el checks for packages modifications on startup using find and it can make things slower. Or maybe even set it to never to check whether modification checking is the problem or not.

[dertuxmalwieder]

emacs 27 starts noticeable faster.

Could straight.el profit from the new early-init.el files?

[raxod502]

Not directly. You can improve Emacs startup by moving all graphical frame customizations into early-init.el, but this is orthogonal to straight.el. In my configuration, for simplicity I actually run all startup code from early-init.el, but this has the key disadvantage that startup messages are not displayed until startup is complete or an error is encountered.