Saltstack pillar clone on fresh install

[pguinardco]

Description of Issue

On the build of a fresh salt master, gitfs backed pillars do not populate data until after the salt master has started and is serving traffic. During this time empty pillar data is sent out until pillar data is populated on disk. Pillars from external systems work as intended during this time.

At a minimum an intended behavior of blocking startup until all salt components are started should be a tunable option. This is a change in behavior since 2019.2.3.

Setup

ext_pillar:
  - git:
    - master [email protected]:our-repo.git:
      - root: pillar
      - pubkey: /etc/salt/master-secrets/salt-master-ssh-deployment-key.pub
      - privkey: /etc/salt/master-secrets/salt-master-ssh-deployment-key

Steps to Reproduce Issue

1. Stop salt master
   systemctl stop salt-master
2. Remove gitfs data from local machine:
   rm -rf /var/cache/salt/master/git_pillar/*
3. Start salt master
   systemctl start salt-master
4. Get pillar data:

# salt-run pillar.show_pillar --output json                                                                                                                                                                                                      [0/930]
[ERROR   ] Failed to checkout master from git_pillar remote 'master [email protected]:our-repo.git': remote ref does not exist
{}

4. Wait for this to appear in the logs, 20200407173007685862 fails but 20200407173019115563 passes:

[DEBUG   ] Gathering reactors for tag salt/run/20200407173007685862/new
[DEBUG   ] Gathering reactors for tag salt/run/20200407173007685862/ret
[DEBUG   ] git_pillar received 12483 objects for remote 'master [email protected]:our-repo'
[DEBUG   ] Removed update lock for git_pillar remote 'master [email protected]:our-repo'
[DEBUG   ] Gathering reactors for tag salt/run/20200407173019115563/new
[DEBUG   ] Gathering reactors for tag salt/run/20200407173019115563/ret

5. Once the git_pillar remote has ran once after the salt master has started everything is ready to go. Run the following command again:

# salt-run pillar.show_pillar pillarenv=linux --output json
{
    "pillartest": {
        "plaintext": "integration-test-pass"
    }
}

Versions Report

Salt Version:
           Salt: 3000.1

Dependency Versions:
           cffi: 1.11.5
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.1
        libgit2: 1.0.0
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: 1.2.0
         Python: 3.6.8 (default, Nov 21 2019, 19:31:34)
   python-gnupg: 0.4.4
         PyYAML: 3.12
          PyZMQ: 17.0.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: centos 8.1.1911 Core
         locale: UTF-8
        machine: x86_64
        release: 4.19.76-linuxkit
         system: Linux
        version: CentOS Linux 8.1.1911 Core

[sathieu]

#56356 ?

[pguinardco]

I've built this and ran our tests against this. #56356 does fix the issue!