[BUG] Jobs fail when pillar rendering takes more than 60 seconds

[dwoz]

Description
When a minion requests the master to render pillar data, if the request takes more than 60 seconds jobs fail with a traceback.

   The minion function caused an exception: Traceback (most recent call last):
      File "/home/dan/src/salt/salt/minion.py", line 1935, in _thread_return
        return_data = minion_instance._execute_job_function(
      File "/home/dan/src/salt/salt/minion.py", line 1894, in _execute_job_function
        return_data = self.executors[fname](opts, data, func, args, kwargs)
      File "/home/dan/src/salt/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/home/dan/src/salt/salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/home/dan/src/salt/salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/home/dan/src/salt/salt/executors/direct_call.py", line 10, in execute
        return func(*args, **kwargs)
      File "/home/dan/src/salt/salt/loader/lazy.py", line 149, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/home/dan/src/salt/salt/loader/lazy.py", line 1228, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/home/dan/src/salt/salt/loader/lazy.py", line 1243, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/home/dan/src/salt/salt/modules/state.py", line 1352, in sls
        st_ = salt.state.HighState(
      File "/home/dan/src/salt/salt/state.py", line 4769, in __init__
        self.state = State(
      File "/home/dan/src/salt/salt/state.py", line 768, in __init__
        self.opts["pillar"] = self._gather_pillar()
      File "/home/dan/src/salt/salt/state.py", line 833, in _gather_pillar
        return pillar.compile_pillar()
      File "/home/dan/src/salt/salt/pillar/__init__.py", line 353, in compile_pillar
        ret_pillar = self.channel.crypted_transfer_decode_dictentry(
      File "/home/dan/src/salt/salt/utils/asynchronous.py", line 125, in wrap
        raise exc_info[1].with_traceback(exc_info[2])
      File "/home/dan/src/salt/salt/utils/asynchronous.py", line 131, in _target
        result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
      File "/home/dan/src/salt/salt/ext/tornado/ioloop.py", line 459, in run_sync
        return future_cell[0].result()
      File "/home/dan/src/salt/salt/ext/tornado/concurrent.py", line 249, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
      File "/home/dan/src/salt/salt/ext/tornado/gen.py", line 1064, in run
        yielded = self.gen.throw(*exc_info)
      File "/home/dan/src/salt/salt/channel/client.py", line 171, in crypted_transfer_decode_dictentry
        ret = yield self.transport.send(
      File "/home/dan/src/salt/salt/ext/tornado/gen.py", line 1056, in run
        value = future.result()
      File "/home/dan/src/salt/salt/ext/tornado/concurrent.py", line 249, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
      File "/home/dan/src/salt/salt/ext/tornado/gen.py", line 1064, in run
        yielded = self.gen.throw(*exc_info)
      File "/home/dan/src/salt/salt/transport/zeromq.py", line 914, in send
        ret = yield self.message_client.send(load, timeout=timeout)
      File "/home/dan/src/salt/salt/ext/tornado/gen.py", line 1056, in run
        value = future.result()
      File "/home/dan/src/salt/salt/ext/tornado/concurrent.py", line 249, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
      File "/home/dan/src/salt/salt/ext/tornado/gen.py", line 1064, in run
        yielded = self.gen.throw(*exc_info)
      File "/home/dan/src/salt/salt/transport/zeromq.py", line 624, in send
        recv = yield future
      File "/home/dan/src/salt/salt/ext/tornado/gen.py", line 1056, in run
        value = future.result()
      File "/home/dan/src/salt/salt/ext/tornado/concurrent.py", line 249, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 4, in raise_exc_info
    salt.exceptions.SaltReqTimeoutError: Message timed out

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

master config

worker_threads: 4
ext_pillar:
  - cmd_json: "python3 /home/dan/src/salt/foo.py"

foo.py

import time
time.sleep(55)
print("{\"foo\": \"bar\"}")

Run any state against 4 minions and one will result in the above traceback.

[dwoz]

This was also reported in #63824

[dwoz]

Fixed in 3006.3