Fix MkdirAll* and its usage

[kolyshkin]

A couple of fixes; similar to moby/moby#35618

1. idtools/mkdirAs: error out if dir exists as file

os.MkdirAll() function returns "not a directory" error in case a
directory to be created already exists but is not a directory
(e.g. a file). MkdirAllNewAs function do not replicate the behavior.

This is a bug since it is expected to ensure the required directory
exists and is indeed a directory, and return an error otherwise.

2. fix MkdirAll usage

This subtle bug keeps lurking in because error checking for Mkdir()
and MkdirAll() is slightly different wrt EEXIST/IsExist:

• for Mkdir(), IsExist error should (usually) be ignored
  (unless you want to make sure directory was not there before)
  as it means "the destination directory was already there";

• for MkdirAll(), IsExist error should NEVER be ignored.

This commit removes ignoring the IsExist error, as it should not
be ignored.

For more details, a quote from my opencontainers/runc#162 (July 2005):

---

TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.

Quoting MkdirAll documentation:

MkdirAll creates a directory named path, along with any necessary
parents, and returns nil, or else returns an error. If path
is already a directory, MkdirAll does nothing and returns nil.

This means two things:

1. If a directory to be created already exists, no error is
   returned.

2. If the error returned is IsExist (EEXIST), it means there exists
   a non-directory with the same name as MkdirAll need to use for
   directory. Example: we want to MkdirAll("a/b"), but file "a"
   (or "a/b") already exists, so MkdirAll fails.

The above is a theory, based on quoted documentation and my UNIX
knowledge.

3. In practice, though, current MkdirAll implementation [1] returns
   ENOTDIR in most of cases described in Add a configuration file #2, with the exception when
   there is a race between MkdirAll and someone else creating the
   last component of MkdirAll argument as a file. In this very case
   MkdirAll() will indeed return EEXIST.

Because of #1, IsExist check after MkdirAll is not needed.

Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.

Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.

[1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go

---

[rhatdan]

LGTM

[rhatdan]

@kolyshkin Needs a rebase
@giuseppe @vrothberg @nalind @saschagrunert @TomSweeneyRedHat PTAL

[vrothberg]

LGTM

[saschagrunert]

LGTM. but this needs rebase

[TomSweeneyRedHat]

LGTM

[kolyshkin]

rebased

[rhatdan]

LGTM