[SPARK-44923][PYTHON][BUILD] Some directories should be cleared when …

…regenerating files

### What changes were proposed in this pull request?
The pr aims to fix some bug in regenerating pyspark docs in certain scenarios.

### Why are the changes needed?
- The following error occurred while I was regenerating the pyspark document.
   <img width="1001" alt="image" src="https://github.com/apache/spark/assets/15246973/548abd63-4349-4267-b1fe-a293bd1e7f3e">

- We can simply reproduce this problem as follows:
 1.git reset --hard 3f380b9
    <img width="1416" alt="image" src="https://github.com/apache/spark/assets/15246973/5ab9c8fc-5835-4ced-8d92-9d5e020b262a">
 2.make clean html, (at this point, it is successful.)
    <img width="1000" alt="image" src="https://github.com/apache/spark/assets/15246973/5c3ce07f-cbe8-4177-ae22-b16c3fc62e01">
3.git pull, (at this point, the function `chr` has been deleted, but the previously generated file(`pyspark.sql.functions.chr.rst`) will not be deleted.)
4.make clean html, (at this point, it is failed.)
    <img width="1001" alt="image" src="https://github.com/apache/spark/assets/15246973/548abd63-4349-4267-b1fe-a293bd1e7f3e">

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
1.Pass GA.
2.Manually test.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#42622 from panbingkun/SPARK-44923.

Authored-by: panbingkun <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>

python/docs/source/conf.py

@@ -33,22 +33,16 @@
 
 # Remove previously generated rst files. Ignore errors just in case it stops
 # generating whole docs.
-shutil.rmtree(
-    "%s/reference/api" % os.path.dirname(os.path.abspath(__file__)), ignore_errors=True)
-shutil.rmtree(
-    "%s/reference/pyspark.pandas/api" % os.path.dirname(os.path.abspath(__file__)),
-    ignore_errors=True)
-try:
-    os.mkdir("%s/reference/api" % os.path.dirname(os.path.abspath(__file__)))
-except OSError as e:
-    if e.errno != errno.EEXIST:
-        raise
-try:
-    os.mkdir("%s/reference/pyspark.pandas/api" % os.path.dirname(
-        os.path.abspath(__file__)))
-except OSError as e:
-    if e.errno != errno.EEXIST:
-        raise
+gen_rst_dirs = ["reference/api", "reference/pyspark.pandas/api",
+    "reference/pyspark.sql/api", "reference/pyspark.ss/api"]
+for gen_rst_dir in gen_rst_dirs:
+    absolute_gen_rst_dir = "%s/%s" % (os.path.dirname(os.path.abspath(__file__)), gen_rst_dir)
+    shutil.rmtree(absolute_gen_rst_dir, ignore_errors=True)
+    try:
+        os.mkdir(absolute_gen_rst_dir)
+    except OSError as e:
+        if e.errno != errno.EEXIST:
+            raise
 
 # -- General configuration ------------------------------------------------