Disable realpath cache for ZTS builds #15600

g-ra · 2024-08-27T16:50:11Z

Description

What happened?

Description:
I'm encountering an issue where file writing operations using fwrite occasionally fail when running under FrankenPHP. Sometimes files are not physically created on the server, but the filesize and file_exists functions report them as existing correctly. The code is designed to handle image uploads encoded in Base64, and it works reliably in development environments (e.g., Docker for Windows, Docker for Mac). However, under FrankenPHP in production, files are often missing, even though all the code logic suggests they should have been created successfully.

public function base64ToFile(string $base64, string $fileName, string $storage = Files::STORAGE_PATH): array
{
    if (!is_dir($storage)) {
        $this->imageService->makeDirPath($storage);
    }

    $resultUploadFile = [];

    // Replace spaces with "+"
    $base64 = str_replace(' ', '+', $base64);

    // Split the data by comma
    $data = explode(',', $base64);

    $this->logger->critical('Uploading image: ' . $fileName, ['base64' => $base64, 'data' => $data]);

    // Ensure the Base64 string is properly formatted
    if (!isset($data[1])) {
        throw new LogicException('Invalid Base64 string format.');
    }

    $outputFile = $storage . '/' . $fileName;

    // Attempt to open the file for writing
    $ifp = fopen($outputFile, 'wb');

    if (!$ifp) {
        throw new LogicException('Failed to create file: ' . $outputFile);
    }

    // Attempt to decode the Base64 data
    $decodedData = base64_decode($data[1], true);

    if ($decodedData === false) {
        fclose($ifp);
        throw new LogicException('Failed to decode Base64 data.');
    }

    // Attempt to write the decoded data to the file
    $fwrite = fwrite($ifp, $decodedData);

    if ($fwrite === false) {
        fclose($ifp);
        throw new LogicException('Failed to write data to file: ' . $outputFile);
    }

    // Attempt to close the file
    $close = fclose($ifp);

    if (!$close) {
        throw new LogicException('Failed to close file: ' . $outputFile);
    }

    // Verify the file exists and is not empty
    if (file_exists($outputFile) && filesize($outputFile) > 0) {
        $resultUploadFile = [
            'name' => $fileName,
            'path' => $outputFile,
            'size' => filesize($outputFile),
            'mimeType' => mime_content_type($outputFile),
        ];
    } else {
        throw new LogicException('File does not exist or is zero-sized: ' . $outputFile);
    }

    return $resultUploadFile;
}

Issue: Intermittent file creation failures with fwrite. Sometimes files are not physically created on the server, but the filesize and file_exists functions report them as existing correctly.
Expected Behavior: The file should be consistently created with the correct size, matching the decoded Base64 data.
Observed Behavior: The file is either not created physically on the server, without throwing any exceptions or errors in some cases.
The code above shows that we try to handle all possible errors in production and log them (Grafana, Sentry), but no exceptions are being thrown inside the code, the file sizes are reported correctly, and the file is said to exist. However, in reality, the file is physically missing. How could this happen? Where could the file be stuck? Is fwrite being altered by FrankenPHP?

What complicates things is that we cannot reproduce this error in the development environment—10 out of 10 messages are processed without errors. The issue only occurs on the production server, about every other time. The configurations are the same as those specified in the API Platform documentation.

We have verified that base64_decode is functioning correctly; the content is decoded properly.

Versions:
FrankenPHP: 1.2.4
API Platform: 3.3.11
PHP: 8.3.10

Build Type

ENV CGO_ENABLED=1 XCADDY_SETCAP=1
ENV XCADDY_GO_BUILD_FLAGS="-ldflags \"-w -s -extldflags '-Wl,-z,stack-size=0x80000'\""
ENV CGO_CFLAGS="-fPIC" \
    CGO_LDFLAGS="-pie" \
    GOFLAGS="-buildmode=pie"
RUN xcaddy build \
	--output /usr/local/bin/frankenphp \
	--with github.com/dunglas/frankenphp=./ \
	--with github.com/dunglas/frankenphp/caddy=./caddy/ \
	# Mercure and Vulcain are included in the official build, but feel free to remove them
	--with github.com/dunglas/caddy-cbrotli \
	--with github.com/dunglas/mercure/caddy \
	--with github.com/dunglas/vulcain/caddy \
	# Add extra Caddy modules here
	--with github.com/darkweak/souin/plugins/caddy@fa61769ccc4499e8c72d994cd1b866d80bca3f7f \
	--with github.com/darkweak/souin@fa61769ccc4499e8c72d994cd1b866d80bca3f7f

Worker Mode

No

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

details

Relevant log output

No response

PHP Version

PHP 8.3.10

Operating System

No response

The text was updated successfully, but these errors were encountered:

cmb69 · 2024-08-27T19:10:30Z

This might indeed be a FrankenPHP issue. Maybe @dunglas can have a look at this?

dunglas · 2024-08-27T19:19:23Z

This is possible but unlikely. This may be a ZTS issue, but FrankenPHP itself does nothing specific regarding files / the FS.

cmb69 · 2024-08-27T19:45:24Z

Hmm, might be a realpath cache issue. @g-ra, please disable the realpath cache (realpath_cache_size=0 should do) for testing purposes, and try again.

g-ra · 2024-08-28T06:49:13Z

it seems the setting worked for us, I don't quite understand how this parameter will affect the performance of our application?

we initially assumed that the problem was in docker volumes, but it seems we were wrong and you were right, will this bug be fixed in future versions?

cmb69 · 2024-08-28T13:13:38Z

Well, the realpath cache is a feature, since it caches expensive realpath() lookups. That works fine for NTS (non thread-safe) builds, if no external programs delete files. It may not work well for ZTS (thread-safe) builds, because one thread wouldn't notice file deletions by another thread (since the realpath cache is stored for each thread, using TLS). Long running processes (like you have with FrankenPHP) may make that worse. Now the classic use-case for ZTS is are multi-threaded Web servers, and there often open_basedir is set, in which case the realpath cache is disabled. Since nowadays ZTS builds are used for other purposes (e.g. FrankenPHP and PECL/parallel), it might make sense to disable the realpath cache by default (or even unconditionally) for ZTS builds.

cmb69 · 2024-08-28T13:15:37Z

Oh, forgot, maybe SAPIs/extension which inherently use on ZTS builds could do that themselves, or at least document the issue.

dunglas · 2024-08-28T19:54:17Z

Thanks for the explanation @cmb69. It's the first time an issue related to realpath cache has been reported to our bug tracker. Maybe could we improve the documentation to explain how the user can clear this cache if necessary. Do you know if calling clearstatcache() will clear the cache for all threads?

Disabling entirely the cache will likely have a negative performance impact on most users.

cmb69 · 2024-08-28T20:20:18Z

Calling clearstatcache(true) will only clear the realpath cache of the current thread. (Note that without the true, only the stat cache will be cleared, which contains only the stat info of the last accessed file).

If users delete files which are in the realpath cache of other threads, and having the realpath cache enabled, can only work if they apply some locking (flock() on an always existing file may work or not). Or users may likely be able to prevent such issues by using different paths according to the thread ID (not sure if that is exposed by PHP). Or we might (optinally) move the realpath cache to SHM, akin to #15603 (comment).

g-ra added Bug Status: Needs Triage labels Aug 27, 2024

g-ra mentioned this issue Aug 27, 2024

Intermittent File Write Failures with fwrite dunglas/frankenphp#1007

Open

cmb69 added Status: Needs Feedback and removed Status: Needs Triage labels Aug 27, 2024

github-actions bot added Status: Needs Triage and removed Status: Needs Feedback labels Aug 28, 2024

cmb69 added Feature and removed Bug labels Aug 28, 2024

cmb69 changed the title ~~Intermittent File Write Failures with fwrite~~ Disable realpath cache for ZTS builds Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable realpath cache for ZTS builds #15600

Disable realpath cache for ZTS builds #15600

g-ra commented Aug 27, 2024 •

edited by cmb69

Loading

cmb69 commented Aug 27, 2024

dunglas commented Aug 27, 2024

cmb69 commented Aug 27, 2024

g-ra commented Aug 28, 2024

cmb69 commented Aug 28, 2024

cmb69 commented Aug 28, 2024

dunglas commented Aug 28, 2024

cmb69 commented Aug 28, 2024

Disable realpath cache for ZTS builds #15600

Disable realpath cache for ZTS builds #15600

Comments

g-ra commented Aug 27, 2024 • edited by cmb69 Loading

Description

What happened?

Build Type

Worker Mode

Operating System

CPU Architecture

PHP configuration

Relevant log output

PHP Version

Operating System

cmb69 commented Aug 27, 2024

dunglas commented Aug 27, 2024

cmb69 commented Aug 27, 2024

g-ra commented Aug 28, 2024

cmb69 commented Aug 28, 2024

cmb69 commented Aug 28, 2024

dunglas commented Aug 28, 2024

cmb69 commented Aug 28, 2024

g-ra commented Aug 27, 2024 •

edited by cmb69

Loading