Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for generator. #166

Open
Legion112 opened this issue Jun 8, 2022 · 5 comments
Open

Support for generator. #166

Legion112 opened this issue Jun 8, 2022 · 5 comments

Comments

@Legion112
Copy link

Instead of loading all item in mermory ->rows or using the readSteam function would be nice to have a iterator interface to be able to loop all items without loading them to memory.

Something like:

$client->select('SELECT * FORM table');
foreach($client->interator() as $row) {
    echo $row['first_column] . ',' . $row['second_column];
}
@isublimity
Copy link
Contributor

Add

// Statement Iterator
$state=$this->client->select('SELECT (number+1) as nnums FROM system.numbers LIMIT 5');
foreach ($state as $key=>$value) {
    echo $value['nnums'];
}

@Legion112
Copy link
Author

Could you explain how this gonna work in detail will it use chunking to load only a few record into application memory?
How to configure the size of chunk?

@ahfeel
Copy link

ahfeel commented Feb 20, 2023

I have quite the same issue @Legion112 , did you find a clean solution ? I seems the right way is to use a streamReader but I can't find a simple way to process each record properly without going into complex stream buffering stuff. Would there be a simple way to achieve that @isublimity ?

Thanks !

@Legion112
Copy link
Author

Hm.. I haven't finished that feature in my application. (We cancel the creation of features which require this API)
I guess this has to be part of this library interface.
The library should handle all the pagination logic or the use of pointers.

@ahfeel
Copy link

ahfeel commented Feb 20, 2023

Thanks for your feedback @Legion112 :) Here's a quick class I've written to make the job easier (we use JSON format, so your QUERY must have FORMAT JSONEachRow):

<?php

class JsonRowStreamReader extends \ClickHouseDB\Transport\StreamRead
{
    private $stream;
    private $objectCallback;

    public function __construct()
    {
        $this->stream = fopen('php://memory','r+');

        parent::__construct($this->stream);

        parent::closure(function($ch, $packet) {
            static $temporaryChunk = '';

            $len = strlen($packet);
            $last_char = $packet[$len - 1];
            $lines = explode("\n", $temporaryChunk . $packet);

            if ($last_char == "\n") {
                array_pop($lines);
                $temporaryChunk = '';
            } else {
                $temporaryChunk = array_pop($lines);
            }

            foreach ($lines as $row) {
                $object = json_decode($row, true);
                if (isset($this->objectCallback) && is_callable($this->objectCallback)) {
                    call_user_func($this->objectCallback, [$object]);
                }
            }

            return $len;
        });
    }

    public function closure(callable $callable)
    {
        $this->objectCallback = $callable;
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants