consistent design of streams, strings and their interfaces

[StefanKarpinski]

This overlaps somewhat with issue #50 in that streams and strings can be viewed as collections that contain characters. However, it seems worth making a separate issue. We currently also make no distinction between streams of characters and strings, but there is a major difference: a stream can only be iterated forwards, while a string provides random access. Many of our string operations could actually be stream operations, thereby allowing much more efficient usage in cases where the data is being read incrementally.

[JeffBezanson]

The only stream we have AFAIK is IOStream. And they aren't especially tied to characters. We could add a StringStream and StringBuilder.
Could you explain your last sentence more? Doing string operations with s[i] seems efficient to me. A stream strikes me as less efficient because it involves state (i.e. the current position).

[StefanKarpinski]

True for the moment, but you could certainly generate characters using some other process that isn't an IOStream. It might make sense to have Stream{T} be an abstract type that has a subset of the operations allowed on collections. If you check the code in string.j a lot of it looks like this:

function print_escaped(s::String, q::Bool, xmax::Char)
    if q; print('"'); end
    i = start(s)
    while !done(s,i)
        c, j = next(s,i)
        c == '\0'     ? print(escape_nul(s,j)) :
        c == '\\'     ? print("\\\\") :
        c == '\e'     ? print(L"\e") :
   q && c == '"'      ? print("\\\"") :
        c == '$'      ? print(L"\$") :
        iswprint(c)   ? print(c) :
        7 <= c <= 13  ? print('\\', "abtnvfr"[c-6]) :
        c <= xmax     ? print(L"\x", hex(c,2)) :
        c <= '\uffff' ? print(L"\u", hex(c,4)) :
                        print(L"\U", hex(c,8))
        i = j
    end
    if q; print('"'); end
end

Nowhere does one require the string to be escaped to be indexable by integers — it can just produce them one at a time. Part of the reason I opened this pair of issues was to figure out these distinctions. Some things are iterable, others are indexable; some can be iterated forward and backwards, some only forwards.

Also, note that indexing through a string using an index i is identical to iterating it — i is the state.

[JeffBezanson]

The real "stream" part of this example is the printing --- it writes to a global current output stream which has implicit state. There will always be a choice between storing bytes into an array and going through streams. The former has lower overhead for small data, and the latter can avoid using temporary memory when ultimately writing to an I/O device.

There are three concepts: indexable (string), stream, and iterable. Indexable implies iterable. A stream is neither indexable nor iterable, but can be wrapped in an iterator such as LineIterator.

[StefanKarpinski]

You mean a stream as in a literal IOStream object which just represents an ios_stream structure? Ok, this seems sensible, but is worth setting down in writing, which I can do. I also need to actually implement all these iterators, which is probably more to the point.

[JeffBezanson]

No, I do imagine an abstract Stream of which IOStream is an instance. But given that abstract types for Indexable and Iterable don't exist either, this isn't a big deal. If/when we add multiple inheritance, all of these would become formal types. Until then we can talk about all of them in terms of interfaces (which exist, but aren't declared).

[JeffBezanson]

Was there a typo in the issue number?

[StefanKarpinski]

Uh, yeah. Sorry.

[JeffBezanson]

We decided there's nothing to do here.