Can't print sparse vectors/matrices

[lrq3000]

After assigning a variable to a sparse vector/matrix, it's only possible to print one entry, but no range nor the whole structure.

With Julia v0.3 prerelease from today, IJulia v0.1.12, Anaconda updated today (IPython v2.1.0) on Windows 7 32-bits.

Eg:

A = sprand(2, 4, 0.5);
print(A[1, 2]) # ok, prints the value
print(A[:,1]); # error
print(A); # error

Here is the full error stack for print(A):

2014-06-23 15:14:12.076 [NotebookApp] CRITICAL | Malformed message: ['stdout', '
<IDS|MSG>', 'c3008313757b289351fb0262451fd556bc7741511d4d9e970833d8419d69a232',
'{"msg_id":"94409fd6-8e55-4d12-b3f8-884d4e7a9d5c","msg_type":"stream","session":
"8786A7AAAF964FB0A1D949A6B238DD41","username":"username"}', '{"msg_id":"6BD7B780
93BA4C9888CCC235658183AF","msg_type":"execute_request","session":"8786A7AAAF964F
B0A1D949A6B238DD41","username":"username"}', '{}', '{"data":"[2, 2]  =  true\\n\
t[1, 4]  =  true","name":"stdout"}']
Traceback (most recent call last):
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\html\base\zmqhandle
rs.py", line 99, in _on_zmq_reply
    msg = self._reserialize_reply(msg_list)
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\html\base\zmqhandle
rs.py", line 82, in _reserialize_reply
    msg = self.session.unserialize(msg_list)
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\kernel\zmq\session.
py", line 828, in unserialize
    message['content'] = self.unpack(msg_list[4])
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\kernel\zmq\session.
py", line 84, in <lambda>
    json_unpacker = lambda s: jsonapi.loads(s)
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\zmq\utils\jsonapi.py", line
 71, in loads
    return jsonmod.loads(s, **kwargs)
  File "C:\Users\slarroqu\Anaconda\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Users\slarroqu\Anaconda\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\slarroqu\Anaconda\lib\json\decoder.py", line 381, in raw_decode

    obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 1 column 27 (char 26)

[stevengj]

There are tab characters in the sparse-matrix printed output. @Carreau, would that cause problems with the latest IPython?

[Carreau]

Hum, @minrk is more comfortable for message spec than I am. And it looks like message spec.

[minrk]

I think JSON (at least Python JSON) escapes tab characters, so \t should be \\t:

In [6]: json.dumps("[2, 2]  =  true\n\t[1, 4]  =  true")
Out[6]: '"[2, 2]  =  true\\n\\t[1, 4]  =  true"'

                                       v
In [7]: json.loads('"[2, 2]  =  true\\n\\t[1, 4]  =  true"')
Out[7]: u'[2, 2]  =  true\n\t[1, 4]  =  true'
                                       v
In [8]: json.loads('"[2, 2]  =  true\\n\t[1, 4]  =  true"')
ValueError: Invalid control character at: line 1 column 19 (char 18)

[stevengj]

Hmm, that doesn't seem to be the problem: Julia's JSON.print function also escapes tab characters as the literal string \t in strings.

[minrk]

But the message that Python is receiving does not have an escaped tab. Julia is not sending a string with an escaped tab character, it's sending a raw tab character.

[stevengj]

It looks escaped on my machine, with IPython 2.0.0 (where sparse-matrix display works fine ... I haven't tried IPython 2.1 yet).

Just ran IJulia.set_verbose(true) so that I can see the JSON messages we are sending, and the output looks fine:

SENDING IPython Msg [ idents pyout ] {
  header = ["msg_id"=>"8e75b726-1c7f-40be-912c-bc670fef8c40","msg_type"=>"pyout","username"=>"username","session"=>"64F06F7DF08645D6909D5E3A60B24825"],
  metadata = Dict{String,Any}(),
  content = ["data"=>["text/plain"=>"2x4 sparse matrix with 5 Float64 entries:\n\t[1, 1]  =  0.298676\n\t[2, 2]  =  0.839125\n\t[1, 3]  =  0.674468\n\t[1, 4]  =  0.35683\n\t[2, 4]  =  0.346931"],"metadata"=>Dict{Any,Any}(),"execution_count"=>2]
}

Note that this is printing the actual string (rather than the repr as in your examples above), so the backslashes are not doubled. The JSON string contains \n and \t as literal strings, not as control characters.

What would have broken this in IPython 2.1?

[stevengj]

Just tried with IPython 2.1 from Anaconda, and it still works fine for me.

Maybe we just need to tag a new JSON release? @lrq3000, can you try Pkg.checkout("JSON") (and then restart IJulia) to see if that fixes the problem? Though it looks like JSON was updated fairly recently (JuliaIO/JSON.jl#73) so maybe you just need Pkg.update()?

[lrq3000]

How can I provide you with more informations on this problem? My install is
fresh, if it's working for Steven it should also work for me... that's
weird.

EDIT: didn't see your edit by mail, I'm trying this right now.

2014-06-23 22:19 GMT+02:00 Steven G. Johnson [email protected]:

Just tried with IPython 2.1 from Anaconda, and it still works fine for me.

—
Reply to this email directly or view it on GitHub
#200 (comment).

[lrq3000]

I tried to Pkg.checkout("JSON") and Pkg.update(), but I was already up-to-date. In the doubt I updated Julia to the last commit, but the same error persists.

[stevengj]

Could you run it IJulia.set_verbose(true) and then try printing the sparse matrix? That way we can see if the pyout message IJulia is sending on your machine is somehow different from the one on my machine (above).

[lrq3000]

Ok I activated verbosity but nothing gets printed in the notebook, only in the console:

EXECUTING print(A)
SENDING IPython Msg [ idents pyin ] {
  header = ["msg_id"=>"ee83c2b5-316b-4f78-81c0-76ead4e46cee","msg_type"=>"pyin",
"session"=>"72EE732B5FFB4841B738C8DCD4FA8D91","username"=>"username"],
  metadata = Dict{String,Any}(),
  content = ["execution_count"=>3,"code"=>"print(A)"]
}
SENDING IPython Msg [ idents status ] {
  header = ["msg_id"=>"80ba5b25-96f7-4d18-91c2-3d2cb9da463d","msg_type"=>"status
","session"=>"f3e07ab5-be77-4a98-9644-241d419588ee","username"=>"jlkernel"],
  metadata = Dict{String,Any}(),
  content = ["execution_state"=>"busy"]
}
SENDING IPython Msg [ idents stdout ] {
  header = ["msg_id"=>"4f95bc0e-a628-46c4-933c-b6474ae56bb9","msg_type"=>"stream
","session"=>"72EE732B5FFB4841B738C8DCD4FA8D91","username"=>"username"],
  metadata = Dict{String,Any}(),
  content = ["data"=>"\n\t","name"=>"stdout"]
}
SENDING IPython Msg [ idents stdout ] {
  header = ["msg_id"=>"32d3fff6-7ab9-44ce-a189-9c8b5cd78029","msg_type"=>"stream
","session"=>"72EE732B5FFB4841B738C8DCD4FA8D91","username"=>"username"],
  metadata = Dict{String,Any}(),
  content = ["data"=>"[1, 2]  =  true\n\t[2, 2]  =  true\n\t[1, 3]  =  true\n\t[
1, 4]  =  true\n\t[2, 4]  =  true","name"=>"stdout"]
}
SENDING IPython Msg [ idents 72EE732B5FFB4841B738C8DCD4FA8D91 ] {
  header = ["msg_id"=>"a77b3db9-c156-49f3-afe7-dec26397d750","msg_type"=>"execut
e_reply","session"=>"72EE732B5FFB4841B738C8DCD4FA8D91","username"=>"username"],
  metadata = Dict{String,Any}(),
  content = ["execution_count"=>3,"user_expressions"=>Dict{Any,Any}(),"status"=>
"ok","user_variables"=>Dict{Any,Any}(),"payload"=>None[]]
}2
014-06-24 16:42:26.105 [NotebookApp] CRITICAL | Malformed message: ['stdout', '<
IDS|MSG>', '72813b78e0612b140fb34b52a6c89736704b055c0f7b1569d7e7e484e36b5509', '
{"msg_id":"4f95bc0e-a628-46c4-933c-b6474ae56bb9","msg_type":"stream","session":"
72EE732B5FFB4841B738C8DCD4FA8D91","username":"username"}', '{"msg_id":"6CE21CF5B
3AF4A2788F3C1A46F6A5B11","msg_type":"execute_request","session":"72EE732B5FFB484
1B738C8DCD4FA8D91","username":"username"}', '{}', '{"data":"\\n\t","name":"stdou
t"}']
Traceback (most recent call last):
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\html\base\zmqhandle
rs.py", line 99, in _on_zmq_reply
    msg = self._reserialize_reply(msg_list)
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\html\base\zmqhandle
rs.py", line 82, in _reserialize_reply
    msg = self.session.unserialize(msg_list)
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\kernel\zmq\session.
py", line 828, in unserialize
    message['content'] = self.unpack(msg_list[4])
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\IPython\kernel\zmq\session.
py", line 84, in <lambda>
    json_unpacker = lambda s: jsonapi.loads(s)
  File "C:\Users\slarroqu\Anaconda\lib\site-packages\zmq\utils\jsonapi.py", line
 71, in loads
    return jsonmod.loads(s, **kwargs)
  File "C:\Users\slarroqu\Anaconda\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Users\slarroqu\Anaconda\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\slarroqu\Anaconda\lib\json\decoder.py", line 381, in raw_decode

    obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 1 column 27 (char 26)

[stevengj]

Is there a newline in the "data" field of

SENDING IPython Msg [ idents stdout ] {
  header = ["msg_id"=>"32d3fff6-7ab9-44ce-a189-9c8b5cd78029","msg_type"=>"stream
","session"=>"72EE732B5FFB4841B738C8DCD4FA8D91","username"=>"username"],
  metadata = Dict{String,Any}(),
  content = ["data"=>"[1, 2]  =  true\n\t[2, 2]  =  true\n\t[1, 3]  =  true\n\t[
1, 4]  =  true\n\t[2, 4]  =  true","name"=>"stdout"]
}

or is that an artifact of how you copied and pasted it?

[lrq3000]

This is probably an artifact because I'm using the Windows pseudo-dos console, but I can't resize the window (to my knowledge). I can retry with powershell, to clear things up, this may remove the artifact.

[stevengj]

cc: @quinnj

[stevengj]

I just tried a Windows 8 machine and was able to reproduce the problem. It seems to have been triggered by a recent update to the JSON module. In particular, git bisect says that the first commit with the problem is this commit by @kmsquire.

[stevengj]

Okay, the problem seems to be that this commit used isprint to determine whether a character needed to be escaped, but isprint('\t') returns true on Windows and false on MacOS.

[stevengj]

Seems to be fixed (or at least worked around) by JuliaIO/JSON.jl@0f038d9

[lrq3000]

I've just updated and this fixed the issue for me! Thank's a lot Steven!