RFC: change lowering of ccall cconvert arguments and add Ref{T} type #9986

vtjnash · 2015-01-31T22:22:51Z

continuation of #9913, but targeted for merging into master

(note: this is part of the implementation of Julep https://gist.github.com/vtjnash/7296634)

JeffBezanson · 2015-02-01T03:36:37Z

Overall this is really good. I think we can rename cconvert_gcroot to cconvert; the name is ugly and there aren't really any meaningful definitions of cconvert as it is, so we might as well merge them. There is also too much complexity in the several subtypes of Ref. They are somewhat redundant with SubArray, and I think it is easy enough to simply avoid anything like pointer(allocate_array(), n). I liked the PR better when it was a net deletion of code:)

vtjnash · 2015-02-01T03:53:42Z

yeah, it may make more sense to just use SubArray instead. part of the intent of this PR is that you can replace most usages of pointer with Ref. essentially, Ref is intended to act as a gc-safe version of Ptr for almost anywhere

there is one meaningful cconvert call, which is cconvert(Ptr{T}, Ref{T}). there may be lots of valid cconvert_gcroot functions

JeffBezanson · 2015-02-01T04:20:03Z

Yes, the fact that cconvert_gcroot is the more useful function to define is part of the problem: the good function has the ugly name.

I'm not sure it's worth trying to replace Ptr with Ref. A Ref is not a pointer, and is not as efficient as a pointer, so nobody will be fooled.

vtjnash · 2015-02-01T06:39:24Z

I'm not sure it's worth trying to replace Ptr with Ref. A Ref is not a pointer, and is not as efficient as a pointer, so nobody will be fooled

but a Ref is much more versatile and is often easy to allocate on the stack. i'm hoping to fool people someday

JeffBezanson · 2015-02-01T07:25:21Z

base/abstractarray.jl

@@ -6,6 +6,17 @@ typealias AbstractVecOrMat{T} Union(AbstractVector{T}, AbstractMatrix{T})

 ## Basic functions ##

+# convert Arrays to pointer arrays for ccall
+cconvert_gcroot{P<:Ptr}(::Type{Ptr{P}}, a::Array{P}) = a
+function cconvert_gcroot{P<:Ptr}(::Type{Ptr{P}}, a::Array)


Should go in array.jl as it is Array-specific.

vtjnash · 2015-02-01T21:57:40Z

actually, there might be a few more useful calls for the method currently named cconvert, namely those with a first argument of Ptr{T} for some T. in general, it's not ideal to allow arrays and strings and such to be implicitly converted to Ptr on assignment. those operations should probably only be allowed in the context of ccall. however, it is fine to allow them to implicitly convert to a Ref object.

vtjnash · 2015-02-02T23:04:13Z

They are somewhat redundant with SubArray, and I think it is easy enough to simply avoid anything like pointer(allocate_array(), n)

i'm more concerned about whether it remains valid under all code transforms. for example, in the following somewhat-contrived example, the string has become unrooted by the time it reached function i, after the inlining pass has taken care of optimizing away the function call to g(x) and corresponding gc-root for x:

f() = g("hello world")
g(x) = i(h(x))
@noinline h(x) = pointer(x)
@noinline i(x) = ccall(:write, Cint, (Ptr{Uint8},), x)

JeffBezanson · 2015-02-02T23:48:08Z

But if you can still call pointer, adding an extra layer of Ref doesn't
fix this. Granted, if nobody ever called pointer we'd be fine, but that's
true with or without Ref.

vtjnash · 2015-02-03T00:13:30Z

true, but i would suggest most usages of pointer that exist now could be replaced by a Ref object construction instead. then the conversion to a Ptr would only implicitly happen during the ccall construction, when codegen is able to ensure, under all possible code transformations, that there has been a gc-root constructed (even though that gc-root may be actually on the stack and thus transiently destroyed very shortly thereafter via #8134)

vtjnash · 2015-02-03T00:17:46Z

do you want me to merge this separately, or is it acceptable if I continue this under #7906 and interleave these two changes when it gets merge

JeffBezanson · 2015-02-11T19:29:36Z

base/pcre.jl

@@ -100,6 +101,7 @@ end
 function study(regex::Ptr{Void}, options::Integer)
    # NOTE: options should always be zero in current PCRE
    errstr = Array(Ptr{UInt8},1)
+    errstr[1] = C_NULL
    extra = ccall((:pcre_study, :libpcre), Ptr{Void},
                  (Ptr{Void}, Int32, Ptr{Ptr{UInt8}}),
                  regex, options, errstr)


Looks like bug fixes in this file? Could you apply these to master?

It's not exactly a bug. Pcre should zero this field, but I had errors in the way I lowered ccall and this helped me get further

vtjnash · 2015-02-16T11:49:09Z

i think this is ready to merge once CI checks pass

vtjnash · 2015-02-17T01:12:48Z

hm, i'm not really sure why travis linux is the only place this PR seems to fail

JeffBezanson · 2015-02-17T17:06:28Z

Looks like one of those mysterious hangs. Heck, if it even works on windows, it must be right, right? :)

JeffBezanson · 2015-02-17T17:20:20Z

Yay for net deletion.
I'm worried that removing so many conversions to Ptr is a pretty big breaking change.
Needs documentation / comments

JeffBezanson · 2015-02-17T17:29:49Z

base/inference.jl

+            return Any
+        end
+        t = rt.parameters[1]
+        if isa(t,DataType) && is((t::DataType).name,Ref.name)


What does it mean if a ccall returns a Ref?

See gist or doc updates on the jn/ccall3 PR. Some of these updates are primarily only meaningful in that context

vtjnash · 2015-02-21T23:35:20Z

Looks like one of those mysterious hangs. Heck, if it even works on windows, it must be right, right? :)

that sounds great, but it seemed to happen on every commit build, so i don't know what to make of it or how to repeat it

edit: restarted again and it still failed. note that the missing tests appear to be parallel and random (consistently)

vtjnash · 2015-02-22T07:44:47Z

now that I found the bug (affecting travis, but not source builds) and fixed it in fdad916, i should be able to merge this now, pending a green-light from travis

I'm worried that removing so many conversions to Ptr is a pretty big breaking change.

perhaps, but that was sort of the point of deleting them

Needs documentation / comments

it's written, and will merged as part of #7906

bikeshedding @JeffBezanson

do you want me to change the names of any of these methods (Base.cconvert, Base.cconvert_gcroot)?

vtjnash · 2015-02-22T18:30:49Z

src/jltypes.c

@@ -3244,21 +3244,26 @@ void jl_init_types(void)
                                        jl_any_type, jl_null, 32);

    tv = jl_tuple1(tvar("T"));
+    jl_ref_type =


@JeffBezanson should we explicitly disable gc in this jl_init_types function? it would be bad if it somehow ran, even though it shouldn't have managed to allocate that much memory by here

GC is already disabled at this point in the init process. I guess we could assert at the top of the function that GC is disabled.

nah, just wanted to make sure i wasn't missing something

JeffBezanson · 2015-02-22T20:38:50Z

I don't think we can just sneak in removing all these conversions to Ptr. cc @StefanKarpinski @ViralBShah

Also, convert and cconvert is a much better set of names than convert, cconvert, and cconvert_gcroot. In fact that last name is so ugly that I veto it.

JeffBezanson · 2015-02-22T20:55:21Z

Part of the problem with the name is that in reality, cconvert_gcroot is a relatively high-level function that converts a julia structure to a different one more suitable for consumption by C code. This behavior is the main purpose, and it's not directly related to the GC. Sure its result will be rooted, but so is everything passed to a ccall.

JeffBezanson · 2015-03-04T18:46:14Z

But cconvert could make a Ref for anything that needed to be converted to Ptr, allowing conversions to Ptr to be defined only for Ref. This at least seems possible.

As it stands, this has ccall depend on four functions plus convert, and it creates garbage where we didn't previously. So I'm not sure this is enough of a net improvement.

vtjnash · 2015-03-04T18:56:45Z

But you are proposing making more garbage Ref objects to avoid defining one functio no? (Also, I don't count the ptr_arg functions since this will also deprecate all that machinery)

JeffBezanson · 2015-03-04T19:09:23Z

Not necessarily. For objects like Vector{Int} that don't need much translation, cconvert can still just return a Ptr, and have the following convert be a no-op. If more translation is needed, cconvert can return a Ref or other object that gets rooted, followed by the actual conversion to Ptr by convert.

vtjnash · 2015-03-04T19:44:30Z

But what would convert(::String) return? Plus that means I need to add another variable to hold your new proposed gc root

JeffBezanson · 2015-03-04T20:27:14Z

Converting a string could return a Ref to the Vector{UInt8} inside a ByteString.

Wouldn't this be the same variable currently used for cconvert_gcroot?

vtjnash · 2015-03-06T00:01:36Z

that is true for ByteString, but it doesn't generalize particularly well. for example, what would you do for PyObject?

vtjnash · 2015-03-07T07:13:05Z

base/refpointer.jl

+Base.getindex(b::RefArray) = b.x[b.i]
+
+Base.setindex!(b::RefValue, x) = (b.x = x; b)
+Base.setindex!(b::RefArray, x) = (b.x[b.i] = x; b)


should i undefine the setindex! operation for a RefArray with roots? thoughts?

Yes definitely. Arguably getindex is not needed either.

well, there does need to be some way of turning a Ref back into a value

i was thinking more along the lines of adding:
Base.setindex!{T}(b::RefArray{T,Void}, x) = (b.x[b.i] = x; b)
Base.setindex!{T}(b::RefArray{T}, x) = error("this Ref has associated GC-roots and does not know how to do setindex! for them")

-1
I don't know why you'd need to setindex through a ref, and if you did that error message is pretty scary.

to set an initial value, or populate an output of a function call, if used as an output argument

…eversably

ccall(:a, B, (C,), d) is now lowered as: d#1 = cconvert_gcroot(C, d) Expr(:call, :ccall, :a, B, (C,), cconvert(C, d#1), d#1) This allows implementation of the special-case ccall lowering logic in Julia and still create a gcroot for it as needed the Ref{T} type is the new supertype of Ptr{T}. the ccall emission turns a Ref{T} into a Ptr{T}, as if the &addressOf operator had been applied. the specific rules for this will be moved from the current gist to the documentation at a later point.

…6618)

… required references

vtjnash · 2015-03-07T08:15:06Z

this is mostly just waiting for CI to turn green to merge now

this also makes changes such that cconvert is mostly unchanged from its functionality before this change. cconvert is expected to typically call convert. in some cases (such as when intending to create a Ptr for C-compatiblity), it should defer the actual unsafe step of making the Ptr to unsafe_convert. any code intending to allow conversion of an object to a Ptr should define the operation as a method of unsafe_convert after this commit, rather than the current behavior of making them a method of convert

change lowering of ccall cconvert arguments to provide user flexibility in the final convert step in ccall, and add Ref{T} type for general-purpose output argument usage

JeffBezanson · 2015-03-07T19:18:45Z

Maybe we should remove the pointer function as well.

JeffBezanson · 2015-03-07T19:43:05Z

This seems to make the system image significantly larger (more than 1MB, about 6%). Probably due to the lowering of ccall I'd guess.

vtjnash · 2015-03-07T21:17:43Z

that's probably true. i removed the pointertoref intrinsic, but i'm thinking it might be necessary to reverse that and remove jl_value_ptr, to give a more concise AST after lowering

vtjnash · 2015-03-07T21:19:03Z

Maybe we should remove the pointer function as well.

i'm on board with that, although it would need to be deprecated for a time first

JeffBezanson · 2015-03-07T21:31:10Z

I wouldn't have thought the increase was due to jl_value_ptr, just the extra unsafe_convert call. pointertoref wasn't used that much.

vtjnash · 2015-03-07T21:54:57Z

perhaps not then. the lowering of Ref gets a bit verbose right now, for something that eventually turns into a single llvm bitcast. but it's true that base doesn't use it much right now

vtjnash changed the title ~~WIP: change lowering of ccall cconvert arguments and add Ref{T} type~~ RFC: change lowering of ccall cconvert arguments and add Ref{T} type Jan 31, 2015

vtjnash mentioned this pull request Jan 31, 2015

Make by-value struct passing work properly #7906

Merged

23 tasks

JeffBezanson reviewed Feb 1, 2015
View reviewed changes

JeffBezanson reviewed Feb 11, 2015
View reviewed changes

vtjnash force-pushed the jn/ccall_cconvert2 branch from 52ce45a to 0904f36 Compare February 16, 2015 11:48

JeffBezanson reviewed Feb 17, 2015
View reviewed changes

vtjnash mentioned this pull request Feb 22, 2015

LoadError with --inline=no #10238

Closed

vtjnash force-pushed the jn/ccall_cconvert2 branch 3 times, most recently from b667509 to 4cd0665 Compare February 22, 2015 07:40

vtjnash reviewed Feb 22, 2015
View reviewed changes

JeffBezanson mentioned this pull request Mar 6, 2015

Thread safety tracker #10421

Closed

vtjnash force-pushed the jn/ccall_cconvert2 branch from a7acb50 to dad34a7 Compare March 7, 2015 07:07

vtjnash reviewed Mar 7, 2015
View reviewed changes

vtjnash added 5 commits March 7, 2015 02:33

add missing argument verification in some rand! calls

51fc652

eliminate pointertoref Intrinsic and just use jl_value_ptr function r…

c6ee2f9

…eversably

remove unused Ref subtypes. disallow implicit convert to Ptr{T} (ref #…

571096b

…6618)

add an optional "roots" field to RefArray for tracking any additional…

9edf5a4

… required references

vtjnash force-pushed the jn/ccall_cconvert2 branch from dad34a7 to 3659629 Compare March 7, 2015 08:14

vtjnash force-pushed the jn/ccall_cconvert2 branch from 3659629 to 721f3e2 Compare March 7, 2015 08:22

vtjnash merged commit 3387e77 into master Mar 7, 2015

vtjnash deleted the jn/ccall_cconvert2 branch March 7, 2015 19:03

JeffBezanson mentioned this pull request Mar 8, 2015

WARNING: convert(::Type{Ptr}, ::CUDArt.CudaPtr{Float32}) methods should be converted to be methods of unsafe_convert #10437

Closed

stevengj mentioned this pull request Mar 8, 2015

using PyCall not working in Julia 0.4 master JuliaPy/PyCall.jl#119

Closed

jiahao mentioned this pull request Mar 8, 2015

latest ccall changes break ZMQ.jl JuliaInterop/ZMQ.jl#71

Closed

RFC: change lowering of ccall cconvert arguments and add Ref{T} type #9986

RFC: change lowering of ccall cconvert arguments and add Ref{T} type #9986

Conversation

vtjnash commented Jan 31, 2015

JeffBezanson commented Feb 1, 2015

vtjnash commented Feb 1, 2015

JeffBezanson commented Feb 1, 2015

vtjnash commented Feb 1, 2015

Choose a reason for hiding this comment

vtjnash commented Feb 1, 2015

vtjnash commented Feb 2, 2015

JeffBezanson commented Feb 2, 2015

vtjnash commented Feb 3, 2015

vtjnash commented Feb 3, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtjnash commented Feb 16, 2015

vtjnash commented Feb 17, 2015

JeffBezanson commented Feb 17, 2015

JeffBezanson commented Feb 17, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtjnash commented Feb 21, 2015

vtjnash commented Feb 22, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeffBezanson commented Feb 22, 2015

JeffBezanson commented Feb 22, 2015

JeffBezanson commented Mar 4, 2015

vtjnash commented Mar 4, 2015

JeffBezanson commented Mar 4, 2015

vtjnash commented Mar 4, 2015

JeffBezanson commented Mar 4, 2015

vtjnash commented Mar 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtjnash commented Mar 7, 2015

JeffBezanson commented Mar 7, 2015

JeffBezanson commented Mar 7, 2015

vtjnash commented Mar 7, 2015

vtjnash commented Mar 7, 2015

JeffBezanson commented Mar 7, 2015

vtjnash commented Mar 7, 2015