java.lang.IllegalStateException: Pointer native already mapped to Proxy interface to native function

[hakanai]

I get an exception when I attempt to upgrade to JNA 4.1:

Exception in thread "main" java.lang.ExceptionInInitializerError
    at com.nuix.JnaExceptionTest.main(JnaExceptionTest.java:13)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.IllegalStateException: Pointer native@0x7fff917299d0 already mapped to Proxy interface to native function@0x7fff917299d0 (com.nuix.JnaExceptionTest$CFDictionaryKeyCallBacks$PlaceholderCallback)
    at com.sun.jna.CallbackReference.getCallback(CallbackReference.java:122)
    at com.sun.jna.CallbackReference.getCallback(CallbackReference.java:105)
    at com.sun.jna.Pointer.getValue(Pointer.java:430)
    at com.sun.jna.Structure.readField(Structure.java:669)
    at com.sun.jna.Structure.read(Structure.java:537)
    at com.nuix.JnaExceptionTest$CFDictionaryValueCallBacks.<init>(JnaExceptionTest.java:105)
    at com.nuix.JnaExceptionTest$CFDictionaryValueCallBacks.<clinit>(JnaExceptionTest.java:90)
    ... 6 more

I have narrowed the issue down to a few classes and removed the bits which are not necessary to reproduce the issue:

import com.sun.jna.*;
import com.sun.jna.win32.StdCallFunctionMapper;

import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class JnaExceptionTest {
    public static void main(String[] args) {
        new CFDictionaryValueCallBacks();
    }

    public static interface CoreFoundation extends Library {

        Map<Object, Object> defaultOptions = new HashMap<Object, Object>() {{
            put(Library.OPTION_FUNCTION_MAPPER, new StdCallFunctionMapper());
        }};

        public static final NativeLibrary JNA_NATIVE_LIB = NativeLibrary.getInstance("CoreFoundation", defaultOptions);
        public static final CoreFoundation INSTANCE = (CoreFoundation)
            Native.loadLibrary("CoreFoundation", CoreFoundation.class, defaultOptions);

        CFMutableDictionary CFDictionaryCreateMutable(CFAllocator allocator, NativeLong capacity,
                                                      CFDictionaryKeyCallBacks keyCallBacks,
                                                      CFDictionaryValueCallBacks valueCallBacks);

    }

    public static class CFMutableDictionary extends PointerType {
        public CFMutableDictionary() {
        }

        public CFMutableDictionary(Pointer pointer) {
            super(pointer);
        }
    }

    public static class CFAllocator extends PointerType {
        public CFAllocator() {
        }

        public CFAllocator(Pointer pointer) {
            super(pointer);
        }
    }

    public static class CFDictionaryKeyCallBacks extends Structure {
        public static final CFDictionaryKeyCallBacks kCFCopyStringDictionaryKeyCallBacks = new CFDictionaryKeyCallBacks(
            CoreFoundation.JNA_NATIVE_LIB.getGlobalVariableAddress("kCFCopyStringDictionaryKeyCallBacks"));

        public static final CFDictionaryKeyCallBacks kCFTypeDictionaryKeyCallBacks = new CFDictionaryKeyCallBacks(
            CoreFoundation.JNA_NATIVE_LIB.getGlobalVariableAddress("kCFTypeDictionaryKeyCallBacks"));

        public NativeLong version;
        public PlaceholderCallback /*CFDictionaryRetainCallBack*/ retain;
        public PlaceholderCallback /*CFDictionaryReleaseCallBack*/ release;
        public PlaceholderCallback /*CFDictionaryCopyDescriptionCallBack*/ copyDescription;
        public PlaceholderCallback /*CFDictionaryEqualCallBack*/ equal;
        public PlaceholderCallback /*CFDictionaryHashCallBack*/ hash;

        public CFDictionaryKeyCallBacks() {
        }

        public CFDictionaryKeyCallBacks(Pointer p) {
            super(p);
            read();
        }

        @Override
        protected List<String> getFieldOrder() {
            return Arrays.asList("version", "retain", "release", "copyDescription", "equal", "hash");
        }

        public static interface PlaceholderCallback extends Callback {
            void callback();
        }
    }

    public static class CFDictionaryValueCallBacks extends Structure {
        public static final CFDictionaryValueCallBacks kCFTypeDictionaryValueCallBacks = new CFDictionaryValueCallBacks(
            CoreFoundation.JNA_NATIVE_LIB.getGlobalVariableAddress("kCFTypeDictionaryValueCallBacks"));

        // Structure fields which I don't know what they do yet.
        public NativeLong version;
        public PlaceholderCallback /*CFDictionaryRetainCallBack*/ retain;
        public PlaceholderCallback /*CFDictionaryReleaseCallBack*/ release;
        public PlaceholderCallback /*CFDictionaryCopyDescriptionCallBack*/ copyDescription;
        public PlaceholderCallback /*CFDictionaryEqualCallBack*/ equal;

        public CFDictionaryValueCallBacks() {
        }

        public CFDictionaryValueCallBacks(Pointer p) {
            super(p);
            read();
        }

        @Override
        protected List<String> getFieldOrder() {
            return Arrays.asList("version", "retain", "release", "copyDescription", "equal");
        }

        public static interface PlaceholderCallback extends Callback {
            void callback();
        }
    }
}

There are a lot of fields and methods in here where deleting that one method makes the exception no longer happen, but I can't figure out a pattern...

[twall]

You’ve got a case where a pointer value is mapped to a Java callback, then that same pointer shows up with an attempt to map it to a different callback type.

On May 12, 2014, at 9:31 PM, Trejkaz (pen name) [email protected] wrote:

I get an exception when I attempt to upgrade to JNA 4.1:

Exception in thread "main" java.lang.ExceptionInInitializerError
at com.nuix.JnaExceptionTest.main(JnaExceptionTest.java:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.IllegalStateException: Pointer native@0x7fff917299d0 already mapped to Proxy interface to native function@0x7fff917299d0 (com.nuix.JnaExceptionTest$CFDictionaryKeyCallBacks$PlaceholderCallback)
at com.sun.jna.CallbackReference.getCallback(CallbackReference.java:122)
at com.sun.jna.CallbackReference.getCallback(CallbackReference.java:105)
at com.sun.jna.Pointer.getValue(Pointer.java:430)
at com.sun.jna.Structure.readField(Structure.java:669)
at com.sun.jna.Structure.read(Structure.java:537)
at com.nuix.JnaExceptionTest$CFDictionaryValueCallBacks.(JnaExceptionTest.java:105)
at com.nuix.JnaExceptionTest$CFDictionaryValueCallBacks.(JnaExceptionTest.java:90)
... 6 more

I have narrowed the issue down to a few classes and removed the bits which are not necessary to reproduce the issue:

import com.sun.jna.*;
import com.sun.jna.win32.StdCallFunctionMapper;

import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class JnaExceptionTest {
public static void main(String[] args) {
new CFDictionaryValueCallBacks();
}

public static interface CoreFoundation extends Library {

    Map<Object, Object> defaultOptions = new HashMap<Object, Object>() {{
        put(Library.OPTION_FUNCTION_MAPPER, new StdCallFunctionMapper());
    }};

    public static final NativeLibrary JNA_NATIVE_LIB = NativeLibrary.getInstance("CoreFoundation", defaultOptions);
    public static final CoreFoundation INSTANCE = (CoreFoundation)
        Native.loadLibrary("CoreFoundation", CoreFoundation.class, defaultOptions);

    CFMutableDictionary CFDictionaryCreateMutable(CFAllocator allocator, NativeLong capacity,
                                                  CFDictionaryKeyCallBacks keyCallBacks,
                                                  CFDictionaryValueCallBacks valueCallBacks);

}

public static class CFMutableDictionary extends PointerType {
    public CFMutableDictionary() {
    }

    public CFMutableDictionary(Pointer pointer) {
        super(pointer);
    }
}

public static class CFAllocator extends PointerType {
    public CFAllocator() {
    }

    public CFAllocator(Pointer pointer) {
        super(pointer);
    }
}

public static class CFDictionaryKeyCallBacks extends Structure {
    public static final CFDictionaryKeyCallBacks kCFCopyStringDictionaryKeyCallBacks = new CFDictionaryKeyCallBacks(
        CoreFoundation.JNA_NATIVE_LIB.getGlobalVariableAddress("kCFCopyStringDictionaryKeyCallBacks"));

    public static final CFDictionaryKeyCallBacks kCFTypeDictionaryKeyCallBacks = new CFDictionaryKeyCallBacks(
        CoreFoundation.JNA_NATIVE_LIB.getGlobalVariableAddress("kCFTypeDictionaryKeyCallBacks"));

    public NativeLong version;
    public PlaceholderCallback /*CFDictionaryRetainCallBack*/ retain;
    public PlaceholderCallback /*CFDictionaryReleaseCallBack*/ release;
    public PlaceholderCallback /*CFDictionaryCopyDescriptionCallBack*/ copyDescription;
    public PlaceholderCallback /*CFDictionaryEqualCallBack*/ equal;
    public PlaceholderCallback /*CFDictionaryHashCallBack*/ hash;

    public CFDictionaryKeyCallBacks() {
    }

    public CFDictionaryKeyCallBacks(Pointer p) {
        super(p);
        read();
    }

    @Override
    protected List<String> getFieldOrder() {
        return Arrays.asList("version", "retain", "release", "copyDescription", "equal", "hash");
    }

    public static interface PlaceholderCallback extends Callback {
        void callback();
    }
}

public static class CFDictionaryValueCallBacks extends Structure {
    public static final CFDictionaryValueCallBacks kCFTypeDictionaryValueCallBacks = new CFDictionaryValueCallBacks(
        CoreFoundation.JNA_NATIVE_LIB.getGlobalVariableAddress("kCFTypeDictionaryValueCallBacks"));

    // Structure fields which I don't know what they do yet.
    public NativeLong version;
    public PlaceholderCallback /*CFDictionaryRetainCallBack*/ retain;
    public PlaceholderCallback /*CFDictionaryReleaseCallBack*/ release;
    public PlaceholderCallback /*CFDictionaryCopyDescriptionCallBack*/ copyDescription;
    public PlaceholderCallback /*CFDictionaryEqualCallBack*/ equal;

    public CFDictionaryValueCallBacks() {
    }

    public CFDictionaryValueCallBacks(Pointer p) {
        super(p);
        read();
    }

    @Override
    protected List<String> getFieldOrder() {
        return Arrays.asList("version", "retain", "release", "copyDescription", "equal");
    }

    public static interface PlaceholderCallback extends Callback {
        void callback();
    }
}

}

There are a lot of fields and methods in here where deleting that one method makes the exception no longer happen, but I can't figure out a pattern...

—
Reply to this email directly or view it on GitHub.

[hakanai]

I wonder how that can happen. The reverse is definitely true, I'm using that PlaceholderCallback to avoid defining (i.e. looking up and manually converting over) all the other callbacks. But any given callback pointer seems like it could only be used with one interface (because the only interface is PlaceholderCallback.)

[twall]

I believe the issue is that JNA associates one pointer with one callback object; you can’t have the same callback object mapped to multiple callback pointers.

On Jun 19, 2014, at 9:15 PM, Trejkaz (pen name) [email protected] wrote:

I wonder how that can happen. The reverse is definitely true, I'm using that PlaceholderCallback to avoid defining (i.e. looking up and manually converting over) all the other callbacks. But any given callback pointer seems like it could only be used with one interface (because the only interface is PlaceholderCallback.)

—
Reply to this email directly or view it on GitHub.

[twall]

When you use a callback object, JNA creates a native trampoline that effectively routes execution from a native function pointer to your callback object. Once it has done so, it caches the result (so that we don’t create multiple trampolines).

The gap here is that you want a new trampoline (native function pointer) for each unique interface on the callback object, which isn’t currently supported (it could be, although there might need be some internal plumbing to make sure the current callback signature is always available).

On Jun 20, 2014, at 8:37 AM, Timothy Wall [email protected] wrote:

I believe the issue is that JNA associates one pointer with one callback object; you can’t have the same callback object mapped to multiple callback pointers.

On Jun 19, 2014, at 9:15 PM, Trejkaz (pen name) [email protected] wrote:

I wonder how that can happen. The reverse is definitely true, I'm using that PlaceholderCallback to avoid defining (i.e. looking up and manually converting over) all the other callbacks. But any given callback pointer seems like it could only be used with one interface (because the only interface is PlaceholderCallback.)

—
Reply to this email directly or view it on GitHub.

[wolftobias]

I`ve a similar problem, but I use definitly different callback classes for each callback.

public class ssl_method_st extends Structure {
public int version;
/* C type : ssl_new_callback* /
public ssl_new_callback ssl_new;
/* C type : ssl_clear_callback* /
public ssl_clear_callback ssl_clear;
/* C type : ssl_free_callback* /
public ssl_free_callback ssl_free;
/* C type : ssl_accept_callback* /
public ssl_accept_callback ssl_accept;
/* C type : ssl_connect_callback* /
public ssl_connect_callback ssl_connect;
/* C type : ssl_read_callback* /
public ssl_read_callback ssl_read;
/* C type : ssl_peek_callback* /
public ssl_peek_callback ssl_peek;
/* C type : ssl_write_callback* /
public ssl_write_callback ssl_write;
/* C type : ssl_shutdown_callback* /
public ssl_shutdown_callback ssl_shutdown;
/* C type : ssl_renegotiate_callback* /
public ssl_renegotiate_callback ssl_renegotiate;
/* C type : ssl_renegotiate_check_callback* /
public ssl_renegotiate_check_callback ssl_renegotiate_check;
/* C type : ssl_get_message_callback* /
public ssl_get_message_callback ssl_get_message;
/* C type : ssl_read_bytes_callback* /
public ssl_read_bytes_callback ssl_read_bytes;
/* C type : ssl_write_bytes_callback* /
public ssl_write_bytes_callback ssl_write_bytes;
/* C type : ssl_dispatch_alert_callback* /
public ssl_dispatch_alert_callback ssl_dispatch_alert;
/* C type : ssl_ctrl_callback* /
public ssl_ctrl_callback ssl_ctrl;
/* C type : ssl_ctx_ctrl_callback* /
public ssl_ctx_ctrl_callback ssl_ctx_ctrl;
/* C type : get_cipher_by_char_callback* /
public get_cipher_by_char_callback get_cipher_by_char;
/* C type : put_cipher_by_char_callback* /
public put_cipher_by_char_callback put_cipher_by_char;
/* C type : ssl_pending_callback* /
public ssl_pending_callback ssl_pending;
/* C type : num_ciphers_callback* /
public num_ciphers_callback num_ciphers;
/* C type : get_cipher_callback* /
public get_cipher_callback get_cipher;
/* C type : get_ssl_method_callback* /
public get_ssl_method_callback get_ssl_method;
/* C type : get_timeout_callback* /
public get_timeout_callback get_timeout;
/
* Extra SSLv3/TLS stuff

* C type : ssl3_enc_method*
/
public ssl3_enc_method ssl3_enc;
/* C type : ssl_version_callback* /
public ssl_method_st.ssl_version_callback ssl_version;
/* C type : ssl_callback_ctrl_callback* /
public ssl_method_st.ssl_callback_ctrl_callback ssl_callback_ctrl;
/* C type : ssl_ctx_callback_ctrl_callback* */
public ssl_method_st.ssl_ctx_callback_ctrl_callback ssl_ctx_callback_ctrl;

public static interface ssl_new_callback extends Callback {
    int callback(SSL s, int i);
};

public static interface ssl_clear_callback extends Callback {
    void callback(SSL s);
};

public static interface ssl_free_callback extends Callback {
    void callback(SSL s);
};

public static interface ssl_accept_callback extends Callback {
    int callback(SSL s);
};

public static interface ssl_connect_callback extends Callback {
    int callback(SSL s);
};

public static interface ssl_read_callback extends Callback {
    int callback(SSL s, Pointer buf, int len);
};

public static interface ssl_peek_callback extends Callback {
    int callback(SSL s, Pointer buf, int len);
};

public static interface ssl_write_callback extends Callback {
    int callback(SSL s, Pointer buf, int len);
};

public static interface ssl_shutdown_callback extends Callback {
    int callback(SSL s);
};

public static interface ssl_renegotiate_callback extends Callback {
    int callback(SSL s);
};

public static interface ssl_renegotiate_check_callback extends Callback {
    int callback(SSL s);
};

public static interface ssl_get_message_callback extends Callback {
    NativeLong callback(SSL s, int st1, int stn, int mt, NativeLong max,
            IntByReference ok);
};

public static interface ssl_read_bytes_callback extends Callback {
    int callback(SSL s, int type, Pointer buf, int len, int peek);
};

public static interface ssl_write_bytes_callback extends Callback {
    int callback(SSL s, int type, Pointer buf_, int len);
};

public static interface ssl_dispatch_alert_callback extends Callback {
    int callback(SSL s);
};

public static interface ssl_ctrl_callback extends Callback {
    NativeLong callback(SSL s, int cmd, NativeLong larg, Pointer parg);
};

public static interface ssl_ctx_ctrl_callback extends Callback {
    NativeLong callback(SSL_CTX ctx, int cmd, NativeLong larg, Pointer parg);
};

public static interface get_cipher_by_char_callback extends Callback {
    SSL_CIPHER callback(Pointer ptr);
};

public static interface put_cipher_by_char_callback extends Callback {
    int callback(SSL_CIPHER cipher, Pointer ptr);
};

public static interface ssl_pending_callback extends Callback {
    int callback(SSL s);
};

public static interface num_ciphers_callback extends Callback {
    int callback();
};

public static interface get_cipher_callback extends Callback {
    SSL_CIPHER callback(int ncipher);
};

public static interface get_ssl_method_callback extends Callback {
    ssl_method_st callback(int version);
};

public static interface get_timeout_callback extends Callback {
    NativeLong callback();
};

public static interface ssl_version_callback extends Callback {
    int callback();
};

public static interface ssl_callback_ctrl_callback extends Callback {
    NativeLong callback(SSL s, int cb_id, Pointer fp);
};

public static interface ssl_ctx_callback_ctrl_callback extends Callback {
    NativeLong callback(SSL_CTX s, int cb_id, Pointer fp);
};

public ssl_method_st() {
    super();
}

protected List<?> getFieldOrder() {
    return Arrays.asList("version", "ssl_new", "ssl_clear", "ssl_free",
            "ssl_accept", "ssl_connect", "ssl_read", "ssl_peek",
            "ssl_write", "ssl_shutdown", "ssl_renegotiate",
            "ssl_renegotiate_check", "ssl_get_message", "ssl_read_bytes",
            "ssl_write_bytes", "ssl_dispatch_alert", "ssl_ctrl",
            "ssl_ctx_ctrl", "get_cipher_by_char", "put_cipher_by_char",
            "ssl_pending", "num_ciphers", "get_cipher", "get_ssl_method",
            "get_timeout", "ssl3_enc", "ssl_version", "ssl_callback_ctrl",
            "ssl_ctx_callback_ctrl");
}

public ssl_method_st(Pointer peer) {
    super(peer);
}

public static class ByReference extends ssl_method_st implements
        Structure.ByReference {

};

public static class SSL_METHOD extends ssl_method_st {
    public static class ByReference extends SSL_METHOD implements
            Structure.ByReference {
    };

    public SSL_METHOD(Pointer address) {
        super(address);
    }

    public SSL_METHOD() {
        super();
    }
};

}

Caused by: java.lang.IllegalStateException: Pointer native@0x4a64bc0 already mapped to Proxy interface to native function@0x4a64bc0 (com.sun.jna.openssl.ssl_method_st$ssl_accept_callback)
at com.sun.jna.CallbackReference.getCallback(CallbackReference.java:122)
at com.sun.jna.CallbackReference.getCallback(CallbackReference.java:105)
at com.sun.jna.Pointer.getValue(Pointer.java:430)
at com.sun.jna.Structure.readField(Structure.java:669)
at com.sun.jna.Structure.read(Structure.java:537)
at com.sun.jna.Structure.autoRead(Structure.java:1898)
at com.sun.jna.Structure.conditionalAutoRead(Structure.java:506)
at com.sun.jna.Function.invoke(Function.java:418)
at com.sun.jna.Function.invoke(Function.java:315)
at com.sun.jna.Library$Handler.invoke(Library.java:212)
at com.sun.proxy.$Proxy4.SSLv23_client_method(Unknown Source)
at main.java.de.gematik.provider.jsse.NativeCrypto.SSL_CTX_new(NativeCrypto.java:543)
at main.java.de.gematik.provider.jsse.AbstractSessionContext.(AbstractSessionContext.java:51)
at main.java.de.gematik.provider.jsse.ClientSessionContext.(ClientSessionContext.java:40)
at main.java.de.gematik.provider.jsse.SSLContextImpl.(SSLContextImpl.java:52)
at main.java.de.gematik.provider.jsse.OpenSSLContextImpl.(OpenSSLContextImpl.java:31)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:357)
at java.lang.Class.newInstance(Class.java:310)
at java.security.Provider$Service.newInstance(Provider.java:1221)

[elect86]

I get a similar error, but with a clear wrong pointer

http://stackoverflow.com/questions/40157199/jna-pointer-already-mapped-to-proxy-interface

[twall]

You could conceivably work around this by avoiding the "magic" value if it's found. Override Structure.read() something like this:

public void read() {
    Memory old = getPointer();
    Memory m = autoAllocate(size());
    // horribly inefficient, but it'll do
    m.write(0, old.getByteArray(0, size()), 0, size());
    useMemory(m);
    // Zero out the problematic callbacks
    for (field : problematic_fields) {
        m.setPointer(field_offset, null);
    }
    super.read();
    useMemory(old);
}

[fpapai]

We use JNA in my company and this is an outstanding issue for us (JNA associates one pointer with one callback object; you can’t have the same callback object mapped to multiple callback pointers).
Any intention fixing this bug?

[matthiasblaesing]

@fpapai

Any intention fixing this bug?

Do you? JNA is driven by its users - if noone cares enough to do something, it won't be fixed.

[fpapai]

@matthiasblaesing
Thanks. I get it. We'll see.

[matthiasblaesing]

Fix was merged to master. @fpapai implemented a fix, which just got merged.