Great effort!

[vkolotov]

Hi @hypfvieh,

just noticed your project. I use tinyb library, it is cool but hard to modify and support. Some features are missing from it as well. I'm wondering if you are going to add notification support in bluez-dbus, e.g. characteristic notifications and some other (like tinyb does)?

Thanks,
Vlad

[hypfvieh]

Hi,

I'm not so sure which kind of notification support you mean.

I just updated the source to support the DBus 'PropertiesChanged' signal, so you can register a callback which gets all property updates from DBus.

I also added the 'startNotify()', 'stopNotifiy()' calls to the GattCharacteristics wrapper class.

As far as I understand the bluez documentation and several other sources on the web, when calling startNotify() all changed properties (like modified GATT values) should be announced by the 'PropertiesChanged' signal of DBus.

So I guess, to get changed attributes it should be suitable to added a PropertiesChanged callback listener and check the received callback objects for the path/devicename/adapter name you want to observe.

If this is not what you mean, please explain.
Also feel free to issue a pull request to add the missing feature, if you could write it yourself.

[vkolotov]

Hi @hypfvieh,

looks like it is exactly what I need.

I'm creating a plugin for OpenHAB project to add a support for Bluetooth Smart devices. I also created a library to read/write GATT characteristics: https://github.com/vkolotov/bluetooth-gatt-parser.
That's why I'm interested in your project. It's got some advantages over tinyb library since it does not have any dependencies on native code. I might want to make use of it.

Thanks,
Vlad

[hypfvieh]

Hi @vkolotov,

always glad to help :-)

I'm also working on a OpenHAB plugin (not yet released to the public).

In my case it is for LED stripes of the german manufacturer Paulmann.
They already have a Android/iOS app, but I don't like it and I also don't want to run around in my living room to find my damn smartphone only to change the brightness of my lights ...

So I wanted to use openHAB + Rasberry 3 (which supports BLE without additional hardware).
This was the reason why I started this small project. I also found tinyb and used it first.
But I wasn't happy with the compile-native-stuff thing (makes the openHAB plugin harder to use and also requires lots of additional stuff to be installed only for one-time use).

The next thing I would like to get rid of is the dependency on libmatthew which requires native library (for unix sockets) as well.
Sadly I did not find any library out there which implements unix sockets 'native' in java without a C-library in the middle.

Maybe I will take a look at the unix socket definition when I find some spare time.
I'm not sure if it's possible or if I'm able to write something java-native. But we'll see.

[vkolotov]

Hi @hypfvieh,

Sounds good.

The binding I'm working on (not published yet as well, 80% done) is a plugin which is supposed to be as generic as possible, e.g. it is basically a manager of BLE devices capable of reading and writing GATT characteristics. GATT Blueooth SIG xml files are used as an input to the plugin, those ones which you can freely download from the Bluetooth SIG website, like that: https://www.bluetooth.com/specifications/gatt/viewer?attributeXmlFile=org.bluetooth.characteristic.heart_rate_measurement.xml

By default the plugin can read and write all (almost all, because not all specifications are well-defined) "approved" GATT specifications. However, it is possible to add some custom GATT services and characteristics by specifying a folder with GATT XML files (through the binding settings).

The binding is also taking care of the "unstable" and "unreliable" nature of the bluetooth protocol, e.g. linked devices and characteristics are being monitored and connections are being restored in case of any issues.

The problem you are working on (the binding for BLE enabled LED strip) seems to me very similar to what I do, to be more precise to when the plugin does. If you believe we could cooperate our effort on crating a generic BLE binding that can solve your problem as well, then feel free to contact me and discuss this :)

gtalk: [email protected]

Thanks,
Vlad

[vkolotov]

Ok. I've started to create a transport for OpenHab bluetooth plugin using this awesome library.

What I do is: I'm just copying TinyB transport library and renaming it to DBus transport hoping that TinyB and your library have more or less the same API. The very first thing that I came across is a lack of enable* methods (comparing to TinyB lib). For example: BluetoothGattCharacteristic.enableValueNotifications.

This method subscribes to GATT notifications of a characteristic.

What would be the analogue to that?

UPDATE:
I can see that there is BluetoothGattCharacteristic.startNotify method, but what it actually does? How do I supply a listener or something to start receiving messages?

Thanks

[hypfvieh]

Yes startNotify() is one part of the puzzle.
This enables bluez signal notification. To handle the notification you have to register a SignalHandler on the DbusConnection.

Sadly, if you register a listener, you will get all kinds of signals, not only the signals of the GattCharacteristics, so you have to filter what's interesting for you.

Here is some test code I wrote some time ago to play with callbacks in DBus, maybe you can use something of it.

package org.github.hypfvieh.sandbox.bluez;

import java.util.HashMap;
import java.util.Map;

import org.bluez.Adapter1;
import org.bluez.Device1;
import org.bluez.GattCharacteristic1;
import org.bluez.GattManager1;
import org.bluez.GattProfile1;
import org.bluez.exceptions.BluezFailedException;
import org.bluez.exceptions.BluezNotAuthorizedException;
import org.bluez.exceptions.BluezNotReadyException;
import org.freedesktop.dbus.DBusPath;
import org.freedesktop.dbus.connections.impl.DBusConnection;
import org.freedesktop.dbus.connections.impl.DBusConnection.DBusBusType;
import org.freedesktop.dbus.exceptions.DBusException;
import org.freedesktop.dbus.handlers.AbstractInterfacesAddedHandler;
import org.freedesktop.dbus.handlers.AbstractInterfacesRemovedHandler;
import org.freedesktop.dbus.handlers.AbstractPropertiesChangedHandler;
import org.freedesktop.dbus.interfaces.DBusInterface;
import org.freedesktop.dbus.interfaces.ObjectManager;
import org.freedesktop.dbus.interfaces.Properties;
import org.freedesktop.dbus.interfaces.Properties.PropertiesChanged;
import org.freedesktop.dbus.types.Variant;

public class SimpleBluez implements DBusInterface, ObjectManager {

    private GattProfile1Impl     profile;
    private DBusConnection       connection;

    private Map<String, Device1> btDevices = new HashMap<>();

    public SimpleBluez() throws DBusException {
        // open connection to bluez on SYSTEM Bus
        connection = DBusConnection.getConnection(DBusBusType.SYSTEM);
        // create profile to export
        profile = new GattProfile1Impl("/com/github/hypfvieh/bluez/Paulmannprofile");
    }

    public void register() throws DBusException {

        connection.exportObject(getObjectPath(), this);

        addPropertiesChangedListener();

        addInterfacesAddedListener();

        addInterfacesRemovedListener();

        // get the GattManager to register new profile
        GattManager1 gattmanager = connection.getRemoteObject("org.bluez", "/org/bluez/hci0", GattManager1.class);

        System.out.println("Registering: " + this.getObjectPath());

        // register profile
        gattmanager.RegisterApplication(new DBusPath(this.getObjectPath()), new HashMap<>());

    }

    private void addInterfacesRemovedListener() throws DBusException {
        connection.addSigHandler(InterfacesRemoved.class,
                new AbstractInterfacesRemovedHandler() {
                    @Override
                    public void handle(InterfacesRemoved _s) {
                        if (_s != null) {
                            if (_s.getInterfaces().contains(Device1.class.getName())) {
                                System.out.println("Bluetooth device removed: " + _s.getSignalSource());
                                btDevices.remove(_s.getPath());
                            }

                            System.out.println("InterfaceRemoved ----> " + _s.getInterfaces());
                        }

                    }

                });
    }

    private void addInterfacesAddedListener() throws DBusException {
        connection.addSigHandler(InterfacesAdded.class,
                new AbstractInterfacesAddedHandler() {

                    @Override
                    public void handle(InterfacesAdded _s) {
                        if (_s != null) {
                            Map<String, Map<String, Variant<?>>> interfaces = _s.getInterfaces();
                            interfaces.entrySet().stream().filter(e -> e.getKey().equals(Device1.class.getName()))
                                    .forEach(e -> {
                                        Variant<?> address = e.getValue().get("Address");
                                        if (address != null && address.getValue() != null) {
                                            System.out.println("Bluetooth device added: " + address.getValue());
                                            String p = _s.getSignalSource().getPath();
                                            try {
                                                Device1 device1 =
                                                        connection.getRemoteObject("org.bluez", p, Device1.class);
                                                btDevices.put(p, device1);
                                            } catch (DBusException _ex) {
                                                // TODO Auto-generated catch block
                                                _ex.printStackTrace();
                                            }
                                        }
                                    });

                            interfaces.entrySet().stream()
                                    .filter(e -> e.getKey().equals(GattCharacteristic1.class.getName())).forEach(e -> {
                                        System.out.println("New characteristics: " + e.getValue());
                                    });
                            // System.out.println("InterfaceAdded ----> " + _s.getInterfaces());
                        }

                    }

                });
    }

    private void addPropertiesChangedListener() throws DBusException {
        connection.addSigHandler(PropertiesChanged.class,
                new AbstractPropertiesChangedHandler() {

                    @Override
                    public void handle(PropertiesChanged _s) {
                        if (_s != null) {

                            if (!_s.getPath().contains("/org/bluez")
                                    && !_s.getPath().contains(getClass().getPackage().getName())) { // filter all events
                                                                                                    // not belonging to
                                                                                                    // bluez
                                return;
                            }

                            // if (_s.get)
                            System.err.println("PropertiesChanged:----> " + _s.getPropertiesChanged());
                            if (!_s.getPropertiesRemoved().isEmpty())
                                System.err.println("PropertiesRemoved:----> " + _s.getPropertiesRemoved());
                        }
                    }

                });
    }

    @Override
    public boolean isRemote() {
        return false;
    }

    @Override
    public String getObjectPath() {
        return "/" + getClass().getName().replace(".", "/");
    }

    @Override
    public Map<DBusPath, Map<String, Map<String, Variant<?>>>> GetManagedObjects() {
        System.out.println("GetManagedObjects Called");
        Map<DBusPath, Map<String, Map<String, Variant<?>>>> outerMap = new HashMap<>();

        outerMap.put(new DBusPath(profile.getObjectPath()), profile.getProperties());

        return outerMap;
    }

    protected void scan(int _i) {
        System.out.println("Scanning for " + _i + " seconds");
        Adapter1 adapter = null;
        try {
            adapter = connection.getRemoteObject("org.bluez", "/org/bluez/hci0", Adapter1.class);
            adapter.StartDiscovery();
            Thread.sleep(_i * 1000);

        } catch (DBusException | InterruptedException _ex) {
            // TODO Auto-generated catch block
            _ex.printStackTrace();
        } finally {
            if (adapter != null) {
                try {
                    adapter.StopDiscovery();
                } catch (BluezNotReadyException | BluezFailedException | BluezNotAuthorizedException _ex) {
                    // TODO Auto-generated catch block
                    _ex.printStackTrace();
                }
            }
        }
        System.out.println("Scanning for finished");
    }

    /*
     * =================================================================
     *
     * STATIC STUFF
     *
     * =================================================================
     */

    public static void main(String[] args) {
        Thread thread = new Thread("MyThread") {
            private boolean running = true;

            @Override
            public void run() {
                System.out.println("Init");
                SimpleBluez simpleBluez = null;
                try {
                    simpleBluez = new SimpleBluez();
                    System.out.println("Registering");
                    simpleBluez.register();
                    System.out.println("Waiting");
                    while (running) {
                        try {
                            Thread.sleep(500L);
                        } catch (InterruptedException _ex) {
                            running = false;
                        }
                    }
                } catch (Exception _ex) {
                    // TODO Auto-generated catch block
                    _ex.printStackTrace();
                } finally {
                    running = false;
                    System.out.println("Terminating");
                    if (simpleBluez != null) {
                        simpleBluez.connection.disconnect();
                    }
                }

            }

        };

        thread.start();
    }

    static class ObjectManagerHandler implements ObjectManager {

        @Override
        public boolean isRemote() {
            return false;
        }

        @Override
        public String getObjectPath() {
            return "/";
        }

        @Override
        public Map<DBusPath, Map<String, Map<String, Variant<?>>>> GetManagedObjects() {
            System.err.println(this.getClass() + " Getmanagedobjects called");
            return null;
        }

    }

    static class GattProfile1Impl implements GattProfile1, Properties {
        private boolean                              released;
        private String                               path;

        private Map<String, Map<String, Variant<?>>> properties = new HashMap<>();

        public GattProfile1Impl(String _path) {
            released = false;
            path = _path;

            Map<String, Variant<?>> map = new HashMap<>();
            map.put("UUIDs", new Variant<>(new String[] {
                    "0000ffb0-0000-1000-8000-00805f9b34fb"
            }));

            properties.put(GattProfile1.class.getName(), map);
        }

        @Override
        public boolean isRemote() {
            return false;
        }

        public Map<String, Map<String, Variant<?>>> getProperties() {
            return properties;
        }

        @Override
        public String getObjectPath() {
            return path;
        }

        /**
         * {@inheritDoc}
         */
        @Override
        public void Release() {
            released = true;
        }

        public boolean isReleased() {
            System.out.println("released called");
            return released;
        }

        @Override
        public <A> A Get(String _interface_name, String _property_name) {
            System.out.println("Get called");
            // Variant<?> variant = properties.get(_interface_name).get(_property_name);
            return null; //
        }

        @Override
        public <A> void Set(String _interface_name, String _property_name, A _value) {
            System.out.println("Set called");

        }

        @Override
        public Map<String, Variant<?>> GetAll(String _interface_name) {
            System.out.println("queried for: " + _interface_name);
            return properties.get(_interface_name);
        }

    }

}

[Edit]: Updated sample code

[mtrewartha]

Hey @hypfvieh, so are there any plans to beef up the notification support? It looks like the method you describe above utilizes D-Bus, which works fine for low volume use cases. However, we rely on characteristic notifications to move a lot of data from a peripheral to our central at a fast pace, sometimes sending several MiB of data via notifications from a single characteristic over the course of a few mins. For that type of use case, it looks like BlueZ has AcquireNotify, which returns a FileDescriptor that can be used to read notification data at a faster pace.

Forgetting about speed for a second, I wanted to try and get a prototype working even if it was slow. I was able to cobble something together using the connection.addSigHandler(...) stuff you show above, but it seems like some notifications are just getting lost in the mix, overwritten before they can be read or somethfing. I know the notifications are being sent properly via BLE because I've got an Android app that sees them all without issue.

I then tried using bluetoothGattCharacteristic.rawGattCharacteristic.AcquireNotify(...), but that just resulted in this:

 org.freedesktop.dbus.errors.NoReply: No reply within specified time
        at org.freedesktop.dbus.RemoteInvocationHandler.executeRemoteMethod(RemoteInvocationHandler.java:158)
        at org.freedesktop.dbus.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:226)
        at com.sun.proxy.$Proxy24.AcquireNotify(Unknown Source)
        ...

Maybe I'm doing something wrong or not setting something up correctly?

[hypfvieh]

The notification stuff is some heavy topic (at least for me). Difficult to test, difficult to use.

I didn't try using AcquireNotify as I don't have any device which supports that feature (afaik).
As the documentation says, this feature is optional and has to be support by the device:

Only works with characteristic that has NotifyAcquired
which relies on notify Flag and no other client have
called StartNotify.

So maybe it does not work, because your device is not supporting this or some other client already blocks that features.

Have you tried to use the AcquireNotify method by testing it with the bluez commandline util?

It may also be possible that this feature is not implemented correctly in bluez-dbus currently.
As said before I didn't test or use it, so a bug is a valid option ;)

Whats worrying me is that you said you are 'loosing' some signals. That is really a problem and should never happen as long as dbus and bluez work like expected.

Could you explain the scenario you used to trigger this issue?
Maybe I can create some sort of unit-test to reproduce (and hopefully fix) this issue.

[mtrewartha]

I know the device supports notifications because it's a custom device we've developed and we've used the notifications with great success in the past. We're able to receive and read all of them via our Android app, so BlueZ should have no issues seeing them. That said, I don't think AcquireNotify is a feature of BLE notifications in general, but rather a feature of BlueZ (that allows you to use file descriptors to get notifications instead of D-Bus).

I haven't tried the command line utils yet, but that was going to be my next step, just to eliminate that as a piece of the puzzle.

Here's what the scenario looks like for us. Essentially, we're trying to offload a bunch of data (that spans a certain date range) from our custom peripheral:

1. The central connects to our custom peripheral and says "Hey, I'm going to need data covering date range X through Y" by writing the X and Y to a pair of start and end characteristics.
2. The central subscribes to notifications from a "data" characteristic that will contain the above mentioned data.
3. The central subscribes to notifications from a start/stop characteristic that will indicate when data notifications have stopped.
4. The central writes a "start" value to the start/stop characteristic that tells the custom peripheral to start triggering the data notifications.
5. Until the central receives a notification from the start/stop characteristic indicating that data notifications are complete, it receives the data notifications and caches the data from them in a buffer.
6. Once the start/stop characteristic sends a notification saying that data notifications have finished, the central wraps up the buffer and says "Hey, here's all that data we requested from dates X to Y."

So, importantly, we receive a lot of notifications in a very short amount of time from the data characteristic. I can't miss a single one of those, because each contains a small piece of the data we're offloading. It seems like some get passed through to the AbstractPropertiesChangedHandler, but some don't. I'll investigate a little more, just to verify that BlueZ is even seeing all of them, then report back.

[hypfvieh]

@miketrewartha
I guess that your feature is not working as dbus-java was not supporting the usage of FileDescriptor.
DBus supports a special type for file descriptors (UNIX_FD, DBus data type 'h', see DBus Spec).

I just added the FileDescriptor stuff to dbus-java in the latest commit.
Maybe you grab that version and retry with your device .. At least in the unit test it seems to work ;)

[mtrewartha]

@hypfvieh in the last couple days, I begrudgingly switched over to TinyB to see if it would work. It didn't, which led me to believe maybe BlueZ's D-Bus interface just doesn't support this high-throughput use case very well. However, I did some more debugging with my firmware engineer and we discovered that the Intel NUC I'm developing on was just randomly disconnecting and that was the cause of my issue. I switched over to a new BLE dongle and the issue suddenly disappeared! That said, I'm not using this library anymore. If I get a chance, I may switch back and give your new stuff a shot. If I do, I'll report back for sure. Thanks for the help!

[falmanna]

Hi @vkolotov , were you successful with replacing tinyb with this lib in your openhab binding?

[moontide]

@hypfvieh

Thanks for the great project, I can get the track changed information of org.bluez.MediaPlayer1.

I also want to get the track changed information of firefox media player, exposed by org.mpris.MediaPlayer2.Player , i think.

I can see information about org.mpris.MediaPlayer2.Player in dbus-monitor command output.

signal time=1679021922.354447 sender=:1.81 -> destination=(null destination) serial=605 path=/org/mpris/MediaPlayer2; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.mpris.MediaPlayer2.Player"
   array [
      dict entry(
         string "Metadata"
         variant             array [
               dict entry(
                  string "mpris:trackid"
                  variant                      object path "/org/mpris/MediaPlayer2/firefox"
               )
               dict entry(
                  string "xesam:title"
                  variant                      string "Romance"
               )
               dict entry(
                  string "xesam:album"
                  variant                      string "La Voix Des Anges"
               )
               dict entry(
                  string "xesam:artist"
                  variant                      array [
                        string "Dominica"
                     ]
               )
            ]
      )
   ]
   array [
   ]

But I can't get this information in dbus-java. I printed _s.getPropertiesChanged() after if (_s!=null) { ( https://github.com/hypfvieh/sandbox/blob/master/src/main/java/com/github/hypfvieh/sandbox/bluez/SimpleBluez.java#L132 ), but I didn't see anything about mpris. Am I doing it wrong?

[hypfvieh]

@moontide Please open a new ticket for that, thanks.