Recipient store has moved into db - scli no longer starts

[maximbaz]

scli used to read some info on startup from recipient store files, but with signal-cli 0.11 the data has been moved to the db, causing the scli to fail to launch.

AsamK/signal-cli@862c2fe

[maximbaz]

I feel like this is a very hacky patch made out of desperation to get back a working desktop version of Signal, but I want to leave it here as a reference point, on how to for example get (some of?) the necessary info from database. Hope it helps!

Patch

diff --git a/scli b/scli
index 07811b6..592a220 100755
--- a/scli
+++ b/scli
@@ -18,6 +18,7 @@ import shutil
 import signal as signal_ipc
 import subprocess
 import sys
+import sqlite3
 import tempfile
 import textwrap
 from abc import ABC, abstractmethod
@@ -1073,17 +1074,21 @@ class SignalData:
         try:
             indivs = self._get_recipients()
         except FileNotFoundError:
-            indivs = self._get_recipients_v1()
+            try:
+                indivs = self._get_recipients_v1()
+            except KeyError:
+                indivs = self._get_recipients_v2()
 
         groups = []
-        for g in get_nested(self._data, 'groupStore', 'groups', default=()):
-            if is_group_v2(g):
-                group_id = g['groupId']
-                cached_name = self._get_group_v2_cache_name(group_id)
-                g['name'] = cached_name or group_id[:10] + '[..]'
-            if g.get('archived') or not g.get('name'):
-                continue
-            g['name'] = strip_non_printable_chars(g['name'])
+        account_db_file = os.path.join(self._file_path + '.d', 'account.db')
+        con = sqlite3.connect(account_db_file)
+        cur = con.cursor()
+        for group_id, group_data in cur.execute("select group_id, group_data from group_v2"):
+            g = {
+                "groupId": base64.b64encode(group_id).decode('utf-8'),
+                "group_id": base64.b64encode(group_id).decode('utf-8'),
+                "name": strip_non_printable_chars(str(group_data)[:10] + '[..]'),
+            }
             groups.append(g)
 
         return indivs, groups
@@ -1196,6 +1201,34 @@ class SignalData:
             profile_names[num] = profile_name
         return profile_names
 
+    def _get_recipients_v2(self):
+        account_db_file = os.path.join(self._file_path + '.d', 'account.db')
+        con = sqlite3.connect(account_db_file)
+        cur = con.cursor()
+        indivs = []
+        for number, given_name, family_name, profile_given_name, profile_family_name in cur.execute("select number, given_name, family_name, profile_given_name, profile_family_name from recipient"):
+            if not number:
+                continue
+
+            name = ''
+            for name_part in (given_name, family_name):
+                if name_part:
+                    name += name_part
+
+            profile_name = ''
+            for name_part in (profile_given_name, profile_family_name):
+                if name_part:
+                    profile_name += name_part
+
+            cont = {
+                "number": number,
+                "name": strip_non_printable_chars(name) if name else None,
+                "profile_name": strip_non_printable_chars(profile_name) if profile_name else None,
+            }
+
+            indivs.append(cont)
+        return indivs
+
     @property
     def own_num(self):
         return self._data['username']

UPDATE: can be applied using $ patch -Np1 -i ./patch.patch (there will be a slight offset in line numbers, if you are applying on master, as the patch is based on #198)

[moppman]

I haven't upgraded to signal.cli 0.11 (or signal-cli 0.11.1) yet, but looking at the commit from the OP, it looks like the recipient store migration to the database is not easily revertible (unless you have a recipients-store backup file), so until maximbaz' patch (thanks!!) has been applied, we should probably put a big fat warning in the readme not to upgrade signal-cli, right ? 😅

[mark2185]

What about those that have already gotten their socks wet? 🙃

[maximbaz]

I lost some messages because my first attempt at the patch didn't account for groups also being moved (in another commit in signal-cli repo), but after that, the above patch seems to work fine - no guarantees though 😜 Except today signal-cli broke for different reason anyway, returning 499 code...

[vargn]

I've updated signal-cli to 0.11.2 and made sure it recieves messages and daemon runs, but still scli throws an error on start. How would I go about to apply the patch mentioned in this thread?

  File "/usr/bin/scli", line 1029, in parse_data_file
    indivs = self._get_recipients()
  File "/usr/bin/scli", line 1089, in _get_recipients
    with open(recipients_store_file, encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/.local/share/signal-cli/data/+46xxxxxxxxx.d/recipients-store'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/scli", line 5271, in <module>
    main()
  File "/usr/bin/scli", line 5253, in main
    coord = Coordinate()
  File "/usr/bin/scli", line 4514, in __init__
    self._contacts = Contacts(sigdata)
  File "/usr/bin/scli", line 1219, in __init__
    self.reload()
  File "/usr/bin/scli", line 1222, in reload
    indivs_dicts, groups_dicts = self.sigdata.parse_data_file()
  File "/usr/bin/scli", line 1031, in parse_data_file
    indivs = self._get_recipients_v1()
  File "/usr/bin/scli", line 1126, in _get_recipients_v1
    for cont in self._data['contactStore']['contacts']:
KeyError: 'contactStore'

[moppman]

@zer0lean
You need to apply maximbaz' patch from the second post in this thread.
In signal-cli v0.11 the recipients store has been migrated to the accounts.db database. This is why your scli complains about a missing recipients store file.

[mark2185]

To be fair, I couldn't apply the patch through git apply on master, for some reason. It's as if we're not both on master, I sure haven't made any changes though.

And after editing it manually, it still didn't solve the issue, but I didn't press any further, I used signal in a different way today.

[maximbaz]

Oh I completely forgot, the patch is actually on top of #198 as I rely on that feature daily 🙂
You can apply with $ patch -Np1 -i ./patch.patch, it would take care of the slight offset in the line numbers 👍

[vargn]

@moppman Yes, this I know. What I was asking is how I do this.

@maximbaz Please eli5 how I apply this patch the easiest way. Do I run $ patch -Np1 -i ./patch.patch in a terminal? That doesn't seem to make any sense to me. Do I edit the scli binary manually? Clearly i'm missing something here and there's an opportunity for me to learn. Hopefully you can point me in the right direction? :)

[maximbaz]

Sure!

• Clone this repo, open terminal and cd into the cloned folder.
• Copy the patch above and save it in a new file called issue-202.patch in the same folder.

(At this moment you are in terminal in a folder, where there is scli binary from latest master and the patch.)

• Now run patch -Np1 -i ./issue-202.patch and the scli binary in this folder will be updated.
• You can already run it using ./scli, do that to see if it works.
• If hopefully it does, probably you want to replace your system installation of scli with the patched one. Find where it is (executed which scli, it's likely in /usr/bin/scli), and then replace using sudo cp ./scli /usr/bin/scli).

Let me know if this makes sense!

[vargn]

Let me know if this makes sense!

This makes total sense now. I've never applied a patch like this before but there's a first time for everything.

Hunk #1 succeeded at 18 with fuzz 1.
Hunk #2 succeeded at 1029 (offset -45 lines).
Hunk #3 succeeded at 1156 (offset -45 lines).

Big thanks for a quick reply and thorough guide.

[maximbaz]

Sure thing! Did it actually work? I have confirmation from one other person that everything seems fine, it would be good to have one more, to reassure people who are otherwise blocked that they can safely use this patch, until the proper cleaner fix is developed.

[vargn]

I should've been more clear, yes it works!

[exquo]

I will make a fix for this as soon as I can. Unfortunately, I probably would not be able to attend to this for at least a few days..
In the meantime, @maximbaz's patch can be used (thanks!)

[exquo]

This problem stems from relying on signal-cli's internal file structure, rather than its external API. If we want to avoid this in the future, we should only interact with signal-cli through its documented calls, and not try to access its internal data (whose format changes often enough).

[Lager80]

I tried the patch as well, but I got this error.
Traceback (most recent call last): File "./scli", line 5359, in <module> main() File "./scli", line 5333, in main coord = Coordinate() File "./scli", line 4594, in __init__ self._contacts = Contacts(sigdata) File "./scli", line 1297, in __init__ self.reload() File "./scli", line 1300, in reload indivs_dicts, groups_dicts = self.sigdata.parse_data_file() File "./scli", line 1080, in parse_data_file indivs = self._get_recipients_v2() File "./scli", line 1209, in _get_recipients_v2 for number, given_name, family_name, profile_given_name, profile_family_name in cur.execute("select number, given_name, family_name, profile_given_name, profile_family_name from recipient"): sqlite3.DatabaseError: malformed database schema (recipient) - near "STRICT": syntax error

[exquo]

I have made a fix in the hotfix/202 branch. It should now work with the newer versions of signal-cli.
Please let me know if you notice any issues with these changes. If there aren't any problems, I'll merge it into the main branch and make a release shortly.
Sorry it took a while for me to get to this!

[maximbaz]

Works great for me, thanks! Nice solution with the contacts cache, by the way 😉

[moppman]

Another 👍 from me, hotfix/202 is working nicely with signal-cli 0.11.4

[Lager80]

With this Hotfix I get the following error:
File "/opt/scli/scli", line 993, in proc_callback
members_ids = set(proc.output.removeprefix(" array [")[:-1].split())
AttributeError: 'str' object has no attribute 'removeprefix'

[maximbaz]

Could it be that your python is older than 3.9?
README does currently promise python support >= 3.7, so this needs to be solved in any case 😉

[Lager80]

You were right, I was still using python3.8 updating the version fixed everything. The Hotfix is working for me now as well.

[exquo]

Thanks @Lager80 for reporting the error, and thanks @maximbaz for the quick troubleshooting!
The README indeed requires only Python >=3.7, and we should try to not raise this requirement if we don't have to.
I have pushed a commit that replaces the removeprefix() method with a simpler construction. It should now run on python 3.7 again.

[freed00m]

When will you merge it? @exquo

People that installed it from AUR can't run scli until merged.

 ~ $ scli
Traceback (most recent call last):
  File "/usr/bin/scli", line 1029, in parse_data_file
    indivs = self._get_recipients()
  File "/usr/bin/scli", line 1089, in _get_recipients
    with open(recipients_store_file, encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/frdm/.local/share/signal-cli/data/609968.d/recipients-store'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/scli", line 5271, in <module>
    main()
  File "/usr/bin/scli", line 5253, in main
    coord = Coordinate()
  File "/usr/bin/scli", line 4514, in __init__
    self._contacts = Contacts(sigdata)
  File "/usr/bin/scli", line 1219, in __init__
    self.reload()
  File "/usr/bin/scli", line 1222, in reload
    indivs_dicts, groups_dicts = self.sigdata.parse_data_file()
  File "/usr/bin/scli", line 1031, in parse_data_file
    indivs = self._get_recipients_v1()
  File "/usr/bin/scli", line 1126, in _get_recipients_v1
    for cont in self._data['contactStore']['contacts']:
KeyError: 'contactStore'

[moppman]

When will you merge it?

The fix has been merged 3 weeks ago 8ccae9f
Or did you mean a new release? Then again, it's a single python script after all. You could just copy scli from the master branch to your bin folder.

[freed00m]

Ha you're right, the scli-git AUR package is bonkers :D :O I thought it suppose to take latest git I am confused by pkgbuilds again