Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSX PKG signing #1340

Closed
totaam opened this issue Oct 13, 2016 · 37 comments
Closed

OSX PKG signing #1340

totaam opened this issue Oct 13, 2016 · 37 comments

Comments

@totaam
Copy link
Collaborator

totaam commented Oct 13, 2016

Follow up form #641. (see also #840)

Links:

Once the certificate have magically appeared in Xcode and the keychain (..), we can sign the Xpra.app bundle, the PKG and DMG files:

codesign --deep --force --verify --verbose --sign "Developer ID Application: Antoine Martin (XXXXXXXXXX)" Xpra.app
codesign --deep --force --verify --verbose --sign "Developer ID Application: Antoine Martin (XXXXXXXXXX)" Xpra.dmg
productsign --sign "Developer ID Installer: Antoine Martin (XXXXXXXXXX)" Xpra.pkg Xpra-signed.pkg

The signature can be verified with:

codesign --verify --verbose=2 Xpra.app
spctl --assess --verbose=4 --type=execute Xpra.app
spctl --assess --verbose=4 --type=execute Xpra.dmg
spctl --assess --verbose=4 --type=install Xpra-signed.pkg

But this fails for signing "Xpra.app" if I keep the Xpra_NoDock.app nested inside the Xpra.app, it complains:

image/Xpra.app: invalid Info.plist (plist or signature have been modified)
In subcomponent: /Users/osx/Xpra-svn/trunk/osx/image/Xpra.app/Contents/Xpra_NoDock.app

Removing the nested app fixes this, but there must be something I can change in the Info.plist (or elsewhere?) to be able to bundle it.

@smo: what am I missing?

@totaam
Copy link
Collaborator Author

totaam commented Oct 13, 2016

@totaam
Copy link
Collaborator Author

totaam commented Oct 14, 2016

2016-10-14 06:05:06: antoine commented


Despite the hopelessly unhelpful and misleading error message from the codesign tool (..), I've narrowed it down through trial and error (mostly error):

  • removing all symlinks from the subapp passes
  • removing Contents/MacOS from the subapp passes
  • using the Info.plist from the main app passes

Bringing the Info.plist closer to the main one "fixes" things, done in r14152.

Along the way I found that changing CFBundleExecutable is not possible in the subapp!? It's fortunate that we don't use this executable directly.

Now onto app submission. Let's hope it is as much fun as this, not.

@totaam
Copy link
Collaborator Author

totaam commented Oct 14, 2016

2016-10-14 13:32:50: antoine commented


r14154 + r14155 adds hooks to do the signing as part of the app / dmg / pkg scripts.

So until I move to a newer build host, the process is:

  • build the app (currently on a 10.5.8 VM):
./make-app.sh
  • copy the "Xpra.app" over to the machine doing the signing, ie:
rsync -rplogt --delete "image/Xpra.app" osx@osx1010:/Users/osx/Xpra-svn/trunk/osx/image/
  • login to the signing VM and unlock the keychain:
security unlock-keychain -p THEACCOUNTPASSWORD

(see User interaction is not allowed: trying to sign an OSX app using codesign)

  • run the remaining part of the build:
cd Xpra-svn/trunk/osx
export CODESIGN_KEYNAME="Antoine Martin"
./sign-app.sh && ./make-DMG.sh && ./make-PKG.sh 

Appstore fun:

  • follow steps in comment:2
  • install ApplicationLoader 3.0
  • watch in despair as the tool fails with yet another set of cryptic messages - for which there does not seem to be any reference or documentation anywhere, some have been fixed (ie: r14160 for "ITMS-90242"), most have not:
  • "ITMS-90251" : The product archive is invalid. Javascript may not be used in the installer.
  • "ITMS-90230" : Invalid product archive metadata. Error in keyPath [product-metadata.product-identifier]...
  • "ITMS-90230" : Invalid product archive metadata. Error in keyPath [product-metadata.product-version]...
  • "ITMS-90237" : The product archive package's signature is invalid. Ensure that it is signed with your "3rd Party Mac Developer Installer" certificate
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Path "Applications/Xpra.app/Contents/Resources/lib/python2.7/numpy/f2py/tests/src/assumed_shape/.f2py_f2cmap" in Payload is missing from Bom.
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra.app" has identifier "Xpra" in PackageInfo but "org.xpra.Xpra" in Payload
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra.app" has version CFBundleVersion="1.3.0" in PackageInfo, but CFBundleShortVersionString="1.0", CFBundleVersion="1.0" in Payload
  • "ITMS-90000" : This bundle is invalid. Package "org.xpra.pkg": Bundle at "Appplications/Xpra.app/Contents/Xpra_NoDock.app" in Payload not declared in PackageInfo
  • "ITMS-90000" : Bad Bundle Executable. You must include a valid CFBundleExecutable key in your bundle's information property list file
  • "ITMS-90000" : Unsupported toolchain. Packages submitted to the App Store must be created either through Xcode, or using the productbuild tool..

@totaam
Copy link
Collaborator Author

totaam commented Oct 15, 2016

2016-10-15 08:03:01: antoine commented


More fixes:

What remains:

  • for javascript: About Distribution Definition Files: Important: For applications distributed on the App Store, use Xcode to make your installer package. If you need to specify system requirements, use the Pre-install Requirements Property List in the Xcode build settings. For more details, see the productbuild man page. - not sure how to fix this one
  • product-metadata issues: Distribution XML Reference: product - sadly, adding <product id="org.xpra.Xpra" version="$VERSION" /> to the Distribution file makes things worse, Application Loader then claims not to find the file you click on: The file \U201cXpra-1-1.0-[r14164](../commit/021494b3d6745a72116ff5bccd2da6dada5c3051)\U201d couldn\U2019t be opened because there is no such file (this error is just garbage)
  • CFBundleExecutable could be caused by the lack of signature on "Xpra_Launcher"?
$ ls -al@ image/Xpra.app/Contents/MacOS/
total 24
drwxr-xr-x   5 osx  staff   170 14 Oct 20:00 .
drwxr-xr-x  10 osx  staff   340 14 Oct 20:03 ..
-rwxr-xr-x@  1 osx  staff  2752 10 Oct 21:44 PythonExecWrapper
	com.apple.cs.CodeDirectory	 141 
	com.apple.cs.CodeRequirements	 180 
	com.apple.cs.CodeSignature	8527 
-rwxr-xr-x@  1 osx  staff   399 10 Oct 21:44 Xpra
	com.apple.cs.CodeDirectory	 128 
	com.apple.cs.CodeRequirements	 164 
	com.apple.cs.CodeSignature	8526 
-rwxr-xr-x   1 osx  staff   292 14 Oct 20:00 Xpra_Launcher

See also:

"/Applications//Application Loader.app/Contents/Frameworks/ITunesSoftwareService.framework/Support/altool" -v -f ./image/Xpra-*.pkg -u 'THEAPPLEID' -p 'THEPASSWORD'

@totaam
Copy link
Collaborator Author

totaam commented Oct 15, 2016

2016-10-15 09:41:53: antoine commented


Alternatively using productbuild:

productbuild --component ./image/Xpra.app \
    /Applications \
    ./image/Xpra-1.0.pkg  \
    --sign "3rd Party Mac Developer Installer"

The resulting installer package lacks any icons or text but seems to install OK.
Only one error remains for "Application Loader" upload for this one: ITMS-90259: Bad Bundle Executable...

Or even:

pkgbuild --analyze --root ./image/Xpra.app ./image/XpraComponents.plist
#edit XpraComponents.plist and set `BundleIsRelocatable` to false
pkgbuild --root ./image/Xpra.app \
    --component-plist ./image/XpraComponents.plist \
    --install-location "/Applications" \
    ./Xpra.pkg
#generate the Distribution.xml file:
productBuild --synthesize --package ./Xpra.pkg ./Distribution.xml
# modify it using the same data as found in our make-PKG.sh script, and:
# change "#base.pkg" to "#Xpra.pkg"
# add the product id and version line
productbuild --distribution ./Distribution.xml --package-path . \
    --sign "3rd Party Mac Developer Installer" \
    ./Xpra-Installer.pkg

Same "bad bundle executable" error, and also missing the 512x512 icons...


More links:

@totaam
Copy link
Collaborator Author

totaam commented Oct 15, 2016

2016-10-15 21:06:35: antoine commented


I can get a little bit further by:

  • replacing the "MacOS" shell scripts with a python binary (temporarily) - scripts should work though, as long as they are more than 28 characters long (this is not a joke: [http://superuser.com/a/463673/58851])
  • deleting the "no dock" sub-app (temporarily)
  • deleting the "Helpers" (temporarily)
  • signing all "dylib", "so", "MacOS/*" and "Resources/bin" files using:
codesign --force --verbose --sign "3rd Party Mac Developer Application" --entitlements ./Xpra.entitlements  THEFILE

The entitlements file was added in r14166. (it's not clear if the same entitlements should be used for all the files: I doubt the system is capable of isolating the code running in libraries from its entrypoint)

  • signing the "image/Xpra.app" after that

Then you need to make an upload to itunes... which takes a lot of time and effort. And then you get some more errors via email! (this is some kind of bad joke) Invalid Signature: (..) satisfies its Designated Requirement test-requirement: code failed to satisfy specified code requirement(s) - could they have worded it more ambiguously perhaps?
Before I started signing the "so" / "dylib" / etc, the error was different for those: code object is not signed at all In architecture: i386).
Rinse and repeat until you find the right combination.
But you can't delete versions on there, so you have to bump the CFBundleVersion every time. How convenient.


More links:

@totaam
Copy link
Collaborator Author

totaam commented Oct 16, 2016

2016-10-16 16:05:09: antoine uploaded file osx-non-relocatable.patch (1.1 KiB)

to make the package non relocatable

@totaam
Copy link
Collaborator Author

totaam commented Oct 16, 2016

2016-10-16 17:23:20: antoine uploaded file osx-make-appstore-pkg.sh (4.5 KiB)

work in progress to generate a PKG compatible with the appstore

@totaam
Copy link
Collaborator Author

totaam commented Oct 16, 2016

2016-10-16 17:25:28: antoine commented


A big part of the difficulty is that it's non-trivial to embed shell scripts... Because shiny. Or something. (and also because it takes way too long to test anything)

With the patch attached, the codesign step shows:

(..)
image/Xpra.app/Contents/Helpers/Bug_Report: replacing existing signature
image/Xpra.app/Contents/Helpers/Bug_Report: signed generic [Bug_Report]
(..)

Note that is uses "generic" and not "signed Mach-O thin (i386)" like it does for all the binaries. And then much much later (..), you get an email (..) saying the build cannot be uploaded:

org.xpra.Xpra.pkg/Payload/Xpra.app: code object is not signed at all
In subcomponent: org.xpra.Xpra.pkg/Payload/Xpra.app/Contents/Helpers/Bug_Report

See:

Maybe we can change the wrappers so that the "MacOS" binary executes the real "Python" executable (renamed if needed) with the script as argument instead. (and move the scripts somewhere else..)

Still TODO:

  • fix sound subapp
  • postinstall script with symlinks to /usr/bin
  • agent
    etc.

@totaam
Copy link
Collaborator Author

totaam commented Oct 17, 2016

2016-10-17 11:05:48: antoine commented


A gem in [http://prod.lists.apple.com/archives/xcode-users/2013/Oct/msg00067.html]:
In addition to my entitlements file I needed to create an info.plist file and embed it into the binary by adding the following linker flag:

-sectcreate __TEXT __info_plist SandboxUnixCommand/SandboxUnixCommand-Info.plist

Well, obviously... ahem. Magic TEXT section values!
More info here: Gimmedebugah: how to embedded a Info.plist into arbitrary binaries

@totaam
Copy link
Collaborator Author

totaam commented Oct 17, 2016

2016-10-17 12:33:04: antoine uploaded file osx-make-appstore-pkg-v2.sh (6.2 KiB)

updated patch

@totaam
Copy link
Collaborator Author

totaam commented Oct 17, 2016

2016-10-17 14:22:56: antoine commented


Lots more nasty tricks to beat this mess into shape:

  • r14167 + r14174: dynamically populate version / revision / buildno during build
  • r14168: fix typo
  • r14169 + r14171 + r14172 + r14173 + r14176 + r14180: move things aroung to make codesign happy (some of these may not be mandatory - trial and error..)
  • r14170: we don't use the webcam yet, remove the entitlement
  • r14175: we need this weird entitlement!?
  • r14177: fix code stripping (backport in 14182)
  • r14178: fix "Bug Report" launch script (backport in 14181)
  • r14183: appstore packaging script

@totaam
Copy link
Collaborator Author

totaam commented Oct 18, 2016

2016-10-18 16:19:30: antoine commented


At last a package that both runs and uploads to the appstore:

@totaam
Copy link
Collaborator Author

totaam commented Oct 20, 2016

2016-10-20 06:14:47: antoine uploaded file appstore.png (78.2 KiB)

appstore view
appstore.png

@totaam
Copy link
Collaborator Author

totaam commented Oct 20, 2016

2016-10-20 06:15:22: antoine commented


as shown on the [https://itunesconnect.apple.com/] appstore control page:
[[Image(appstore.png)]]

@totaam
Copy link
Collaborator Author

totaam commented Oct 20, 2016

2016-10-20 12:54:34: antoine commented


Sound sub-app fixes:

  • r14223: move binaries to "/Resources/bin"
  • r14224 + r14226: run the "sandbox inherit" version of the sound binary
  • r14225, r14222: minor related tweaks

@totaam
Copy link
Collaborator Author

totaam commented Oct 23, 2016

2016-10-23 07:53:25: antoine commented


Some path fixes and updates:

  • r14266 (backport in 14267)
  • r14268
  • r14269: generate an SSL cert (but postinstall script is still not used in appstore version..)

Ideally, we would store our preferences in /Library/Preferences, but:

  • there is no API to get the path to this directory (sigh), see NSSearchPathDirectory
  • we are only supposed to use XML plist files in there? (sigh)

Then, we could handle Migrating an App to a Sandbox and have the preferences moved for us to the user sandbox.

@totaam
Copy link
Collaborator Author

totaam commented Oct 28, 2016

2016-10-28 11:02:27: antoine commented


I don't think this is going to happen anytime soon. Re-scheduling.

Their complaints after reviewing the app:

  • load session file dialog can trigger errors if the folders are not accessible due to sandboxing (ie: the home folder is not), not much we can do here since the file selection dialog is GTK's - we could just remove the load and save options for apple users (simplicity at its best), or use a different file selection tool (native somehow?), or remove the sanbox restrictions? (yuk)
  • name of the app doesn't match ("Launcher" vs "Xpra") and "Quit" is "Disconnect Launcher" - no biggie, this could be changed easily enough
  • we require sandboxing exceptions:
  • com.apple.coreservices.launchservicesd is needed to for pyobjc "coreservices" - not much we can do about this - maybe try to silence / ignore the errors this causes, or report the pyobjc issue upstream
  • com.apple.systemuiserver.screencapture is needed for using xpra as a server - they unhelpfully suggested: We encourage you to investigate other ways of implementing the desired functionality.

@totaam
Copy link
Collaborator Author

totaam commented Nov 19, 2016

2016-11-19 06:25:55: antoine changed title from OSX PKG signing - appstore to OSX PKG signing

@totaam
Copy link
Collaborator Author

totaam commented Nov 19, 2016

2016-11-19 06:25:55: antoine commented


Moving the appstore packaging to a new ticket: #1366

This purpose of this ticket is now only about signing the PKG.
At least some of the latest PKG builds should be signed... I should make it easier to know which PKG are signed and which ones are not. (requires using a newer VM to sign.. which won't boot right now on my systems.. even with the Intel one..)

I'm not entirely sure what the actual difference is for users when installing signed vs unsigned packages. I've paid the fee for the privilege of having Apple sign the key. Does this make it easier to install the PKG on OSX versions with the more stringent security settings?

@totaam
Copy link
Collaborator Author

totaam commented Nov 21, 2016

2016-11-21 16:35:21: antoine commented


Success. I have fixed my signing 10.10.x VM (it needs vbox 5.0 to boot, fails with 5.1 - fixes for running 5.0 on Fedora 25: Error: nothing provides libvpx.so.3()(64bit) on Fedora 25 and Runtime/r0drv/linux: 4.8 mod_timer_pinned fix).

Uploaded both DMG and PKG r14467 builds signed with my developer key.
@afarr: do these signed builds help with installation on newer versions of macosx? (you may need to turn the "stricter installation checks" setting back on again)

@totaam
Copy link
Collaborator Author

totaam commented Dec 2, 2016

2016-12-02 23:35:52: afarr changed owner from afarr to antoine

@totaam
Copy link
Collaborator Author

totaam commented Dec 2, 2016

2016-12-02 23:35:52: afarr commented


Odd.

I have an r14467 .dmg installed in my usual odd place - deleting and re-installing into the more usual /Applications directory works as usual, and likewise with the r14492 .dmg.

Double clicking on them, however, they fail the "Verifying 'Xpra'" validation test, "because it is from an unidentified developer".

I can't find any sign of the r14467 pkg though, or .dmg, in either the /dists or the /beta directories (so, where did I get the r14467 .dmg I already have?).

Trying to install the r14492 .pkg I get the usual security/unsigned (unknown developer) issue, but I'm not sure you signed that one.

I think you'll need to post another signed .pkg or even .dmg for me to check.

@totaam
Copy link
Collaborator Author

totaam commented Dec 6, 2016

2016-12-06 16:34:58: antoine commented


1.0 should be signed, just released today.
(the other ones disappeared because I needed to free some diskspace)


Note to self since this is not obvious, this is how we make the signed DMG / PKG:

  • run ./make-app.sh on the build VM
  • copy the Xpra.app to the signing VM
  • unlock the keys on the singing VM: security unlock-keychain -p THEACCOUNTPASSWORD
  • run ./make-DMG.sh && ./make-PKG.sh on the signing VM

@totaam
Copy link
Collaborator Author

totaam commented Dec 9, 2016

2016-12-09 01:46:40: afarr commented


Checking in xpra.org/beta/osx ... the 1.0 r14492 pkg (http://xpra.org/beta/osx/Xpra-1.0-[r14492](../commit/3e5fb41d1498a516d1602a98296b4329af368770).pkg) as well as the xpra.pkg (http://xpra.org/beta/osx/Xpra.pkg) both, once downloaded, trigger a security warning when trying to install (“Xpra.pkg” can’t be opened because it is from an unidentified developer.).

They indicate that they were posted/updated 11/27.

Handing this back to you... and chuckling that my username seems to have become 'alas' (I feel three breaths closer to Elizabethan-dom ... three breathless breaths, whatever that means).

@totaam
Copy link
Collaborator Author

totaam commented Dec 26, 2016

2016-12-26 09:46:56: antoine changed owner from antoine to afarr

@totaam
Copy link
Collaborator Author

totaam commented Dec 26, 2016

2016-12-26 09:46:56: antoine commented


1.0 was released, meaning it is no longer in the beta area! Try here instead: [http://xpra.org/dists/osx/x86/].
(and the files in the beta area will soon disappear altogether to make way for 2.0 builds)

As for 'alas', that's your "name" in your account settings. The trac update now shows the name rather username. You should be able to change it yourself from your account profile - but it does have a nice ring to it, doesn't it?

@totaam
Copy link
Collaborator Author

totaam commented Jan 7, 2017

2017-01-07 11:26:43: antoine commented


1.0.1 is also signed.

@totaam
Copy link
Collaborator Author

totaam commented Jan 9, 2017

2017-01-09 18:02:27: afarr uploaded file ticket1340_xpra_1-0-1-r14723-pkg-double-click-install-response_10-12-and-10-10-5.png (84.5 KiB)

unknown developer error when installing .pkg
ticket1340_xpra_1-0-1-r14723-pkg-double-click-install-response_10-12-and-10-10-5.png

@totaam
Copy link
Collaborator Author

totaam commented Jan 9, 2017

2017-01-09 18:06:36: afarr commented


Err — alas... I seem to be triggering the same "unknown developer" error with the new 1.0.1 (Xpra-1.0.1-14723.pkg) that I was mentioning getting with 1.0 in comment:20 (which I also repeated with Xpra-1.0-14502).

I got the errors with the new 10.12.1 super-shiny mac OS, and then confirmed I'm also getting it with a 10.10.5 significantly-less-shiny mac.

Same message with 1.0 and 1.0.1, on 10.12.1 & 10.10.5.

[[Image( ticket1340_xpra_1-0-1-14723-pkg-double-click-install-response_10-12-and-10-10-5.png)]]

Apple must have more hoops for you?

@totaam
Copy link
Collaborator Author

totaam commented Mar 12, 2017

2017-03-12 09:16:58: antoine commented


Does it work any better with the beta 2.0 builds?
Those are built and signed on the same system.

@totaam
Copy link
Collaborator Author

totaam commented Jun 30, 2017

2017-06-30 18:47:44: maxmylyn changed owner from afarr to antoine

@totaam
Copy link
Collaborator Author

totaam commented Jun 30, 2017

2017-06-30 18:47:44: maxmylyn commented


Sadly, the latest 2.1 betas are still complaining about unidentified developers.

@totaam
Copy link
Collaborator Author

totaam commented Jun 30, 2017

2017-06-30 18:48:30: maxmylyn uploaded file 1340 unsigned 2.1.png (50.5 KiB)

1340 unsigned 2.1.png

@totaam
Copy link
Collaborator Author

totaam commented Jun 30, 2017

2017-06-30 19:19:06: antoine commented


The certificate expired a few months back so I stopped signing the builds, you may want to try a build made around the time of comment:25 instead. (2.0.x)

@totaam
Copy link
Collaborator Author

totaam commented Jul 14, 2017

2017-07-14 21:55:06: maxmylyn commented


Retried with a build from mid March, and still got the error, and that was the oldest 2.0 build available in the betas folder (dists had nothing relevant unless I missed something that obvious..which is possible).

Either way, based on the conversations we've had, I'm going to close this - if we want to revisit OSX signing, we can re-open this, but in the meantime, closing.

(As an aside, I think we need a "crickets" resolved option, but that's just me)

@totaam totaam closed this as completed Jul 14, 2017
@totaam
Copy link
Collaborator Author

totaam commented Aug 20, 2020

2020-08-20 05:29:08: antoine commented


See also #2015, #2441.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant