The future of the Objective-C crate ecosystem

[madsmtm]

Introduction

Hey @SSheldon, I really like your collection of Objective-C crates:

• objc
• objc-encode
• objc_exception
• objc_id
• objc-foundation
• block
• uikit
• (dispatch)

For better discoverability, maintainability, interoperability and usability (I will get to each of these), I would like to propose moving objc-encode, objc_exception, objc_id and objc-foundation (and maybe block and uikit? But probably not dispatch) into this repository.

Discoverability

objc (and by extension, it's dependencies) is popular and therefore easily discoverable, but we should strive to make the other crates receive some 👀 as well.

As you noted in SSheldon/rust-objc-id#4 (comment):

Not having wider adoption has slowed it down

As a very concrete example, I reimplemented almost the entirety of objc_id in #98 before realising that it existed.

Maintainability

We might want to change Sel to be an opaque struct like most of the other types, or add UnwindSafe bound to objc_exception::try - Making these changes are far easier to coordinate in a "monorepo".

Also, if a better way to do a general thing is discovered, it can easily be applied everywhere - for example e86d26a (making stuff opaque structs instead of uninhabited enums) haven't been done in objc_exception (see SSheldon/rust-objc-exception#9).

Finally, we'd only need to maintain stuff like CI in one place.

Interoperability

A lesser point, since objc-encode, objc_exception, objc and objc_id all form a dependency graph, but block for example needs Class but has to create it's own type to not depend on objc.

Also, it would be cool if Id<Block, Owned>/Id<Block, Shared> instead of RcBlock was a thing, that might be able to make Block::call (+ Block::call_mut) and impl FnMut/Fn for Block safe.

Usability

Users don't want to consider all the above crates, and whether they're fit for their use cases; e.g. winit reimplements objc_foundation::NSString and objc::rc::StrongPtr, partially for historical reasons, but I suspect also because there haven't been a clear user story on how to to these higher-level things.

fruity exists for a reason; users want easy and safe. (Though I think @nvzqz took the wrong approach creating everything from scratch, I'll probably contact them at some point in the future to see if we can combine efforts).

Concrete implementation

Make this a cargo workspace like core-foundation-rs, and move the crates in here (and maybe for objc_id just merge it into the objc crate?).

Additionally, we should make some more cargo features to make it easy to choose which parts of the library to compile (e.g. an rc feature, a message feature and so on).

I can make the PRs, but you'll have to mark the old repositories as archived and such.

Drawbacks and alternatives

• Git history (solvable) and GitHub issues (less so) would get mangled up.
• There's less room for exploration, having something as a separate crate let's you easier (also solvable, we could have "unstable_x" features or just have the crate in the workspace).
• The discoverability issue could be mitigated with better linking to the different repositories in documentation.

Outtro

We can talk about it over a quicker medium if you prefer.

Also, if you're loosing interest in the project(s) and need a hand with the maintainer role, I'll gladly help out.

[simlay]

Also, if you're loosing interest in the project(s) and need a hand with the maintainer role, I'll gladly help out.

I've been watching this repo for a couple years now and no shade to @SSheldon but I think he could use a co-maintainer.

I added mediocre objective-c support to rust-bindgen last year and used that to make uikit-sys which uses a bunch of the objc repos. I think more maintenance would really add to the macOS and iOS rust ecosystem.

[SSheldon]

I've become disillusioned that much of this is meaningful/valuable. Originally I was really frustrated by Objective-C's lack of safety and I thought Rust could provide a much safer way to deal with that. But Swift had just been announced and has since gotten a ton more adoption and has incredible interop with Objective-C. Trying to write safe wrappers around Foundation and UIKit proved to be a massive undertaking, and I'm not sure why anyone would want to use them rather than just writing Swift. The widest adoption I've seen this crate ecosystem used for has been writing small unsafe shims to call Objective-C, like game devs use for graphics stuff. So writing safe wrappers over ObjC to make it fit Rust's model was a fun exercise for a while, but I can't see an endgame where using this is a better idea than writing Swift. It seems like the goal should be trying to minimize the amount of interaction that Rust and ObjC have rather than trying to drive more ObjC from Rust.

[chrisballinger]

Now that Swift has a stable ABI, I wonder if it's worth investigating directly bridging between Rust and Swift instead of relying on an Objective-C middle layer. Seems like it's possible with a little more compiler support: https://internals.rust-lang.org/t/using-swift-abi-from-rust/6307/22

[simlay]

Now that Swift has a stable ABI, I wonder if it's worth investigating directly bridging between Rust and Swift instead of relying on an Objective-C middle layer. Seems like it's possible with a little more compiler support: https://internals.rust-lang.org/t/using-swift-abi-from-rust/6307/22

There's actually been a start on this: rust-lang/rust#64582 as well as some initial layout for a swift-bindgen.

[madsmtm]

I've become disillusioned that much of this is meaningful/valuable.

I'm sad to hear that, but I've been there myself, it's fully understandable!

Trying to write safe wrappers around Foundation and UIKit proved to be a massive undertaking

You're probably right in this, though this is by far not my primary objective - see below.

and I'm not sure why anyone would want to use them rather than just writing Swift.

For me: Because Rust applications can be cross-platform.

The widest adoption I've seen this crate ecosystem used for has been writing small unsafe shims to call Objective-C, like game devs use for graphics stuff.

This is actually exactly why I came here; because I want to help improve this situation, and these use cases safer!

As an example, currently winit has their own objc_id::Id-like implementation, and still has quite a lot of memory leaks because people forget to use it.

They also have their own NSString helpers, so while it may not be needed to write wrappers on the entirety of Foundation and UIKit, it would be really nice to have safe wrappers around the commonly used things like NSString and NSArray.

EDIT: Ooops, I already mentioned the stuff about winit.

[madsmtm]

worth investigating directly bridging between Rust and Swift instead of relying on an Objective-C middle layer

Definitely interesting, though I think there's merit in having both this library and the Swift bridging. Actually, would it even work with frameworks (like Foundation) originally written in Objective-C? Like, does there exist some "Swift-Foundation" framework shim that calls the Objective-C Foundation framework or what?

Also, Swift requires at least OS X Mavericks 10.9, where Rust (and by extension, probably this library) would work on versions as low as OS X Lion 10.7, see rust-lang/release-team#2.

And a minor note: For the few people that use it, the GNUStep support that this library has probably wouldn't work with Swift.

[madsmtm]

@SSheldon I'm sorry to press on but at some point I and others need to know how to move forwards.

It is totally cool if you consider this library feature complete and good enough as-is, in that case I'll just take my suggestions somewhere else / create a fork and move on.
But I would much rather have the entire community benefit from improvements I and others may find, so if it's because you're burnt out of the projects then I hope we can find an agreement that can work for you. Maybe someone else takes on the maintainer role for a while, or we move the projects under some GitHub organization like @rust-windowing, or something!

Wish you all the best!

[SSheldon]

Hey @madsmtm! I don't feel that any of the open PRs currently merit the undertaking of users updating to a new breaking version of objc. But for example, I think the adoption of extern types could be a compelling reason, when they stabilize. If there are severe issues in objc or big improvements to be made, I am still interested in making those changes or reviewing and accepting PRs for them, but unfortunately I don't know of anything that currently rises to that level. This could be a reflection of the disillusionment I referenced before, and I'm sorry if that's the case.

Perhaps I'm too conservative here, and I recognize this risks hurting the project by stifling innovation and discouraging contributions. It's hard for me to shift my mindset and think of a passion project in that way.

If you have a vision for ways that the rust-objc ecosystem could be significantly better than it is currently, perhaps the best way forward is a fork. I'd love to see how all of your ideas can work together, and if there's enough demand from the rust community for these improvements perhaps that does mean it's time for me step down as objc maintainer.

[madsmtm]

I fully understand and very much respect that you want to be conservative on updating objc! When we do get there, we could perhaps also utilize the semver trick to minimize breakage. I'll probably also volunteer to update some of the biggest dependencies like winit, cocoa, clipboard and such.

But wanting to be conservative and only making major version bumps when absolutely required doesn't have to stifle innovation and discourage contribution!
A concrete suggestion: You could make a branch called 0.3 or something, and all breaking changes (and most bigger changes) would just go into that, until we at some point (e.g. when/if extern types are stabilized) release a new version. You could even release alpha/beta/pre-release versions (e.g. 0.3.0-alpha.0), so that users could test it out and pin it in their own test crates before the final version is released.
This would be vastly better than me making a fork and making an infinitude of changes that we'd then maybe at some point agree should go back into this crate. With a branch, you and others could review each change separately, so that we keep the code quality high. And equally importantly for me, I'd know that the code I write is not in vain 😉.

But no, I don't think I have many radical new ideas; I simply have a desire to fix soundness issues, make the library safer and make it more user friendly (better documentation and more examples is a big one in here), to in the end encourage a bigger ecosystem of crates using native macOS features.

And by no means do I want to push you off the maintainer pedestal, it's been great having someone criticize my work so far, forcing me to formulate my reasoning (and actually discovering a flaw in some of my previous reasoning). So thank you!

[madsmtm]

Alright, I yield, I'll take my suggestions elsewhere / create a fork.

On a related note, how would you feel (a license is one thing, being considerate is another 😉) about me copying code snippets semi-verbatim / with a lot of inspiration from these crates to fruity or a similar crate? (I haven't actually talked to @nvzqz yet, so this might not be relevant).

And how would you prefer attribution?

• git author / GitHub co-author?
• An overall mention in the README?
• A mention near the specific code where heavy inspiration is taken?
• Something else?

[madsmtm]

Alright, so I went and made a fork (still quite WIP), and merged the linked projects into that: objc2. The repository contains the following new crates:

• objc2 (objc_id and objc-exception has been merged into this)
• block2
• objc2-encode
• objc2-foundation

These crates contain fixes to a lot of upstream issues, you can see which ones in madsmtm/objc2#22. Unfortunately this requires some gnarly breaking changes, especially in how Encode works, so if you want to try it out I suggest you follow the migration guide in the README.

In addition, I've created two new -sys crates, objc-sys and block-sys, which is used to specify the linking to system libraries. This has been a huge boost for proper GNUStep support!
I've used these names because I would like to use them in fruity, objc, block and others at some point, but if you, @SSheldon, feel that the names are "yours", I'll be happy to transfer ownership.

I'm hoping to mature these crates over time, and start integrating them into some of the bigger projects (e.g. winit, metal-rs, core-foundation-rs) in the future, because I believe my fork contains important soundness and usability fixes. However, my wish is not to divide the ecosystem, and I still hope to merge some (if not most) of my changes back into this project.

Thanks for your time!

P.S. I'm sorry I haven't released the fork earlier, I hadn't decided on a name for all the crates yet, and didn't want to spend effort on a later crate-renaming.

[madsmtm]

Oh, and a final thing, @SSheldon, then I won't bother you again for a while: Would you be willing to let me relicense your work under a dual MIT/Apache-2.0 license? Mostly to bring the crates more in-line with other Rust projects.

I'd also like to remove the copyright notice from the MIT license (see rust-lang/rust@2a8807e), it would be nice if you could agree to that as well?

I will of course be getting permission from other contributors before I make the final relicensing, see madsmtm/objc2#23.

[comex]

worth investigating directly bridging between Rust and Swift instead of relying on an Objective-C middle layer

Definitely interesting, though I think there's merit in having both this library and the Swift bridging. Actually, would it even work with frameworks (like Foundation) originally written in Objective-C? Like, does there exist some "Swift-Foundation" framework shim that calls the Objective-C Foundation framework or what?

Belated answer, but: sort of.

There is /usr/lib/swift/libswiftFoundation.dylib, which contains some manually-written wrappers and extensions for Foundation APIs; the corresponding Swift "header file" is <sdk>/usr/lib/swift/Foundation.swiftmodule/<arch>.swiftinterface.¹

However, that is only for APIs that need an extra level of customization. Many of the APIs that become available when you import Foundation in Swift instead just come through Swift's Objective-C importer. A large fraction of these are given custom names for Swift, but these custom names are not wrapper methods; instead, the Objective-C importer uses a YAML file, <sdk>/System/Library/Frameworks/Foundation.framework/Headers/Foundation.apinotes, which contains a long list of mappings from Objective-C names to Swift names. (Swift custom names can alternately be provided in the header files themselves. And beyond naming, there are some other customizations available, which can be provided in the apinotes file, the headers themselves, or both. It's a bit of a mess.)

At the binary level, APIs imported with the Objective-C importer do not have wrappers in libswiftFoundation.dylib. Instead, compiled Swift binaries access them the same way that Objective-C binaries do. Classes and global variables are imported directly from /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (using their Objective-C names), while methods are invoked with objc_msgSend (using their Objective-C selector names).

All of the above is not specific to Foundation but also applies to other system frameworks.

Footnotes

1. e.g. /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/swift/Foundation.swiftmodule/x86_64.swiftinterface ↩

[madsmtm]

compiled Swift binaries access them the same way that Objective-C binaries do.

Yeah, I suspected this would be the case, which means that a Rust/Swift bridge would not be able to access Foundation APIs without doing essentially what objc already does.
Thanks a lot though for the thorough walkthrough (that's a lot of "-oughs" 😉), I did not know about the .apinotes, that might be useful for enhancing bindgen's Objective-C support.

[Josh015]

EDIT: Okay, maybe I should read more carefully next time. Sorry.

For anyone arriving at this thread, see this post further up about objc2:
#101 (comment)

[SSheldon]

@Josh015 regarding a lack of updates to this crate over the last 2 years: are there any issues or bugs you would've liked to see addressed?

[Josh015]

No it's fine. I shot my mouth off before knowing the whole story and made an ass of myself. I'm sorry.

[SSheldon]

😆 you all good man. I definitely haven't been very active here, just wanted to double check if there were some new issues I had missed!