Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metal support for Swift #3078

Merged
merged 6 commits into from
Sep 9, 2023
Merged

Metal support for Swift #3078

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
89a96fd
Metal support for Swift
kchro3 Sep 8, 2023
ce92d75
update
kchro3 Sep 8, 2023
99f85e7
add a toggle for arm/arm64
kchro3 Sep 8, 2023
690c794
set minimum versions for all platforms
kchro3 Sep 8, 2023
24013b3
update to use newLibraryWithURL
kchro3 Sep 9, 2023
d58af4d
bump version
kchro3 Sep 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions Package.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,27 @@ import PackageDescription

let package = Package(
name: "llama",
platforms: [.macOS(.v11)],
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tried iOS but I do wonder if we are OK by limiting this package to macOS only. Can llamacpp run on iOS, tvOS or even watchOS @ggerganov?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can run even on a refrigerator 😄

Jokes aside - I see no reason to limit this to just macOS

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure why, but the compiler was not happy if platforms was not included, although that could be a metal thing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't have a way to test on watchOS or tvOS... i can set it to the minimum non-deprecated version and if someone tries to cross that bridge, we could update it?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you mind sharing the compiler error?

Copy link
Contributor Author

@kchro3 kchro3 Sep 9, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey folks, I would appreciate help on this. I'm seeing in my build logs that it's compiling the .metal file even if I put it in resources.

For example, if I do:

#if arch(arm) || arch(arm64)
let platforms: [SupportedPlatform]? = [
    .macOS(.v11),
    .iOS(.v11),
    .watchOS(.v4),
    .tvOS(.v11)
]
let exclude: [String] = []
let resources: [Resource]? = [
    .copy("ggml-metal.metal"),
    .copy("README.md")  // just to validate that files get copied
]
let additionalSources: [String] = ["ggml-metal.m"]
let additionalSettings: [CSetting] = [
    .unsafeFlags(["-fno-objc-arc"]),
    .define("GGML_SWIFT"),
    .define("GGML_USE_METAL")
]
#else

I still see the default.metallib in my resources:

ls /Users/.../Library/Developer/Xcode/DerivedData/.../Build/Products/Debug/....app/Contents/Resources/llama_llama.bundle/Contents/Resources/
README.md		default.metallib
Screenshot 2023-09-08 at 6 37 22 PM

Could it be because the file is in the project root and the target path is "."? tried copying it into a new directory & excluding the original, but it still compiled...

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if you exclude ggml-metal.metal like it is done in master right now, but then add it in the resources section. That should do it I think

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://developer.apple.com/documentation/packagedescription/target/exclude#discussion

I think it doesn't work because exclude takes precedence over resources. For example, I pushed a branch https://github.com/ggerganov/llama.cpp/pull/3091/files, and my build logs don't show the .metal file:

Screenshot 2023-09-08 at 7 57 54 PM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://developer.apple.com/library/archive/documentation/3DDrawing/Conceptual/MTLBestPracticesGuide/FunctionsandLibraries.html

Just a sanity check, but this documentation is saying that .metal files get automatically compiled, and that seems to be the case from what I've tried. Is there a specific reason why we need to compile from source?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://developer.apple.com/library/archive/documentation/3DDrawing/Conceptual/MTLBestPracticesGuide/FunctionsandLibraries.html

Just a sanity check, but this documentation is saying that .metal files get automatically compiled, and that seems to be the case from what I've tried.

Got it, it looks like a better way if we use it on Xcode.

products: [
.library(name: "llama", targets: ["llama"]),
],
targets: [
.target(
name: "llama",
path: ".",
exclude: ["ggml-metal.metal"],
sources: [
"ggml.c",
"llama.cpp",
"ggml-alloc.c",
"k_quants.c"
"k_quants.c",
"ggml-metal.m",
],
publicHeadersPath: "spm-headers",
cSettings: [
.unsafeFlags(["-Wno-shorten-64-to-32"]),
.unsafeFlags(["-fno-objc-arc"]),
.define("GGML_SWIFT"),
.define("GGML_USE_METAL"),
Copy link

@pkrmf pkrmf Sep 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this change forcing everyone to use metal? Can this be a flag defined by the customer instead? For instance, I can't run llamacpp with metal ON with my old MacBook and an AMD card

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kchro3 Let's address this comment and we can merge

Copy link
Contributor Author

@kchro3 kchro3 Sep 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how does it look now? i don't have an old macbook, but i was able to build it if i switched the if/else condition

# in Package.swift
#if arch(x86_64)  // instead of arch(arm) || arch(arm64)

// demo that it's not using metal anymore
llm_load_tensors: ggml ctx size =    0.12 MB
llm_load_tensors: mem required  = 7500.97 MB (+  400.00 MB per state)
...................................................................................................
llama_new_context_with_model: kv self size  =  400.00 MB
llama_new_context_with_model: compute buffer total size =   75.47 MB


 what is the capital of japanToken received in Swift: 

Token received in Swift: The
Token received in Swift:  capital

.define("GGML_USE_K_QUANTS"),
.define("GGML_USE_ACCELERATE")
],
Expand Down
13 changes: 10 additions & 3 deletions ggml-metal.m
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,12 +141,19 @@ @implementation GGMLMetalClass

ctx->d_queue = dispatch_queue_create("llama.cpp", DISPATCH_QUEUE_CONCURRENT);

#if 0
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought it would be ok to replace this since it looked unfinished.

// compile from source string and show compile log
#ifdef GGML_SWIFT
// load the default.metallib file
{
NSError * error = nil;

ctx->library = [ctx->device newLibraryWithSource:msl_library_source options:nil error:&error];
NSBundle * bundle = [NSBundle bundleForClass:[GGMLMetalClass class]];
NSString * llamaBundlePath = [bundle pathForResource:@"llama_llama" ofType:@"bundle"];
NSBundle * llamaBundle = [NSBundle bundleWithPath:llamaBundlePath];
NSString * libPath = [llamaBundle pathForResource:@"default" ofType:@"metallib"];

// Load the metallib file into a Metal library
ctx->library = [ctx->device newLibraryWithFile:libPath error:&error];

if (error) {
metal_printf("%s: error: %s\n", __func__, [[error description] UTF8String]);
return NULL;
Expand Down