llm.c discussions

[karpathy]

🔥 llm.c 🔥

Turning on discussions feature as a place for people to ask, share and engage, without having to create Issues.

[Missper]

Can I get a job studying this?

[fate-ubw]

If you want be a programmer who need to deep into cuda, this project might be help. C will better than python to understand how cuda works

[Missper]

My current job is java programmer, but the money is too less, so I want find another job even not programmer

[lordpba]

don't follow money follow your passions, money will come

[StoyanStAtanasov]

Follow the money and the passion will come (because you feel more useful). I think this will work for more people.

[forrestmckee]

Are there any plans to go through this in a video format similar to NN Zero to Hero?

[karpathy]

Yes, actually this is one of the major reasons I wrote this. It's the next piece of code I need for continuing the series.

Imo people have to understand in detail what PyTorch for them does when you do things like .to(device), or torch.cuda.synchronize() , and also how tensors work and what is .contiguous(), and etc. And eventually float16, bfloat16, etc. All of these calls get into the internals of tensors, what they are, how/where they are stored and manipulated.

TLDR yes. I want to finish llm.c and let it settle a bit, then we'll build it.

[lordpba]

please explain to us how you can do so many things and top quality

[dagelf]

Practice, practice, practice. Spend time doing it... iterate! Question yourself. Try new things. Try to understand things you did in the past. 😄 That's how!

[zhouwg]

thanks for your llm.c. could you add more comments in code to explain details for AI beginners?

[dagelf]

You can paste functions in GPT4 or Claude3 and it does a fair job of explaining, even adding comments and answering questions. It's not fool proof, but it can really help you think.

[arseniyturin]

My generated text during training:

<|endoftext|>I was so upright that I would have never heard you had any talk. I have heard you sometimes datts you, Eats could or should be choked, crows and
Fearsome snakes say it right.

<|endoftext|>Second Servingman:
I flagged him down in a half dozen ships AND don

QQ

Will we get a prompt to ask gpt-2 questions and get answers?

[mizuruwu]

Hi karpathy,
how much minimum GPU memory do i need?

[zocterminal]

If anyone wants to play around with this on an amazon AWS cloud machine, here's a newbie friendly article that shows how to set that up.

[karpathy]

wow, that looks really involved.
Personally I use https://lambdalabs.com/service/gpu-cloud which looks much much easier, but they don't always have GPUs available
I also tried Google Colab PRO but sadly it's so laggy and weird and really militant in shutting down my instance the second it looks inactive.
I'm familiar with (but haven't used yet) Lightning AI studios, another potential option.

[zocterminal]

I think many of those may be just wrappers around AWS, like when you instantiate a machine there, they will go to AWS and instantiate one of their canned installations (probably with a markup). AWS is a steep learning curve if you never used it (like with learning everything), but once it's set up and you know what to do it's not too hard to setup a cuda machine (the hardest part was to figure out how and which Linux works with the NVIDIA cuda drivers and how to set that up ... these installations are very brittle).

[lordpba]

I am trying Lighting AI, I found it interesting, in the Team there is also Sebastian Raschka that is doing an awesome job at teaching, I am a newbie into the AI world, but thanks to Sebastian and Andrej I have the opportunity to learn many exciting things!

[dagelf]

Here are some more options, in no particular order: (@zocterminal Self/P2P/Own DC is not hard, maybe 1 wrapper here)

Hyperstack: https://www.hyperstack.cloud/gpu-pricing
Latitude.sh: https://www.latitude.sh/pricing
Brev: https://brev.dev/pricing
Replicate: https://replicate.com/
Lepton AI: https://dashboard.lepton.ai/
Hugging Face: https://huggingface.co/pricing
Vast AI: https://cloud.vast.ai/create/
OVHCloud: https://www.ovhcloud.com/
Paperspace: https://www.paperspace.com/
Vultr: https://www.vultr.com/
G Core: https://gcorelabs.com/
Genesis Cloud: https://www.genesiscloud.com/pricing
Lambda Labs: https://lambdalabs.com/
Tensor Dock: https://tensordock.com/
Microsoft Azure: https://azure.microsoft.com/
IBM Cloud: https://www.ibm.com/cloud
Google Cloud Platform (GCP): https://cloud.google.com/
NVIDIA GPU Cloud (NGC): https://www.nvidia.com/en-us/gpu-cloud/
Runpod IO (Serverless): https://runpod.io/
Banana Dev: https://www.banana.dev/
CoreWeave: https://coreweave.com/
Modal: https://modal.com/
Lightning AI: https://lightning.ai/pricing

Comparison sites:
Holori https://app.holori.com/compare (PS The total cost of AWS is generally about 3x of what is quoted anywhere, AWS is a roach motel. "dark pattern" UX for lock in. Need third party tools to decommission easily.)
Vantage https://instances.vantage.sh/

Hardware:
https://gpuprices.us/
https://www.cpuscout.com/
https://diskprices.com/
https://listofdisks.com/
https://shucks.top/
https://www.productchart.com/monitors/

[bijouvj]

I am trying this on a x86 Linux system and running into a weird issue. I see this:
step 1: train loss 5.356186 (took 4222.497864 ms)
Error: must forward with targets before backward

On debugging, I found that the mean_loss is sometimes NaN, but making the backward conditional on a NaN check (using != ) still ends up with this problem. My feeling is that this has something to do with the loader, but not 100% sure.

Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8480CL
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual

[rosslwheeler]

You need to modify your CFLAGS - check out the comments here.

#19

[jtscm]

Thank you for the llm.c project. One way to learn how llm.c works is by rewriting the code. This is why I rewrote llm.c into a simple command line utility that runs GPT-2 inference. The link is: https://github.com/jtscm/iim.c/

[fate-ubw]

great! follow your repo

[joshcarp]

Still got some work to do but I've got an initial implementation in go: https://github.com/joshcarp/llm.go

[DongqiShen]

Thanks for your great project. Could you please share how you debug and profile the code? That will be helpful. Thanks again.

[StoyanStAtanasov]

I would like to see an implementation where you start with a small transformer and teach it the ABCs and then grow it with more layers and then feed it more complex information until it gets big and with only a small set of data you teach it new tasks.
@karpathy Do you think this could be a good strategy?

[youseai]

Can someone recommend any book or course where I can start my journey of understanding LLMs from a very scratch level?

[fate-ubw]

Actually the best way to learn LLM is debug huggingfece transformer in inference and training period. I have debug transformer line by line use pdb which took me 5 month. But I learn a lot by doing this, remember debug the code is always better to learn from video or book. If huggingfece transformer is diffcult to you, miniGPT and nanoGPT is a better repo to start

[ehzawad]

How did you keep track of all the functions calls or jumps and variables and states? Did you need to use any other Python packages other than PDB? @fate-ubw

[fate-ubw]

Pdb is all you need. If you haven't learn how to debug by pdb. please learn pdb degger which is a really good tool. Pdb can tell jump, track function calling, print variables, interact mode to run code and even debugger after your code break down(pdb.pm() function)

[ehzawad]

Thanks man. I probably underestimated the power of Pdb before!

[chinthysl]

Current build on master fails due to C++17 features. Not sure it's my system or others also faces the same issue.
train_gpt2.cu(905): error: namespace "std" has no member "bool_constant" std::bool_constant<Atomic>)
train_gpt2.cu(964): warning #2912-D: constexpr if statements are a C++17 feature if constexpr (!Atomic) {

Setting nvcc to cpp17 fixes it for me.
NVCC_FLAGS = -std=c++17 -O3 -t=0 --use_fast_math

[rosslwheeler]

Got it - will look into it and see if why it's different. Thanks.

[rosslwheeler]

@chinthysl - can you please run this? And let me know the output? Thanks.

g++ -dM -E -x c++ - < /dev/null | grep __cplusplus

[chinthysl]

@rosslwheeler Here's the output from the above command - #define __cplusplus 201703L

[rosslwheeler]

@chinthysl - it's saying that your gcc IS built using the default of C++ 17 (same as Ubuntu 22.04). So, it doesn't make sense that it's asking you to explicitly add C++17 to your compilations since it's (in theory based on the output) already set to that. I'll need to do some more digging.

[chinthysl]

@rosslwheeler at some point it wasn't building without -std=c++17 for me. But now it's building without any issues.
I have bunch of compiler and cuda versions available. May be something went wrong in my environment setup.
Thank you very much for further looking into this issue.