Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Readme #3

Merged
merged 2 commits into from
May 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,48 @@
[![Aqua](https://raw.githubusercontent.com/JuliaTesting/Aqua.jl/master/badge.svg)](https://github.com/JuliaTesting/Aqua.jl)


![Spy+snowman = Spehulak](docs/assets/spehulak.png)

> [!WARNING]
> This package is still under development and is not yet ready for use.

Spehulak.jl is a package for GenAI observability - helping you understand what's happening inside your GenAI model. It provides a set of tools for inspecting and evaluating the traces saved via PromptingTools.jl
Spehulak.jl is a package for GenAI observability - helping you understand what's happening inside your GenAI model. It provides a set of tools for inspecting and evaluating the traces saved via PromptingTools.jl.

## Quick Start

Run the file `main.jl`, eg, `julia --project -t auto main.jl`.

Open the browser and navigate to `http://localhost:8000`.

As an example, load the file `log/conversation...` in this folder.

## Frequently Asked Questions

**What can it do?**
Browse any serialized LLM conversations. If you used traced messages, we will display all the relevant metadata as well (model, templates, template versions, etc.).

**How to automatically save/log my LLM conversations?**
See [PromptingTools.jl docs](https://svilupp.github.io/PromptingTools.jl/dev/frequently_asked_questions#Automatic-Logging-/-Tracing)

Or generate a few toy examples
```julia
using PromptingTools
using PromptingTools: TracerSchema, SaverSchema, OpenAISchema, pprint

schema = OpenAISchema() |> TracerSchema |> SaverSchema

# Let's create a multi-turn conversation
conversation = aigenerate(schema,:JuliaExpertAsk; ask="Write a function to convert vector of strings into pig latin", model="gpt4o", return_all=true)
pprint(conversation)

# Let's create one more turn -- notice that we provide the past conversation as a kwarg
conversation = aigenerate(schema,"That's too complicated. Please simplify it. Think step by step first"; conversation, model="gpt4o", return_all=true)
pprint(conversation)

```

You should see a few files created in the folder `log/` - try opening them with Spehulak!

**What's Spehulak?**

"Špehulak" (pronounced "shpeh-hoo-lahk") blends "spy" and "snowman," riffing on the Czech words "špeh" for spy and "sněhulák" for snowman. It’s a fun name for an observability platform because it’s all about keeping a watchful eye, like a spy, while the snowman part plays into being cool under pressure as you sift through heaps of data plus being invisible/in the background, not being in the way. It’s perfect for diving deep into the inner workings of your LLM apps!
Binary file added docs/assets/spehulak.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[{"object":{"content":"You are a world-class Julia language programmer with the knowledge of the latest syntax. Your communication is brief and concise. You're precise and answer only when you're confident in the high quality of your answer.","variables":[],"_type":"systemmessage"},"from":null,"to":null,"viewers":[],"time_received":"2024-05-29T20:05:38.816","time_sent":"2024-05-29T20:05:11.754","model":"gpt4t","parent_id":"##parent#225","thread_id":"##thread#226","run_id":1901966690,"meta":{"template_name":"JuliaExpertAsk","template_version":"1"},"_type":"tracermessage"},{"object":{"content":"# Question\n\nWrite a function to convert vector of strings into pig latin","variables":["ask"],"_type":"usermessage"},"from":null,"to":null,"viewers":[],"time_received":"2024-05-29T20:05:38.816","time_sent":"2024-05-29T20:05:11.754","model":"gpt4t","parent_id":"##parent#227","thread_id":"##thread#228","run_id":-992446581,"meta":{"template_name":"JuliaExpertAsk","template_version":"1"},"_type":"tracermessage"},{"object":{"content":"Here's how you can write a function in Julia to convert a vector of strings into Pig Latin:\n\n```julia\nfunction pig_latin(words::Vector{String})\n pig_words = String[]\n for word in words\n if length(word) > 0 && isletter(word[1])\n # Find the first vowel in the word\n idx = findfirst(c -> occursin(c, \"aeiouAEIOU\"), word)\n if idx !== nothing\n # Move the preceding characters to the end and add \"ay\"\n transformed = word[idx:end] * word[1:idx-1] * \"ay\"\n else\n # If no vowel, keep the word as is and add \"ay\" at the end\n transformed = word * \"ay\"\n end\n push!(pig_words, transformed)\n else\n push!(pig_words, word)\n end\n end\n return pig_words\nend\n```\n\nThis function:\n- Checks if the word starts with a letter.\n- Finds the first vowel in the word.\n- Rearranges the word by moving the part before the first vowel to the end and adds \"ay\".\n- If a word has no vowels, it just appends \"ay\".\n- Ignores non-letter starting words by adding them unchanged to the result list.\n\nFor example:\n```julia\nwords = [\"hello\", \"world\", \"smile\", \"julia\"]\nresult = pig_latin(words)\nprintln(result) # Output: [\"ellohay\", \"orldway\", \"ilesmay\", \"uliajay\"]\n```","status":200,"tokens":[66,328],"elapsed":26.361568667,"cost":0.0105,"log_prob":null,"finish_reason":"stop","run_id":-1122367774,"sample_id":null,"_type":"aimessage"},"from":null,"to":null,"viewers":[],"time_received":"2024-05-29T20:05:38.816","time_sent":"2024-05-29T20:05:11.754","model":"gpt4t","parent_id":"##parent#229","thread_id":"##thread#230","run_id":-1191167018,"meta":{"template_name":"JuliaExpertAsk","template_version":"1"},"_type":"tracermessage"},{"object":{"content":"That's too complicated. Please simplify it. Think step by step first","variables":[],"_type":"usermessage"},"from":null,"to":null,"viewers":[],"time_received":"2024-05-29T20:06:02.935","time_sent":"2024-05-29T20:05:46.972","model":"gpt4t","parent_id":"##parent#231","thread_id":"##thread#232","run_id":-324452990,"meta":{},"_type":"tracermessage"},{"object":{"content":"Sure, let's simplify the process of converting a word to Pig Latin and implement it step-by-step in a function.\n\n1. **Check for the first vowel in the word**: This is needed to know where to split the word.\n2. **Split and rearrange**: Move the initial consonant cluster (all consonants before the first vowel) to the end of the word.\n3. **Append \"ay\"**: After rearranging, append \"ay\" to the word.\n\nLet's implement this in a function that handles a single word, and then extend it to handle a vector of strings:\n\n```julia\n# Convert a single word to Pig Latin\nfunction to_pig_latin(word::String)\n # Find the first vowel index\n idx = findfirst(c -> c in \"aeiouAEIOU\", word)\n if idx === nothing\n return word * \"ay\"\n elseif idx > 1\n # Rearrange and append 'ay'\n return word[idx:end] * word[1:idx-1] * \"ay\"\n else\n # starts with a vowel, add \"ay\" at the end\n return word * \"ay\"\n end\nend\n\n# Convert a vector of strings to Pig Latin\nfunction pig_latin_vector(words::Vector{String})\n return [to_pig_latin(word) for word in words]\nend\n```\n\nUsage example for the function:\n```julia\nwords = [\"hello\", \"world\", \"algorithm\", \"school\", \"eat\"]\nresult = pig_latin_vector(words)\nprintln(result) # Output: [\"ellohay\", \"orldway\", \"algorithmay\", \"oolschay\", \"eatay\"]\n```\n\nThis code defines a function `to_pig_latin` to handle individual words, using simple steps for converting to Pig Latin, and `pig_latin_vector` to map this conversion over a vector of strings.","status":200,"tokens":[416,395],"elapsed":15.832970458,"cost":0.016010000000000003,"log_prob":null,"finish_reason":"stop","run_id":-1783399986,"sample_id":null,"_type":"aimessage"},"from":null,"to":null,"viewers":[],"time_received":"2024-05-29T20:06:02.935","time_sent":"2024-05-29T20:05:46.972","model":"gpt4t","parent_id":"##parent#233","thread_id":"##thread#234","run_id":-1504361254,"meta":{},"_type":"tracermessage"}]
Loading