Github Action for LLM Prompt Evaluation

This Github Action uses promptfoo to produce a before/after view of edit prompts.

When you change a prompt, an eval will automatically be posted on the pull request:

The provided link opens the promptfoo web viewer, which allows you to interactively explore the before vs. after:

Configuration

The action can be configured using the following inputs:

Parameter	Description	Required
github-token	The Github token. Used to authenticate requests to the Github API.	Yes
prompts	The glob patterns for the prompt files. These patterns are used to find the prompt files that the action should evaluate.	Yes
openai-api-key	The API key for OpenAI. Used to authenticate requests to the OpenAI API.	No
azure-openai-api-key	The API key for Azure OpenAI. Used to authenticate requests to the Azure OpenAI API.	No
cache-path	The path to the cache. This is where the action stores temporary data.	No

Here is a generic Github Action configuration using "typpo/promptfoo-action@v1" with a cache step:

name: 'Prompt Evaluation'
on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read # Required for actions/checkout
      pull-requests: write # Ability to post comments on Pull Requests
    steps:
      # Required for promptfoo-action's git usage
      - uses: actions/checkout@v4 

      # This cache is optional, but you'll save money and time by setting it up!
      - name: Set up promptfoo cache
        uses: actions/cache@v3
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-v1
          restore-keys: |
            ${{ runner.os }}-promptfoo-

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          prompts: 'prompts/**/*.json'
          cache-path: ~/.cache/promptfoo

If you are using an OpenAI model, remember to create the secret in Repository Settings > Secrets and Variables > Actions > New repository secret.

For more information on how to set up the promptfoo config, see documentation.