Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Structured output improvements #286

Merged
merged 12 commits into from
Sep 16, 2024
Merged

Structured output improvements #286

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
9b294a6
Adding an "install dependencies" section and updating the intro wording.
Giom-V Sep 12, 2024
82ffe21
Rephrasing text to make it eeasier to understand.
Giom-V Sep 12, 2024
924a359
Adding a "Next Steps" section
Giom-V Sep 12, 2024
0f9a09f
Formatting
Giom-V Sep 12, 2024
2394ade
typo
Giom-V Sep 12, 2024
bc0a271
Using Flash instead of Pro
Giom-V Sep 12, 2024
e1a4bc2
Formatting
Giom-V Sep 12, 2024
38eae2b
Updating output
Giom-V Sep 12, 2024
51c2391
Relative links
Giom-V Sep 13, 2024
7eb58db
Wording
Giom-V Sep 16, 2024
ddf5d43
Wording
Giom-V Sep 16, 2024
7facfd3
Wording
Giom-V Sep 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 35 additions & 5 deletions quickstarts/Enum.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

136 changes: 111 additions & 25 deletions quickstarts/JSON_mode.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
"\n",
"<table align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/JSON_mode.ipynb\"><img src=\"../images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/JSON_mode.ipynb\"><img src=\"https://github.com/google-gemini/cookbook/blob/0f9a09fd0db1b54cc1af0f87f11f88b493e0cb3b/images/colab_logo_32px.png?raw=1\" />Run in Google Colab</a>\n",
" </td>\n",
"</table>"
]
Expand All @@ -54,23 +54,42 @@
"source": [
"The Gemini API can be used to generate a JSON output if you set the schema that you would like to use.\n",
"\n",
"**Note**: JSON schemas are only supported by Gemini 1.5 Pro right now."
"Two methods are available. You can either set the desired output in the prompt or supply a schema to the model separately."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "857d8bf104ed"
},
"source": [
"### Install dependencies"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"metadata": {
"id": "qLuL9m7KhvxR"
},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/164.0 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m163.8/164.0 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m164.0/164.0 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/725.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m716.8/725.4 kB\u001b[0m \u001b[31m38.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m725.4/725.4 kB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25h"
]
}
],
"source": [
"!pip install -U -q \"google-generativeai>=0.7.2\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 3,
"metadata": {
"id": "ATIbQM0NHhkj"
},
Expand All @@ -89,14 +108,14 @@
"id": "B-axqBTM8Lbd"
},
"source": [
"## Configure your API key\n",
"### Configure your API key\n",
"\n",
"To run the following cell, your API key must be stored in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) for an example."
"To run the following cell, your API key must be stored in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](../quickstarts/Authentication.ipynb) for an example."
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 4,
"metadata": {
"id": "d6lYXRcjthKV"
},
Expand All @@ -114,14 +133,14 @@
"id": "hD3qXcOTRD3z"
},
"source": [
"## Activate JSON Mode\n",
"## Set your constrained output in the prompt\n",
"\n",
" Activate JSON mode by specifying `respose_mime_type` in the `generation_config` parameter:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 5,
"metadata": {
"id": "i5Rod-lXRIhf"
},
Expand All @@ -137,12 +156,12 @@
"id": "4071a6143d31"
},
"source": [
"For this first example just describe the schema you want back:"
"For this first example just describe the schema you want back in the prompt:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 6,
"metadata": {
"id": "K8ezjNb0RJ6Y"
},
Expand All @@ -156,7 +175,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 7,
"metadata": {
"id": "ggudoxK8RMlb"
},
Expand All @@ -176,7 +195,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 8,
"metadata": {
"id": "b99ee66972f5"
},
Expand Down Expand Up @@ -205,7 +224,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 9,
"metadata": {
"id": "WLDPREpmSMu5"
},
Expand Down Expand Up @@ -244,23 +263,25 @@
"id": "K9nIks0R-tIa"
},
"source": [
"## Generate JSON from schema\n",
"## Supply the schema to the model directly\n",
"\n",
"While `gemini-1.5-flash` models only accept a text description of the JSON you want back, `gemini-1.5-pro` models support \"controlled generation\" (aka \"constrained decoding\"). This allows you to pass a schema object (or a python type equivalent) and the output will strictly follow that schema.\n",
"The newest models (1.5 and beyond) allow you to pass a schema object (or a python type equivalent) directly and the output will strictly follow that schema.\n",
"\n",
"Following the same example as the previous section, here's that recipe type:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 10,
"metadata": {
"id": "JiIxKaLl4R0f"
},
"outputs": [],
"source": [
"class Recipe(typing.TypedDict):\n",
" recipe_name: str"
" recipe_name: str\n",
" recipe_description: str\n",
" recipe_ingredients: list[str]"
]
},
{
Expand All @@ -274,16 +295,16 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 11,
"metadata": {
"id": "8oe-tL8MDGtx"
},
"outputs": [],
"source": [
"model = genai.GenerativeModel(model_name=\"models/gemini-1.5-pro-latest\")\n",
"model = genai.GenerativeModel(model_name=\"models/gemini-1.5-flash-latest\")\n",
"\n",
"result = model.generate_content(\n",
" \"List a few popular cookie recipes\",\n",
" \"List a few imaginative cookie recipes along with a one-sentence description as if you were a gourmet restaurant and their main ingredients\",\n",
" generation_config=genai.GenerationConfig(\n",
" response_mime_type=\"application/json\",\n",
" response_schema = list[Recipe]),\n",
Expand All @@ -293,7 +314,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 12,
"metadata": {
"id": "slYcVAcqaDQY"
},
Expand All @@ -302,12 +323,77 @@
"name": "stdout",
"output_type": "stream",
"text": [
"[{\"recipe_name\": \"Chocolate Chip Cookies\"}, {\"recipe_name\": \"Peanut Butter Cookies\"}, {\"recipe_name\": \"Oatmeal Raisin Cookies\"}, {\"recipe_name\": \"Sugar Cookies\"}, {\"recipe_name\": \"Snickerdoodles\"}] \n"
"[\n",
" {\n",
" \"recipe_description\": \"A delicate and fragrant cookie infused with the bright citrus notes of bergamot and the warming spice of cardamom.\",\n",
" \"recipe_ingredients\": [\n",
" \"Bergamot zest\",\n",
" \"Cardamom pods\",\n",
" \"Almond flour\",\n",
" \"Butter\",\n",
" \"Sugar\"\n",
" ],\n",
" \"recipe_name\": \"Bergamot Cardamom Sabl\\u00e9s\"\n",
" },\n",
" {\n",
" \"recipe_description\": \"A luxurious and decadent cookie with a rich chocolate flavor and a hint of salty caramel.\",\n",
" \"recipe_ingredients\": [\n",
" \"Dark chocolate\",\n",
" \"Sea salt\",\n",
" \"Caramel sauce\",\n",
" \"Butter\",\n",
" \"Flour\"\n",
" ],\n",
" \"recipe_name\": \"Salted Caramel Chocolate Chunk Cookies\"\n",
" },\n",
" {\n",
" \"recipe_description\": \"A whimsical and colorful cookie inspired by a summer garden, featuring a blend of fresh herbs and bright citrus.\",\n",
" \"recipe_ingredients\": [\n",
" \"Lavender flowers\",\n",
" \"Lemon zest\",\n",
" \"Rosemary\",\n",
" \"Butter\",\n",
" \"Sugar\",\n",
" \"Flour\"\n",
" ],\n",
" \"recipe_name\": \"Lavender Lemon Rosemary Shortbread\"\n",
" }\n",
"]\n"
]
}
],
"source": [
"print(result.text)"
"print(json.dumps(json.loads(result.text), indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "239af844c650"
},
"source": [
"It is the recommended method if you're using a compatible model."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5ef9efdb62d3"
},
"source": [
"## Next Steps\n",
"### Useful API references:\n",
"\n",
"Check the [structured ouput](https://ai.google.dev/gemini-api/docs/structured-output) documentation or the [`GenerationConfig`](https://ai.google.dev/api/generate-content#generationconfig) API reference for more details\n",
"\n",
"### Related examples\n",
"\n",
"* The constrained output is used in the [Text summarization](../examples/json_capabilities/Text_Summarization.ipynb) example to provide the model a format to summarize a story (genre, characters, etc...)\n",
"* The [Object detection](../examples/Object_detection.ipynb) examples are using the JSON constrained output to uniiformize the output of the detection.\n",
"\n",
"### Continue your discovery of the Gemini API\n",
"\n",
"JSON is not the only way to constrain the output of the model, you can also use an [Enum](../quickstarts/Enum.ipynb). [Function calling](../quickstarts/Function_calling.ipynb) and [Code execution](../quickstarts/Code_Execution.ipynb) are other ways to enhance your model by either using your own functions or by letting the model write and run them."
]
}
],
Expand Down
22 changes: 17 additions & 5 deletions quickstarts/rest/JSON_mode_REST.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,13 @@
"id": "JMNKdTpTGZET"
},
"source": [
"This notebook provides a code example that shows you how to get started with JSON mode using `curl`.\n",
"The Gemini API can be used to generate a JSON output if you set the schema that you would like to use. This notebook provides a code example that shows you how to get started with JSON mode using `curl`.\n",
"\n",
"You can run this in Google Colab, or you can copy/paste the `curl` commands into your terminal.\n",
"\n",
"To run this notebook, your API key must be stored it in a Colab Secret named GOOGLE_API_KEY. If you are running in a different environment, you can store your key in an environment variable. See [Authentication](https://github.com/google-gemini/gemini-api-cookbook/blob/main/quickstarts/Authentication.ipynb) to learn more."
"### Authentication\n",
"\n",
"To run this notebook, your API key must be stored it in a Colab Secret named GOOGLE_API_KEY. If you are running in a different environment, you can store your key in an environment variable. See [Authentication](../../quickstarts/Authentication.ipynb) to learn more."
]
},
{
Expand Down Expand Up @@ -120,7 +122,7 @@
"source": [
"%%bash\n",
"\n",
"curl \"https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro-latest:generateContent?key=$GOOGLE_API_KEY\" \\\n",
"curl \"https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-latest:generateContent?key=$GOOGLE_API_KEY\" \\\n",
"-H 'Content-Type: application/json' \\\n",
"-d '{\n",
" \"contents\": [{\n",
Expand Down Expand Up @@ -151,9 +153,19 @@
"id": "jXkRYBhbB_b2"
},
"source": [
"## Learning more\n",
"## Next Steps\n",
"### Useful API references:\n",
"\n",
"Check the [structured ouput](https://ai.google.dev/gemini-api/docs/structured-output) documentation or the [`GenerationConfig`](https://ai.google.dev/api/generate-content#generationconfig) API reference for more details \n",
"\n",
"### Related examples\n",
"\n",
"* The constrained output is used in the [Text summarization](../../examples/json_capabilities/Text_Summarization.ipynb) example to provide the model a format to summarize a story (genre, characters, etc...)\n",
"* The [Object detection](../../examples/Object_detection.ipynb) examples are using the JSON constrained output to normalize the output of the detection.\n",
"\n",
"### Continue your discovery of the Gemini API\n",
"\n",
"See the [JSON mode documentation](https://ai.google.dev/docs/gemini_api_overview#json) and the [REST API reference](https://ai.google.dev/api/rest/v1beta/GenerationConfig) for `generationConfig` to learn more.\n"
"JSON is not the only way to constrain the output of the model, you can also use an [Enum](../../quickstarts/Enum.ipynb). [Function calling](../../quickstarts/Function_calling.ipynb) and [Code execution](../../quickstarts/Code_Execution.ipynb) are other ways to enhance your model by using your own functions or by letting the model write and run them."
]
}
],
Expand Down
Loading