(EAI-383) Fetch & Refine Top Jira Tickets #539

nlarew · 2024-10-17T01:00:35Z

Jira: (EAI-383) Fetch & Refine Top Jira Tickets

Changes

Adds a new generateJiraPromptResponse command to the artifact generator

Usage

mongodb-ai generateJiraPromptResponse --issue EAI-383

Example output

[
  {
    "issueKey": "EAI-383",
    "prompt": "How can the top 250 Jira tickets be utilized in chatbot development?",
    "response": "The top 250 Jira tickets can be utilized in chatbot development by transforming them into a FAQ or prompt/answer format. This process involves identifying the most frequently accessed tickets, which likely contain valuable information or common issues faced by users. By refactoring these tickets into a structured format, the chatbot can provide quick and accurate responses to user queries, thereby enhancing its robustness and efficiency. This approach helps in addressing common questions and issues, improving user experience, and reducing the need for human intervention in resolving repetitive queries."
  },
  {
    "issueKey": "EAI-383",
    "prompt": "What is a method to create a FAQ from Jira tickets?",
    "response": "To create a FAQ from Jira tickets, follow these steps:\n\n1. **Identify High-Traffic Tickets**: Start by selecting the top Jira tickets with the most traffic or relevance. This ensures that the FAQ addresses the most common and important issues.\n\n2. **Analyze and Categorize**: Review each ticket to understand the core issue or question. Group similar tickets together to identify common themes or topics.\n\n3. **Extract Key Information**: For each ticket, extract the main question or problem and the corresponding solution or answer. Ensure that the information is clear and concise.\n\n4. **Refine and Simplify**: Simplify the language to make it accessible to a broader audience. Remove any technical jargon that might not be necessary for understanding.\n\n5. **Format as Q&A**: Structure the information in a question and answer format. This makes it easy for users to find the information they need quickly.\n\n6. **Review and Validate**: Have subject matter experts review the FAQ to ensure accuracy and completeness. This step is crucial to maintain the reliability of the information.\n\n7. **Publish and Update**: Once validated, publish the FAQ in an accessible location. Regularly update it based on new tickets or changes in existing issues to keep it relevant.\n\nBy following these steps, you can effectively convert Jira tickets into a useful FAQ that enhances the robustness of a chatbot or any other support tool."
  },
  {
    "issueKey": "EAI-383",
    "prompt": "What approach can be taken to convert Jira tickets into a prompt/answer format?",
    "response": "To convert Jira tickets into a prompt/answer format, follow these steps:\n\n1. **Identify Key Information**: Extract essential details from each ticket, such as the issue summary, description, and any relevant comments or resolutions.\n\n2. **Categorize Tickets**: Group similar tickets to identify common themes or issues. This helps in creating generalized questions that can cover multiple tickets.\n\n3. **Formulate Questions**: Based on the ticket content, draft clear and concise questions that a user might ask. Ensure these questions reflect the core issue or topic of the ticket.\n\n4. **Draft Answers**: Use the information from the ticket to create comprehensive answers. Include any solutions, workarounds, or insights provided in the ticket.\n\n5. **Review and Refine**: Ensure the questions and answers are accurate, clear, and free of jargon. Simplify technical language where possible to make it accessible to a broader audience.\n\n6. **Test with Chatbot**: Integrate the prompt/answer pairs into the chatbot and conduct testing to ensure they are triggered correctly and provide useful responses.\n\n7. **Iterate Based on Feedback**: Gather feedback from users interacting with the chatbot and refine the prompt/answer pairs as needed to improve clarity and usefulness.\n\nBy following these steps, you can effectively transform Jira tickets into a format that enhances chatbot functionality and user experience."
  },
  {
    "issueKey": "EAI-383",
    "prompt": "How can we improve the robustness of a chatbot using Jira tickets?",
    "response": "To improve the robustness of a chatbot using Jira tickets, you can follow these steps:\n\n1. **Identify High-Traffic Tickets**: Start by selecting the top 250 Jira tickets that have received the most traffic. These tickets likely contain valuable information and common issues that users encounter.\n\n2. **Analyze and Extract Information**: Carefully analyze these tickets to extract key information, such as common questions, issues, and solutions. This will help in understanding the frequent concerns and queries of users.\n\n3. **Refactor into FAQ Format**: Convert the extracted information into a FAQ format. This involves creating clear and concise questions and answers based on the content of the tickets. Ensure that the language is user-friendly and easy to understand.\n\n4. **Create Prompt/Answer Pairs**: For chatbot integration, develop prompt/answer pairs from the FAQs. This will allow the chatbot to recognize user queries and provide accurate responses based on the refined information.\n\n5. **Integrate with Chatbot**: Implement these prompt/answer pairs into the chatbot’s database or knowledge base. Ensure that the chatbot is trained to recognize variations of the questions to provide relevant answers.\n\n6. **Test and Refine**: Conduct thorough testing to ensure that the chatbot responds accurately and efficiently to user queries. Gather feedback and make necessary adjustments to improve the chatbot’s performance.\n\nBy following these steps, you can leverage the information from high-traffic Jira tickets to enhance the chatbot’s ability to handle user queries effectively, thereby improving its robustness and user satisfaction."
  },
  {
    "issueKey": "EAI-383",
    "prompt": "How can Jira tickets be used to enhance a chatbot's performance?",
    "response": "Jira tickets can be used to enhance a chatbot's performance by analyzing and transforming them into a structured FAQ or prompt/answer format. This process involves identifying the most frequently accessed or high-traffic tickets, which likely contain valuable information or common issues faced by users. By converting these tickets into a format that a chatbot can understand and respond to, the chatbot can provide quick and accurate answers to user queries.\n\nHere's how this can be done:\n\n1. **Data Collection**: Gather the top Jira tickets with the most traffic. These tickets often represent common questions or issues that users encounter.\n\n2. **Content Analysis**: Analyze the content of these tickets to identify key questions and answers. This involves understanding the problem described in the ticket and the solution or information provided.\n\n3. **Refinement**: Refactor the information into clear and concise FAQ entries or prompt/answer pairs. This step ensures that the information is easily digestible and can be effectively used by the chatbot.\n\n4. **Integration**: Integrate these FAQs into the chatbot's knowledge base. This allows the chatbot to access this information when responding to user queries.\n\n5. **Testing and Optimization**: Test the chatbot's responses to ensure accuracy and relevance. Continuously optimize the FAQ entries based on user feedback and new data.\n\nBy following these steps, a chatbot can leverage the insights from Jira tickets to improve its ability to assist users, thereby enhancing its overall performance and user satisfaction."
  }
]

packages/mongodb-artifact-generator/src/chat/makeGenerateResponse.ts

mongodben · 2024-10-18T14:52:20Z

packages/mongodb-artifact-generator/src/chat/makeSummarizer.test.ts

+  deployment: OPENAI_CHAT_COMPLETION_DEPLOYMENT,
+};
+
+// TODO: Convert this into a .eval file


can you do as part of this PR or make a follow up ticket?

if doing follow up, it looks like there's some commented out stuff that could be removed

thoughts on this @nlarew?

packages/mongodb-artifact-generator/src/commands/generateJiraPromptResponse.ts

mongodben · 2024-10-18T19:23:05Z

packages/mongodb-artifact-generator/src/commands/generateJiraPromptResponse.ts

+      .option("maxInputLength", {
+        type: "number",
+        demandOption: false,
+        description:
+          "The maximum number of issues to process in this run. Any additional issues are skipped.",
+      })


why is this functionality included?

Purely for testing purposes - e.g. run my updated prompt on the first 6 of the top issues list.

packages/mongodb-artifact-generator/src/commands/generateJiraPromptResponse.ts

mongodben · 2024-10-18T19:27:55Z

packages/mongodb-artifact-generator/src/commands/generateJiraPromptResponse.ts

+        [prompt: string, response: string][]
+      >();
+
+      const generateResponse = makeGenerateResponse({


imo this should be evaluated but also for this specific project 🤷

https://jira.mongodb.org/browse/EAI-519

mongodben · 2024-10-18T19:28:14Z

packages/mongodb-artifact-generator/src/commands/generateJiraPromptResponse.ts

+
+      // // Generate a list of N questions/prompts for each issue
+      const promptsByIssueKey = new Map<string, string[]>();
+      const generatePrompts = makeGeneratePrompts({


think should be evaluated but also 🤷

https://jira.mongodb.org/browse/EAI-519

i think there's some ticket confusion here. looks like you updated the name of an existing ticket plus also created a new one https://jira.mongodb.org/browse/EAI-579 (tho the title on that is wrong i think?)

Not sure I see the issue - 519 is for this prompt-response project & 579 is for more generic RAG helpers like "summarize this input" (that are currently defined in/only used in the artifact generator)

packages/mongodb-artifact-generator/src/commands/generateJiraPromptResponse.ts

packages/mongodb-artifact-generator/src/chat/utils.ts

mongodben

couple small things throughout.

also some suggestions for bigger changes that i'm not sure are worth implementing even if more correct/maintainable. all these have 🤷 emojis

packages/mongodb-artifact-generator/src/Config.ts

mongodben

lgtm. pls see/address few new comments. nothing blocking

Co-authored-by: Ben Perlmutter <[email protected]>

nlarew added 3 commits October 16, 2024 19:26

(EAI-383) Fetch & Refine Top 250 Jira Tickets

d6b7124

better issue input

baf6ce7

add env var

39dc5d1