Overview

AI Snipping Tool

A Chrome extension that takes custom screenshots, extracts text, and allows users to ask questions from the extracted content.

Chrome Extension popup

Step 1: Take the screenshot

Step 2: Copy the obtained text or proceed further

Step 3: Fill Google Gemini API key and ask related question
Prerequisite: Google Gemini API

Steps to get Gemini API key

• Sign up at https://ai.google.dev/aistudio

• Click on Get API key

Built With

👉 Tech Stacks:

Getting Started 👩‍💻

⚠️Prerequisites

• Recommended to have the latest version of Google Chrome
• Make sure to switch on Chrome's developer mode

To get a local copy up and running follow these simple steps

Installation

1. Open Git Bash and change directory

   cd path

2. Clone the repo

 git clone https://github.com/gitgoap/AI-Snipping-Tool.git

3. Open Google Chrome

4. Click 3 dots at top right corner

5. Go to Extensions> Manage Extensions

6. Click Load Unpack and select the repo where you saved initially while cloning

Contribution Guide

• Make sure to raise an issue before raising a Pull Request.
• Mention the issue number (Eg: #4) while raising a Pull Request in the description.

Star History

Problem Statement

We often need to type out text from videos, images, or thumbnails on websites. This can be a tedious and error-prone process, especially if the text is long or has tricky stuff like website links, technical words, code examples, or math equations. This issue comes up on YouTube, where useful info is shown in videos or thumbnail images.

Some of the scenarios where this problem arises:

• Manually typing the displayed link is inefficient and prone to errors if it can't be copied.

• Copying the references given in the presentation footer is difficult

• Copying code directly to the editor is not possible

• Copying text from documents shared as images on social media is tricky
• People may require this text to feed into LLMs to get a summary

Working

Our Chrome extension is utilizing Tesseract OCR for character recognition in the image:

1. The extension's background script worker.js listens for user actions, such as clicking the extension's icon or using the context menu.
2. When the user initiates a screenshot, the extension captures the visible tab using chrome.tabs.captureVisibleTab and gets the screenshot data as a PNG image.
3. The extension then injects several scripts (helper.js, response.js, elements.js, and custom-elements.min.js) into the current tab's context. These scripts are responsible for rendering the OCR result and handling the OCR process.
4. The ocr-result custom element, defined in elements.js, is used to perform the OCR operation. This custom element utilizes the tesseract.js library, which is a pure JavaScript port of the Tesseract OCR engine.
5. The ocr-result element is configured with user preferences such as language, accuracy, and other settings fetched from the Chrome extension's local storage.
6. The captured screenshot data (PNG image) is passed to the ocr-result element, which then runs the Tesseract OCR engine on the image to extract the text.
7. The extracted text is rendered within the ocr-result element, allowing the user to view and interact with the OCR results.

Tesseract OCR: It is an open-source optical character recognition (OCR) engine developed by Google. It is designed to extract text from images and documents without a text layer, outputting the document in various formats such as plain text, HTML, PDF, and more. Tesseract supports recognition of over 100 languages "out of the box" and is highly customizable.

Future Scope / Features

• YouTube Video Screenshot: Capture text in the current YouTube video frame with just one click of a popup button.

• Answers questions just with screenshots: This feature is possible by utilizing the multimodal feature of Google Gemini.

  

Our Contributors ❤️

Thank you for contributing to our repository.