Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Update npm dependencies and add normalize-url package #109

Merged
merged 6 commits into from
Jul 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import { Translation } from "./Translation";
interface ContentWithTranslationsProps {
content: string;
sourceTextInfoWithTranslations: SourceTextInfoWithTranslations[];
targetLanguage: string;
userId: number | null;
}

Expand Down
7 changes: 4 additions & 3 deletions web/app/routes/reader.$encodedUrl/components/Translation.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,14 @@ export function Translation({
return (
<div
className="relative group"
onMouseEnter={() => setTimeout(() => setIsHovered(true), 100)}
onMouseLeave={() => setIsHovered(false)}
onMouseEnter={() =>
!isHovered && setTimeout(() => setIsHovered(true), 500)
}
ttizze marked this conversation as resolved.
Show resolved Hide resolved
>
<div className="w-full notranslate mt-2 pt-2 border-t border-gray-200">
{sanitizedAndParsedText}
</div>
<div className="absolute top-0 left-0 right-0 z-10 opacity-0 invisible border bg-slate-50 dark:bg-gray-900 rounded-xl shadow-xl dark:shadow-white/10 group-hover:opacity-100 group-hover:visible transition-all duration-500 ease-in-out">
<div className="absolute top-0 left-0 right-0 z-10 opacity-0 invisible border bg-white dark:bg-gray-900 rounded-xl shadow-xl dark:shadow-white/10 group-hover:opacity-100 group-hover:visible transition-all duration-500 ease-in-out">
{isHovered && (
<AddAndVoteTranslations
bestTranslationWithVote={bestTranslationWithVote}
Expand Down
8 changes: 5 additions & 3 deletions web/app/routes/reader.$encodedUrl/route.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import type { ActionFunctionArgs, LoaderFunctionArgs } from "@remix-run/node";
import { useParams } from "@remix-run/react";
import { typedjson, useTypedLoaderData } from "remix-typedjson";
import { Header } from "~/components/Header";
import { normalizeAndSanitizeUrl } from "~/utils/normalize-and-sanitize-url.server";
import { getTargetLanguage } from "~/utils/target-language.server";
import { authenticator } from "../../utils/auth.server";
import { ContentWithTranslations } from "./components/ContentWithTranslations";
Expand All @@ -15,15 +16,17 @@ import { actionSchema } from "./types";

export const loader = async ({ params, request }: LoaderFunctionArgs) => {
const targetLanguage = await getTargetLanguage(request);

const { encodedUrl } = params;

if (!encodedUrl) {
throw new Response("Missing URL parameter", { status: 400 });
}

const safeUser = await authenticator.isAuthenticated(request);
const safeUserId = safeUser?.id;
const normalizedUrl = normalizeAndSanitizeUrl(encodedUrl);
const pageData = await fetchLatestPageVersionWithTranslations(
decodeURIComponent(encodedUrl),
normalizedUrl,
safeUserId ?? 0,
targetLanguage,
);
Expand Down Expand Up @@ -107,7 +110,6 @@ export default function ReaderView() {
sourceTextInfoWithTranslations={
pageData.sourceTextInfoWithTranslations
}
targetLanguage={targetLanguage}
userId={safeUser?.id ?? null}
/>
ttizze marked this conversation as resolved.
Show resolved Hide resolved
</article>
ttizze marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
6 changes: 4 additions & 2 deletions web/app/routes/translate/route.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { Header } from "~/components/Header";
import { getTranslateUserQueue } from "~/feature/translate/translate-user-queue";
import { validateGeminiApiKey } from "~/feature/translate/utils/gemini";
import { authenticator } from "~/utils/auth.server";
import { normalizeAndSanitizeUrl } from "~/utils/normalize-and-sanitize-url.server";
import { getTargetLanguage } from "~/utils/target-language.server";
import { GeminiApiKeyForm } from "./components/GeminiApiKeyForm";
import { URLTranslationForm } from "./components/URLTranslationForm";
Expand Down Expand Up @@ -75,10 +76,11 @@ export async function action({ request }: ActionFunctionArgs) {
};
}
const targetLanguage = await getTargetLanguage(request);
const normalizedUrl = normalizeAndSanitizeUrl(submission.value.url);
// Start the translation job in background
const queue = getTranslateUserQueue(safeUser.id);
const job = await queue.add(`translate-${safeUser.id}`, {
url: submission.value.url,
url: normalizedUrl,
targetLanguage,
apiKey: dbUser.geminiApiKey,
userId: safeUser.id,
Expand All @@ -88,7 +90,7 @@ export async function action({ request }: ActionFunctionArgs) {
return {
intent,
lastResult: submission.reply({ resetForm: true }),
url: submission.value.url,
url: normalizedUrl,
};
}
default: {
Expand Down
15 changes: 15 additions & 0 deletions web/app/utils/normalize-and-sanitize-url.server.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import normalizeUrl from "normalize-url";

export function normalizeAndSanitizeUrl(inputUrl: string): string {
let decodedUrl: string;
try {
decodedUrl = decodeURIComponent(inputUrl);
} catch {
decodedUrl = inputUrl;
}

return normalizeUrl(decodedUrl, {
stripHash: true,
removeQueryParameters: true,
});
}
Comment on lines +3 to +15

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image description CodeRabbit

normalizeAndSanitizeUrl関数内で、URLのデコードに失敗した場合のエラーハンドリングが不十分です。具体的なエラーをキャッチしてログに記録するか、適切なエラーメッセージを返すべきです。

	try {
		decodedUrl = decodeURIComponent(inputUrl);
	} catch (error) {
		console.error(`Failed to decode URL: ${inputUrl}`, error);
		decodedUrl = inputUrl;
	}

2 changes: 1 addition & 1 deletion web/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
"scripts": {
"build": "remix vite:build",
"install-and-build": "bun install && bun run build",
"deploy": "bun run build && wrangler pages deploy",
"dev": "remix vite:dev",
"start": "remix-serve ./build/server/index.js",
"typecheck": "tsc",
Expand Down Expand Up @@ -47,6 +46,7 @@
"isomorphic-dompurify": "^2.13.0",
"lucide-react": "^0.408.0",
"next-themes": "^0.3.0",
"normalize-url": "^8.0.1",
"react": "^19.0.0-beta-26f2496093-20240514",
"react-dom": "^19.0.0-beta-26f2496093-20240514",
"react-icons": "^5.2.1",
ttizze marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
72 changes: 72 additions & 0 deletions web/scripts/normalize-page-urls.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
import { PrismaClient } from "@prisma/client";
import { normalizeAndSanitizeUrl } from "../app/utils/normalize-and-sanitize-url.server";

const prisma = new PrismaClient();

async function normalizeUrls() {
const batchSize = 100;
let processedPageCount = 0;
let processedPageVersionCount = 0;

try {
// Normalize Page URLs
while (true) {
const pages = await prisma.page.findMany({
take: batchSize,
skip: processedPageCount,
select: { id: true, url: true },
});

if (pages.length === 0) break;

for (const page of pages) {
const normalizedUrl = normalizeAndSanitizeUrl(page.url);
if (normalizedUrl !== page.url) {
await prisma.page.update({
where: { id: page.id },
data: { url: normalizedUrl },
});
console.log(`Updated Page URL: ${page.url} -> ${normalizedUrl}`);
}
}

processedPageCount += pages.length;
console.log(`Processed ${processedPageCount} pages`);
}
Comment on lines +11 to +35

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image description CodeRabbit

この部分のコードは、ページURLを正規化およびサニタイズするためにバッチ処理を行っていますが、データベース操作のパフォーマンスを向上させるためにトランザクションを使用することを検討してください。これにより、複数の更新操作を一括でコミットでき、パフォーマンスが向上します。

		await prisma.$transaction(async (prisma) => {
			while (true) {
				const pages = await prisma.page.findMany({
					take: batchSize,
					skip: processedPageCount,
					select: { id: true, url: true },
				});

				if (pages.length === 0) break;

				for (const page of pages) {
					const normalizedUrl = normalizeAndSanitizeUrl(page.url);
					if (normalizedUrl !== page.url) {
						await prisma.page.update({
							where: { id: page.id },
							data: { url: normalizedUrl },
						});
						console.log(`Updated Page URL: ${page.url} -> ${normalizedUrl}`);
					}
				}

				processedPageCount += pages.length;
				console.log(`Processed ${processedPageCount} pages`);
			}
		});


// Normalize PageVersion URLs
while (true) {
const pageVersions = await prisma.pageVersion.findMany({
take: batchSize,
skip: processedPageVersionCount,
select: { id: true, url: true },
});

if (pageVersions.length === 0) break;

for (const pageVersion of pageVersions) {
const normalizedUrl = normalizeAndSanitizeUrl(pageVersion.url);
if (normalizedUrl !== pageVersion.url) {
await prisma.pageVersion.update({
where: { id: pageVersion.id },
data: { url: normalizedUrl },
});
console.log(
`Updated PageVersion URL: ${pageVersion.url} -> ${normalizedUrl}`,
);
}
}

processedPageVersionCount += pageVersions.length;
console.log(`Processed ${processedPageVersionCount} page versions`);
}
Comment on lines +37 to +62

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image description CodeRabbit

この部分のコードも同様に、ページバージョンURLを正規化およびサニタイズするためにバッチ処理を行っていますが、データベース操作のパフォーマンスを向上させるためにトランザクションを使用することを検討してください。

		await prisma.$transaction(async (prisma) => {
			while (true) {
				const pageVersions = await prisma.pageVersion.findMany({
					take: batchSize,
					skip: processedPageVersionCount,
					select: { id: true, url: true },
				});

				if (pageVersions.length === 0) break;

				for (const pageVersion of pageVersions) {
					const normalizedUrl = normalizeAndSanitizeUrl(pageVersion.url);
					if (normalizedUrl !== pageVersion.url) {
						await prisma.pageVersion.update({
							where: { id: pageVersion.id },
							data: { url: normalizedUrl },
						});
						console.log(
							`Updated PageVersion URL: ${pageVersion.url} -> ${normalizedUrl}`,
						);
					}
				}

				processedPageVersionCount += pageVersions.length;
				console.log(`Processed ${processedPageVersionCount} page versions`);
			}
		});


console.log("URL normalization complete");
} catch (error) {
console.error("Error during URL normalization:", error);
} finally {
await prisma.$disconnect();
}
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image description CodeRabbit

ファイルの最後に改行を追加してください。

normalizeUrls();
+ 


normalizeUrls();
Loading