feat: use pinned IPFS container version, other improvements (#1006)

* feat: many CUs in worker * Apply automatic changes * use nox with multiCU * up * improve workers remove, rename property in provider config * fix * Apply automatic changes * fix * strip ansi colors from connector messages * add provider name to provider info * improve tx error * return descriptions * Apply automatic changes * add new line * remove price per worker epoch mentions * Apply automatic changes * fix(deps,deal): update multicu deal-ts-clients, fix deal init balance calc * Apply automatic changes * fix account switching, update default curl CID * fix offer update false positives * chore(versions): set nox 0.26.1 * update versions.json * Update cli/src/versions.json * feat: chunk offer create (#1004) * feat: chunk offer create * add chunking to update offer * fix(deal-ts-clients): rename Multicall import to IMulticall (#1005) * fix(deal-ts-clients): rename import * fix(deps): revert version, keep fix * feat: create only one ipfs client * debug all * use specific container version * use a different ipfs version * push cli version, don't allow to send tx if wallet is different * show warnings instead of failing on offer create * improve contributing doc * fix --------- Co-authored-by: shamsartem <[email protected]> Co-authored-by: folex <[email protected]> Co-authored-by: folex <[email protected]> Co-authored-by: Alexey Proshutinskiy <[email protected]>
fluencelabs · Aug 29, 2024 · 3095b26 · 3095b26
1 parent 4cc06c0
commit 3095b26
Show file tree

Hide file tree

Showing 8 changed files with 140 additions and 99 deletions.
diff --git a/cli/CONTRIBUTING.md → CONTRIBUTING.md b/cli/CONTRIBUTING.md → CONTRIBUTING.md
@@ -14,14 +14,14 @@ When you contribute, you have to be aware that your contribution is covered by *
 
 ## Guidelines for contributors
 
--   CLI uses `yarn` as a package manager. To install yarn you need Node.js 18 installed and then run `corepack enable`. Then you run `yarn` to install all the dependencies and you need to also run `yarn before-build` before moving any further
+-   CLI repo consists of a yarn monorepo (that uses turborepo for caching) which contains everything in the `./packages` dir + a separate yarn package for CLI itself is needed in the `./cli` dir (because oclif framework that doesn't work as part of the monorepo). To install yarn you need Node.js 18 installed and then run `corepack enable`. Then you run `yarn` to install all the dependencies for the monorepo and `yarn build` to build everything. You will also have to separately run `yarn` and `yarn build` inside `cli` directory
 -   To run tests locally you need to do the following:
     1. Install and run [Docker](https://docs.docker.com/get-docker/)
-    1. run `yarn`
-    1. run e.g. for linux: `DEBUG=fcli:*,deal-ts-clients:* CI=true yarn test-linux-x64` which will lint and check the code, build it, package it, prepare the tests and run them. For fish shell: `env DEBUG="fcli:*,deal-ts-clients:*" CI="true" yarn test-linux-x64`. `DEBUG` env variable is needed to see debug logs of Fluence CLI and deal-ts-clients lib (you can also inspect js-client logs like by adding `fluence:*`, libp2p also uses a similar system, check out their docs). `CI=true` is needed so tests look just like in CI without seeing spinners that might duplicate in the output
+    1. run `yarn` and `yarn build` top level and run `yarn` in `cli` dir
+    1. run e.g. for linux in `cli` dir: `DEBUG=fcli:*,deal-ts-clients:* yarn test-linux-x64` which will lint and check the code, build it, package it, prepare the tests and run them. For fish shell: `env DEBUG="fcli:*,deal-ts-clients:*" yarn test-linux-x64`. `DEBUG` env variable is needed to see debug logs of Fluence CLI and deal-ts-clients lib (you can also inspect js-client logs like by adding `fluence:*`, libp2p also uses a similar system, check out their docs).
 -   **First** commit in your PR or PR title must use [Conventional Commits](https://www.conventionalcommits.org/) (optionally end your commit message with: `[fixes DXJ-000 DXJ-001]`. Where `DXJ-000` and `DXJ-001` are ids of the Linear issues that you were working on)
--   To use Fluence CLI in the development mode, run: `./bin/dev.js` (types are not checked in development mode because it's faster and more convenient to work with. Use typechecking provided by your IDE during development)
--   To use Fluence CLI in the production mode, run `yarn build` first, then run: `./bin/run.js`. If you want to make sure you are running the actual package the users will use, do `yarn pack-*` command for your platform and architecture (this is used for tests as well)
+-   To use Fluence CLI in the development mode, run: `./cli/bin/dev.js` (types are not checked in development mode because it's faster and more convenient to work with. Use typechecking provided by your IDE during development)
+-   To use Fluence CLI in the production mode, run `yarn build` first, then run: `./cli/bin/run.js`. If you want to make sure you are running the actual package the users will use, do `yarn pack-*` command for your platform and architecture (this is used for tests as well)
 -   Don't name arguments or flags with names that contain underscore symbols because autogenerated links in markdown will not work
 -   Don't export anything from command files except for the command itself. If you need to share code between commands - create a separate file
 -   Avoid using `this` in commands except for inside `initCli` function. This style is easier to understand and there will be less stuff to refactor if instead of using methods on command object you simply use separate functions which can later be moved outside into a separate module for reuse in other commands

diff --git a/cli/src/lib/chain/offer/offer.ts b/cli/src/lib/chain/offer/offer.ts
@@ -78,22 +78,25 @@ export type OffersArgs = {
 };
 
 export async function createOffers(flags: OffersArgs) {
-  const offers = await resolveOffersFromProviderConfig(flags);
+  const allOffers = await resolveOffersFromProviderConfig(flags);
   const { dealClient } = await getDealClient();
   const market = dealClient.getMarket();
   const usdc = dealClient.getUSDC();
   const providerArtifactsConfig = await initNewProviderArtifactsConfig();
   const fluenceEnv = await ensureFluenceEnv();
 
-  const alreadyCreatedOffers = offers.filter(({ offerName }) => {
-    const { id } =
-      providerArtifactsConfig.offers[fluenceEnv]?.[offerName] ?? {};
+  const [alreadyCreatedOffers, offers] = splitErrorsAndResults(
+    allOffers,
+    (offer) => {
+      const { id } =
+        providerArtifactsConfig.offers[fluenceEnv]?.[offer.offerName] ?? {};
 
-    return id !== undefined;
-  });
+      return id === undefined ? { result: offer } : { error: offer };
+    },
+  );
 
   if (alreadyCreatedOffers.length > 0 && flags.force !== true) {
-    commandObj.error(
+    commandObj.warn(
       `You already created the following offers: ${alreadyCreatedOffers
         .map(({ offerName }) => {
           return offerName;
@@ -176,69 +179,75 @@ export async function createOffers(flags: OffersArgs) {
 
     const REGISTER_OFFER_TITLE = `Register offer: ${offerName}`;
 
-    if (
-      allCUs.length <= GUESS_NUMBER_OF_CU_THAT_FIT_IN_ONE_TX &&
-      computePeersFromProviderConfig.length <=
-        GUESS_NUMBER_OF_CP_THAT_FIT_IN_ONE_TX
-    ) {
-      const offerRegisterTxReceipt = await sign({
-        validateAddress: assertProviderIsRegistered,
-        title: REGISTER_OFFER_TITLE,
-        method: market.registerMarketOffer,
-        args: [
-          minPricePerCuPerEpochBigInt,
-          usdcAddress,
-          effectorPrefixesAndHash,
-          computePeersFromProviderConfig,
-          minProtocolVersion ?? versions.protocolVersion,
-          maxProtocolVersion ?? versions.protocolVersion,
-        ],
-      });
-
-      registeredMarketOffers.push(getOfferIdRes(offerRegisterTxReceipt));
-    } else {
-      const offerRegisterTxReceipt = await sign({
-        validateAddress: assertProviderIsRegistered,
-        title: REGISTER_OFFER_TITLE,
-        method: market.registerMarketOffer,
-        args: [
-          minPricePerCuPerEpochBigInt,
-          usdcAddress,
-          effectorPrefixesAndHash,
-          [],
-          minProtocolVersion ?? versions.protocolVersion,
-          maxProtocolVersion ?? versions.protocolVersion,
-        ],
-      });
-
-      const offerIdRes = getOfferIdRes(offerRegisterTxReceipt);
-      registeredMarketOffers.push(offerIdRes);
-
-      if ("error" in offerIdRes) {
-        continue;
-      }
+    try {
+      if (
+        allCUs.length <= GUESS_NUMBER_OF_CU_THAT_FIT_IN_ONE_TX &&
+        computePeersFromProviderConfig.length <=
+          GUESS_NUMBER_OF_CP_THAT_FIT_IN_ONE_TX
+      ) {
+        const offerRegisterTxReceipt = await sign({
+          validateAddress: assertProviderIsRegistered,
+          title: REGISTER_OFFER_TITLE,
+          method: market.registerMarketOffer,
+          args: [
+            minPricePerCuPerEpochBigInt,
+            usdcAddress,
+            effectorPrefixesAndHash,
+            computePeersFromProviderConfig,
+            minProtocolVersion ?? versions.protocolVersion,
+            maxProtocolVersion ?? versions.protocolVersion,
+          ],
+        });
+
+        registeredMarketOffers.push(getOfferIdRes(offerRegisterTxReceipt));
+      } else {
+        const offerRegisterTxReceipt = await sign({
+          validateAddress: assertProviderIsRegistered,
+          title: REGISTER_OFFER_TITLE,
+          method: market.registerMarketOffer,
+          args: [
+            minPricePerCuPerEpochBigInt,
+            usdcAddress,
+            effectorPrefixesAndHash,
+            [],
+            minProtocolVersion ?? versions.protocolVersion,
+            maxProtocolVersion ?? versions.protocolVersion,
+          ],
+        });
+
+        const offerIdRes = getOfferIdRes(offerRegisterTxReceipt);
+        registeredMarketOffers.push(offerIdRes);
+
+        if ("error" in offerIdRes) {
+          continue;
+        }
 
-      const { offerId } = offerIdRes.result;
-
-      for (const cp of computePeersFromProviderConfig) {
-        for (const [i, unitIds] of Object.entries(
-          chunk(cp.unitIds, GUESS_NUMBER_OF_CU_THAT_FIT_IN_ONE_TX),
-        )) {
-          if (i === "0") {
-            await sign({
-              title: `Add compute peer ${cp.name} (${cp.peerIdBase58})\nto offer ${offerName} (${offerId})`,
-              method: market.addComputePeers,
-              args: [offerId, [{ ...cp, unitIds }]],
-            });
-          } else {
-            await sign({
-              title: `Add ${numToStr(unitIds.length)} compute units\nto compute peer ${cp.name} (${cp.peerIdBase58})\nfor offer ${offerName} (${offerId})`,
-              method: market.addComputeUnits,
-              args: [cp.peerId, unitIds],
-            });
+        const { offerId } = offerIdRes.result;
+
+        for (const cp of computePeersFromProviderConfig) {
+          for (const [i, unitIds] of Object.entries(
+            chunk(cp.unitIds, GUESS_NUMBER_OF_CU_THAT_FIT_IN_ONE_TX),
+          )) {
+            if (i === "0") {
+              await sign({
+                title: `Add compute peer ${cp.name} (${cp.peerIdBase58})\nto offer ${offerName} (${offerId})`,
+                method: market.addComputePeers,
+                args: [offerId, [{ ...cp, unitIds }]],
+              });
+            } else {
+              await sign({
+                title: `Add ${numToStr(unitIds.length)} compute units\nto compute peer ${cp.name} (${cp.peerIdBase58})\nfor offer ${offerName} (${offerId})`,
+                method: market.addComputeUnits,
+                args: [cp.peerId, unitIds],
+              });
+            }
           }
         }
       }
+    } catch (e) {
+      commandObj.warn(
+        `Error when creating offer ${offerName}: ${stringifyUnknown(e)}`,
+      );
     }
   }
 
@@ -250,7 +259,7 @@ export async function createOffers(flags: OffersArgs) {
   );
 
   if (offerIdErrors.length > 0) {
-    commandObj.error(
+    commandObj.warn(
       `When getting ${OFFER_ID_PROPERTY} property from event ${MARKET_OFFER_REGISTERED_EVENT_NAME}:\n\n${offerIdErrors.join(
         ", ",
       )}`,

diff --git a/cli/src/lib/configs/project/dockerCompose.ts b/cli/src/lib/configs/project/dockerCompose.ts
@@ -200,7 +200,7 @@ async function genDockerCompose(): Promise<LatestConfig> {
     ),
     services: {
       [IPFS_CONTAINER_NAME]: {
-        image: "ipfs/kubo",
+        image: "ipfs/kubo:v0.27.0",
         ports: [`${IPFS_PORT}:${IPFS_PORT}`, "4001:4001"],
         environment: {
           IPFS_PROFILE: "server",

diff --git a/cli/src/lib/localServices/ipfs.ts b/cli/src/lib/localServices/ipfs.ts
@@ -19,6 +19,7 @@ import { access, readFile } from "node:fs/promises";
 
 import type { IPFSHTTPClient } from "ipfs-http-client";
 
+import { jsonStringify } from "../../common.js";
 import { commandObj } from "../commandObj.js";
 import { FS_OPTIONS } from "../const.js";
 import { dbg } from "../dbg.js";
@@ -30,17 +31,27 @@ import { setTryTimeout, stringifyUnknown } from "../helpers/utils.js";
 
 /* eslint-disable @typescript-eslint/no-unsafe-return, @typescript-eslint/no-unsafe-assignment,  @typescript-eslint/no-unsafe-member-access, @typescript-eslint/no-unsafe-call, @typescript-eslint/restrict-template-expressions  */
 
-export async function createIPFSClient(multiaddrString: string) {
-  const [{ multiaddr, protocols }, { create }] = await Promise.all([
-    import("@multiformats/multiaddr"),
-    import("ipfs-http-client"),
-  ]);
+let ipfsClient: Promise<IPFSHTTPClient> | null = null;
 
-  return create(
-    multiaddr(multiaddrString)
-      .decapsulateCode(protocols("p2p").code)
-      .toOptions(),
-  );
+export function createIPFSClient(multiaddrString: string) {
+  if (ipfsClient === null) {
+    ipfsClient = (async () => {
+      const [{ multiaddr, protocols }, { create }] = await Promise.all([
+        import("@multiformats/multiaddr"),
+        import("ipfs-http-client"),
+      ]);
+
+      const createOptions = multiaddr(multiaddrString)
+        .decapsulateCode(protocols("p2p").code)
+        .toOptions();
+
+      dbg(`creating ipfs client with options: ${jsonStringify(createOptions)}`);
+      dbg(`multiaddr: ${multiaddrString}`);
+      return create(createOptions);
+    })();
+  }
+
+  return ipfsClient;
 }
 
 async function upload(
@@ -49,7 +60,6 @@ async function upload(
 ): Promise<string> {
   try {
     const ipfsClient = await createIPFSClient(multiaddr);
-    dbg(`created ipfs client`);
 
     const { cid } = await ipfsClient.add(content, {
       pin: true,

diff --git a/cli/src/lib/server.ts b/cli/src/lib/server.ts
@@ -229,5 +229,9 @@ export async function returnToCLI() {
 }
 
 function ping() {
-  void sendEvent({ tag: "ping", addressUsedByCLI: addressFromConnector });
+  void sendEvent({
+    tag: "ping",
+    addressUsedByCLI: addressFromConnector,
+    CLIVersion: commandObj.config.version,
+  });
 }
diff --git a/packages/cli-connector/src/App.tsx b/packages/cli-connector/src/App.tsx
@@ -125,6 +125,7 @@ export function App({
 
   const CLIDisconnectedTimeout = useRef<number>();
   const gotFirstMessage = useRef(false);
+  const cliVersion = useRef<string | null>(null);
 
   useEffect(() => {
     const events = new EventSource("/events");
@@ -160,6 +161,15 @@ export function App({
           sendTransaction(msg.payload.transactionData);
         }
       } else if (msg.tag === "ping") {
+        if (cliVersion.current === null) {
+          cliVersion.current = msg.CLIVersion;
+        }
+
+        if (cliVersion.current !== msg.CLIVersion) {
+          // reload the page to get the latest frontend version
+          window.location.href = window.location.href;
+        }
+
         setAddressUsedByCLI(msg.addressUsedByCLI);
         setIsCLIConnected(true);
         clearTimeout(CLIDisconnectedTimeout.current);
@@ -207,7 +217,16 @@ export function App({
     return <h1>Return to your terminal in order to continue</h1>;
   }
 
-  const isSendTxButtonEnabled = !isPending && !isLoading && !isSuccess;
+  const hasSwitchedAccountDuringCommandExecution =
+    addressUsedByCLI !== null &&
+    address !== undefined &&
+    addressUsedByCLI !== address;
+
+  const isSendTxButtonEnabled =
+    !isPending &&
+    !isLoading &&
+    !isSuccess &&
+    !hasSwitchedAccountDuringCommandExecution;
 
   return (
     <>
@@ -249,18 +268,16 @@ export function App({
           smallScreen: "none",
         }}
       />
-      {addressUsedByCLI !== null &&
-        address !== undefined &&
-        addressUsedByCLI !== address && (
-          <div className="error">
-            The account address sent to CLI when command execution started:
-            {" \n"}
-            <b>{addressUsedByCLI}</b> is different from the current account
-            address:{" \n"}
-            <b>{address}</b>. Please switch back to the{" \n"}
-            <b>{addressUsedByCLI}</b> account or rerun the CLI command
-          </div>
-        )}
+      {hasSwitchedAccountDuringCommandExecution && (
+        <div className="error">
+          The account address sent to CLI when command execution started:
+          {" \n"}
+          <b>{addressUsedByCLI}</b> is different from the current account
+          address:{" \n"}
+          <b>{address}</b>. Please switch back to the{" \n"}
+          <b>{addressUsedByCLI}</b> account or rerun the CLI command
+        </div>
+      )}
       {isConnected && address !== undefined && (
         <>
           {isExpectingAddress && (

diff --git a/packages/cli-connector/src/index.css b/packages/cli-connector/src/index.css
@@ -75,7 +75,7 @@ summary {
 }
 
 .button_disabled {
-  cursor: wait;
+  cursor: not-allowed;
   opacity: 0.5;
 }
 

diff --git a/packages/common/src/index.ts b/packages/common/src/index.ts
@@ -82,6 +82,7 @@ export type CLIToConnectorMsg =
   | {
       tag: "ping";
       addressUsedByCLI: string | null;
+      CLIVersion: string;
     }
   | {
       tag: "returnToCLI";