4th milestone features (#79)

* Milestone 4 - draft: TODO * User stats endpoint, test and possible rework of response mapping * Add unit tests * Add proxy endpoints for: - user/register-address - auth/signin/web3 * Adds test coverage, minor fixes * Adds additional proxies for 4 milestone * Reworks reputation oracle gateway test suite, adds necessary mappings * adds missing await * Review fixes * Adds /user/register-address handling * Changes endpoint name * Adds import fixes * Adds controller and module to the main app module * Updates README.md (#125) * Updates README.md: Adds Caching section, Changes in signature type * README.md updates * README.md updates * fix(app/user/worker): remove chain id validation (#114) * fix(app/user/worker): remove chain id validation * Test fix and JWT User data align * Updates README.md: Adds Caching section, Changes in signature type * Removes commented code * Changes after hcaptcha api check * Adds interceptor for axios requests * Fix/signature (#126) * Updates README.md (#125) * Updates README.md: Adds Caching section, Changes in signature type * README.md updates * README.md updates * fix(app/worker): add signature field to req body * update(app): revert readme * fix(app/register-address): fix test --------- Co-authored-by: macnablocky <[email protected]> Co-authored-by: Kacper Koza <[email protected]> * Fixes undefined in POST data, fixes unit tests * Adds Resign job feature, small changes in the gateway config (#134) * Fixes * Fixes in dependencies --------- Co-authored-by: Kacper Koza <[email protected]> Co-authored-by: maciej.nabialek <[email protected]> Co-authored-by: macnablocky <[email protected]> * Fixes restore password * Fixes restore password part 2. * review fixes --------- Co-authored-by: KacperKoza343 <[email protected]> Co-authored-by: Kacper Koza <[email protected]> Co-authored-by: Bartosz Solka <[email protected]>
blockydevs · Jun 20, 2024 · 179358e · 179358e
1 parent 035e9b1
commit 179358e
Show file tree

Hide file tree

Showing 114 changed files with 5,812 additions and 3,638 deletions.
diff --git a/packages/apps/human-app/server/.env.example b/packages/apps/human-app/server/.env.example
@@ -4,16 +4,27 @@ REPUTATION_ORACLE_URL= # string
 REPUTATION_ORACLE_ADDRESS= # string
 REDIS_HOST= # string, example: localhost
 REDIS_PORT= # number, example: 6379
+RPC_URL= # string
+CHAIN_IDS_ENABLED= # number array, example: 80002,80001
+HCAPTCHA_LABELING_STATS_API_URL= # string
+HCAPTCHA_LABELING_VERIFY_API_URL= # string
+HCAPTCHA_LABELING_API_KEY= # string
+DAILY_HMT_SPENT_KEY= # string
+IS_AXIOS_REQUEST_LOGGING_ENABLED= #string, true if enabled, disabled otherwise
+# CACHE TTL VALUES
 CACHE_TTL_ORACLE_DISCOVERY= # number, example: 43200
 CACHE_TTL_ORACLE_STATS= # number, example: 900
 CACHE_TTL_USER_STATS= # number, example: 86400
+CACHE_TTL_EXCHANGE_ORACLE_URL= # number: example 86400
+CACHE_TTL_HCAPTCHA_USER_STATS= # number: example 86400
+CACHE_TTL_DAILY_HMT_SPENT= # number: example 86400
+# E2E TESTING
 E2E_TESTING_EMAIL_ADDRESS= # string
 E2E_TESTING_PASSWORD= # string
 E2E_TESTING_EXCHANGE_ORACLE_URL= # string
 E2E_TESTING_ESCROW_ADDRESS= # string
 E2E_TESTING_ESCROW_CHAIN_ID= # number
-RPC_URL= # string
-CORS_ENABLED= # boolean, example: true
+# CORS
 CORS_ALLOWED_ORIGIN= # string example: http://localhost:5173
 CORS_ALLOWED_HEADERS= # string, example: 'Content-Type,Accept'
-CHAIN_IDS_ENABLED= # number array, example: 80002,80001
+CORS_ENABLED= # boolean, example: true
diff --git a/packages/apps/human-app/server/README.md b/packages/apps/human-app/server/README.md
@@ -69,9 +69,28 @@ Models are used to define the shape and responsibilities of the data:
 
 ### Additional Information Regarding Project Structure
 
-- **Mappers**: There are two types of mappers. The first type is domain-specific, used in each module
-to distinguish between DTO and Command datatypes. The second type is layer-specific and part of gateway integration. 
-This division of responsibilities is intentional, as gateways serve as the gathering point between many domain purposes. 
-Domain-specific mappers in this context would be overly convoluted and difficult to configure.
-- **Gateway Configuration**: The configuration of the gateways' destination points is located 
-in the `gateway-config.service.ts` file.
+  - **Mappers**: There are two types of mappers. The first type is domain-specific, used in each module
+  to distinguish between DTO and Command datatypes. The second type is layer-specific and part of gateway integration. 
+  This division of responsibilities is intentional, as gateways serve as the gathering point between many domain purposes. 
+  Domain-specific mappers in this context would be overly convoluted and difficult to configure.
+  - **Gateway Configuration**: The configuration of the gateways' destination points is located 
+  in the `gateway-config.service.ts` file.
+  - **Caching**: Cache persistence functionality was introduced in this project for optimization purposes. Redis was chosen
+  as the in-memory storage. The TTL can be checked and changed in the `.env.example` file, all `TTL` values are in seconds:
+    * `REDIS_HOST` - URL of the Redis host
+    * `REDIS_PORT` - port on which Redis is hosted
+    * `CACHE_TTL_ORACLE_DISCOVERY` - time of persisting found oracles
+    * `CACHE_TTL_ORACLE_STATS` - time of persisting statistics of the given oracle
+    * `CACHE_TTL_DAILY_HMT_SPENT` - time of persisting statistics of global daily HMT expenditures
+    * `CACHE_TTL_USER_STATS` - time of persisting statistics of the given user
+    * `CACHE_TTL_HCAPTCHA_USER_STATS` - time of persisting statistics related to h-captcha tasks for given user
+    * `CACHE_TTL_EXCHANGE_ORACLE_URL` - time of persisting exchange oracle URL
+
+    Caching is used for persisting the exchange oracle URL as well as responses from the following endpoints:
+    * `/h-captcha/daily-hmt-spent`
+    * `/h-captcha/user-stats`
+    * `/oracles`
+    * `/statistics/stats`
+    * `/statistics/stats/assignment`
+
+    Redis config may be found in: `./packages/apps/human-app/server/src/common/config/cache-factory.config.ts`
diff --git a/packages/apps/human-app/server/package.json b/packages/apps/human-app/server/package.json
@@ -29,15 +29,20 @@
     "@nestjs/common": "^10.2.7",
     "@nestjs/config": "^3.1.1",
     "@nestjs/core": "^10.2.8",
+    "@nestjs/jwt": "^10.2.0",
+    "@nestjs/passport": "^10.0.3",
     "@nestjs/platform-express": "^10.3.8",
     "@nestjs/swagger": "^7.1.13",
+    "@types/passport-jwt": "^4.0.1",
     "cache-manager": "^5.4.0",
     "cache-manager-redis-store": "^3.0.1",
     "ioredis": "^5.3.2",
     "class-transformer": "^0.5.1",
     "class-validator": "^0.14.1",
     "ethers": "^6.12.1",
     "joi": "^17.12.2",
+    "passport": "^0.7.0",
+    "passport-jwt": "^4.0.1",
     "reflect-metadata": "^0.1.13",
     "rxjs": "^7.2.0"
   },

diff --git a/packages/apps/human-app/server/src/app.module.ts b/packages/apps/human-app/server/src/app.module.ts
@@ -27,9 +27,15 @@ import { PasswordResetModule } from './modules/password-reset/password-reset.mod
 import { DisableOperatorModule } from './modules/disable-operator/disable-operator.module';
 import { KycProcedureModule } from './modules/kyc-procedure/kyc-procedure.module';
 import { PrepareSignatureModule } from './modules/prepare-signature/prepare-signature.module';
+import { HCaptchaModule } from './modules/h-captcha/h-captcha.module';
+import { HCaptchaLabelingModule } from './integrations/h-captcha-labeling/h-captcha-labeling.module';
+import { HCaptchaController } from './modules/h-captcha/h-captcha.controller';
 import { EscrowUtilsModule } from './integrations/escrow/escrow-utils.module';
 import Joi from 'joi';
 import { ChainId } from '@human-protocol/sdk';
+import { RegisterAddressController } from './modules/register-address/register-address.controller';
+import { RegisterAddressModule } from './modules/register-address/register-address.module';
+import { InterceptorModule } from './common/interceptors/interceptor.module';
 
 @Module({
   imports: [
@@ -44,6 +50,9 @@ import { ChainId } from '@human-protocol/sdk';
         REDIS_PORT: Joi.number().required(),
         REDIS_HOST: Joi.string().required(),
         RPC_URL: Joi.string().required(),
+        HCAPTCHA_LABELING_STATS_API_URL: Joi.string().required(),
+        HCAPTCHA_LABELING_VERIFY_API_URL: Joi.string().required(),
+        HCAPTCHA_LABELING_API_KEY: Joi.string().required(),
         CHAIN_IDS_ENABLED: Joi.string()
           .custom((value) => {
             const chainIds = value.split(',');
@@ -79,7 +88,11 @@ import { ChainId } from '@human-protocol/sdk';
     DisableOperatorModule,
     KycProcedureModule,
     PrepareSignatureModule,
+    HCaptchaModule,
+    HCaptchaLabelingModule,
     EscrowUtilsModule,
+    RegisterAddressModule,
+    InterceptorModule,
   ],
   controllers: [
     AppController,
@@ -89,6 +102,8 @@ import { ChainId } from '@human-protocol/sdk';
     OracleDiscoveryController,
     JobAssignmentController,
     StatisticsController,
+    HCaptchaController,
+    RegisterAddressController,
   ],
   exports: [HttpModule],
 })

diff --git a/packages/apps/human-app/server/src/common/config/environment-config.service.ts b/packages/apps/human-app/server/src/common/config/environment-config.service.ts
@@ -1,10 +1,13 @@
 import { ConfigService } from '@nestjs/config';
 import { Injectable } from '@nestjs/common';
+const DEFAULT_CACHE_TTL_HCAPTCHA_USER_STATS = 12 * 60 * 60;
 const DEFAULT_CACHE_TTL_ORACLE_STATS = 12 * 60 * 60;
 const DEFAULT_CACHE_TTL_USER_STATS = 15 * 60;
 const DEFAULT_CACHE_TTL_ORACLE_DISCOVERY = 24 * 60 * 60;
+const DEFAULT_CACHE_TTL_DAILY_HMT_SPENT = 24 * 60 * 60;
 const DEFAULT_CORS_ALLOWED_ORIGIN = 'http://localhost:5173';
 const DEFAULT_CORS_ALLOWED_HEADERS = 'Content-Type, Accept';
+const DEFAULT_CACHE_TTL_EXCHANGE_ORACLE_URL = 24 * 60 * 60;
 @Injectable()
 export class EnvironmentConfigService {
   constructor(private configService: ConfigService) {}
@@ -18,7 +21,14 @@ export class EnvironmentConfigService {
     return this.configService.getOrThrow<string>('REPUTATION_ORACLE_URL');
   }
   get reputationOracleAddress(): string {
-    return this.configService.getOrThrow<string>('REPUTATION_ORACLE_ADDRESS');
+    return this.configService
+      .getOrThrow<string>('REPUTATION_ORACLE_ADDRESS')
+      .toLowerCase();
+  }
+  get axiosRequestLoggingEnabled(): boolean {
+    return (
+      this.configService.get('IS_AXIOS_REQUEST_LOGGING_ENABLED') === 'true'
+    );
   }
   get cachePort(): number {
     return this.configService.getOrThrow<number>('REDIS_PORT');
@@ -32,13 +42,27 @@ export class EnvironmentConfigService {
       DEFAULT_CACHE_TTL_ORACLE_STATS,
     );
   }
-
+  get dailyHmtSpentKey(): string {
+    return this.configService.getOrThrow('DAILY_HMT_SPENT_KEY');
+  }
   get cacheTtlUserStats(): number {
     return this.configService.get<number>(
       'CACHE_TTL_USER_STATS',
       DEFAULT_CACHE_TTL_USER_STATS,
     );
   }
+  get cacheTtlDailyHmtSpent(): number {
+    return this.configService.get<number>(
+      'CACHE_TTL_DAILY_HMT_SPENT',
+      DEFAULT_CACHE_TTL_DAILY_HMT_SPENT,
+    );
+  }
+  get cacheTtlHCaptchaUserStats(): number {
+    return this.configService.get<number>(
+      'CACHE_TTL_HCAPTCHA_USER_STATS',
+      DEFAULT_CACHE_TTL_HCAPTCHA_USER_STATS,
+    );
+  }
 
   get cacheTtlOracleDiscovery(): number {
     return this.configService.get<number>(
@@ -64,6 +88,25 @@ export class EnvironmentConfigService {
       DEFAULT_CORS_ALLOWED_HEADERS,
     );
   }
+  get cacheTtlExchangeOracleUrl(): number {
+    return this.configService.get<number>(
+      'CACHE_TTL_EXCHANGE_ORACLE_URL',
+      DEFAULT_CACHE_TTL_EXCHANGE_ORACLE_URL,
+    );
+  }
+  get hcaptchaLabelingStatsApiUrl(): string {
+    return this.configService.getOrThrow<string>(
+      'HCAPTCHA_LABELING_STATS_API_URL',
+    );
+  }
+  get hcaptchaLabelingVerifyApiUrl(): string {
+    return this.configService.getOrThrow<string>(
+      'HCAPTCHA_LABELING_VERIFY_API_URL',
+    );
+  }
+  get hcaptchaLabelingApiKey(): string {
+    return this.configService.getOrThrow<string>('HCAPTCHA_LABELING_API_KEY');
+  }
   get chainIdsEnabled(): string[] {
     const chainIds = this.configService.getOrThrow<string>('CHAIN_IDS_ENABLED');
     return chainIds.split(',').map((id) => id.trim());

diff --git a/packages/apps/human-app/server/src/common/config/gateway-config.service.ts b/packages/apps/human-app/server/src/common/config/gateway-config.service.ts
@@ -1,9 +1,16 @@
 import { Injectable } from '@nestjs/common';
 import { ExternalApiName } from '../enums/external-api-name';
-import { GatewayConfig, Gateways } from '../interfaces/endpoint.interface';
+import {
+  HCaptchaLabelingStatsEndpoints, HCaptchaLabelingVerifyEndpoints,
+  ReputationOracleEndpoints,
+} from '../enums/reputation-oracle-endpoints';
+import {
+  GatewayConfig,
+  GatewayEndpointConfig,
+  Gateways,
+} from '../interfaces/endpoint.interface';
 import { EnvironmentConfigService } from './environment-config.service';
 import { HttpMethod } from '../enums/http-method';
-import { ReputationOracleEndpoints } from '../enums/reputation-oracle-endpoints';
 
 @Injectable()
 export class GatewayConfigService {
@@ -68,7 +75,54 @@ export class GatewayConfigService {
               method: HttpMethod.POST,
               headers: this.JSON_HEADER,
             },
-          },
+            [ReputationOracleEndpoints.ENABLE_LABELING]: {
+              endpoint: '/user/register-labeler',
+              method: HttpMethod.POST,
+              params: {
+                api_key: this.envConfig.hcaptchaLabelingApiKey,
+              },
+            },
+            [ReputationOracleEndpoints.OPERATOR_SIGNIN]: {
+              endpoint: '/auth/web3/signin',
+              method: HttpMethod.POST,
+              headers: this.JSON_HEADER,
+            },
+            [ReputationOracleEndpoints.REGISTER_ADDRESS]: {
+              endpoint: '/user/register-address',
+              method: HttpMethod.POST,
+              headers: this.JSON_HEADER,
+            },
+          } as Record<ReputationOracleEndpoints, GatewayEndpointConfig>,
+        },
+        [ExternalApiName.HCAPTCHA_LABELING_STATS]: {
+          url: this.envConfig.hcaptchaLabelingStatsApiUrl,
+          endpoints: {
+            [HCaptchaLabelingStatsEndpoints.USER_STATS]: {
+              endpoint: '/support/labeler/', // email to append as url param
+              method: HttpMethod.GET,
+              params: {
+                api_key: this.envConfig.hcaptchaLabelingApiKey,
+              },
+            },
+            [HCaptchaLabelingStatsEndpoints.DAILY_HMT_SPENT]: {
+              endpoint: '/requester/daily_hmt_spend',
+              method: HttpMethod.GET,
+              params: {
+                api_key: this.envConfig.hcaptchaLabelingApiKey,
+                actual: false,
+              },
+            },
+          } as Record<HCaptchaLabelingStatsEndpoints, GatewayEndpointConfig>,
+        },
+        [ExternalApiName.HCAPTCHA_LABELING_VERIFY]: {
+          url: this.envConfig.hcaptchaLabelingVerifyApiUrl,
+          endpoints: {
+            [HCaptchaLabelingVerifyEndpoints.TOKEN_VERIFY]: {
+              endpoint: '/siteverify',
+              method: HttpMethod.POST,
+              // params in this method are dynamic
+            },
+          } as Record<HCaptchaLabelingVerifyEndpoints, GatewayEndpointConfig>,
         },
       },
     };

diff --git a/packages/apps/human-app/server/src/common/config/gateway-config.types.ts b/packages/apps/human-app/server/src/common/config/gateway-config.types.ts
@@ -0,0 +1,10 @@
+import {
+  HCaptchaLabelingStatsEndpoints,
+  HCaptchaLabelingVerifyEndpoints,
+  ReputationOracleEndpoints,
+} from '../enums/reputation-oracle-endpoints';
+
+export type GatewayEndpoints =
+  | HCaptchaLabelingStatsEndpoints
+  | ReputationOracleEndpoints
+  | HCaptchaLabelingVerifyEndpoints;
diff --git a/packages/apps/human-app/server/src/common/config/params-decorators.ts b/packages/apps/human-app/server/src/common/config/params-decorators.ts
@@ -1,8 +1,24 @@
 import { createParamDecorator, ExecutionContext } from '@nestjs/common';
+import { jwtDecode } from 'jwt-decode';
+import { JwtUserData } from '../interfaces/jwt-token.model';
 
 export const Authorization = createParamDecorator(
   (data: unknown, ctx: ExecutionContext) => {
     const request = ctx.switchToHttp().getRequest();
     return request.headers['authorization'];
   },
-);
+);
+export const JwtPayload = createParamDecorator(
+  (data: unknown, ctx: ExecutionContext): any => {
+    const request = ctx.switchToHttp().getRequest();
+    const token = request.headers['authorization']?.split(' ')[1];
+    if (!token) return null;
+    try {
+      const decoded = jwtDecode(token);
+      return decoded as JwtUserData;
+    } catch (error) {
+      console.error('Error in decoding token: ', error);
+      return null;
+    }
+  },
+);
diff --git a/...mon/config/gateway-config.service.mock.ts → ...onfig/spec/gateway-config-service.mock.ts b/...mon/config/gateway-config.service.mock.ts → ...onfig/spec/gateway-config-service.mock.ts
@@ -52,6 +52,48 @@ export const gatewayConfigServiceMock = {
         method: 'POST',
         headers: { 'Content-Type': 'application/json' },
       },
+      ENABLE_LABELING: {
+        endpoint: '/labeler/register',
+        method: 'POST',
+        params: { api_key: 'mock-api-key' },
+      },
+      OPERATOR_SIGNIN: {
+        endpoint: '/auth/web3/signin',
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        params: {},
+      },
+      REGISTER_ADDRESS: {
+        endpoint: '/user/register-address',
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+      },
+    },
+  }),
+};
+
+export const hCaptchaGatewayConfigServiceMock = {
+  getConfig: jest.fn().mockReturnValue({
+    url: 'https://api.example.com',
+    endpoints: {
+      TOKEN_VERIFY: {
+        method: 'POST',
+        endpoint: '/siteverify',
+        headers: {},
+        params: {},
+      },
+      DAILY_HMT_SPENT: {
+        method: 'GET',
+        endpoint: '/requester/daily_hmt_spend',
+        headers: {},
+        params: { api_key: 'mock-api-key', actual: false },
+      },
+      USER_STATS: {
+        method: 'GET',
+        endpoint: '/support/labeler/',
+        headers: {},
+        params: { api_key: 'mock-api-key' },
+      },
     },
   }),
 };
diff --git a/packages/apps/human-app/server/src/common/enums/external-api-name.ts b/packages/apps/human-app/server/src/common/enums/external-api-name.ts
@@ -1,3 +1,5 @@
 export enum ExternalApiName {
   REPUTATION_ORACLE = 'REPUTATION_ORACLE',
+  HCAPTCHA_LABELING_STATS = 'HCAPTCHA_LABELING_STATS',
+  HCAPTCHA_LABELING_VERIFY = 'HCAPTCHA_LABELING_VERIFY',
 }