feat: add datasource config in import module #509

hetao92 · 2023-03-16T11:03:16Z

What type of PR is this?

bug
feature
enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

nianiaJR · 2023-03-20T15:28:31Z

app/config/service.ts

+    return _delete(`/api/datasources/${id}`)(undefined, config);
+  },
+  batchDeleteDatasource: (payload, config?) => {
+    return _delete(`/api/datasources`)(undefined, { data: payload });


recommand: undefined => {}?

nianiaJR · 2023-03-20T15:29:04Z

app/interfaces/import.ts

+  endpoint: string;
+  accessKey: string;
+  accessSecret: string;
+  bucket?: string;


this is required now?

nianiaJR · 2023-03-20T15:29:38Z

app/interfaces/import.ts

+  accessSecret: string;
+  bucket?: string;
+  token?: string;
+  key: string;


what are the token and key?

they are the parameters of the importer, key is the file path

nianiaJR · 2023-03-20T15:30:11Z

app/interfaces/import.ts

+  keyFile?: string;
+  keyData?: string;
+  passPhrase?: string;
+}


what are they used for?

Keep the configuration definition the same as the nebula importer, but it is not currently used in the studio

If they are not used in our app, you could not pre-define them now, maybe we will never use these fields, it's little redundant.

nianiaJR · 2023-03-20T15:46:41Z

app/pages/Import/DatasourceList/DatasourceConfig/PlatformConfig/S3ConfigForm.tsx

+        <FormItem name="platform" rules={[{ required: true, message: intl.get('formRules.platformRequired') } ]}>
+          <Select placeholder={intl.get('import.selectPlatform')}>
+            <Select.Option value="aws">AWS S3</Select.Option>
+            <Select.Option value="aliyun">OSS</Select.Option>


Aliyun OSS?

S3 is a standard， also should consider the local s3 service

nianiaJR · 2023-03-20T15:50:20Z

app/pages/Import/TaskCreate/SchemaConfig/FileMapping/FileSelectModal.tsx

recommend to use one component per file, easy to understand

nianiaJR · 2023-03-20T15:52:30Z

server/api/studio/internal/service/datasource.go

-			return nil, ecode.WithBadRequest(err, "json stringify error")
+			return nil, ecode.WithErrorMessage(ecode.ErrBadRequest, err, "json stringify error")


WithBadRequest is more short and proper?

nianiaJR · 2023-03-20T15:54:55Z

server/api/studio/internal/service/datasource.go

+				bucket = pathParts[1]
+			}
+		} else {
+			// Format: https://<bucket-name>.s3.<region>.amazonaws.com


can it also work for aliyun oss?

I split the oss logic out from here

nianiaJR · 2023-03-20T15:56:26Z

server/api/studio/internal/service/datasource.go

 	}
 	defer conn.Close()

 	// create an SFTP client session
 	client, err := sftp.NewClient(conn)
 	if err != nil {
-		return ecode.WithBadRequest(err, "failed to create SFTP session")
+		return ecode.WithErrorMessage(ecode.ErrBadRequest, err, "failed to create SFTP session")


ecode.WithBadRquest is the short-mode of ecode.WithErrorMessage(ecode.ErrBadRequest....

its not same

@veezhang why are they different? I previously think that the WithBadRequest is the shortcut of ht WithErrorMessage(ecode.ErrBadRequest...

nianiaJR

LGTM

huaxiabuluo · 2023-03-27T06:54:22Z

app/interfaces/datasource.ts

+export enum IDatasourceType {
+  's3' = 's3',
+  'sftp' = 'sftp',
+  'local' = 'local'
+}
+export enum IS3Platform {
+  'aws' = 'aws',
+  'oss' = 'oss',
+  'customize' = 'customize',
+  'tecent' = 'cos'
+}


export enum DatasourceType { S3 = 's3', SFTP = 'sftp', Local = 'local' } export enum S3Platform { AWS = 'aws', OSS = 'oss', Tecent = 'cos', Customize = 'customize', }

huaxiabuluo · 2023-03-27T07:11:43Z

app/interfaces/datasource.ts

+export interface IDatasourceAdd {
+  name: string;
+  type: IDatasourceType;
+  platform?: string;
+  s3Config?: {
+    accessKey: string;
+    accessSecret?: string;
+    endpoint: string;
+    bucket: string;
+    region?: string;
+  };
+  sftpConfig?: {
+    host: string;
+    port: number;
+    username: string;
+    password: string;
+  };
+}
+export interface IDatasourceUpdate extends IDatasourceAdd {
+  id: number;
+}
+
+export interface IDatasourceItem {
+  id?: number;
+  name: string;
+  type: IDatasourceType;
+  platform?: string;
+  createTime?: string;
+  s3Config?: {
+    accessKey: string;
+    accessSecret?: string;
+    endpoint: string;
+    bucket: string;
+    region?: string;
+  };
+  sftpConfig?: {
+    host: string;
+    port: number;
+    username: string;
+    password: string;
+  };
+}


repetitive content between IDatasourceAdd and IDatasourceItem

huaxiabuluo · 2023-03-27T10:12:35Z

app/pages/Import/DatasourceList/DatasourceConfig/PlatformConfig/index.tsx

+  const submit = async (values: IDatasourceItem) => {
+    const _type = values.type || type;
+    setLoading(true);
+    if(mode === 'create') {
+      const flag = await addDataSource({
+        type: _type,
+        name: '',
+        ...values
+      });
+      setLoading(false);
+      flag && (message.success(intl.get('schema.createSuccess')), onConfirm());
+    } else {
+      let _config;
+      switch (_type) {
+        case IDatasourceType.s3:
+          _config = values.s3Config;
+          _config.accessSecret === tempPwd && delete _config.accessSecret;
+          break;
+        case IDatasourceType.sftp:
+          _config = values.sftpConfig;
+          _config.password === tempPwd && delete _config.password;
+          break;
+        default:
+          break;
+      }
+      const flag = await updateDataSource({
+        id: data.id,
+        type: _type,
+        name: '',
+        ...values
+      });
+      setLoading(false);
+      flag && (message.success(intl.get('common.updateSuccess')), onConfirm());
+    }
+  };


huaxiabuluo · 2023-03-27T10:13:48Z

app/pages/Import/DatasourceList/RemoteList/index.tsx

+const cloudKeys = [
+  {
+    title: 'ipAddress',
+    key: ['s3Config', 'endpoint']
+  },
+  {
+    title: 'bucketName',
+    key: ['s3Config', 'bucket']
+  },
+  {
+    title: 'accessKeyId',
+    key: ['s3Config', 'accessKey']
+  },
+  {
+    title: 'region',
+    key: ['s3Config', 'region']
+  },
+  {
+    title: 'createTime',
+    key: 'createTime',
+    render: data => data && dayjs(data).format('YYYY-MM-DD HH:mm:ss')
+  },
+];
+const sftpKeys = [{
+  title: 'ipAddress',
+  key: ['sftpConfig', 'host'],
+  render: (_, row) => `${row.sftpConfig.host}:${row.sftpConfig.port}`
+},
+{
+  title: 'account',
+  key: ['sftpConfig', 'username']
+},
+{
+  title: 'createTime',
+  key: 'createTime',
+  render: data => data && dayjs(data).format('YYYY-MM-DD HH:mm:ss')
+}
+];


const cloudColumns: TableColumnType<IDatasourceItem>[] = [ { title: <Translation>{(i) => i.get('import.ipAddress')}</Translation>, dataIndex: ['s3Config', 'endpoint'], }, { title: <Translation>{(i) => i.get('import.bucketName')}</Translation>, dataIndex: ['s3Config', 'bucket'], }, { title: <Translation>{(i) => i.get('import.accessKeyId')}</Translation>, dataIndex: ['s3Config', 'accessKey'], }, { title: <Translation>{(i) => i.get('import.region')}</Translation>, dataIndex: ['s3Config', 'region'], }, { title: <Translation>{(i) => i.get('import.createTime')}</Translation>, dataIndex: 'createTime', render: (data) => data && dayjs(data).format('YYYY-MM-DD HH:mm:ss'), }, ]; const sftpColumns: TableColumnType<IDatasourceItem>[] = [ { title: <Translation>{(i) => i.get('import.ipAddress')}</Translation>, dataIndex: ['sftpConfig', 'host'], render: (_, row) => `${row.sftpConfig.host}:${row.sftpConfig.port}`, }, { title: <Translation>{(i) => i.get('import.account')}</Translation>, dataIndex: ['sftpConfig', 'username'], }, { title: <Translation>{(i) => i.get('import.createTime')}</Translation>, dataIndex: 'createTime', render: (data) => data && dayjs(data).format('YYYY-MM-DD HH:mm:ss'), }, ];

huaxiabuluo · 2023-03-27T10:14:24Z

app/pages/Import/DatasourceList/RemoteList/index.tsx

+  const columns = columnKeys[type].map(item => ({
+    title: intl.get(`import.${item.title}`),
+    dataIndex: item.key,
+    render: item.render
+  })).concat(({
+    title: intl.get('common.operation'),
+    key: 'operation',
+    render: (_, item: IDatasourceItem) => (<div className={styles.operation}>
+      <Button className="primaryBtn" onClick={() => editItem(item)}>
+        <Icon type="icon-studio-btn-detail" />
+      </Button>
+      <Popconfirm
+        onConfirm={() => deleteItem(item.id)}
+        title={intl.get('common.ask')}
+        okText={intl.get('common.confirm')}
+        cancelText={intl.get('common.cancel')}
+      >
+        <Button className="warningBtn">
+          <Icon type="icon-studio-btn-delete" />
+        </Button>
+      </Popconfirm>
+    </div>)
+  } as any));


const tableColumns: TableColumnType<IDatasourceItem>[] = useMemo(() => { const columns = type === IDatasourceType.s3 ? cloudColumns : sftpColumns; return columns.concat({ title: intl.get('common.operation'), key: 'operation', render: (_, item) => ( <div className={styles.operation}> <Button className="primaryBtn" onClick={() => editItem(item)}> <Icon type="icon-studio-btn-detail" /> </Button> <Popconfirm onConfirm={() => deleteItem(item.id)} title={intl.get('common.ask')} okText={intl.get('common.confirm')} cancelText={intl.get('common.cancel')} > <Button className="warningBtn"> <Icon type="icon-studio-btn-delete" /> </Button> </Popconfirm> </div> ), }); }, [type, intl]);

huaxiabuluo · 2023-03-27T10:38:30Z

app/pages/Import/TaskCreate/SchemaConfig/FileMapping/FileSelect.tsx

+    const _path = path.slice(0, -1).split('/');
+    _path.pop();
+    const newPath = _path.join('/').length ? _path.join('/') + '/' : '';


/** * - `/a/b/c/?` => `/a/b/` * - `a/b/?` => `a` * - `a/?` => `` */ const parentPath = path.replace(/(\/|^)[^/]+\/?$/, '$1');

huaxiabuluo · 2023-03-27T10:43:40Z

app/pages/Import/TaskCreate/SchemaConfig/FileMapping/FileSelect.tsx

+  const handleConfirm = useCallback(async () => {
+    if(activeItem && activeId === IDatasourceType.local) {
+      // select local file
+      onConfirm(activeItem, state);
+    } else {
+      setState({ loading: true });
+      const _path = `${path === '/' ? '' : path}${activeItem.name}`;
+      const data = await previewFile({ id: activeId, path: _path });
+      const item = {
+        name: activeItem.name,
+        withHeader: false,
+        delimiter: ',',
+        sample: data.contents.join('\r\n'),
+        path: _path,
+        datasourceId: activeId === IDatasourceType.local ? null : activeId,
+      } as any;
+      setState({ loading: false });
+      onConfirm(item, state);
+    }
+  }, [activeItem, activeId, path]);


early return, unnecessary else

& fix some issues

datasource model

nianiaJR

LGTM

nianiaJR reviewed Mar 20, 2023

View reviewed changes

nianiaJR previously approved these changes Mar 26, 2023

View reviewed changes

huaxiabuluo reviewed Mar 27, 2023

View reviewed changes

hetao92 dismissed nianiaJR’s stale review via 5d01122 March 27, 2023 10:56

huaxiabuluo reviewed Mar 27, 2023

View reviewed changes

hetao92 added 15 commits March 29, 2023 08:09

feat: add datasource in import

f640d09

feat: add multiple datasources

37cccc1

mod: update page

eafd96b

feat: add importer config

972210e

feat: add load directory of datasource

c669d30

mod: update file select logic before import

81333b9

feat: support concat id & edit config

383c268

feat: get config from db before import

ddaf73c

& fix some issues

mod: add platform in config && modify

3df8030

datasource model

mod: adjust the region rules in the config rules

fa0933c

feat: support cos platform

a44521f

fix: fix style

4b5432f

mod: code review

7e3e03e

mod: code review

43b45ae

feat: add template validate tips

6d10007

hetao92 force-pushed the hetao-datasource branch from b77ed0a to 6d10007 Compare March 30, 2023 03:45

mod: code review

793b82e

mod: update i18n

7802bb2

nianiaJR approved these changes Mar 31, 2023

View reviewed changes

nianiaJR merged commit 63fcc5d into vesoft-inc:master Mar 31, 2023

wey-gu mentioned this pull request Apr 1, 2023

Weekly Report 2023-03-31 vesoft-inc/nebula-community#394

Closed

This was linked to issues Apr 3, 2023

support multiple types of data sources in the studio #82

Closed

support clear logs of data import #83

Closed

The advice about upload #188

Closed

hetao92 deleted the hetao-datasource branch April 3, 2023 06:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add datasource config in import module #509

feat: add datasource config in import module #509

hetao92 commented Mar 16, 2023 •

edited

Loading

nianiaJR Mar 20, 2023

nianiaJR Mar 20, 2023

nianiaJR Mar 20, 2023

hetao92 Mar 21, 2023

nianiaJR Mar 20, 2023

hetao92 Mar 21, 2023

nianiaJR Mar 26, 2023

nianiaJR Mar 20, 2023

nianiaJR Mar 20, 2023

nianiaJR Mar 20, 2023

nianiaJR Mar 20, 2023

nianiaJR Mar 20, 2023

nianiaJR Mar 20, 2023

hetao92 Mar 23, 2023

nianiaJR Mar 20, 2023

hetao92 Mar 21, 2023

nianiaJR Mar 26, 2023

nianiaJR left a comment

huaxiabuluo Mar 27, 2023

huaxiabuluo Mar 27, 2023

huaxiabuluo Mar 27, 2023 •

edited

Loading

huaxiabuluo Mar 27, 2023

huaxiabuluo Mar 27, 2023

huaxiabuluo Mar 27, 2023

huaxiabuluo Mar 27, 2023

nianiaJR left a comment

		return nil, ecode.WithBadRequest(err, "json stringify error")
		return nil, ecode.WithErrorMessage(ecode.ErrBadRequest, err, "json stringify error")

feat: add datasource config in import module #509

feat: add datasource config in import module #509

Conversation

hetao92 commented Mar 16, 2023 • edited Loading

What type of PR is this?

What problem(s) does this PR solve?

Issue(s) number:

Description:

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nianiaJR left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huaxiabuluo Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nianiaJR left a comment

Choose a reason for hiding this comment

hetao92 commented Mar 16, 2023 •

edited

Loading

huaxiabuluo Mar 27, 2023 •

edited

Loading