Kubemq -hadoop target connector allows services using kubemq server to access hadoop service.
The following required to run the -hadoop target connector:
- kubemq cluster
- hadoop active server
- kubemq-targets deployment
hadoop target connector configuration properties:
Properties Key | Required | Description | Example |
---|---|---|---|
address | yes | hadoop address | "localhost:9000" |
user | no | hadoop user | "my_user" |
Example:
bindings:
- name: kubemq-query--hadoop
source:
kind: kubemq.query
name: kubemq-query
properties:
address: "kubemq-cluster:50000"
client_id: "kubemq-query--hadoop-connector"
auth_token: ""
channel: "query..hadoop"
group: ""
auto_reconnect: "true"
reconnect_interval_seconds: "1"
max_reconnects: "0"
target:
kind: storage.hadoop
name: hadoop
properties:
_key: "id"
_secret_key: 'json'
region: "region"
token: ""
downloader: "true"
uploader: "true"
Read File:
Metadata Key | Required | Description | Possible values |
---|---|---|---|
file_path | yes | path to file | "/test/foo2.txt" |
method | yes | type of method | "read_file" |
Example:
{
"metadata": {
"method": "read_file",
"file_path": "/test/foo2.txt"
},
"data": null
}
Write File:
Metadata Key | Required | Description | Possible values |
---|---|---|---|
file_path | yes | path to file | "/test/foo2.txt" |
method | yes | type of method | "write_file" |
file_mode | no | os permission mode default(0777) | "0777" |
data | yes | file as byte array | "TXkgZXhhbXBsZSBmaWxlIHRvIHVwbG9hZA==" |
Example:
{
"metadata": {
"method": "write_file",
"file_path": "/test/foo2.txt"
},
"data": "TXkgZXhhbXBsZSBmaWxlIHRvIHVwbG9hZA=="
}
Remove File:
Metadata Key | Required | Description | Possible values |
---|---|---|---|
file_path | yes | path to file | "/test/foo2.txt" |
method | yes | type of method | "remove_file" |
Example:
{
"metadata": {
"method": "remove_file",
"file_path": "/test/foo2.txt"
},
"data": null
}
Rename File:
Metadata Key | Required | Description | Possible values |
---|---|---|---|
file_path | yes | new path to file | "/test/foo3.txt" |
old_file_path | yes | new path to file | "/test/foo2.txt" |
method | yes | type of method | "rename_file" |
Example:
{
"metadata": {
"method": "rename_file",
"file_path": "/test/foo3.txt",
"old_file_path": "/test/foo2.txt"
},
"data": null
}
Make Dir :
Metadata Key | Required | Description | Possible values |
---|---|---|---|
file_path | yes | new path to file | "/test_folder" |
file_mode | no | os permission mode default(0777) | "0777" |
method | yes | type of method | "mkdir" |
Example:
{
"metadata": {
"method": "mkdir",
"file_path": "/test_folder"
},
"data": null
}
Stat :
Metadata Key | Required | Description | Possible values |
---|---|---|---|
file_path | yes | new path to file | "/test/foo3.txt" |
method | yes | type of method | "stat" |
Example:
{
"metadata": {
"method": "stat",
"file_path": "/test/foo2.txt"
},
"data": null
}