Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update: aws ec2 deployment guide #18360

Merged
merged 1 commit into from
Oct 25, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
132 changes: 45 additions & 87 deletions docs/deploying-airbyte/on-aws-ec2.md
Original file line number Diff line number Diff line change
@@ -1,140 +1,98 @@
# On AWS (EC2)
# Deploy on AWS (Amazon EC2)

:::info
This page guides you through deploying Airbyte Open Source on an Amazon EC2 instance by setting up the deployment environment, installing and starting Airbyte, and connecting it to the Amazon EC2 instance.

The instructions have been tested on `Amazon Linux 2 AMI (HVM)`
> INFO
>
> The instructions have been tested on Amazon Linux 2 AMI (HVM)

:::
## Requirements

## Create a new instance
- To test Airbyte, we recommend a `t2.medium` instance
- To deploy Airbyte in a production environment, we recommend a `t2.large` instance
- Make sure your Docker Desktop app is running to ensure all services run smoothly

- Launch a new instance
[Create and download an SSH key to connect to the instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html)

![](../.gitbook/assets/aws_ec2_launch.png)
## Set up the environment

- Select instance AMI
1. Create an Amazon EC2 instance.

![](../.gitbook/assets/aws_ec2_ami.png)
To connect to your instance, run the following command on your local terminal:

- Select instance type
- For testing out Airbyte, a `t2.medium` instance is likely sufficient. Airbyte uses a lot of disk space with images and logs, so make sure to provision at least 30GBs of disk per node.
- For long-running Airbyte installations, we recommend a `t2.large` instance.

![](../.gitbook/assets/aws_ec2_instance_type.png)

- `Next: Configure Instance Details`
- You can tune parameters or keep the defaults
- `Next: Add Storage`
- You can tune parameters or keep the defaults
- `Next: Add Tags`
- You can tune parameters or keep the defaults
- `Next: Configure Security Groups`
- We are going to allow network for `ssh`

![](../.gitbook/assets/aws_ec2_security_group.png)

- `Review and Launch`
- `Launch`
- Create a ssh key so you can connect to the instance
- Download the key \(and don't lose it or you won't be able to connect to the instance\)

![](../.gitbook/assets/aws_ec2_ssh_key.png)

- `Launch Instances`

![](../.gitbook/assets/aws_ec2_instance_view.png)

- Wait for the instance to become `Running`

## Install environment

:::info

Note: The following commands will be entered either on your local terminal or in your ssh session on the instance terminal. The comments above each command block will indicate where to enter the commands.

:::

- Connect to your instance

```bash
# In your workstation terminal
SSH_KEY=~/Downloads/airbyte-key.pem # or wherever you've downloaded the key
INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP
```
SSH_KEY=~/Downloads/dataline-key-airbyte.pem # the file path you downloaded the key
INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP # you can find your IP address in the EC2 console under the Instances tab
chmod 400 $SSH_KEY # or ssh will complain that the key has the wrong permissions
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP # connect to the aws ec2 instance AMI and the your private IP address
```

- Install `docker`
2. To install Docker, run the following command in your SSH session on the instance terminal:

```bash
# In your ssh session on the instance terminal
```
sudo yum update -y
sudo yum install -y docker
sudo service docker start
sudo usermod -a -G docker $USER
```

- Install `docker-compose`
3. To install `docker-compose`, run the following command in your ssh session on the instance terminal:

```bash
# In your ssh session on the instance terminal
```
sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version
```

- Close the ssh connection to ensure the group modification is taken into account
4. To close the SSH connection, run the following command in your SSH session on the instance terminal:

```bash
# In your ssh session on the instance terminal
```
logout
```

## Install & start Airbyte
## Install and start Airbyte

- Connect to your instance
In your local terminal, run the following commands:

```bash
# In your workstation terminal
1. Connect to your instance

```
ssh -i $SSH_KEY ec2-user@$INSTANCE_IP
```

- Install Airbyte
2. Install Airbyte

```bash
# In your ssh session on the instance terminal
```
mkdir airbyte && cd airbyte
wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
docker-compose up -d
docker-compose up -d # run the Docker container
```

## Connect to Airbyte

:::danger

For security reasons, we strongly recommend to not expose Airbyte on Internet available ports. Future versions will add support for SSL & Authentication.

:::
> DANGER
>
> We strongly recommend not exposing Airbyte on Internet available ports for security reasons.

- Create ssh tunnel for port 8000
1. Create an SSH tunnel for port 8000

:::info
> INFO
>
> If you want to use different ports, modify `API_URL` in your .env file and restart Airbyte.
> Run the following commands in your workstation terminal from the downloaded key folder:

If you want to use different ports or change the HTTP basic auth username and password, you will need to modify `API_URL` in your `.env` file and restart Airbyte.

:::

```bash
```
# In your workstation terminal
SSH_KEY=~/Downloads/dataline-key-airbyte.pem
ssh -i $SSH_KEY -L 8000:localhost:8000 -N -f ec2-user@$INSTANCE_IP
```

- Just visit [http://localhost:8000](http://localhost:8000) in your browser and start moving some data!
2. Visit `http://localhost:8000` to verify the deployment.

## Pushing Airbyte logs to CloudWatch
## Get Airbyte logs in CloudWatch

If you want to get your logs from your Airbyte Docker containers in CloudWatch, simply follow [this](https://aws.amazon.com/pt/premiumsupport/knowledge-center/cloudwatch-docker-container-logs-proxy/) guide to do so.
Follow this [guide](https://aws.amazon.com/pt/premiumsupport/knowledge-center/cloudwatch-docker-container-logs-proxy/) to get your logs from your Airbyte Docker containers in CloudWatch.

## Troubleshooting

If you encounter any issues, just connect to our [Slack](https://slack.airbyte.io). Our community will help! We also have a [FAQ](../troubleshooting/on-deploying.md) section in our docs for common problems.
If you encounter any issues, reach out to our community on [Slack](https://slack.airbyte.com/).