Skip to content

Commit

Permalink
Integrating PostgreSQL with YugaByte distributed transactions; adding…
Browse files Browse the repository at this point in the history
… PostgreSQL to yb-ctl and the release package

Summary:
Integrating PostgreSQL with YugaByte distributed transactions.
- PgTxnManager manages the TransactionManager, YBSession, and Transaction instances. There is at most one active transaction at a time per PostgreSQL backend process (i.e. per conneciton).
- PgTxnManager's member functions BeginTransaction() / CommitTransaction() / AbortTransaction() are invoked from PostgreSQL's BEGIN / COMMIT / ROLLBACK.

Adding PostgreSQL to yb-ctl and the release package.
- yb-ctl is our local cluster management script. Now, running `bin/yb-ctl create --enable_postgres` will start the PostgreSQL server in addition to yb-master and yb-tserver processes.
- Modifying yb_release.py and related Python libraries so that PostgreSQL is included as a top-level directory inside the YugaByte release package. On Linux the `bin/post_install.sh` script inside the release package will now also fix RPATHs and the interpreter path in PostgreSQL binaries.
- yb-ctl works with PostgreSQL from both the local directory and from the release package.

Development efficiency improvements:
- Adding a script called `dev_postgres.sh` to manage the local PostgreSQL server running outside of the YB cluster during development.
- An experimental mode of running our PostgreSQL tests against a PostgreSQL server running in a non-YugaByte mode, which should be identical in behavior to regular PostgreSQL. This can be enabled by setting the YB_USE_VANILLA_POSTGRES_IN_TEST environment variable to 1. This is useful to validate our tests.
- In case a PostgreSQL test gets stuck, we automatically cause core dumps of its backend processes and dump stack traces.

Bug fixes:
- Adding a missing thread pool shutdown in `TransactionManager` destructor.
- Fixing a bug in generating a combined `compile_commands.json` file in `build_postgres.py`.
- "Common daemon flags" in our Java test suite were actually only applied to tablet servers. Most importantly, the `--yb_num_shards_per_tserver` was not taking effect. This could improve test stability across the board due to using a lower number of tablets, especially in ASAN/TSAN.

Other changes:
- Extending the C wrapper DSL in pggate to support functions with no arguments. PgTxnManager is actually the first real use of this macro-based DSL framework.
- Adding a test utility to BaseYBTest that can run a given Runnable in a separate thread with a timeout. This is used for catching query timeouts in PostgreSQL tests.
- Unifying accessing environment variables and Java system properties in the EnvAndSysPropertyUtil utility class. E.g. we can specify that we only want to collect the list of tests by setting either the YB_COLLECT_TESTS_ONLY environment variable or the yb.collect.tests.only Java system property.
- During our automatic conversion of a YugaByte C++ Status to a YBCStatus structure, we used to allocate memory for the YBCStatus using the PostgreSQL memory context. However, sometimes we need to remember the YBCStatus object across memory context changes. Therefore, we now allocate it using malloc initially, and HandleYBStatus re-allocates it on the memory context.

Caveats:

- Currently, PostgreSQL's own locking is still enabled by default, because system tables are still stored in PostgreSQL's storage engine. We also put transaction support in PostgreSQL under a flag (`YB_PG_TRANSACTIONS_ENABLED`) for this reason. This flag is enabled in unit tests.
- There are still some intermittent failures in TestPgTransactions#testBasicTransaction and TestPgTransactions#testTransactionConflicts. SQL queries get stuck for more than 30 seconds between the PostgreSQL and tablet server layers. This will be fixed shortly after this diff lands.

Test Plan:
Jenkins

Testing yb-ctl (both from the local build directory and from the package distribution in /tmp/yb-dist):

```
bin/yb-ctl destroy; bin/yb-ctl create --enable_postgres
bin/psql -h localhost -p 5433 -U postgres

create table t (k int primary key, v int);

Inspect http://localhost:7000/tables to make sure the table got created.

In psql:

insert into t (k, v) values (10, 20);
select * from t;
```

Building the package locally:

```
cd ~/code/yugabyte
./yb_build.sh release
rm -rf /tmp/yb-dist
./yb_release --build_target /tmp/yb-dist --force --skip_build --verbose
```

Make sure we don't rely on build output still being there -- move it away temporarily before testing yb-ctl:
```
mv build/debug-gcc-dynamic-enterprise-ninja build/debug-gcc-dynamic-enterprise-ninja-moved-away
cd /tmp/yb-dist
```

Reviewers: neil, robert, sergei, mihnea

Reviewed By: mihnea

Subscribers: bharat, karthik, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D5518
  • Loading branch information
mbautin committed Oct 13, 2018
1 parent 1851ac9 commit 3f536a8
Show file tree
Hide file tree
Showing 73 changed files with 2,183 additions and 535 deletions.
2 changes: 2 additions & 0 deletions .arclint
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@
"(^src/yb/yql/pggate/if_macros_c_wrapper_decl[.]h$)",
"(^src/yb/yql/pggate/if_macros_c_wrapper_impl[.]h$)",
"(^src/yb/yql/pggate/if_macros_c_pg_wrapper_inl[.]h$)",
"(^src/yb/yql/pggate/pg_api_example_if[.]h$)",
"(^src/yb/yql/pggate/pg_txn_manager_if[.]h$)",
"(^src/yb/yql/pggate/pggate_if[.]h$)"
],
"include": [
Expand Down
128 changes: 128 additions & 0 deletions bin/dev_postgres.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
#!/usr/bin/env bash

set -euo pipefail

. "${BASH_SOURCE%/*}"/../build-support/common-test-env.sh

print_usage() {
cat <<-EOT
${0##*/}: manage a development PostgreSQL server (running outside of a YB tserver)
Usage: ${0##*/} [options] [<command>]
Options:
-h, --help
Print usage
--recreate
Destroy and re-create the local cluster and PostgreSQL data directory
Commands:
postgres - start the local YugaByte PostgreSQL server
psql - start psql and connect to the local YugaByte PostgreSQL server
EOT
}

set_common_postgres_env() {
export PGDATA=/tmp/ybpgdata

# Needed when running on our LDAP/NFS-based GCP devservers.
export YB_PG_FALLBACK_SYSTEM_USER_NAME=$USER

export YB_ENABLED_IN_POSTGRES=1

export FLAGS_pggate_master_addresses=127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100
YB_PG_PORT=54320

YB_PG_BIN_DIR=$YB_SRC_ROOT/build/latest/postgres/bin
}

print_usage_and_exit() {
print_usage
exit 1
}

parse_cmd_line_args() {
recreate=false
command=""
while [[ $# -gt 0 ]]; do
case "$1" in
-h|--help)
print_usage_and_exit ;;
--recreate)
recreate=true
;;
psql|postgres)
if [[ -n $command && $command != $1 ]]; then
fatal "Different commands specified: '$command' and '$1'"
fi
command=$1
;;
*)
log "Invalid option: $1"
exit 1
esac
shift
done

if [[ -z $command ]]; then
print_usage
log "No command specified"
exit 1
fi
}

kill_postgres_server() {
set +e
( set -x; pkill -f "postgres --port=$YB_PG_PORT" )
set -e
}

cleanup() {
kill_postgres_server
}

manage_postgres_server() {
kill_postgres_server
if "$recreate"; then
heading "Re-creating the local cluster and PostgreSQL directory"
rm -rf "$PGDATA"
"$YB_SRC_ROOT/bin/yb-ctl" destroy
"$YB_SRC_ROOT/bin/yb-ctl" create
else
"$YB_SRC_ROOT/bin/yb-ctl" start
fi

if "$recreate" || [[ ! -d $PGDATA ]]; then
heading "Running initdb"
$YB_PG_BIN_DIR/initdb -U postgres
fi

cd "$PGDATA"

set -m # enable job control
heading "Starting the PostgreSQL server"
trap cleanup EXIT
"$YB_PG_BIN_DIR/postgres" --port=$YB_PG_PORT &

heading "Bringing the PostgreSQL server back into foreground"
fg %1
}

start_psql() {
"$YB_PG_BIN_DIR/psql" --port=$YB_PG_PORT -h localhost -U postgres
}

# -------------------------------------------------------------------------------------------------
# Main script
# -------------------------------------------------------------------------------------------------

set_common_postgres_env
parse_cmd_line_args "$@"

case $command in
postgres)
manage_postgres_server
;;
psql)
start_psql
;;
*)
fatal "Unknown command: $command"
esac
6 changes: 6 additions & 0 deletions bin/repeat_unit_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,9 @@ EOT
delete_tmp_files() {
# Be extra careful before we nuke a directory.
if [[ -n ${TEST_TMPDIR:-} && $TEST_TMPDIR =~ ^/tmp/ ]]; then
set +e
rm -rf "$TEST_TMPDIR"
set -e
fi
}

Expand Down Expand Up @@ -318,7 +320,9 @@ $RANDOM.$RANDOM.$RANDOM.$$
pushd "$log_dir"
test_log_path="$YB_SUREFIRE_REPORTS_DIR.tar.gz"
tar czf "$test_log_path" "${YB_SUREFIRE_REPORTS_DIR##*/}"
set +e
rm -rf "$YB_SUREFIRE_REPORTS_DIR"
set -e
popd
else
if [[ $pass_or_fail == "PASSED" ]]; then
Expand All @@ -338,7 +342,9 @@ $RANDOM.$RANDOM.$RANDOM.$$
else
rm -f "$raw_test_log_path"
if "$is_java_test"; then
set +e
rm -rf "$YB_SUREFIRE_REPORTS_DIR"
set -e
fi
fi

Expand Down
Loading

0 comments on commit 3f536a8

Please sign in to comment.