Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Integrating PostgreSQL with YugaByte distributed transactions; adding…
… PostgreSQL to yb-ctl and the release package Summary: Integrating PostgreSQL with YugaByte distributed transactions. - PgTxnManager manages the TransactionManager, YBSession, and Transaction instances. There is at most one active transaction at a time per PostgreSQL backend process (i.e. per conneciton). - PgTxnManager's member functions BeginTransaction() / CommitTransaction() / AbortTransaction() are invoked from PostgreSQL's BEGIN / COMMIT / ROLLBACK. Adding PostgreSQL to yb-ctl and the release package. - yb-ctl is our local cluster management script. Now, running `bin/yb-ctl create --enable_postgres` will start the PostgreSQL server in addition to yb-master and yb-tserver processes. - Modifying yb_release.py and related Python libraries so that PostgreSQL is included as a top-level directory inside the YugaByte release package. On Linux the `bin/post_install.sh` script inside the release package will now also fix RPATHs and the interpreter path in PostgreSQL binaries. - yb-ctl works with PostgreSQL from both the local directory and from the release package. Development efficiency improvements: - Adding a script called `dev_postgres.sh` to manage the local PostgreSQL server running outside of the YB cluster during development. - An experimental mode of running our PostgreSQL tests against a PostgreSQL server running in a non-YugaByte mode, which should be identical in behavior to regular PostgreSQL. This can be enabled by setting the YB_USE_VANILLA_POSTGRES_IN_TEST environment variable to 1. This is useful to validate our tests. - In case a PostgreSQL test gets stuck, we automatically cause core dumps of its backend processes and dump stack traces. Bug fixes: - Adding a missing thread pool shutdown in `TransactionManager` destructor. - Fixing a bug in generating a combined `compile_commands.json` file in `build_postgres.py`. - "Common daemon flags" in our Java test suite were actually only applied to tablet servers. Most importantly, the `--yb_num_shards_per_tserver` was not taking effect. This could improve test stability across the board due to using a lower number of tablets, especially in ASAN/TSAN. Other changes: - Extending the C wrapper DSL in pggate to support functions with no arguments. PgTxnManager is actually the first real use of this macro-based DSL framework. - Adding a test utility to BaseYBTest that can run a given Runnable in a separate thread with a timeout. This is used for catching query timeouts in PostgreSQL tests. - Unifying accessing environment variables and Java system properties in the EnvAndSysPropertyUtil utility class. E.g. we can specify that we only want to collect the list of tests by setting either the YB_COLLECT_TESTS_ONLY environment variable or the yb.collect.tests.only Java system property. - During our automatic conversion of a YugaByte C++ Status to a YBCStatus structure, we used to allocate memory for the YBCStatus using the PostgreSQL memory context. However, sometimes we need to remember the YBCStatus object across memory context changes. Therefore, we now allocate it using malloc initially, and HandleYBStatus re-allocates it on the memory context. Caveats: - Currently, PostgreSQL's own locking is still enabled by default, because system tables are still stored in PostgreSQL's storage engine. We also put transaction support in PostgreSQL under a flag (`YB_PG_TRANSACTIONS_ENABLED`) for this reason. This flag is enabled in unit tests. - There are still some intermittent failures in TestPgTransactions#testBasicTransaction and TestPgTransactions#testTransactionConflicts. SQL queries get stuck for more than 30 seconds between the PostgreSQL and tablet server layers. This will be fixed shortly after this diff lands. Test Plan: Jenkins Testing yb-ctl (both from the local build directory and from the package distribution in /tmp/yb-dist): ``` bin/yb-ctl destroy; bin/yb-ctl create --enable_postgres bin/psql -h localhost -p 5433 -U postgres create table t (k int primary key, v int); Inspect http://localhost:7000/tables to make sure the table got created. In psql: insert into t (k, v) values (10, 20); select * from t; ``` Building the package locally: ``` cd ~/code/yugabyte ./yb_build.sh release rm -rf /tmp/yb-dist ./yb_release --build_target /tmp/yb-dist --force --skip_build --verbose ``` Make sure we don't rely on build output still being there -- move it away temporarily before testing yb-ctl: ``` mv build/debug-gcc-dynamic-enterprise-ninja build/debug-gcc-dynamic-enterprise-ninja-moved-away cd /tmp/yb-dist ``` Reviewers: neil, robert, sergei, mihnea Reviewed By: mihnea Subscribers: bharat, karthik, yql Differential Revision: https://phabricator.dev.yugabyte.com/D5518
- Loading branch information
