Skip to content

Commit

Permalink
Feature: Optimize local framework logging output
Browse files Browse the repository at this point in the history
  • Loading branch information
peggiezhu committed Oct 25, 2023
1 parent daa9de2 commit 56f2279
Show file tree
Hide file tree
Showing 25 changed files with 1,262 additions and 76 deletions.
1 change: 1 addition & 0 deletions .github/.codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ coverage:
status:
project:
default:
informational: true
threshold: 1%
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ English | [中文](README.zh_CN.md)
* [custom metrics plugin](./en/custom_metrics.md)
* [prometheus](./en/prometheus_metrics.md)
* Logging
* [local logging tools](./en/local_logging.md)
* [custom logging plugin](./en/custom_logging.md)
* [cls](https://github.com/trpc-ecosystem/cpp-logging-cls/blob/main/README.md)
* Tracing
Expand Down
1 change: 1 addition & 0 deletions docs/README.zh_CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
* [开发自定义metrics插件](./zh/custom_metrics.md)
* [prometheus](./zh/prometheus_metrics.md)
* Logging插件
* [本地日志工具](./zh/local_logging.md)
* [开发自定义logging插件](./zh/custom_logging.md)
* [cls](https://github.com/trpc-ecosystem/cpp-logging-cls/blob/main/README.zh_CN.md)
* Tracing插件
Expand Down
403 changes: 403 additions & 0 deletions docs/en/local_logging.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/en/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ You can use it:

Conventional framework performance test data is only performance data in relatively simple scenarios, and does not represent good performance in real and different business scenarios. Considering that Tencent has many different business scenarios, the requirements for framework performance are also different, for example:

- business accesses gateway scenarios: the feature is that the business logic is light, hign qps, large number of connections (long/short connections), heavy network io operations, and asynchronous programming is often used for programming;
- business accesses gateway scenarios: the feature is that the business logic is light, high qps, large number of connections (long/short connections), heavy network io operations, and asynchronous programming is often used for programming;
- recommendation/search scenarios: the feature is that the business logic is heavy, the qps is not large, each request needs to be calculated in parallel, pay attention to the long tail delay, and the programming often use synchronous programming;
- game business scenarios: the feature is that logic very complex and has stateful, large qps, single-threaded programming, and use synchronous programming;
- storage scenarios: the feature is that the business logic is light, large qps, low latency requirement, heavy network io and disk io operations, and asynchronous programming is often used for programming;
Expand Down
Binary file added docs/images/log_design.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
403 changes: 403 additions & 0 deletions docs/zh/local_logging.md

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions trpc/config/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -136,3 +136,13 @@ cc_test(
"@com_google_googletest//:gtest_main",
],
)

cc_test(
name = "config_test",
srcs = ["config_test.cc"],
deps = [
"//trpc/config/testing:mock_config",
"@com_google_googletest//:gtest",
"@com_google_googletest//:gtest_main",
],
)
10 changes: 5 additions & 5 deletions trpc/config/config.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
namespace trpc {

/// @brief The tRPC Config plugin abstract interface definition class, targeting plugin developers.
/// @note This class is reserved for 1.0 compatibility, and new data sources (rainbow, etcd, local cache)
/// @note This class is reserved for 1.0 compatibility, and new data sources
/// should inherit provider_base.h directly.
class Config : public Plugin {
public:
Expand All @@ -31,15 +31,15 @@ class Config : public Plugin {
~Config() override = default;

/// @brief Pulls file configuration.
/// @param config_name - For tconf, pass in the configuration file name; for rainbow, pass in the group_name.
/// @param config_name - Group_name.
/// @param config - The content of the obtained configuration.
/// @param params - For tconf, this parameter is not required; for rainbow, pass in the file name.
/// @param params - File name.
/// @return - true: Loading successful.
/// false: Loading failed.
virtual bool PullFileConfig(const std::string& config_name, std::string* config, const std::any& params) = 0;

/// @brief Pulls a single key-value configuration.
/// @param config_name - The filename of the KV configuration to be loaded, for rainbow pass in the group.
/// @param config_name - The filename of the KV configuration to be loaded, means group.
/// @param key - Input key.
/// @param config - Obtains the value corresponding to the key.
/// @return - true: Loading successful.
Expand All @@ -48,7 +48,7 @@ class Config : public Plugin {
const std::any& params) = 0;

/// @brief Pulls multiple key-value configurations.
/// @param config_name - The filename of the KV configuration to be loaded, for rainbow pass in the group.
/// @param config_name - The filename of the KV configuration to be loaded, means group.
/// @param config - Obtains key-value pairs.
/// @return - true: Loading successful.
/// false: Loading failed.
Expand Down
106 changes: 106 additions & 0 deletions trpc/config/config_test.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
//
//
// Tencent is pleased to support the open source community by making tRPC available.
//
// Copyright (C) 2023 THL A29 Limited, a Tencent company.
// All rights reserved.
//
// If you have downloaded a copy of the tRPC source code from Tencent,
// please note that tRPC source code is licensed under the Apache 2.0 License,
// A copy of the Apache 2.0 License is included in this file.
//
//

#include <any>
#include <map>
#include <string>

#include "gtest/gtest.h"
#include "gmock/gmock.h"
#include "trpc/config/testing/mock_config.h"

namespace trpc::testing {

class AnyEqMatcher : public ::testing::MatcherInterface<const std::any&> {
public:
explicit AnyEqMatcher(const std::any& expected) : expected_(expected) {}

bool MatchAndExplain(const std::any& actual, ::testing::MatchResultListener* listener) const override {
if (!actual.has_value() && !expected_.has_value()) {
return true;
}

if (actual.type() != expected_.type()) {
return false;
}

if (actual.type() == typeid(int)) {
return std::any_cast<int>(actual) == std::any_cast<int>(expected_);
} else if (actual.type() == typeid(std::string)) {
return std::any_cast<std::string>(actual) == std::any_cast<std::string>(expected_);
}

return false;
}

void DescribeTo(::std::ostream* os) const override {
*os << "is equal to the expected value";
}

void DescribeNegationTo(::std::ostream* os) const override {
*os << "is not equal to the expected value";
}

private:
const std::any expected_;
};

inline ::testing::Matcher<const std::any&> AnyEq(const std::any& expected) {
return ::testing::MakeMatcher(new AnyEqMatcher(expected));
}

class TestConfig : public ::testing::Test {
protected:
void SetUp() override {
mock_config_ = MakeRefCounted<MockConfig>();
ASSERT_TRUE(mock_config_ != nullptr);
}

RefPtr<MockConfig> mock_config_;
};

TEST_F(TestConfig, PullFileConfig) {
std::string config_name = "test_config";
std::string config;
std::any params;
EXPECT_CALL(*mock_config_, PullFileConfig(config_name, &config, AnyEq(params)))
.WillOnce(::testing::Return(true));
bool result = mock_config_->PullFileConfig(config_name, &config, params);

ASSERT_TRUE(result);
}

TEST_F(TestConfig, PullKVConfig) {
std::string config_name = "test_config";
std::string key = "test_key";
std::string config;
std::any params;
EXPECT_CALL(*mock_config_, PullKVConfig(config_name, key, &config, AnyEq(params)))
.WillOnce(::testing::Return(true));
bool result = mock_config_->PullKVConfig(config_name, key, &config, params);

ASSERT_TRUE(result);
}

TEST_F(TestConfig, PullMultiKVConfig) {
std::string config_name = "test_config";
std::map<std::string, std::string> config;
std::any params;
EXPECT_CALL(*mock_config_, PullMultiKVConfig(config_name, &config, AnyEq(params)))
.WillOnce(::testing::Return(true));
bool result = mock_config_->PullMultiKVConfig(config_name, &config, params);

ASSERT_TRUE(result);
}

} // namespace trpc::testing
8 changes: 4 additions & 4 deletions trpc/config/default/loader.cc
Original file line number Diff line number Diff line change
Expand Up @@ -78,18 +78,18 @@ ProviderPtr GetProvider(const std::string& instance_name) {
} // namespace detail

LoadOptions WithCodec(const std::string& name) {
return LoadOptions([name = std::move(name)](DefaultConfigPtr& config) {
return LoadOptions([name](DefaultConfigPtr& config) {

Check warning on line 81 in trpc/config/default/loader.cc

View check run for this annotation

Codecov / codecov/patch

trpc/config/default/loader.cc#L81

Added line #L81 was not covered by tests
config::CodecPtr codec = CodecFactory::GetInstance()->Get(name);
TRPC_ASSERT(codec != nullptr && "Codec not found!");
(*config).SetCodec(codec);
config->SetCodec(codec);
});
}

LoadOptions WithProvider(const std::string& name) {
return LoadOptions([name = std::move(name)](DefaultConfigPtr& config) {
return LoadOptions([name](DefaultConfigPtr& config) {

Check warning on line 89 in trpc/config/default/loader.cc

View check run for this annotation

Codecov / codecov/patch

trpc/config/default/loader.cc#L89

Added line #L89 was not covered by tests
config::ProviderPtr provider = ProviderFactory::GetInstance()->Get(name);
TRPC_ASSERT(provider != nullptr && "Provider not found!");
(*config).SetProvider(provider);
config->SetProvider(provider);
});
}

Expand Down
40 changes: 27 additions & 13 deletions trpc/config/default/loader_test.cc
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
//
//
// Tencent is pleased to support the open source community by making tRPC available.
//
// Copyright (C) 2023 THL A29 Limited, a Tencent company.
// All rights reserved.
//
// If you have downloaded a copy of the tRPC source code from Tencent,
// please note that tRPC source code is licensed under the Apache 2.0 License,
// A copy of the Apache 2.0 License is included in this file.
//
//

#include "trpc/config/default/loader.h"

#include "gtest/gtest.h"
Expand Down Expand Up @@ -56,22 +69,23 @@ TEST_F(TestLoader, LoadSuccessful) {
TEST_F(TestLoader, LoadUnsuccessful) {
std::string invalid_configuration_path = "path/to/your/invalid/configuration/file";

EXPECT_DEATH(config::DefaultConfigPtr cfg = config::detail::Load(
invalid_configuration_path,
{config::WithCodec("UnknownCodecPlugin"), config::WithProvider("UnknownProviderPlugin")}),
"assertion failed: codec != nullptr && \"Codec not found!\"");
// Both codec and provider are not registered
EXPECT_DEATH(config::detail::Load(invalid_configuration_path,
{config::WithCodec("UnknownCodecPlugin"),
config::WithProvider("UnknownProviderPlugin")}),
"Codec not found!");

// codec is correct, but provider is not registered
EXPECT_DEATH(config::DefaultConfigPtr cfg = config::detail::Load(
invalid_configuration_path,
{config::WithCodec("TestCodecPlugin"), config::WithProvider("UnknownProviderPlugin")}),
"assertion failed: provider != nullptr && \"Provider not found!\"");
EXPECT_DEATH(config::detail::Load(invalid_configuration_path,
{config::WithCodec("TestCodecPlugin"),
config::WithProvider("UnknownProviderPlugin")}),
"Provider not found!");

// codec is correct, but provider is not registered
EXPECT_DEATH(config::DefaultConfigPtr cfg = config::detail::Load(
invalid_configuration_path,
{config::WithCodec("UnknownCodecPlugin"), config::WithProvider("TestProviderPlugin")}),
"assertion failed: codec != nullptr && \"Codec not found!\"");
// codec is not registered, but provider is correct
EXPECT_DEATH(config::detail::Load(invalid_configuration_path,
{config::WithCodec("UnknownCodecPlugin"),
config::WithProvider("TestProviderPlugin")}),
"Codec not found!");
}

} // namespace trpc::testing
8 changes: 8 additions & 0 deletions trpc/config/testing/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,14 @@ exports_files([
"test_load.toml",
])

cc_library(
name = "mock_config",
hdrs = ["mock_config.h"],
deps = [
"//trpc/config:config",
],
)

cc_library(
name = "config_plugin_testing",
hdrs = ["config_plugin_testing.h"],
Expand Down
46 changes: 46 additions & 0 deletions trpc/config/testing/mock_config.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
//
//
// Tencent is pleased to support the open source community by making tRPC available.
//
// Copyright (C) 2023 THL A29 Limited, a Tencent company.
// All rights reserved.
//
// If you have downloaded a copy of the tRPC source code from Tencent,
// please note that tRPC source code is licensed under the Apache 2.0 License,
// A copy of the Apache 2.0 License is included in this file.
//
//

#pragma once

#include "gmock/gmock.h"
#include "trpc/config/config.h"

namespace trpc::testing {

class MockConfig : public trpc::Config {
public:
MOCK_METHOD(bool, PullFileConfig,
(const std::string& config_name, std::string* config, const std::any& params),
(override));

MOCK_METHOD(bool, PullKVConfig,
(const std::string& config_name, const std::string& key, std::string* config,
const std::any& params),
(override));

MOCK_METHOD(bool, PullMultiKVConfig,
(const std::string&, (std::map<std::string, std::string>*), const std::any&), (override));

MOCK_METHOD(void, SetAsyncCallBack, (const std::any& param), (override));

Check warning on line 35 in trpc/config/testing/mock_config.h

View check run for this annotation

Codecov / codecov/patch

trpc/config/testing/mock_config.h#L35

Added line #L35 was not covered by tests

PluginType Type() const override {
return PluginType::kConfig;

Check warning on line 38 in trpc/config/testing/mock_config.h

View check run for this annotation

Codecov / codecov/patch

trpc/config/testing/mock_config.h#L37-L38

Added lines #L37 - L38 were not covered by tests
}

std::string Name() const override {
return "MockConfig";

Check warning on line 42 in trpc/config/testing/mock_config.h

View check run for this annotation

Codecov / codecov/patch

trpc/config/testing/mock_config.h#L41-L42

Added lines #L41 - L42 were not covered by tests
}
};

} // namespace trpc::testing
4 changes: 2 additions & 2 deletions trpc/config/trpc_conf.h
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ void Destroy();

/// Example usage:
/// @code{.cpp}
/// auto db_cfg = trpc::config::Load("test.yaml", trpc::config::WithCodec("yaml"),
/// trpc::config::WithProvider("rainbow1")); std::string username = db_cfg.GetString("username", "trpc-db1");
/// auto db_cfg = trpc::config::Load("test.yaml", trpc::config::WithCodec("yaml");
/// trpc::config::WithProvider("my_provider1")); std::string username = db_cfg.GetString("username", "trpc-db1");
/// std::string password = db_cfg.GetString("password", "123456");
/// @endcode
template <typename... Args>
Expand Down
4 changes: 2 additions & 2 deletions trpc/config/trpc_conf_compatible.h
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ namespace trpc::config {
/// This method is provided for compatibility with the legacy framework configuration loading method,
/// but is not recommended for use.
/// Usage: first get the configuration content, then use the specific interface to parse it:
/// For example, auto yaml_result = trpc::config::Loadconfig<YAML::Node>(rainbow, myconf.conf);
/// For example, auto yaml_result = trpc::config::LoadConfig<YAML::Node>(my_provider, myconf.conf);
/// @brief Fetches the configuration from the configuration center.
/// @param plugin_name (old)The name of the configuration plugin, like rainbow.
/// @param plugin_name (old)The name of the configuration plugin.
/// @param config_name (old)The name of the configuration file to be loaded.
/// @param params Any type of expandable parameters.
/// @return Returns the configuration in the format of the template type T, such as Json::Value or std::map.
Expand Down
6 changes: 3 additions & 3 deletions trpc/config/trpc_conf_deprecated.h
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class TrpcConf {
public:
/// @brief Loads the text configuration of a file type
/// @param plugin_name The name of the configuration plugin
/// @param config_name The name of the configuration file, "group" for rainbow and the configuration name for tconf
/// @param config_name The Group name
/// @param result The loaded text configuration content
/// @param params The configuration request information
/// @return bool true: if the configuration is loaded successfully
Expand All @@ -54,7 +54,7 @@ class TrpcConf {

/// @brief Loads all key-value pairs of a kv type configuration
/// @param plugin_name The name of the configuration plugin
/// @param config_name The name of the kv configuration file, "group" for rainbow and the configuration name for tconf
/// @param config_name The Group name
/// @param result The loaded key-value pairs content
/// @param params The configuration request information
/// @return bool true: if the configuration is loaded successfully
Expand All @@ -79,7 +79,7 @@ class TrpcConf {

/// @brief Loads a single key-value pair of a kv type configuration
/// @param plugin_name The name of the configuration plugin
/// @param config_name The name of the kv configuration file, "group" for rainbow and the configuration name for tconf
/// @param config_name The Group name
/// @param key The name of the key to be loaded
/// @param result The loaded value of the specified key
/// @param params The configuration request information
Expand Down
Loading

0 comments on commit 56f2279

Please sign in to comment.