Skip to content

Commit

Permalink
refactor: organize tests by endpoint and provide endpoint-specific ma…
Browse files Browse the repository at this point in the history
…ke targets (#566)

This PR introduces to changes:

Groups the tests by integration vs unit tests first, with namespaces; then by service (cache, topics, auth); then by control vs data. This is to ease filtering by the Makefile targets.
Adds Makefile targets for each of the endpoints (cache, control, token, storage). Because we test Windows-only builds as well, we make each of these OS-conditional.
A caller wishing to run all the tests can run make test. On Windows this will run against both .NET Framework 4.62 (which is Windows only) as well as .NET 6.0. On Linux or Mac OS this will only run against .NET 6.0.

Similarly, a caller wishing to run only the control endpoint tests can run make test-control-endpoint (or test-cache-endpoint, test-token-endpoint, test-storage-endpoint for other endpoints).
  • Loading branch information
malandis authored Aug 14, 2024
1 parent cf6ce8d commit ff8a58a
Show file tree
Hide file tree
Showing 21 changed files with 145 additions and 51 deletions.
103 changes: 94 additions & 9 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,28 +1,60 @@
.PHONY: all build build-dotnet6 build-dotnet-framework clean clean-build precommit restore test test-dotnet6 test-dotnet-framework run-examples help
# Note on the structure of this Makefile:
# - We build the project on both .NET 6.0 and .NET Framework 4.62. The latter of which is only available on Windows.
# - We still test on Linux. That means we can't run the .NET Framework tests on Linux, but we need to run both .NET 6.0 and .NET Framework tests on Windows.
# - Because of this, we must conditionally run certain build and test targets based on the operating system.
# - We split the build and test targets are split into two categories: .NET 6.0 and .NET Framework.
# - At the top we detect the operating system and set the appropriate build and test targets.
# - On Windows `make build` (test) runs both .NET 6.0 and .NET Framework build (test) targets.
# - On other operating systems `make build` (test) we only runs the .NET 6.0 build (test) targets.
# - We also have a GRPC_WEB flag that can be set to true to enable gRPC-Web support.
# - The caller can run `make GRPC_WEB=true build` to enable gRPC-Web support.
# - We additionally group the integration tests by endpoint (cache, control, token).
# - This is to allow for more granular testing by endpoint.
# - Similar to `build` and `test` targets, we have `test-cache-endpoint`, `test-control-endpoint`, `test-token-endpoint`, and `test-storage-endpoint` targets
# that are conditionally run based on the operating system.

.PHONY: all build build-dotnet6 build-dotnet-framework clean clean-build precommit restore test \
test-dotnet6 test-dotnet6-integration test-dotnet6-cache-endpoint test-dotnet6-control-endpoint test-dotnet6-token-endpoint \
test-dotnet-framework test-dotnet-framework-integration test-dotnet-framework-cache-endpoint test-dotnet-framework-control-endpoint test-dotnet-framework-token-endpoint \
test-control-endpoint test-cache-endpoint test-token-endpoint test-storage-endpoint \
run-examples help

# Determine the operating system
OS := $(shell uname)

# Set the default .NET version to .NET 6.0
DOTNET_VERSION := net6.0
DOTNET_FRAMEWORK_VERSION := net462
TEST_LOGGER_OPTIONS := --logger "console;verbosity=detailed"

# Windows-specific settings
# This tests if "NT" is in the OS string, which would indicate Windows.
ifneq (,$(findstring NT,$(OS)))
BUILD_TARGETS := build-dotnet6 build-dotnet-framework
TEST_TARGETS := test-dotnet6 test-dotnet-framework
BUILD_TARGETS := build-dotnet6 build-dotnet-framework
TEST_TARGETS := test-dotnet6 test-dotnet-framework
TEST_TARGETS_CACHE_ENDPOINT := test-dotnet6-cache-endpoint test-dotnet-framework-cache-endpoint
TEST_TARGETS_CONTROL_ENDPOINT := test-dotnet6-control-endpoint test-dotnet-framework-control-endpoint
TEST_TARGETS_TOKEN_ENDPOINT := test-dotnet6-token-endpoint test-dotnet-framework-token-endpoint
else
BUILD_TARGETS := build-dotnet6
TEST_TARGETS := test-dotnet6
BUILD_TARGETS := build-dotnet6
TEST_TARGETS := test-dotnet6
TEST_TARGETS_CACHE_ENDPOINT := test-dotnet6-cache-endpoint
TEST_TARGETS_CONTROL_ENDPOINT := test-dotnet6-control-endpoint
TEST_TARGETS_TOKEN_ENDPOINT := test-dotnet6-token-endpoint
endif

# Enable gRPC-Web if requested
GRPC_WEB_FLAG :=
ifeq ($(GRPC_WEB), true)
GRPC_WEB_FLAG := -p:DefineConstants=USE_GRPC_WEB
GRPC_WEB_FLAG := -p:DefineConstants=USE_GRPC_WEB
endif

# Various test filters
CACHE_ENDPOINT_TESTS_FILTER := "FullyQualifiedName~Momento.Sdk.Tests.Integration.Cache.Data|FullyQualifiedName~Momento.Sdk.Tests.Integration.Topics.Data"
CONTROL_ENDPOINT_TESTS_FILTER := "FullyQualifiedName~Momento.Sdk.Tests.Integration.Cache.Control"
TOKEN_ENDPOINT_TESTS_FILTER := "FullyQualifiedName~Momento.Sdk.Tests.Integration.Auth"


## Generate sync unit tests, format, lint, and test
all: precommit

Expand Down Expand Up @@ -68,14 +100,67 @@ test: ${TEST_TARGETS}

## Run unit and integration tests on the .NET 6.0 runtime
test-dotnet6:
@echo "Running tests on .NET 6.0..."
@echo "Running unit and integration tests on the .NET 6.0 runtime..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_VERSION}


## Run integration tests on the .NET 6.0 runtime against the cache endpoint
test-dotnet6-cache-endpoint:
@echo "Running integration tests on the .NET 6.0 runtime against the cache endpoint..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_VERSION} --filter ${CACHE_ENDPOINT_TESTS_FILTER}


## Run integration tests on the .NET 6.0 runtime against the control endpoint
test-dotnet6-control-endpoint:
@echo "Running integration tests on the .NET 6.0 runtime against the control endpoint..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_VERSION} --filter ${CONTROL_ENDPOINT_TESTS_FILTER}


## Run integration tests on the .NET 6.0 runtime against the token endpoint
test-dotnet6-token-endpoint:
@echo "Running integration tests on the .NET 6.0 runtime against the token endpoint..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_VERSION} --filter ${TOKEN_ENDPOINT_TESTS_FILTER}


## Run unit and integration tests on the .NET Framework runtime (Windows only)
test-dotnet-framework:
@echo "Running tests on .NET Framework 4.62 (Windows only)..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f net462
@echo "Running unit and integration tests on the .NET Framework runtime..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_FRAMEWORK_VERSION}


## Run integration tests on the .NET Framework runtime against the cache endpoint (Windows only)
test-dotnet-framework-cache-endpoint:
@echo "Running integration tests on the .NET Framework runtime against the cache endpoint..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_FRAMEWORK_VERSION} --filter ${CACHE_ENDPOINT_TESTS_FILTER}


## Run integration tests on the .NET Framework runtime against the control endpoint (Windows only)
test-dotnet-framework-control-endpoint:
@echo "Running integration tests on the .NET Framework runtime against the control endpoint..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_FRAMEWORK_VERSION} --filter ${CONTROL_ENDPOINT_TESTS_FILTER}


## Run integration tests on the .NET Framework runtime against the token endpoint (Windows only)
test-dotnet-framework-token-endpoint:
@echo "Running integration tests on the .NET Framework runtime against the token endpoint..."
@dotnet test ${TEST_LOGGER_OPTIONS} -f ${DOTNET_FRAMEWORK_VERSION} --filter ${TOKEN_ENDPOINT_TESTS_FILTER}


## Run cache endpoint tests
test-cache-endpoint: ${TEST_TARGETS_CACHE_ENDPOINT}


## Run control endpoint tests
test-control-endpoint: ${TEST_TARGETS_CONTROL_ENDPOINT}


## Run token endpoint tests
test-token-endpoint: ${TEST_TARGETS_TOKEN_ENDPOINT}


## Run storage endpoint tests
test-storage-endpoint:
@echo "Storage tests are not yet implemented."


## Run example applications and snippets
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
using Momento.Sdk.Auth.AccessControl;
using Momento.Sdk.Config;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Auth;

[Collection("AuthClient")]
public class AuthClientCacheTest : IClassFixture<CacheClientFixture>, IClassFixture<AuthClientFixture>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
using Momento.Sdk.Config;
using System.Collections.Generic;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Auth;

[Collection("AuthClient")]
public class AuthClientTopicTest : IClassFixture<CacheClientFixture>, IClassFixture<AuthClientFixture>, IClassFixture<TopicClientFixture>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
using System.Threading.Tasks;
using Momento.Sdk.Auth;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Cache.Control;

[Collection("CacheClient")]
public class CacheControlTest : TestBase
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
using Momento.Sdk.Config;
using Momento.Sdk.Config.Transport;

namespace Momento.Sdk.Tests.Integration;
namespace Momento.Sdk.Tests.Integration.Cache.Data;

public class CacheEagerConnectionTest
{
Expand Down Expand Up @@ -38,7 +38,7 @@ public void CacheClientConstructor_WithChannelsAndMaxConn_Success()
config = config.WithTransportStrategy(config.TransportStrategy
.WithGrpcConfig(grpcConfiguration)
.WithMaxConcurrentRequests(2));

// just validating that we can construct the client wh
var client = new CacheClient(config, authProvider, defaultTtl);
// still 2; clients shouldn't know we are doing 2/10 magic internally
Expand All @@ -52,7 +52,7 @@ public async void CacheClientCreate_EagerConnection_BadEndpoint()
var config = Configurations.Laptop.Latest(loggerFactory);
var authProviderWithBadCacheEndpoint = authProvider.WithCacheEndpoint("cache.cell-external-beta-1.prod.a.momentohq.com:65000");
Console.WriteLine($"Hello developer! We are about to run a test that verifies that the cache client is still operational even if our eager connection (ping) fails. So you will see the test log a warning message about that. It's expected, don't worry!");

await Assert.ThrowsAsync<ConnectionException>(async () => await CacheClient.CreateAsync(config, authProviderWithBadCacheEndpoint, defaultTtl, TimeSpan.FromSeconds(2)));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
using System.Threading.Tasks;
using Momento.Sdk.Internal.ExtensionMethods;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Cache.Data;

[Collection("CacheClient")]
public class CacheDataTest : TestBase
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
using Momento.Sdk.Internal.ExtensionMethods;
using Momento.Sdk.Requests;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Cache.Data;

[Collection("CacheClient")]
public class DictionaryTest : TestBase
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
using Momento.Sdk.Responses;
using Momento.Sdk.Tests;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Cache.Data;

[Collection("CacheClient")]
public class ListTest : TestBase
Expand Down Expand Up @@ -140,7 +140,7 @@ public async Task ListFetchAsync_WithNullEndIndex_HappyPath()
public async Task ListRetainAsync_InvalidIndices_AreError()
{
var listName = Utils.NewGuidString();

// the positive startIndex is larger than the positive endIndex
CacheListRetainResponse fetchResponse = await client.ListRetainAsync(cacheName, listName, 3, 1);
Assert.True(fetchResponse is CacheListRetainResponse.Error, $"Unexpected response: {fetchResponse}");
Expand All @@ -156,7 +156,7 @@ public async Task ListRetainAsync_InvalidIndices_AreError()
Assert.True(fetchResponse is CacheListRetainResponse.Error, $"Unexpected response: {fetchResponse}");
Assert.Equal(MomentoErrorCode.INVALID_ARGUMENT_ERROR, ((CacheListRetainResponse.Error)fetchResponse).ErrorCode);
}

[Fact]
public async Task ListRetainAsync_HappyPath()
{
Expand All @@ -182,24 +182,24 @@ public async Task ListRetainAsync_HappyPath()
Assert.True(fetchResponse is CacheListFetchResponse.Hit, $"Unexpected response: {fetchResponse}");
var hitResponse = (CacheListFetchResponse.Hit)fetchResponse;
Assert.Equal(new string[] { value2, value3, value4 }, hitResponse.ValueListString);

await resetList();
retainResponse = await client.ListRetainAsync(cacheName, listName, 2, -1);
Assert.True(retainResponse is CacheListRetainResponse.Success, $"Unexpected response: {retainResponse}");
fetchResponse = await client.ListFetchAsync(cacheName, listName);
Assert.True(fetchResponse is CacheListFetchResponse.Hit, $"Unexpected response: {fetchResponse}");
hitResponse = (CacheListFetchResponse.Hit)fetchResponse;
Assert.Equal(new string[] { value3, value4, value5 }, hitResponse.ValueListString);

await resetList();
retainResponse = await client.ListRetainAsync(cacheName, listName, -4, -1);
Assert.True(retainResponse is CacheListRetainResponse.Success, $"Unexpected response: {retainResponse}");
fetchResponse = await client.ListFetchAsync(cacheName, listName);
Assert.True(fetchResponse is CacheListFetchResponse.Hit, $"Unexpected response: {fetchResponse}");
hitResponse = (CacheListFetchResponse.Hit)fetchResponse;
Assert.Equal(new string[] { value3, value4, value5 }, hitResponse.ValueListString);
await resetList();

await resetList();
// valid case for a negative startIndex and null endIndex
retainResponse = await client.ListRetainAsync(cacheName, listName, -3, null);
Assert.True(retainResponse is CacheListRetainResponse.Success, $"Unexpected response: {retainResponse}");
Expand All @@ -216,7 +216,7 @@ public async Task ListRetainAsync_HappyPath()
Assert.True(fetchResponse is CacheListFetchResponse.Hit, $"Unexpected response: {fetchResponse}");
hitResponse = (CacheListFetchResponse.Hit)fetchResponse;
Assert.Equal(new string[] { value3, value4, value5, value6 }, hitResponse.ValueListString);

await resetList();
// valid case for null startIndex and positive endIndex
retainResponse = await client.ListRetainAsync(cacheName, listName, null, 1);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
using Momento.Sdk.Tests;
using Xunit.Abstractions;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Cache.Data;

[Collection("CacheClient")]
public class SetTest : TestBase
Expand Down Expand Up @@ -577,7 +577,7 @@ public async Task SetFetchAsync_UsesCachedStringSet_HappyPath()
var set2 = hitResponse.ValueSetString;
Assert.Same(set1, set2);
}

[Theory]
[InlineData(null, "my-set", 100)]
[InlineData("cache", null, 100)]
Expand All @@ -588,7 +588,7 @@ public async Task SetSampleAsync_NullChecks_IsError(string cacheName, string set
Assert.True(response is CacheSetSampleResponse.Error, $"Unexpected response: {response}");
Assert.Equal(MomentoErrorCode.INVALID_ARGUMENT_ERROR, ((CacheSetSampleResponse.Error)response).ErrorCode);
}

[Fact]
public async Task SetSampleAsync_Missing_HappyPath()
{
Expand All @@ -609,17 +609,17 @@ public async Task SetSampleAsync_UsesCachedStringSet_HappyPath()
Assert.True(allElementsResponse is CacheSetSampleResponse.Hit, $"Unexpected response: {allElementsResponse}");
var allElementsHitValues = ((CacheSetSampleResponse.Hit)allElementsResponse).ValueSetString;
Assert.True(allValues.SetEquals(allElementsHitValues), $"Expected sample with with limit matching set size to return the entire set; expected ({String.Join(", ", allValues)}), got ({String.Join(", ", allElementsHitValues)})");

CacheSetSampleResponse limitGreaterThanSetSizeResponse = await client.SetSampleAsync(cacheName, setName, 1000);
Assert.True(limitGreaterThanSetSizeResponse is CacheSetSampleResponse.Hit, $"Unexpected response: {limitGreaterThanSetSizeResponse}");
var limitGreaterThanSetSizeHitValues = ((CacheSetSampleResponse.Hit)limitGreaterThanSetSizeResponse).ValueSetString;
Assert.True(allValues.SetEquals(limitGreaterThanSetSizeHitValues), $"Expected sample with with limit greater than set size to return the entire set; expected ({String.Join(", ", allValues)}), got ({String.Join(", ", limitGreaterThanSetSizeHitValues)})");

CacheSetSampleResponse limitZeroResponse = await client.SetSampleAsync(cacheName, setName, 0);
var emptySet = new HashSet<String>();
var limitZeroHitValues = ((CacheSetSampleResponse.Hit)limitZeroResponse).ValueSetString;
Assert.True(emptySet.SetEquals(limitZeroHitValues), $"Expected sample with with limit zero to return empty set; got ({limitZeroHitValues})");

for (int i = 0; i < 10; i++)
{
CacheSetSampleResponse response = await client.SetSampleAsync(cacheName, setName, allValues.Count - 2);
Expand All @@ -630,7 +630,7 @@ public async Task SetSampleAsync_UsesCachedStringSet_HappyPath()
$"Expected hit values ({String.Join(", ", hitValues)}) to be subset of all values ({String.Join(", ", allValues)}), but it is not!");
}
}


[Fact]
public async Task CacheSetFetchResponse_ToString_HappyPath()
Expand Down Expand Up @@ -675,7 +675,7 @@ public async Task SetDeleteAsync_SetExists_HappyPath()
fetchResponse = await client.SetFetchAsync(cacheName, setName);
Assert.True(fetchResponse is CacheSetFetchResponse.Miss, $"Unexpected response: {fetchResponse}");
}

[Fact]
public async Task SetLengthAsync_SetIsMissing()
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
using System.Threading.Tasks;
using Momento.Sdk.Internal.ExtensionMethods;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration.Cache.Data;

[Collection("CacheClient")]
public class TtlTest : TestBase
Expand Down
2 changes: 1 addition & 1 deletion tests/Integration/Momento.Sdk.Tests/Fixtures.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
using Momento.Sdk.Auth;
using Momento.Sdk.Config;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration;

/// <summary>
/// A cache client fixture.
Expand Down
2 changes: 1 addition & 1 deletion tests/Integration/Momento.Sdk.Tests/TestBase.cs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
using Momento.Sdk.Auth;
using Momento.Sdk.Tests;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration;

public class TestBase
{
Expand Down
7 changes: 4 additions & 3 deletions tests/Integration/Momento.Sdk.Tests/TestCacheClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
using Momento.Sdk.Config;
using Momento.Sdk.Requests;

namespace Momento.Sdk.Tests;
namespace Momento.Sdk.Tests.Integration;

/// <summary>
/// Client call wrapper to make test expectations easier to write.
Expand All @@ -26,7 +26,8 @@ public TestCacheClient(IConfiguration config, ICredentialProvider authProvider,
/// Momento in the same datacenter. From your laptop you'll pretty much always pay 10
/// milliseconds to lightspeed tax, at least in 2023.
/// </summary>
private async Task Quiesce() {
private async Task Quiesce()
{
await Task.Delay(TimeSpan.FromMilliseconds(10));
}

Expand Down Expand Up @@ -316,7 +317,7 @@ public Task<CacheSetFetchResponse> SetFetchAsync(string cacheName, string setNam
{
return ((ICacheClient)client).SetFetchAsync(cacheName, setName);
}

public Task<CacheSetSampleResponse> SetSampleAsync(string cacheName, string setName, int limit)
{
return ((ICacheClient)client).SetSampleAsync(cacheName, setName, limit);
Expand Down
Loading

0 comments on commit ff8a58a

Please sign in to comment.