Merge branch 'master' of https://github.com/Azure/azure-sdk-for-python …

…into http_request_json

* 'master' of https://github.com/Azure/azure-sdk-for-python: (147 commits)
  [text analytics] add perf tests (Azure#17060)
  Add cloud event to core (Azure#16800)
  [Perf] Small fixes to storage-blob (Azure#17055)
  [EG] Regenerate Code (Azure#17053)
  Scrub batch shared keys (Azure#17030)
  [Tables] Add SAS to tables (Azure#16717)
  T2 containerservice 2021 03 03 (Azure#17050)
  Addressing issues with CredScan (Azure#16944)
  Communication chat preview4 (Azure#16905) (Azure#17037)
  remove first query section (Azure#17033)
  [formrecognizer] temp disable sample tests until service bug fixed (Azure#17036)
  [device update] allow device update pylint failures (Azure#17034)
  fix build (Azure#17029)
  update artifact names for ALL packages to align with the actual package name
  Create azure-iot-nspkg (Azure#17026)
  [Communication]: SMS 1:N Messages, Custom Tags, and Idempotence (Azure#16836)
  Fixing credentials to use AAD (Azure#16885)
  T2 deviceupdate 2021 03 02 (Azure#17016)
  T2 cosmosdb 2021 02 23 (Azure#16875)
  T2 datadog 2021 03 02 (Azure#17004)
  ...

.github/CODEOWNERS

@@ -45,7 +45,7 @@
 /sdk/loganalytics/                                                   @alexeldeib
 
 # PRLabel: %Monitor - Exporter
-/sdk/monitor/azure-opentelemetry-exporter-azuremonitor               @rakshith91 @lmazuel @hectorhdzg @lzchen
+/sdk/monitor/azure-monitor-opentelemetry-exporter                    @rakshith91 @lmazuel @lzchen
 
 # PRLabel: %Consumption
 /sdk/consumption/                                                    @sandeepnl

build_package.py

@@ -10,8 +10,8 @@
 import os
 import glob
 import sys
-from subprocess import check_call
 
+from subprocess import check_call
 
 DEFAULT_DEST_FOLDER = "./dist"
 

ci_template.yml

@@ -5,6 +5,7 @@ trigger:
   branches:
     include:
     - master
+    - main
     - hotfix/*
     - release/*
     - restapi*
@@ -16,6 +17,7 @@ pr:
   branches:
     include:
     - master
+    - main
     - feature/*
     - hotfix/*
     - release/*

doc/dev/tests-advanced.md

@@ -1,10 +1,80 @@
 # Setup Python Development Environment - Advanced
 In this document we will provide additional information about the test environments:
 
+- [Test Mixin Classes](#test-mixin-classes)
 - [Resource Preparers](#preparers)
 - [Examples with Preparers](#examples-with-preparers)
 - [mgmt_settings_real.py](#mgmt_settings_real-file)
 
+## Test Mixin Classes
+Many of our test suites use a mixin class to reduce re-writing code in multiple test files. For example, in the Tables test suite there is a `_shared` directory containing two of these mixin classes, a [sync one](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/tables/azure-data-tables/tests/_shared/testcase.py) and an [async version](https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/tables/azure-data-tables/tests/_shared/asynctestcase.py). These classes will often have ways to create connection strings from an account name and key, formulate the account url, configure logging, or validate service responses. In order for these mixin classes to be used by both the functional and unit tests they should inherit from `object`. For example:
+
+```python
+
+class TablesTestMixin(object):
+    def connection_string(self, account, key):
+        return "DefaultEndpointsProtocol=https;AccountName=" + account + ";AccountKey=" + str(key) + ";EndpointSuffix=core.windows.net"
+
+    def account_url(self, account, endpoint_type):
+        """Return an url of storage account.
+        :param str storage_account: Storage account name
+        :param str storage_type: The Storage type part of the URL. Should be "table", or "cosmos", etc.
+        """
+        try:
+            if endpoint_type == "table":
+                return account.primary_endpoints.table.rstrip("/")
+            if endpoint_type == "cosmos":
+                return "https://{}.table.cosmos.azure.com".format(account.name)
+            else:
+                raise ValueError("Unknown storage type {}".format(storage_type))
+        except AttributeError: # Didn't find "primary_endpoints"
+            if endpoint_type == "table":
+                return 'https://{}.{}.core.windows.net'.format(account, endpoint_type)
+            if endpoint_type == "cosmos":
+                return "https://{}.table.cosmos.azure.com".format(account)
+
+    def enable_logging(self):
+        handler = logging.StreamHandler()
+        handler.setFormatter(logging.Formatter(LOGGING_FORMAT))
+        self.logger.handlers = [handler]
+        self.logger.setLevel(logging.INFO)
+        self.logger.propagate = True
+        self.logger.disabled = False
+```
+
+In action this class can be used in functional tests:
+
+```python
+class TestTablesFunctional(AzureTestCase, TablesTestMixin):
+    ...
+    def test_with_mixin(self, account, key):
+        conn_str = self.connection_string(account, key)
+        client = TableClient.from_connection_string(conn_str)
+        client.create_table('first')
+        client.create_table('second')
+        tables = 0
+        for table in client.list_tables():
+            tables += 1
+
+        assert tables == 2
+```
+
+Or can be used in a unit test:
+```python
+class TestTablesUnit(TablesTestMixin):
+    ...
+    def test_valid_url(self):
+        account = "fake_tables_account"
+        credential = "fake_tables_account_key_0123456789"
+
+        url = self.account_url(account, "tables")
+        client = TableClient(account_url=url, credential=credential)
+
+        assert client is not None
+        assert client.account_url == "https://{}.tables.core.windows.net/".format(account)
+```
+
+
 ## Preparers
 The Azure SDK team has created some in house tools to help with easier testing. These additional tools are located in the `devtools_testutils` package that was installed with your `dev_requirements.txt`. In this package are the preparers that will be commonly used throughout the repository to test various resources. A preparer is a way to programmatically create fresh resources to run our tests against and then deleting them after running a test suite. These help guarantee standardized behavior by starting each test group from a fresh resource and account.
 

doc/dev/tests.md

@@ -113,6 +113,7 @@ azure-sdk-for-python\sdk\my-directory\my-library> tox -c ../../../eng/tox/tox.in
 azure-sdk-for-python\sdk\my-directory\my-library> tox -c ../../../eng/tox/tox.ini -e whl
 azure-sdk-for-python\sdk\my-directory\my-library> tox -c ../../../eng/tox/tox.ini -e sdist
 azure-sdk-for-python\sdk\my_directory\my_library> tox -c ../../../eng/tox/tox.ini -e samples
+azure-sdk-for-python\sdk\my_directory\my_library> tox -c ../../../eng/tox/tox.ini -e apistub
 ```
 A quick description of the five commands above:
 * sphinx: documentation generation using the inline comments written in our code
@@ -121,9 +122,10 @@ A quick description of the five commands above:
 * whl: creates a whl package for installing our package
 * sdist: creates a zipped distribution of our files that the end user could install with pip
 * samples: runs all of the samples in the `samples` directory and verifies they are working correctly
+* apistub: runs the [apistubgenerator](https://github.com/Azure/azure-sdk-tools/tree/master/packages/python-packages/api-stub-generator) tool on your code
 
 ## `devtools_testutils` Package
-The Azure SDK team has created some in house tools to help with easier testing. These additional tools are located in the `devtools_testutils` package that was installed with your `dev_requirements.txt`. In this package is the `AzureTestCase` object which every test case object should inherit from. This management object takes care of creating and scrubbing recordings to make sure secrets are not added to the recordings files (and subsequently to the git history) and authenticating clients for test methods.
+The Azure SDK team has created some in house tools to help with easier testing. These additional tools are located in the `devtools_testutils` package that was installed with your `dev_requirements.txt`. In this package is the [`AzureTestCase`](https://github.com/Azure/azure-sdk-for-python/blob/master/tools/azure-sdk-tools/devtools_testutils/azure_testcase.py#L99-L350) object which every test case object should inherit from. This management object takes care of creating and scrubbing recordings to make sure secrets are not added to the recordings files (and subsequently to the git history) and authenticating clients for test methods.
 
 ## Writing New Tests
 SDK tests are based on the `scenario_tests` subpackage located in [`azure-sdk-for-python/tools/azure-devtools/src/azure_devtools`](https://pypi.org/project/azure-devtools/). `scenario_tests` is a general, mostly abstracted framework which provides several useful features for writing SDK tests, ie:
@@ -264,32 +266,64 @@ From your terminal run the `pytest` command to run all the tests that you have w
 
 Your update should run smooth and have green dots representing passing tests. Now if you look at the contents of your `tests` directory there should be a new directory called `recording` with four `.yaml` files. Each `yaml` file is a recording for a single test. To run a test in playback mode change the `testsettings_local.cfg` to `live-mode: false` and rerun the tests with the same command. The test infrastructure will use the automatically created `.yaml` recordings to mock the HTTP traffic and run the tests.
 
+## Functional vs. Unit Tests
 
-## More Test Examples
+The test written above is a functional test, it generates HTTP traffic and sends data to the service. Most of our clients have some client-side validation for account names, formatting, or properties that do not generate HTTP traffic. For unit tests, the best practice is to have a separate test class from the `AzureTestCase` class which tests client side validation methods. For example, the `azure-data-tables` library has client-side validation for the table name and properties of the entity, below is an example of how these could be tested:
 
-This section will demonstrate how to write tests with the `devtools_testutils` package with a few samples to showcase the features of the test framework.
+```python
+import pytest
+from azure.data.tables import TableServiceClient, EntityProperty, EdmType
+
+class TestTablesUnitTest(object):
+
+    def test_invalid_table_name(self):
+        account_name = 'fake_account_name'
+        account_key = 'fake_account_key1234567890'
+        tsc = TableServiceClient(
+            account_url='https://{}.table.core.windows.net/'.format(account_name),
+            credential=account_key
+        )
+
+        invalid_table_name = "bad_table_name" # table name cannot have an '_' character
+
+        with pytest.raises(ValueError):
+            tsc.create_table(invalid_table_name)
+
+    def test_entity_properties(self):
+        ep = EntityProperty('abc', EdmType.STRING)
+        ep = EntityProperty(b'abc', EdmType.BINARY)
+        ep = EntityProperty(1.2345, EdmType.DOUBLE)
 
-### Example 1: Basic Recording and Interactions
+        with pytest.raises(ValueError):
+            ep = EntityProperty(2 ** 75, EdmType.Int64) # Tables can only handle integers up to 2 ^ 63
+```
 
+Async tests need to be marked with a `@pytest.mark.asyncio` to be properly handled. For example:
 ```python
-import os
 import pytest
+from azure.data.tables.aio import TableServiceClient
 
-from azure.data.tables import TableServiceClient
-from devtools_testutils import AzureTestCase
+class TestTablesUnitTest(object):
 
-class ExampleStorageTestCase(AzureTestCase):
+    @pytest.mark.asyncio
+    async def test_invalid_table_name(self):
+        account_name = 'fake_account_name'
+        account_key = 'fake_account_key1234567890'
+        tsc = TableServiceClient(
+            account_url='https://{}.table.core.windows.net/'.format(account_name),
+            credential=account_key
+        )
 
-    def test_create_table(self):
-        credential = self.get_credential(TableServiceClient)
-        client = self.create_client_from_credential(TableServiceClient, credential, account_url=self.account_url)
+        invalid_table_name = "bad_table_name" # table name cannot have an '_' character
 
         with pytest.raises(ValueError):
-            table_name = "an_invalid_table"
-            created = client.create_table(table_name)
+            await tsc.create_table(invalid_table_name)
 ```
 
-This simple tests that the client verifies a valid table name before sending the HTTP request (Azure Data Tables can only be alphanumeric values). This test will have no recording associated with it.
+
+## More Test Examples
+
+This section will demonstrate how to write tests with the `devtools_testutils` package with a few samples to showcase the features of the test framework.
 
 For more information, refer to the [advanced tests notes][advanced_tests_notes] on more advanced scenarios and additional information.
 

doc/eng_sys_checks.md

@@ -48,7 +48,7 @@ Analyze job in both nightly CI and pull request validation pipeline runs a set o
 Change log verification is added to ensure package has valid change log for current version. Guidelines to properly maintain the change log is documented [here](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/)
 
 ## PR Validation Checks
-Each pull request runs various tests using `pytest` in addition to all the tests mentioned above in analyze check. Pull request validation performs 3 different types of test: `whl, sdist and depends`. Following section explains the purpose of each of these tests and how to execute them locally. All pull requests are validated on multiple python versions across different platforms and below is the test matrix for pull request.
+Each pull request runs various tests using `pytest` in addition to all the tests mentioned above in analyze check. Pull request validation performs 3 different types of test: `whl, sdist and depends`. The following section explains the purpose of each of these tests and how to execute them locally. All pull requests are validated on multiple python versions across different platforms. Find the test matrix below.
 
 
 |`Python  Version`|`Platform`  |
@@ -73,8 +73,8 @@ This test installs sdist of the package being tested and runs all tests cases in
 2. Run following command
    ``tox -e sdist -c ../../../eng/tox/tox.ini``
 
-####depends
-This test is to ensure all modules in the package being tested can be successfully imported. This is to ensure all package requirement is properly set in setup.py as well as to ensure modules are imported using valid namespace. This test install the package and it's required packages and executes `from <package-root-namespace> import *`. For e.g. `from azure.core import *`.
+#### depends
+The `depends` check ensures all modules in a target package can be successfully imported. Actually installing and importing will verify that all package requirements are properly set in setup.py and that the `__all__` set for the package is properly defined. This test installs the package and its required packages, then executes `from <package-root-namespace> import *`. For example from `azure-core`, the following would be invoked:  `from azure.core import *`.
 
 Following is the command to run this test environment locally.