From 769ea7985602655a15ee296ebedf2eccd02a4f1a Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Tue, 10 Aug 2021 12:30:22 -0600 Subject: [PATCH 01/12] blueprint for layered profile --- blueprints/layered-profile.md | 274 ++++++++++++++++++++++++++++++++++ 1 file changed, 274 insertions(+) create mode 100644 blueprints/layered-profile.md diff --git a/blueprints/layered-profile.md b/blueprints/layered-profile.md new file mode 100644 index 0000000000..fd27d84326 --- /dev/null +++ b/blueprints/layered-profile.md @@ -0,0 +1,274 @@ + +# Layered Profile + +## Problem Description + +Profiles are unwieldy and dangerous for Operations. + +Currently, we have countless Profiles, in broad categories. For example, "EDGE_123_FOO_ABC_ATS_714-RC0-42" might represent servers which are +1. Edges +2. Amiga 123 machines +3. In the Foo CDN +4. In the ABC datacenter +5. Running ATS 7.14 RC0 +6. Who knows what 42 means? + +Suppose we have Amiga 456 machines, at DEF datacenters, and a Bar CDN, and some servers are running ATS 7.15, 8.0, and 8.1. We make profiles for each server we have fitting all those categories: + +EDGE_123_FOO_ABC_ATS_714-RC0-27, EDGE_456_FOO_ABC_ATS_714-RC0-27, EDGE_123_BAR_ABC_ATS_714-RC0-42, EDGE_456_FOO_DEF_ATS_714-RC1-27, EDGE_123_FOO_DEF_ATS_800-RC4-29 + +- The number of Profiles quickly becomes a combinatorial explosion +- It's nearly impossible for Ops engineers to figure out what Parameters are assigned to all Profiles of a given CDN, machine, or datacenter. +- What about that one random Parameter on a Cloned profile that was changed a year ago? Is it still in place? Should it be? Did it need to be applied to all new Profiles cloned from this one? + +## Proposed Change + +Layered Profiles allow assigning multiple Profiles, in order, to both Delivery Services and Servers. If multiple Profiles have a Parameter with the same Name and Config File, the Parameter from the last Profile in the ordering is applied. + +Layered Profiles is exactly as powerful as the existing Profiles, it doesn't enable any new things. It makes profiles much easier to manage. + +With Layered Profiles, hundreds of Profiles become a few dozen, each representing a logical group. For example, a server might have the Profiles, in order: +1. EDGE +2. AMIGA_123 +3. CDN_FOO +4. DATACENTER_ABC +5. ATS_714 +6. Custom_Hack_42 + +### Traffic Portal Impact + +1. A UI to view all parameters currently applied to a delivery service/server, as well as the profile each parameter came from. A new page, linked from DS and Server pages. For eg: + +| Name |Config File | Value | Profile | +|--------------------------------------------------|--------------------------|--------------------------------------|-----------| +| location | url_sig_myds.config | /opt/trafficserver/etc/trafficserver | EDGE | +| Drive_Prefix | storage.config | /dev/sd | AMIGA_123 | +| error_url | url_sig_myotherds.config | 403 | EDGE | +| CONFIG proxy.config.exec_thread.autoconfig.scale | records.config | FLOAT 1.5 | ATS_714 | + +2. A UI change to add, remove, and reorder (sortable list) profiles for both Delivery Services and Servers, on the existing DS and Server pages. + +### Traffic Ops Impact + +- Traffic Ops will need to add the logic to the below-mentioned existing API endpoints, in order to show/create/update/delete a sorted list of profiles for delivery services and server. +- `/deliveryservices/ GET, POST` +- `/deliveryservices/{id} PUT, DELETE` +- `/server/ GET, POST` +- `/servers/{id} PUT, DELETE` + +#### REST API Impact + +- No new endpoints are required + +**Existing Endpoints** + +- Modify JSON request and response for existing delivery services and servers endpoints. +- JSON **response** with the proposed change will look as follows: + +`/deliveryservices?id=100` +```JSON +{ + "id": 100, + ⋮ + "requested": { + "profileIds": [ + 1234, + 5678, + 9012 + ], + "profileNames": [ + "EDGE", + "ATS8", + "TOP" + ] + ⋮ + } +} +``` + +`/servers?id=5` +```JSON +{ + "id": 5, + ⋮ + "requested": { + "profileIds": [ + 1357, + 2468, + 2356 + ], + "profileNames": [ + "MID", + "TOP", + "PRIMARY" + ], + ⋮ + } +} +``` + +JSON **request** with the proposed change will look as follows: + +`/deliveryservices` +```JSON +{ + "requested": { + "profileIds": [ + 1234, + 5678, + 9012 + ], + "profileNames": [ + "EDGE", + "ATS8", + "TOP" + ], + ⋮ + }, + ⋮ +} +``` + +`/servers` +```JSON +{ + "requested": { + "profileIds": [ + 1357, + 2468, + 2356 + ], + "profileNames": [ + "MID", + "TOP", + "PRIMARY" + ], + ⋮ + }, + ⋮ +} +``` + +The following table describes the top level `layered_profile` object for delivery services: + +| field | type | optionality | description | +| ------------------ | --------------| ----------- | ---------------------------------------------------------------| +| deliveryservice | bigint | required | the delivery service id associated with a given profile | +| profile | bigint | required | the profile id associated with a delivery service | +| order | bigint | required | the order in which a profile is applied to a delivery service | +| lastUpdated | bigint | required | the last time this delivery service was updated | + +The following table describes the top level `layered_profile` object for servers: + +| field | type | optionality | description | +| ----------- | ---------------------| ----------- | ---------------------------------------------------------------| +| server | bigint | required | the server id associated with a given profile | +| profile | bigint | required | the profile id associated with a server | +| order | bigint | required | the order in which a profile is applied to a server | +| lastUpdated | bigint | required | the last time this server was updated | + +**API constraints** +- TO REST APIs will not be backward compatibility and this feature will be a major version (v5.0) change. +- Add `Profiles` key to API endpoints for Server and DeliveryService objects (and ProfileNames, ProfileIDs) + +#### Client Impact +- Existing Go client methods will be updated for the `/deliveryservices` and `/servers` endpoints in order to write TO API tests for the exising endpoints. +- All functions which get Parameters on Delivery Services or Servers must be changed to get the Parameters of all assigned Profiles in order. + +#### Data Model / Database Impact + +- A new Database Table `server_profiles` as described below, will be created: +- A new Database Table `deliveryservice_profiles` as described below, will be created: +```text + Table "traffic_ops.deliveryservice_profiles" + Column | Type | Collation | Nullable | Default +----------------+--------------------------+-----------+----------+-------- +deliveryservice | bigint | | not null | +profile | bigint | | not null | +order | bigint | | not null | +last_updated | timestamp with time zone | | not null | now() +Indexes: +"pk_deliveryservice_profile" PRIMARY KEY(profile) +Foreign-key constraints: +"fk_deliveryservice" FOREIGN KEY (deliveryservice) REFERENCES deliveryservice(id) +``` + +```text + Table "traffic_ops.server_profiles" + Column | Type | Collation | Nullable | Default +---------------+--------------------------+-----------+----------+-------- + server | bigint | | not null | + profile | bigint | | not null | + order | bigint | | not null | + last_updated | timestamp with time zone | | not null | now() +Indexes: + "pk_server_profile" PRIMARY KEY(profile) +Foreign-key constraints: + "fk_server" FOREIGN KEY (server) REFERENCES server(id) +``` + +### ORT Impact +New set of profiles will be created and the profile-parameter relationship will change. + +### Traffic Monitor Impact +No impact + +### Traffic Router Impact +No impact + +### Traffic Stats Impact +No impact + +### Traffic Vault Impact +No impact + +### Documentation Impact +All existing endpoints will need to be updated, along with the documentation explaining `layered_profile`. + +### Testing Impact +Client/API integration tests should be updated to verify the `layered_profile` functionality on existing API `/deliveryservices`, `/servers` endpoints. + +### Performance Impact +We do not anticipate any performance impact with layered_profile approach. + +### Security Impact +We do not anticipate any impact on security. + +### Upgrade Impact +- A Database Migration to: + - drop profile column in existing deliveryservice and server table + - insert existing server and DS profiles along with their order into the new tables(deliveryservice_profiles, server_profiles) +- The new capability can just be added to the `seeds.sql` file. + +### Operations Impact +The profile-parameter relationship will change based on new sets of profile and Operations will have to learn on how to assign profile order to a delivery service and/or server. + +### Developer Impact +Developers will most likely need to use layered_profile, so they'll need to be familiar with the process +of adding, sorting, deleting, debugging and working with layered_profiles. + +## Alternatives +None, except to keep using existing Profiles. + +## Dependencies +None + +## References +None From 03d6a957c43ba44de6cbe73a762f9e8651d10aa8 Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Tue, 10 Aug 2021 12:40:58 -0600 Subject: [PATCH 02/12] capitalization --- blueprints/layered-profile.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/blueprints/layered-profile.md b/blueprints/layered-profile.md index fd27d84326..3724536498 100644 --- a/blueprints/layered-profile.md +++ b/blueprints/layered-profile.md @@ -40,11 +40,11 @@ EDGE_123_FOO_ABC_ATS_714-RC0-27, EDGE_456_FOO_ABC_ATS_714-RC0-27, EDGE_123_BAR_A ## Proposed Change -Layered Profiles allow assigning multiple Profiles, in order, to both Delivery Services and Servers. If multiple Profiles have a Parameter with the same Name and Config File, the Parameter from the last Profile in the ordering is applied. +Layered Profiles allow assigning multiple profiles, in order, to both Delivery Services and Servers. If multiple profiles have a parameter with the same name and config File, the parameter from the last profile in the ordering is applied. -Layered Profiles is exactly as powerful as the existing Profiles, it doesn't enable any new things. It makes profiles much easier to manage. +Layered Profiles is exactly as powerful as the existing profiles, it doesn't enable any new things. It makes profiles much easier to manage. -With Layered Profiles, hundreds of Profiles become a few dozen, each representing a logical group. For example, a server might have the Profiles, in order: +With Layered Profiles, hundreds of profiles become a few dozen, each representing a logical group. For example, a server might have the profiles, in order: 1. EDGE 2. AMIGA_123 3. CDN_FOO From 672007c7bedba83f672b179077baaafa139bc646 Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Thu, 18 Nov 2021 17:03:42 -0700 Subject: [PATCH 03/12] updated blueprint for layered profile to include proposed changed for only servers. Removed all proposed changed for delivery service. --- blueprints/layered-profile.md | 135 ++++++++++------------------------ 1 file changed, 38 insertions(+), 97 deletions(-) diff --git a/blueprints/layered-profile.md b/blueprints/layered-profile.md index 3724536498..e40ccdc655 100644 --- a/blueprints/layered-profile.md +++ b/blueprints/layered-profile.md @@ -40,7 +40,7 @@ EDGE_123_FOO_ABC_ATS_714-RC0-27, EDGE_456_FOO_ABC_ATS_714-RC0-27, EDGE_123_BAR_A ## Proposed Change -Layered Profiles allow assigning multiple profiles, in order, to both Delivery Services and Servers. If multiple profiles have a parameter with the same name and config File, the parameter from the last profile in the ordering is applied. +Layered Profiles allow assigning multiple profiles to Servers. If multiple profiles have a parameter with the same name and config file, the parameter from the last profile in the ordering is applied. Layered Profiles is exactly as powerful as the existing profiles, it doesn't enable any new things. It makes profiles much easier to manage. @@ -54,7 +54,7 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin ### Traffic Portal Impact -1. A UI to view all parameters currently applied to a delivery service/server, as well as the profile each parameter came from. A new page, linked from DS and Server pages. For eg: +1. A UI to view all parameters currently applied to a server, as well as the profile each parameter came from. A new page, linked from Server pages. For eg: | Name |Config File | Value | Profile | |--------------------------------------------------|--------------------------|--------------------------------------|-----------| @@ -63,15 +63,16 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin | error_url | url_sig_myotherds.config | 403 | EDGE | | CONFIG proxy.config.exec_thread.autoconfig.scale | records.config | FLOAT 1.5 | ATS_714 | -2. A UI change to add, remove, and reorder (sortable list) profiles for both Delivery Services and Servers, on the existing DS and Server pages. +2. A UI change to add, remove, and reorder (sortable list) profiles for Servers, on the existing Server pages. +3. Filtering based on Profiles will also need to be updated to take into account the plurality of Profiles. ### Traffic Ops Impact -- Traffic Ops will need to add the logic to the below-mentioned existing API endpoints, in order to show/create/update/delete a sorted list of profiles for delivery services and server. -- `/deliveryservices/ GET, POST` -- `/deliveryservices/{id} PUT, DELETE` +- Traffic Ops will need to add the logic to the below-mentioned existing API endpoints, in order to show/create/update/delete a sorted list of profiles for server. + - `/server/ GET, POST` - `/servers/{id} PUT, DELETE` +- `/servers/details` #### REST API Impact @@ -79,29 +80,9 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin **Existing Endpoints** -- Modify JSON request and response for existing delivery services and servers endpoints. +- Modify JSON request and response for existing servers endpoints. - JSON **response** with the proposed change will look as follows: -`/deliveryservices?id=100` -```JSON -{ - "id": 100, - ⋮ - "requested": { - "profileIds": [ - 1234, - 5678, - 9012 - ], - "profileNames": [ - "EDGE", - "ATS8", - "TOP" - ] - ⋮ - } -} -``` `/servers?id=5` ```JSON @@ -109,11 +90,6 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin "id": 5, ⋮ "requested": { - "profileIds": [ - 1357, - 2468, - 2356 - ], "profileNames": [ "MID", "TOP", @@ -126,35 +102,10 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin JSON **request** with the proposed change will look as follows: -`/deliveryservices` -```JSON -{ - "requested": { - "profileIds": [ - 1234, - 5678, - 9012 - ], - "profileNames": [ - "EDGE", - "ATS8", - "TOP" - ], - ⋮ - }, - ⋮ -} -``` - `/servers` ```JSON { "requested": { - "profileIds": [ - 1357, - 2468, - 2356 - ], "profileNames": [ "MID", "TOP", @@ -166,64 +117,52 @@ JSON **request** with the proposed change will look as follows: } ``` -The following table describes the top level `layered_profile` object for delivery services: - -| field | type | optionality | description | -| ------------------ | --------------| ----------- | ---------------------------------------------------------------| -| deliveryservice | bigint | required | the delivery service id associated with a given profile | -| profile | bigint | required | the profile id associated with a delivery service | -| order | bigint | required | the order in which a profile is applied to a delivery service | -| lastUpdated | bigint | required | the last time this delivery service was updated | - The following table describes the top level `layered_profile` object for servers: | field | type | optionality | description | | ----------- | ---------------------| ----------- | ---------------------------------------------------------------| | server | bigint | required | the server id associated with a given profile | -| profile | bigint | required | the profile id associated with a server | +| profileName | string | required | the profile name associated with a server | | order | bigint | required | the order in which a profile is applied to a server | -| lastUpdated | bigint | required | the last time this server was updated | **API constraints** -- TO REST APIs will not be backward compatibility and this feature will be a major version (v5.0) change. -- Add `Profiles` key to API endpoints for Server and DeliveryService objects (and ProfileNames, ProfileIDs) +- In API 5.0, GET /servers object profile and profileId fields to be arrays, but a PUT or POST with multiple values returns a 400 error. + The UI may or may not display a list (which can only add 1), the client implements handling multiple, and the API is documented to potentially return multiple and how their Parameters must be applied. +- In API 6.0 (or possibly 5.0 in the following TC major version wherein API 4.0 is removed), this feature is actually implemented, and multiple profiles can be assigned, are displayed in the UI, etc. + +The only disadvantage to splitting it across mulitple version is a little delay to get the feature deployed. +But the advantage is that it solves both concerns: +- clients on supported APIs are never given 4xx errors even after the feature is in-use +- we never return or apply partial data that could produce wrong results in production + +- Add `Profiles` key to API endpoints for Server objects (and ProfileNames, ProfileIDs) #### Client Impact -- Existing Go client methods will be updated for the `/deliveryservices` and `/servers` endpoints in order to write TO API tests for the exising endpoints. -- All functions which get Parameters on Delivery Services or Servers must be changed to get the Parameters of all assigned Profiles in order. +- Existing Go client methods will be updated for the `/servers` endpoints in order to write TO API tests for the exising endpoints. +- All functions which get Parameters on Servers must be changed to get the Parameters of all assigned Profiles in order. #### Data Model / Database Impact - A new Database Table `server_profiles` as described below, will be created: -- A new Database Table `deliveryservice_profiles` as described below, will be created: -```text - Table "traffic_ops.deliveryservice_profiles" - Column | Type | Collation | Nullable | Default -----------------+--------------------------+-----------+----------+-------- -deliveryservice | bigint | | not null | -profile | bigint | | not null | -order | bigint | | not null | -last_updated | timestamp with time zone | | not null | now() -Indexes: -"pk_deliveryservice_profile" PRIMARY KEY(profile) -Foreign-key constraints: -"fk_deliveryservice" FOREIGN KEY (deliveryservice) REFERENCES deliveryservice(id) -``` - ```text Table "traffic_ops.server_profiles" Column | Type | Collation | Nullable | Default ---------------+--------------------------+-----------+----------+-------- server | bigint | | not null | - profile | bigint | | not null | + profileName | string | | not null | order | bigint | | not null | - last_updated | timestamp with time zone | | not null | now() Indexes: - "pk_server_profile" PRIMARY KEY(profile) + "pk_server_profile" PRIMARY KEY(profileName, server, order) Foreign-key constraints: - "fk_server" FOREIGN KEY (server) REFERENCES server(id) + "fk_server" FOREIGN KEY (server, profileName) REFERENCES server(id) ``` +All profiles assigned to a given server will have the same values of: +- type +- cdn +- routing_disabled +If any of those differ within the same server, it's probably a mistake. + ### ORT Impact New set of profiles will be created and the profile-parameter relationship will change. @@ -231,7 +170,9 @@ New set of profiles will be created and the profile-parameter relationship will No impact ### Traffic Router Impact -No impact +These changes will affect the Snapshot generation (both crconfig and monitoring). Even though that is more of a TO impact. +Reason being that Snapshots and Monitoring Configurations for a CDN include Profile and Parameter information for the servers, Traffic Monitors, and Traffic Routersz. + ### Traffic Stats Impact No impact @@ -243,7 +184,7 @@ No impact All existing endpoints will need to be updated, along with the documentation explaining `layered_profile`. ### Testing Impact -Client/API integration tests should be updated to verify the `layered_profile` functionality on existing API `/deliveryservices`, `/servers` endpoints. +Client/API integration tests should be updated to verify the `layered_profile` functionality on existing API `/servers` endpoints. ### Performance Impact We do not anticipate any performance impact with layered_profile approach. @@ -253,16 +194,16 @@ We do not anticipate any impact on security. ### Upgrade Impact - A Database Migration to: - - drop profile column in existing deliveryservice and server table - - insert existing server and DS profiles along with their order into the new tables(deliveryservice_profiles, server_profiles) -- The new capability can just be added to the `seeds.sql` file. + - drop profile column in existing server table + - insert existing server and DS profiles along with their order into the new table(server_profiles) ### Operations Impact -The profile-parameter relationship will change based on new sets of profile and Operations will have to learn on how to assign profile order to a delivery service and/or server. +The profile-parameter relationship will change based on new sets of profile and Operations will have to learn on how to assign profile order to a server. ### Developer Impact Developers will most likely need to use layered_profile, so they'll need to be familiar with the process of adding, sorting, deleting, debugging and working with layered_profiles. +When searching for any parameters assigned to a profile for a given server, one will need to look up all profiles assigned to the server and then process all the parameters in order to get the "final" view of the profile. ## Alternatives None, except to keep using existing Profiles. From 316fe3dd70f13dbe7585054b7cbc6100e3468251 Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Fri, 19 Nov 2021 13:12:22 -0700 Subject: [PATCH 04/12] differentiated API endpoints between 5.0 and 6.0 --- blueprints/layered-profile.md | 170 ++++++++++++++++++++++++++++++---- 1 file changed, 151 insertions(+), 19 deletions(-) diff --git a/blueprints/layered-profile.md b/blueprints/layered-profile.md index e40ccdc655..8c232a0d81 100644 --- a/blueprints/layered-profile.md +++ b/blueprints/layered-profile.md @@ -70,7 +70,7 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin - Traffic Ops will need to add the logic to the below-mentioned existing API endpoints, in order to show/create/update/delete a sorted list of profiles for server. -- `/server/ GET, POST` +- `/servers/ GET, POST` - `/servers/{id} PUT, DELETE` - `/servers/details` @@ -81,35 +81,107 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin **Existing Endpoints** - Modify JSON request and response for existing servers endpoints. -- JSON **response** with the proposed change will look as follows: - +#### API 5.0 GET +- JSON **response** with the proposed change will look as follows: `/servers?id=5` ```JSON { "id": 5, ⋮ "requested": { - "profileNames": [ - "MID", - "TOP", - "PRIMARY" + "layeredProfiles": [ + "MID" ], ⋮ } } ``` +`/servers` +```JSON +{ + "requested": { + "layeredProfiles": [ + "MID" + ], + ⋮ + }, + ⋮ +} +``` + +#### API 5.0 POST/PUT JSON **request** with the proposed change will look as follows: +`POST /servers ` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profileId": [10, 11, 15], + ⋮ +} +``` + +`PUT /servers/5` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profileId": [10, 11, 15], + ⋮ +} +``` + +returns a **400** **response** with + +```JSON +{ + "alerts":[ + { + "text":"cannot associate multiple profiles to a server", + "level":"error" + } +]} +``` + +#### API 6.0 GET +- JSON **response** with the proposed change will look as follows: + `/servers?id=5` +```JSON +{ + "id": 5, + ⋮ + "requested": { + "layeredProfiles": [ + "MID", + "AMIGA_123", + "CDN_FOO" + ], + ⋮ + } +} +``` + `/servers` ```JSON { "requested": { - "profileNames": [ - "MID", - "TOP", - "PRIMARY" + "layeredProfiles": [ + "MID", + "AMIGA_123", + "CDN_FOO" ], ⋮ }, @@ -117,13 +189,72 @@ JSON **request** with the proposed change will look as follows: } ``` +#### API 6.0 POST/PUT +JSON **request** with the proposed change will look as follows: + +`POST /servers ` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profileId": [10, 11, 15], + ⋮ +} +``` + +`PUT /servers/5` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profileId": [10, 11, 15], + ⋮ +} +``` + +return following **response** +```JSON +{ "alerts": [ + { + "text": "Server created/updated", + "level": "success" + } +], + "response": { + "cachegroup": "CDN_in_a_Box_Mid", + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "id": 5, + "layeredProfiles": [ + "MID", + "AMIGA_123", + "CDN_FOO" + ], + "profileDesc": "Mid Cache - Apache Traffic Server, Amiga 123 Machine, CDN Foo", + "profileIds": [10, 11, 15], + ⋮ + } +} +``` + The following table describes the top level `layered_profile` object for servers: -| field | type | optionality | description | -| ----------- | ---------------------| ----------- | ---------------------------------------------------------------| -| server | bigint | required | the server id associated with a given profile | -| profileName | string | required | the profile name associated with a server | -| order | bigint | required | the order in which a profile is applied to a server | +| field | type | optionality | description | +| ----------------| ---------------------| ----------- | ---------------------------------------------------------| +| server | bigint | required | the server id associated with a given profile | +| layeredProfiles | string | required | the profile names associated with a server | +| order | bigint | required | the order in which a profile is applied to a server | **API constraints** - In API 5.0, GET /servers object profile and profileId fields to be arrays, but a PUT or POST with multiple values returns a 400 error. @@ -149,12 +280,13 @@ But the advantage is that it solves both concerns: Column | Type | Collation | Nullable | Default ---------------+--------------------------+-----------+----------+-------- server | bigint | | not null | - profileName | string | | not null | + profile_names | text | | not null | order | bigint | | not null | Indexes: - "pk_server_profile" PRIMARY KEY(profileName, server, order) + "pk_server_profile" PRIMARY KEY(profile_names, server, order) Foreign-key constraints: - "fk_server" FOREIGN KEY (server, profileName) REFERENCES server(id) + "fk_server" FOREIGN KEY (server) REFERENCES server(id) + "fk_server" FOREIGN KEY (profile_names) REFERENCES profile(name) ``` All profiles assigned to a given server will have the same values of: From 7c96c9aaabb6072ea001b81dfb1f9a27a1172753 Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Fri, 19 Nov 2021 13:18:08 -0700 Subject: [PATCH 05/12] updated type for layeredProfiles --- blueprints/layered-profile.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blueprints/layered-profile.md b/blueprints/layered-profile.md index 8c232a0d81..d10a734cc0 100644 --- a/blueprints/layered-profile.md +++ b/blueprints/layered-profile.md @@ -253,7 +253,7 @@ The following table describes the top level `layered_profile` object for servers | field | type | optionality | description | | ----------------| ---------------------| ----------- | ---------------------------------------------------------| | server | bigint | required | the server id associated with a given profile | -| layeredProfiles | string | required | the profile names associated with a server | +| layeredProfiles | text | required | the profile names associated with a server | | order | bigint | required | the order in which a profile is applied to a server | **API constraints** From 06931c06061f608737f3526f31ac25454fd306fc Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Mon, 22 Nov 2021 09:21:14 -0700 Subject: [PATCH 06/12] updated text error message --- blueprints/layered-profile.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blueprints/layered-profile.md b/blueprints/layered-profile.md index d10a734cc0..73ef8b3401 100644 --- a/blueprints/layered-profile.md +++ b/blueprints/layered-profile.md @@ -150,7 +150,7 @@ returns a **400** **response** with { "alerts":[ { - "text":"cannot associate multiple profiles to a server", + "text":"cannot associate multiple profiles to a server with current API version but can be done via 5.0+", "level":"error" } ]} From 8ad306fb9995e3578ee66786996c4c88c08f1b79 Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Fri, 10 Dec 2021 10:18:02 -0700 Subject: [PATCH 07/12] updated blueprint based on review comments. --- blueprints/layered-profile-server.md | 349 +++++++++++++++++++++++++++ 1 file changed, 349 insertions(+) create mode 100644 blueprints/layered-profile-server.md diff --git a/blueprints/layered-profile-server.md b/blueprints/layered-profile-server.md new file mode 100644 index 0000000000..0158eb0983 --- /dev/null +++ b/blueprints/layered-profile-server.md @@ -0,0 +1,349 @@ + +# Layered Profile + +## Problem Description + +Profiles are unwieldy and dangerous for Operations. + +Currently, we have countless Profiles, in broad categories. For example, "EDGE_123_FOO_ABC_ATS_714-RC0-42" might represent servers which are +1. Edges +2. Amiga 123 machines +3. In the Foo CDN +4. In the ABC datacenter +5. Running ATS 7.14 RC0 +6. Who knows what 42 means? + +Suppose we have Amiga 456 machines, at DEF datacenters, and a Bar CDN, and some servers are running ATS 7.15, 8.0, and 8.1. We make profiles for each server we have fitting all those categories: + +EDGE_123_FOO_ABC_ATS_714-RC0-27, EDGE_456_FOO_ABC_ATS_714-RC0-27, EDGE_123_BAR_ABC_ATS_714-RC0-42, EDGE_456_FOO_DEF_ATS_714-RC1-27, EDGE_123_FOO_DEF_ATS_800-RC4-29 + +- The number of Profiles quickly becomes a combinatorial explosion +- It's nearly impossible for Ops engineers to figure out what Parameters are assigned to all Profiles of a given CDN, machine, or datacenter. +- What about that one random Parameter on a Cloned profile that was changed a year ago? Is it still in place? Should it be? Did it need to be applied to all new Profiles cloned from this one? + +## Proposed Change + +Layered Profiles allow assigning multiple profiles to Servers. If multiple profiles have a parameter with the same name and config file, the parameter from the last profile in the ordering is applied. + +Layered Profiles is exactly as powerful as the existing profiles, it doesn't enable any new things. It makes profiles much easier to manage. + +With Layered Profiles, hundreds of profiles become a few dozen, each representing a logical group. For example, a server might have the profiles, in order: +1. EDGE +2. AMIGA_123 +3. CDN_FOO +4. DATACENTER_ABC +5. ATS_714 +6. Custom_Hack_42 + +### Traffic Portal Impact + +1. A UI to view all parameters currently applied to a server, as well as the profile each parameter came from. A new page, linked from Server pages. For eg: + +| Name |Config File | Value | Profile | +|--------------------------------------------------|--------------------------|--------------------------------------|-----------| +| location | url_sig_myds.config | /opt/trafficserver/etc/trafficserver | EDGE | +| Drive_Prefix | storage.config | /dev/sd | AMIGA_123 | +| error_url | url_sig_myotherds.config | 403 | EDGE | +| CONFIG proxy.config.exec_thread.autoconfig.scale | records.config | FLOAT 1.5 | ATS_714 | + +2. A UI change to add, remove, and reorder (sortable list) profiles for Servers, on the existing Server pages. +3. Filtering based on Profiles will also need to be updated to take into account the plurality of Profiles. + +### Traffic Ops Impact + +- Add backend logic to the below-mentioned existing API endpoints, in order to show/create/update/delete a sorted list of profiles for server. + +- `/servers/ GET, POST` +- `/servers/{id} PUT, DELETE` +- `/servers/details` +- `/deliveryservices/{{ID}}/servers` +- `/deliveryservices/{{ID}}/servers/eligible` + +#### REST API Impact + +- No new endpoints are required + +**Existing Endpoints** + +- Modify JSON request and response for existing servers endpoints. + +#### API 5.0 GET +- JSON **response** with the proposed change will look as follows: +`/servers?id=5` +```JSON +{ + "id": 5, + ⋮ + "requested": { + "profiles": [ + "MID" + ], + ⋮ + } +} +``` + +`/servers` +```JSON +{ + "requested": { + "profiles": [ + "MID" + ], + ⋮ + }, + ⋮ +} +``` + +#### API 5.0 POST/PUT +JSON **request** with the proposed change will look as follows: + +`POST /servers ` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"], + ⋮ +} +``` + +`PUT /servers/5` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"], + ⋮ +} +``` + +returns a **400** **response** with + +```JSON +{ + "alerts":[ + { + "text":"cannot associate multiple profiles to a server with current API version but can be done via 6.0", + "level":"error" + } +]} +``` + +#### API 6.0 GET +- JSON **response** with the proposed change will look as follows: + `/servers?id=5` +```JSON +{ + "id": 5, + ⋮ + "requested": { + "profiles": [ + "MID", + "AMIGA_123", + "CDN_FOO" + ], + ⋮ + } +} +``` + +`/servers` +```JSON +{ + "requested": { + "profiles": [ + "MID", + "AMIGA_123", + "CDN_FOO" + ], + ⋮ + }, + ⋮ +} +``` + +#### API 6.0 POST/PUT +JSON **request** with the proposed change will look as follows: + +`POST /servers ` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"], + ⋮ +} +``` + +`PUT /servers/5` +```JSON +{ + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "interfaces": [ + ⋮ + ], + ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"], + ⋮ +} +``` + +return following **response** +```JSON +{ "alerts": [ + { + "text": "Server created/updated", + "level": "success" + } +], + "response": { + "cachegroup": "CDN_in_a_Box_Mid", + "cachegroupId": 6, + "cdnId": 2, + ⋮ + "id": 5, + "profiles": [ + "MID", + "AMIGA_123", + "CDN_FOO" + ], + "profileDesc": "Mid Cache - Apache Traffic Server, Amiga 123 Machine, CDN Foo", + "profileIds": [10, 11, 15], + ⋮ + } +} +``` + +The following table describes the top level `layered_profile` object for servers: + +| field | type | optionality | description | +| ----------------| ---------------------| ----------- | ---------------------------------------------------------| +| server | bigint | required | the server id associated with a given profile | +| profiles | text | required | the profile names associated with a server | +| order | bigint | required | the order in which a profile is applied to a server | + +**API constraints** +- In API 5.0, GET /servers object profile and profileId fields to be arrays, but a PUT or POST with multiple values returns a 400 error. + The UI may or may not display a list (which can only add 1), the client implements handling multiple, and the API is documented to potentially return multiple and how their Parameters must be applied. +- In API 6.0 (or possibly 5.0 in the following TC major version wherein API 4.0 is removed), this feature is actually implemented, and multiple profiles can be assigned, are displayed in the UI, etc. + +The only disadvantage to splitting it across mulitple version is a little delay to get the feature deployed. +But the advantage is that it solves both concerns: +- clients on supported APIs are never given 4xx errors even after the feature is in-use +- we never return or apply partial data that could produce wrong results in production + +- Add `Profiles` key to API endpoints for Server objects (and ProfileNames, ProfileIDs) + +#### Client Impact +- Existing Go client methods will be updated for the `/servers` endpoints in order to write TO API tests for the exising endpoints. +- All functions which get Parameters on Servers must be changed to get the Parameters of all assigned Profiles in order. + +#### Data Model / Database Impact + +- A new Database Table `server_profiles` as described below, will be created: +```text + Table "traffic_ops.server_profiles" + Column | Type | Collation | Nullable | Default +---------------+--------------------------+-----------+----------+-------- + server | bigint | | not null | + profile_names | text | | not null | + order | bigint | | not null | +Indexes: + "pk_server_profile" PRIMARY KEY(profile_names, server, order) +Foreign-key constraints: + "fk_server" FOREIGN KEY (server) REFERENCES server(id) + "fk_server" FOREIGN KEY (profile_names) REFERENCES profile(name) +``` + +All profiles assigned to a given server will have the same values of: +- type +- cdn +- routing_disabled +If any of those differ within the same server, it's probably a mistake. + +### ORT Impact +New set of profiles will be created and the profile-parameter relationship will change. + +### Traffic Monitor Impact +No impact + +### Traffic Router Impact +These changes will affect the Snapshot generation (both crconfig and monitoring). Even though that is more of a TO impact. +Reason being that Snapshots and Monitoring Configurations for a CDN include Profile and Parameter information for the servers, Traffic Monitors, and Traffic Routersz. + + +### Traffic Stats Impact +No impact + +### Traffic Vault Impact +No impact + +### Documentation Impact +All existing endpoints will need to be updated, along with the documentation explaining `layered_profile`. + +### Testing Impact +Client/API integration tests should be updated to verify the `layered_profile` functionality on existing API `/servers` endpoints. + +### Performance Impact +We do not anticipate any performance impact with layered_profile approach. + +### Security Impact +We do not anticipate any impact on security. + +### Upgrade Impact +- A Database Migration to: + - drop profile column in existing server table + - insert existing server and DS profiles along with their order into the new table(server_profiles) + +### Operations Impact +The profile-parameter relationship will change based on new sets of profile and Operations will have to learn on how to assign profile order to a server. + +### Developer Impact +Developers will most likely need to use layered_profile, so they'll need to be familiar with the process +of adding, sorting, deleting, debugging and working with layered_profiles. +When searching for any parameters assigned to a profile for a given server, one will need to look up all profiles assigned to the server and then process all the parameters in order to get the "final" view of the profile. + +## Alternatives +None, except to keep using existing Profiles. + +## Dependencies +None + +## References +None From 125e3eb4bf3eb96fe3af5a1ce7bbabed5dde04c4 Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Mon, 13 Dec 2021 09:24:40 -0700 Subject: [PATCH 08/12] Updated based on review comments-1 --- blueprints/layered-profile-server.md | 61 ++--- blueprints/layered-profile.md | 347 --------------------------- 2 files changed, 21 insertions(+), 387 deletions(-) delete mode 100644 blueprints/layered-profile.md diff --git a/blueprints/layered-profile-server.md b/blueprints/layered-profile-server.md index 0158eb0983..955222227a 100644 --- a/blueprints/layered-profile-server.md +++ b/blueprints/layered-profile-server.md @@ -88,13 +88,11 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin - JSON **response** with the proposed change will look as follows: `/servers?id=5` ```JSON -{ - "id": 5, - ⋮ - "requested": { - "profiles": [ - "MID" - ], +{ + "response": [{ + "id": 5, + ⋮ + "profiles": ["MID"], ⋮ } } @@ -102,14 +100,11 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin `/servers` ```JSON -{ - "requested": { - "profiles": [ - "MID" - ], - ⋮ - }, - ⋮ +{ + "response": [{ + "profiles": ["MID"] + ⋮, + } } ``` @@ -162,29 +157,21 @@ returns a **400** **response** with - JSON **response** with the proposed change will look as follows: `/servers?id=5` ```JSON -{ +{ + "response": [{ "id": 5, ⋮ - "requested": { - "profiles": [ - "MID", - "AMIGA_123", - "CDN_FOO" - ], - ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"], + ⋮ } } ``` `/servers` ```JSON -{ - "requested": { - "profiles": [ - "MID", - "AMIGA_123", - "CDN_FOO" - ], +{ + "response": [{ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"], ⋮ }, ⋮ @@ -238,13 +225,7 @@ return following **response** "cdnId": 2, ⋮ "id": 5, - "profiles": [ - "MID", - "AMIGA_123", - "CDN_FOO" - ], - "profileDesc": "Mid Cache - Apache Traffic Server, Amiga 123 Machine, CDN Foo", - "profileIds": [10, 11, 15], + "profiles": ["MID", "AMIGA_123", "CDN_FOO"], ⋮ } } @@ -282,13 +263,13 @@ But the advantage is that it solves both concerns: Column | Type | Collation | Nullable | Default ---------------+--------------------------+-----------+----------+-------- server | bigint | | not null | - profile_names | text | | not null | + profile_name | text | | not null | order | bigint | | not null | Indexes: - "pk_server_profile" PRIMARY KEY(profile_names, server, order) + "pk_server_profile" PRIMARY KEY(profile_name, server, order) Foreign-key constraints: "fk_server" FOREIGN KEY (server) REFERENCES server(id) - "fk_server" FOREIGN KEY (profile_names) REFERENCES profile(name) + "fk_server" FOREIGN KEY (profile_name) REFERENCES profile(name) ``` All profiles assigned to a given server will have the same values of: diff --git a/blueprints/layered-profile.md b/blueprints/layered-profile.md deleted file mode 100644 index 73ef8b3401..0000000000 --- a/blueprints/layered-profile.md +++ /dev/null @@ -1,347 +0,0 @@ - -# Layered Profile - -## Problem Description - -Profiles are unwieldy and dangerous for Operations. - -Currently, we have countless Profiles, in broad categories. For example, "EDGE_123_FOO_ABC_ATS_714-RC0-42" might represent servers which are -1. Edges -2. Amiga 123 machines -3. In the Foo CDN -4. In the ABC datacenter -5. Running ATS 7.14 RC0 -6. Who knows what 42 means? - -Suppose we have Amiga 456 machines, at DEF datacenters, and a Bar CDN, and some servers are running ATS 7.15, 8.0, and 8.1. We make profiles for each server we have fitting all those categories: - -EDGE_123_FOO_ABC_ATS_714-RC0-27, EDGE_456_FOO_ABC_ATS_714-RC0-27, EDGE_123_BAR_ABC_ATS_714-RC0-42, EDGE_456_FOO_DEF_ATS_714-RC1-27, EDGE_123_FOO_DEF_ATS_800-RC4-29 - -- The number of Profiles quickly becomes a combinatorial explosion -- It's nearly impossible for Ops engineers to figure out what Parameters are assigned to all Profiles of a given CDN, machine, or datacenter. -- What about that one random Parameter on a Cloned profile that was changed a year ago? Is it still in place? Should it be? Did it need to be applied to all new Profiles cloned from this one? - -## Proposed Change - -Layered Profiles allow assigning multiple profiles to Servers. If multiple profiles have a parameter with the same name and config file, the parameter from the last profile in the ordering is applied. - -Layered Profiles is exactly as powerful as the existing profiles, it doesn't enable any new things. It makes profiles much easier to manage. - -With Layered Profiles, hundreds of profiles become a few dozen, each representing a logical group. For example, a server might have the profiles, in order: -1. EDGE -2. AMIGA_123 -3. CDN_FOO -4. DATACENTER_ABC -5. ATS_714 -6. Custom_Hack_42 - -### Traffic Portal Impact - -1. A UI to view all parameters currently applied to a server, as well as the profile each parameter came from. A new page, linked from Server pages. For eg: - -| Name |Config File | Value | Profile | -|--------------------------------------------------|--------------------------|--------------------------------------|-----------| -| location | url_sig_myds.config | /opt/trafficserver/etc/trafficserver | EDGE | -| Drive_Prefix | storage.config | /dev/sd | AMIGA_123 | -| error_url | url_sig_myotherds.config | 403 | EDGE | -| CONFIG proxy.config.exec_thread.autoconfig.scale | records.config | FLOAT 1.5 | ATS_714 | - -2. A UI change to add, remove, and reorder (sortable list) profiles for Servers, on the existing Server pages. -3. Filtering based on Profiles will also need to be updated to take into account the plurality of Profiles. - -### Traffic Ops Impact - -- Traffic Ops will need to add the logic to the below-mentioned existing API endpoints, in order to show/create/update/delete a sorted list of profiles for server. - -- `/servers/ GET, POST` -- `/servers/{id} PUT, DELETE` -- `/servers/details` - -#### REST API Impact - -- No new endpoints are required - -**Existing Endpoints** - -- Modify JSON request and response for existing servers endpoints. - -#### API 5.0 GET -- JSON **response** with the proposed change will look as follows: -`/servers?id=5` -```JSON -{ - "id": 5, - ⋮ - "requested": { - "layeredProfiles": [ - "MID" - ], - ⋮ - } -} -``` - -`/servers` -```JSON -{ - "requested": { - "layeredProfiles": [ - "MID" - ], - ⋮ - }, - ⋮ -} -``` - -#### API 5.0 POST/PUT -JSON **request** with the proposed change will look as follows: - -`POST /servers ` -```JSON -{ - "cachegroupId": 6, - "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profileId": [10, 11, 15], - ⋮ -} -``` - -`PUT /servers/5` -```JSON -{ - "cachegroupId": 6, - "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profileId": [10, 11, 15], - ⋮ -} -``` - -returns a **400** **response** with - -```JSON -{ - "alerts":[ - { - "text":"cannot associate multiple profiles to a server with current API version but can be done via 5.0+", - "level":"error" - } -]} -``` - -#### API 6.0 GET -- JSON **response** with the proposed change will look as follows: - `/servers?id=5` -```JSON -{ - "id": 5, - ⋮ - "requested": { - "layeredProfiles": [ - "MID", - "AMIGA_123", - "CDN_FOO" - ], - ⋮ - } -} -``` - -`/servers` -```JSON -{ - "requested": { - "layeredProfiles": [ - "MID", - "AMIGA_123", - "CDN_FOO" - ], - ⋮ - }, - ⋮ -} -``` - -#### API 6.0 POST/PUT -JSON **request** with the proposed change will look as follows: - -`POST /servers ` -```JSON -{ - "cachegroupId": 6, - "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profileId": [10, 11, 15], - ⋮ -} -``` - -`PUT /servers/5` -```JSON -{ - "cachegroupId": 6, - "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profileId": [10, 11, 15], - ⋮ -} -``` - -return following **response** -```JSON -{ "alerts": [ - { - "text": "Server created/updated", - "level": "success" - } -], - "response": { - "cachegroup": "CDN_in_a_Box_Mid", - "cachegroupId": 6, - "cdnId": 2, - ⋮ - "id": 5, - "layeredProfiles": [ - "MID", - "AMIGA_123", - "CDN_FOO" - ], - "profileDesc": "Mid Cache - Apache Traffic Server, Amiga 123 Machine, CDN Foo", - "profileIds": [10, 11, 15], - ⋮ - } -} -``` - -The following table describes the top level `layered_profile` object for servers: - -| field | type | optionality | description | -| ----------------| ---------------------| ----------- | ---------------------------------------------------------| -| server | bigint | required | the server id associated with a given profile | -| layeredProfiles | text | required | the profile names associated with a server | -| order | bigint | required | the order in which a profile is applied to a server | - -**API constraints** -- In API 5.0, GET /servers object profile and profileId fields to be arrays, but a PUT or POST with multiple values returns a 400 error. - The UI may or may not display a list (which can only add 1), the client implements handling multiple, and the API is documented to potentially return multiple and how their Parameters must be applied. -- In API 6.0 (or possibly 5.0 in the following TC major version wherein API 4.0 is removed), this feature is actually implemented, and multiple profiles can be assigned, are displayed in the UI, etc. - -The only disadvantage to splitting it across mulitple version is a little delay to get the feature deployed. -But the advantage is that it solves both concerns: -- clients on supported APIs are never given 4xx errors even after the feature is in-use -- we never return or apply partial data that could produce wrong results in production - -- Add `Profiles` key to API endpoints for Server objects (and ProfileNames, ProfileIDs) - -#### Client Impact -- Existing Go client methods will be updated for the `/servers` endpoints in order to write TO API tests for the exising endpoints. -- All functions which get Parameters on Servers must be changed to get the Parameters of all assigned Profiles in order. - -#### Data Model / Database Impact - -- A new Database Table `server_profiles` as described below, will be created: -```text - Table "traffic_ops.server_profiles" - Column | Type | Collation | Nullable | Default ----------------+--------------------------+-----------+----------+-------- - server | bigint | | not null | - profile_names | text | | not null | - order | bigint | | not null | -Indexes: - "pk_server_profile" PRIMARY KEY(profile_names, server, order) -Foreign-key constraints: - "fk_server" FOREIGN KEY (server) REFERENCES server(id) - "fk_server" FOREIGN KEY (profile_names) REFERENCES profile(name) -``` - -All profiles assigned to a given server will have the same values of: -- type -- cdn -- routing_disabled -If any of those differ within the same server, it's probably a mistake. - -### ORT Impact -New set of profiles will be created and the profile-parameter relationship will change. - -### Traffic Monitor Impact -No impact - -### Traffic Router Impact -These changes will affect the Snapshot generation (both crconfig and monitoring). Even though that is more of a TO impact. -Reason being that Snapshots and Monitoring Configurations for a CDN include Profile and Parameter information for the servers, Traffic Monitors, and Traffic Routersz. - - -### Traffic Stats Impact -No impact - -### Traffic Vault Impact -No impact - -### Documentation Impact -All existing endpoints will need to be updated, along with the documentation explaining `layered_profile`. - -### Testing Impact -Client/API integration tests should be updated to verify the `layered_profile` functionality on existing API `/servers` endpoints. - -### Performance Impact -We do not anticipate any performance impact with layered_profile approach. - -### Security Impact -We do not anticipate any impact on security. - -### Upgrade Impact -- A Database Migration to: - - drop profile column in existing server table - - insert existing server and DS profiles along with their order into the new table(server_profiles) - -### Operations Impact -The profile-parameter relationship will change based on new sets of profile and Operations will have to learn on how to assign profile order to a server. - -### Developer Impact -Developers will most likely need to use layered_profile, so they'll need to be familiar with the process -of adding, sorting, deleting, debugging and working with layered_profiles. -When searching for any parameters assigned to a profile for a given server, one will need to look up all profiles assigned to the server and then process all the parameters in order to get the "final" view of the profile. - -## Alternatives -None, except to keep using existing Profiles. - -## Dependencies -None - -## References -None From 901ad5e25be2884f5d3367abf18ddd105f9b422c Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Mon, 13 Dec 2021 10:29:08 -0700 Subject: [PATCH 09/12] Updated based on review comments-2 --- blueprints/layered-profile-server.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/blueprints/layered-profile-server.md b/blueprints/layered-profile-server.md index 955222227a..ee042498ae 100644 --- a/blueprints/layered-profile-server.md +++ b/blueprints/layered-profile-server.md @@ -288,7 +288,6 @@ No impact These changes will affect the Snapshot generation (both crconfig and monitoring). Even though that is more of a TO impact. Reason being that Snapshots and Monitoring Configurations for a CDN include Profile and Parameter information for the servers, Traffic Monitors, and Traffic Routersz. - ### Traffic Stats Impact No impact @@ -310,7 +309,7 @@ We do not anticipate any impact on security. ### Upgrade Impact - A Database Migration to: - drop profile column in existing server table - - insert existing server and DS profiles along with their order into the new table(server_profiles) + - insert existing server profiles along with their order into the new table(server_profiles) ### Operations Impact The profile-parameter relationship will change based on new sets of profile and Operations will have to learn on how to assign profile order to a server. From 6673277abb2e6157adb3fdf804ffc930ffebe00c Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Tue, 14 Dec 2021 08:51:37 -0700 Subject: [PATCH 10/12] Corrected API version. --- blueprints/layered-profile-server.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/blueprints/layered-profile-server.md b/blueprints/layered-profile-server.md index ee042498ae..bba8fedc10 100644 --- a/blueprints/layered-profile-server.md +++ b/blueprints/layered-profile-server.md @@ -84,7 +84,7 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin - Modify JSON request and response for existing servers endpoints. -#### API 5.0 GET +#### API 4.0 GET - JSON **response** with the proposed change will look as follows: `/servers?id=5` ```JSON @@ -108,7 +108,7 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin } ``` -#### API 5.0 POST/PUT +#### API 4.0 POST/PUT JSON **request** with the proposed change will look as follows: `POST /servers ` @@ -147,13 +147,13 @@ returns a **400** **response** with { "alerts":[ { - "text":"cannot associate multiple profiles to a server with current API version but can be done via 6.0", + "text":"cannot associate multiple profiles to a server with current API version but can be done via 5.0", "level":"error" } ]} ``` -#### API 6.0 GET +#### API 5.0 GET - JSON **response** with the proposed change will look as follows: `/servers?id=5` ```JSON @@ -178,7 +178,7 @@ returns a **400** **response** with } ``` -#### API 6.0 POST/PUT +#### API 5.0 POST/PUT JSON **request** with the proposed change will look as follows: `POST /servers ` @@ -240,9 +240,9 @@ The following table describes the top level `layered_profile` object for servers | order | bigint | required | the order in which a profile is applied to a server | **API constraints** -- In API 5.0, GET /servers object profile and profileId fields to be arrays, but a PUT or POST with multiple values returns a 400 error. +- In API 4.0, GET /servers object profile and profileId fields to be arrays, but a PUT or POST with multiple values returns a 400 error. The UI may or may not display a list (which can only add 1), the client implements handling multiple, and the API is documented to potentially return multiple and how their Parameters must be applied. -- In API 6.0 (or possibly 5.0 in the following TC major version wherein API 4.0 is removed), this feature is actually implemented, and multiple profiles can be assigned, are displayed in the UI, etc. +- In API 5.0 (or in the following TC major version wherein API 4.0 is removed), this feature is actually implemented, and multiple profiles can be assigned, are displayed in the UI, etc. The only disadvantage to splitting it across mulitple version is a little delay to get the feature deployed. But the advantage is that it solves both concerns: @@ -275,7 +275,6 @@ Foreign-key constraints: All profiles assigned to a given server will have the same values of: - type - cdn -- routing_disabled If any of those differ within the same server, it's probably a mistake. ### ORT Impact From 8016a6af9491443f3171836b015ed7596256c219 Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Thu, 16 Dec 2021 14:11:12 -0700 Subject: [PATCH 11/12] Updated to correct JSON struct. --- blueprints/layered-profile-server.md | 62 +++++++--------------------- 1 file changed, 15 insertions(+), 47 deletions(-) diff --git a/blueprints/layered-profile-server.md b/blueprints/layered-profile-server.md index bba8fedc10..7e0d25b023 100644 --- a/blueprints/layered-profile-server.md +++ b/blueprints/layered-profile-server.md @@ -83,7 +83,8 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin **Existing Endpoints** - Modify JSON request and response for existing servers endpoints. - +- **_Note_**: All fields in the response structure for API 4.0/ 5.0 except `profileId`, `profileDesc` and `profile`, remain same. + - In API 4.0/5.0 `profile` changes to `profiles` and in API 5.0 `profileId` and `profileDesc` are no longer part of response structure #### API 4.0 GET - JSON **response** with the proposed change will look as follows: `/servers?id=5` @@ -91,10 +92,8 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin { "response": [{ "id": 5, - ⋮ - "profiles": ["MID"], - ⋮ - } + "profiles": ["MID"] + }] } ``` @@ -103,8 +102,7 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin { "response": [{ "profiles": ["MID"] - ⋮, - } + }] } ``` @@ -116,13 +114,7 @@ JSON **request** with the proposed change will look as follows: { "cachegroupId": 6, "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profiles": ["MID", "AMIGA_123", "CDN_FOO"], - ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"] } ``` @@ -131,13 +123,7 @@ JSON **request** with the proposed change will look as follows: { "cachegroupId": 6, "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profiles": ["MID", "AMIGA_123", "CDN_FOO"], - ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"] } ``` @@ -159,11 +145,9 @@ returns a **400** **response** with ```JSON { "response": [{ - "id": 5, - ⋮ - "profiles": ["MID", "AMIGA_123", "CDN_FOO"], - ⋮ - } + "id": 5, + "profiles": ["MID", "AMIGA_123", "CDN_FOO"] + }] } ``` @@ -171,10 +155,8 @@ returns a **400** **response** with ```JSON { "response": [{ - "profiles": ["MID", "AMIGA_123", "CDN_FOO"], - ⋮ - }, - ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"] + }] } ``` @@ -186,13 +168,7 @@ JSON **request** with the proposed change will look as follows: { "cachegroupId": 6, "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profiles": ["MID", "AMIGA_123", "CDN_FOO"], - ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"] } ``` @@ -201,13 +177,7 @@ JSON **request** with the proposed change will look as follows: { "cachegroupId": 6, "cdnId": 2, - ⋮ - "interfaces": [ - ⋮ - ], - ⋮ - "profiles": ["MID", "AMIGA_123", "CDN_FOO"], - ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"] } ``` @@ -223,10 +193,8 @@ return following **response** "cachegroup": "CDN_in_a_Box_Mid", "cachegroupId": 6, "cdnId": 2, - ⋮ "id": 5, - "profiles": ["MID", "AMIGA_123", "CDN_FOO"], - ⋮ + "profiles": ["MID", "AMIGA_123", "CDN_FOO"] } } ``` From bd072c33ffe478fb8e18ace7bb552d706203a76a Mon Sep 17 00:00:00 2001 From: Rima Shah Date: Wed, 5 Jan 2022 13:29:41 -0700 Subject: [PATCH 12/12] BP has only one API version. --- blueprints/layered-profile-server.md | 78 ++++------------------------ 1 file changed, 10 insertions(+), 68 deletions(-) diff --git a/blueprints/layered-profile-server.md b/blueprints/layered-profile-server.md index 7e0d25b023..6b17b9d4db 100644 --- a/blueprints/layered-profile-server.md +++ b/blueprints/layered-profile-server.md @@ -83,63 +83,10 @@ With Layered Profiles, hundreds of profiles become a few dozen, each representin **Existing Endpoints** - Modify JSON request and response for existing servers endpoints. -- **_Note_**: All fields in the response structure for API 4.0/ 5.0 except `profileId`, `profileDesc` and `profile`, remain same. - - In API 4.0/5.0 `profile` changes to `profiles` and in API 5.0 `profileId` and `profileDesc` are no longer part of response structure -#### API 4.0 GET -- JSON **response** with the proposed change will look as follows: -`/servers?id=5` -```JSON -{ - "response": [{ - "id": 5, - "profiles": ["MID"] - }] -} -``` - -`/servers` -```JSON -{ - "response": [{ - "profiles": ["MID"] - }] -} -``` - -#### API 4.0 POST/PUT -JSON **request** with the proposed change will look as follows: +- **_Note_**: All fields in the response structure for API 4.0 except `profileId`, `profileDesc` and `profile`, remain same. + - In API 4.0 `profile` changes to `profiles` and `profileId` and `profileDesc` are no longer part of response structure -`POST /servers ` -```JSON -{ - "cachegroupId": 6, - "cdnId": 2, - "profiles": ["MID", "AMIGA_123", "CDN_FOO"] -} -``` - -`PUT /servers/5` -```JSON -{ - "cachegroupId": 6, - "cdnId": 2, - "profiles": ["MID", "AMIGA_123", "CDN_FOO"] -} -``` - -returns a **400** **response** with - -```JSON -{ - "alerts":[ - { - "text":"cannot associate multiple profiles to a server with current API version but can be done via 5.0", - "level":"error" - } -]} -``` - -#### API 5.0 GET +#### API 4.0 GET - JSON **response** with the proposed change will look as follows: `/servers?id=5` ```JSON @@ -160,7 +107,7 @@ returns a **400** **response** with } ``` -#### API 5.0 POST/PUT +#### API 4.0 POST/PUT JSON **request** with the proposed change will look as follows: `POST /servers ` @@ -204,20 +151,15 @@ The following table describes the top level `layered_profile` object for servers | field | type | optionality | description | | ----------------| ---------------------| ----------- | ---------------------------------------------------------| | server | bigint | required | the server id associated with a given profile | -| profiles | text | required | the profile names associated with a server | +| profiles | text [] | required | the profile names associated with a server | | order | bigint | required | the order in which a profile is applied to a server | **API constraints** -- In API 4.0, GET /servers object profile and profileId fields to be arrays, but a PUT or POST with multiple values returns a 400 error. - The UI may or may not display a list (which can only add 1), the client implements handling multiple, and the API is documented to potentially return multiple and how their Parameters must be applied. -- In API 5.0 (or in the following TC major version wherein API 4.0 is removed), this feature is actually implemented, and multiple profiles can be assigned, are displayed in the UI, etc. - -The only disadvantage to splitting it across mulitple version is a little delay to get the feature deployed. -But the advantage is that it solves both concerns: -- clients on supported APIs are never given 4xx errors even after the feature is in-use -- we never return or apply partial data that could produce wrong results in production - -- Add `Profiles` key to API endpoints for Server objects (and ProfileNames, ProfileIDs) +- In API 4.0, + - GET `/servers` object, `profiles` field is an array, + - and a PUT or POST `/servers` allows multiple profiles to be assigned, and displayed in the UI, etc. + The UI may or may not display a list (which can only add 1), but the client implements handling multiple, and the API is documented to potentially return multiple and how their parameters must be applied. +- Add `Profiles` key to API endpoints for Server objects #### Client Impact - Existing Go client methods will be updated for the `/servers` endpoints in order to write TO API tests for the exising endpoints.