Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Event endpoint to handle notifications was implemented #296

Merged
merged 6 commits into from
Mar 7, 2019
Merged

Event endpoint to handle notifications was implemented #296

merged 6 commits into from
Mar 7, 2019

Conversation

BraslavskiyAndrey
Copy link
Contributor

No description provided.

schernysh
schernysh suggested changes Mar 1, 2019
View reviewed changes
src/main/java/org/prebid/server/analytics/model/NotificationEvent.java

import lombok.Value;

@Value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use static factory of method for consistency

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated
private static final String VIEW_TYPE = "view";
private static final String WIN_TYPE = "win";
private static final String FORMAT_PARAMETER = "format";
private static final String JPG = "jpg";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's have these renamed to JPG_FORMAT and PNG_FORMAT correspondingly

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated
private static final String FORMAT_PARAMETER = "format";
private static final String JPG = "jpg";
private static final String PNG = "png";
private static final String JPG_HEADER = "image/jpeg";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are rather JPG_CONTENT_TYPE and PNG_CONTENT_TYPE

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated
readTrackingPixel(TRACKING_PIXEL_PNG), readTrackingPixel(TRACKING_PIXEL_JPG));
}

private static byte[] readTrackingPixel(String path) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move this method next to handle to have private methods after public ones

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated
public void handle(RoutingContext context) {
final EventNotificationRequest eventNotificationRequest;
try {
eventNotificationRequest = makeEventRequest(context.getBody());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's correct - it is a GET request and all parameters should be read from query params.

Fixed tech spec to explicitly say it is a GET request

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated

analyticsReporter.processEvent(notificationEvent);

final Map<String, String> queryParams = HttpContext.from(context).getQueryParams();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is not need to create HttpContext for the sake of just reading a query parameter - it could be read from context directly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been addressed somehow?

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated
}

private void validateEventRequestQueryParams(String format) {
if (format != null && ObjectUtils.notEqual(format, JPG) && ObjectUtils.notEqual(format, PNG)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting from here and onwards: this kind of conditions is not scalable enough and may bite you when third or fourth format will be added, especially this one and alike: format.equals(JPG) ? JPG_HEADER : PNG_HEADER.
I suggest creating a map holding all the necessary information about supported format:

final Map<String, TrackingPixel> trackingPixels;
...
private static class TrackingPixel {
    String contentType;
    byte[] content
}

It will be easy to use it then for validation and response construction and it will be much safer when it will come to adding another format/

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated

private void respondWithBadStatus(RoutingContext context, String message) {
context.response().setStatusCode(HttpResponseStatus.BAD_REQUEST.code())
.end(String.format("%s%s", "Request is invalid: ", message));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

String.format("Request is invalid: %s", message) looks more readable, isn't it?

src/main/java/org/prebid/server/spring/config/WebConfiguration.java Outdated
@@ -164,6 +166,7 @@ Router router(CookieHandler cookieHandler,
router.get("/bidders/params").handler(bidderParamHandler);
router.get("/info/bidders").handler(biddersHandler);
router.get("/info/bidders/:bidderName").handler(bidderDetailsHandler);
router.post("/event").handler(notificationEventHandler);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, it is implicit but it should be GET request. Fixed in the tech spec to explicitly say this.

src/test/java/org/prebid/server/handler/NotificationEventHandlerTest.java Outdated
}

@Test
public void shouldThrowIllegalExceptionWhenReporterIsNull() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we stopped writing this kind of tests and gradually removing them from where they still exist.

schernysh
schernysh reviewed Mar 5, 2019
View reviewed changes
src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated

analyticsReporter.processEvent(notificationEvent);

final Map<String, String> queryParams = HttpContext.from(context).getQueryParams();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been addressed somehow?

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated
}

private void validateEventRequestQueryParams(String format) {
if (format != null && ObjectUtils.notEqual(format, JPG_FORMAT) && ObjectUtils.notEqual(format, PNG_FORMAT)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is better to be replaced with if( format != null && trackingPixels.containsKey(format))

src/main/java/org/prebid/server/handler/NotificationEventHandler.java Outdated

private void validateEventRequestQueryParams(String format) {
if (format != null && ObjectUtils.notEqual(format, JPG_FORMAT) && ObjectUtils.notEqual(format, PNG_FORMAT)) {
throw new InvalidRequestException("format when defined should has value of png or jpg.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to pull format parameter values from trackingPixels map

schernysh
schernysh reviewed Mar 5, 2019
View reviewed changes
Copy link
Contributor

@schernysh schernysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add new endpoint description in docs/endpoints directory also

@schernysh schernysh merged commit 3e6f633 into master Mar 7, 2019
@schernysh schernysh deleted the implement-event-notification-endpoint branch March 7, 2019 09:53
@bretg bretg mentioned this pull request Aug 11, 2020
Closed
BraslavskiyAndrey added a commit that referenced this pull request Aug 13, 2020
@BraslavskiyAndrey
Merge pg to master rubicon (#469)
a569127 
* Create stubs for PG services (#57)

* Remove dealsOnly bidder from imp without matching deals (#58)

* Add deals processor to chain after request was fully updated in factory (#61)

* Create stubs for PG services (#57)

* Remove dealsOnly bidder from imp without matching deals (#58)

* Add deals processor to chain after request was fully updated in factory (#61)

* Enhance NetAcuity Geo Location to provide additional information (#66)

* Fill impression with deal info from selected line items (#62)

* Implement fetching data from planner (#71)

* Enrich BidRequest with deals targeting (device, geo, user segments) (#78)

* Line Item post process filtration was implemented (#77)

* Implement getting User Details from User Data Store (#76)

* Implement plan advance and freeze (#79)

* Move ready at to statuses (#81)

* Implement processing win event (#82)

* Send event to user service (#83)

* Basic Authentication implemented for planner communication (#84)

* Add planner delivery progress report sending (#85)

* LineItem to impression prematching implemented (#87)

* Implement targeting definition parsing and create foundation for interpretation (#86)

* Add Maxmind additional geo properties (#72)

* Implement "matches" and "within" matching functions (#89)

* Pg 51degrees deviceInfoService implementation (#63)

* Add new device targeting categories (#90)

* Pg DeviceAtlas deviceInfoService implementation  (#67)

* Move targeting service to a more proper place in package hierarchy (#92)

* Refine setting deals information (#96)

* Add new attributes for DeviceInfo model and their processing (#93)

*  Implement targeting attributes lookup from request and imp (#98)

* Rubicon deals processing (#91)

* Enable deals processing for Rubicon bidder

* Apply model and processing changes

* Fix to use correct field according to tech spec

* Fix post-merge conflicts

* Apply requested changes

* Complete TargetingService implementation (#100)

* Implement deals-related bids validations (#101)

* Implement deals-related bids validations

* Unit tests for deals related validations

* Implement plan expiration and delivery process clean up - first take (#104)

* Change imp.deals[].ext.line to be object instead of array (#106)

* Add lineItemId parameter to event notification callbacks (#108)

* Gathering line items metrics in transaction log for further aggregation (#109)

* Gather line items metrics over the course of bid request processing

* Refine TxnLog model

* Record transaction log metrics at the end of auction

* Unit tests for Line Item transactional log

* Pull utility method

* Fix sentToBidderAsTopMatch metric semantics

* Call DeviceInfo, GeoLocation and User services in DealsProcessor (#110)

* Add line items related debug information in response in debug and deep debug modes (#116)

* Remove readyAt from readiness calculation (#117)

* Send only relevant deals to bidders present in imp (#115)

* Improve report delilvery process (#113)

* Update report delivery process and line item management

* fix after rebasing

* Refactor updating delivery service with new report

* Fixed metrics gathering after resolving merge conflicts.

* Refactored according to pull request review comments

* Implement periodic registration in the Planner (#118)

* Implement periodic registration in the Planner

* Minor improvements

* Reverted clock related change

* Leave only one instance of Clock

* Integrate targeting service (#119)

* Update report delivery process and line item management

* fix after rebasing

* Refactor updating delivery service with new report

* Fixed metrics gathering after resolving merge conflicts.

* Refactored according to pull request review comments

* Targeting Service integration to LineItemService and LineItemManagementService

* Refactored according to pull request comments

* Make sure deals are ordered according to the readiness of line items (#121)

* Make sure deals are ordered according to the readiness of line items

* Make sure deals are ordered according to the readiness of line items -fix after review

* Update current plan from planner (#122)

* Current plan updating from planner implemented

* Fixed concurrent modification issue

* Actualize planner and delivery stats service protocol (#123)

* Actualize planner protocol

* Actualize delivery stats service protocol - partly

* Actualize delivery stats service protocol - refine accountAuctions in LineItemStatus and add extId

* Tidying up

* Add pg-trx-id header to all interactions with planner and delivery stats service

* Send interim delivery progress report on application shutdown (#124)

* Implemented event tracking in delivery progress report (#127)

* Implemented event tracking in delivery progress report

* Fixed log message

* Call LineItemService in DealsProcessor and update request with deals (#128)

* Flat matched line items by bidder in DealsProcessor (#129)

* LineItemLost metric implemented (#130)

* Deals admin endpoint implemented (#132)

* Deals admin endpoint implemented

* Add exception handler to deals status endpoint handler

* Create integration test for PG flow (#136)

* Create integration test for PG flow

* Fix tests

* Fix NPE in test

* Add lineItemId to VAST event tracking url (#137)

* Set start time date to DeliveryProgress (#134)

* Set start time date to DeliveryProgress

* Fix tests after merge

* Track win event for analytics disabled requests (#139)

* Add lineItem for lostToLineItem metric count (#140)

* Add lineItem for lostToLineItem metric count

* Remove unused import

* Fix and improve DealsTest (#141)

* Fix expectation

* Introduce deals.enabled switch to control PG feature (#142)

* Introduce deals.enabled switch to control PG feature

* Adjust "Overall winner" determination applied to targeting keywords (#143)

* Adjust "Overall winner" determination applied to targeting keywords

* Adjust "Overall winner" determination applied to targeting keywords - fix PR

* Request compressed response from planner (#145)

* Request compressed response from planner

* Rename ReportDeliveryService to DeliveryStatsService

* Update auction lost to lineItem metric (#149)

* Update auction lost to lineItem metric

* Replace ObjectUtils library with native one

* Planner response models update (#146)

* Planner response models update.

Add Price model.

Update tests.

Q:
 frequencyCaps.fcapId (spec don`t have)
 deliverySchedule.endTimeStamp , startTimeStamp  (spec only have this)
 Price, also have currency field.

* Remove lineUpdatedTimeStamp
Remove pre plan*

Remove DeliverySchedule.relativePriority

* Additional logging for interaction with Planner and Delivery stats (#147)

*  Replace Vertx object with mock in some tests to avoid waiting before making assertions (#150)

* Replace Vertx object with mock to avoid temporal assertions - PlannerTest

* Replace Vertx object with mock to avoid temporal assertions - CurrencyConversionServiceTest

* Replace Vertx object with mock to avoid temporal assertions - DeliveryStatsServiceTest

* Replace Vertx object with mock to avoid temporal assertions - LineItemManagementServiceTest

* Make DealsTest more robust

* Rename 'hostname' configuration property to avoid clashing with the env variable set by docker (#151)

* Add support for ssl in httpClient. (#153)

* Change plan response schema and reverse setting Accept-Encoding header (#154)

* Change frequency cap period type spelling to match what Planner provides (#155)

*  Determine AWS Fargate Task Id and pass it to the prebid-server as a HOST_ID env variable (#158)

* Determine AWS Fargate Task Id and pass it to the prebid-server as a HOST_ID env variable

* Fix command

* Add fallback - generate uuid for HOST_ID if it couldn't be determined otherwise

* Implement health indicator calculation for Planner (#156)

* Implement health indicator calculation for Planner

* Implement health indicator calculation for Planner - fix

* Refactor of geo location configuration (#159)

* Refactor of geo location configuration

* Refactor of geo location configuration - fix PR

* Lowercase dealsonly flag according to OpenRTB convention (#164)

* Rename response field ext.debug.deals to ext.debug.pgmetrics (#165)

* Rename response field ext.debug.deals to ext.debug.pgmetrics

* Rename response field ext.debug.deals to ext.debug.pgmetrics - fix PR

* Username and password properties field extraction  (#162)

* extract basic auth logic

* Add integration param in analytics module

* separate the parameters for BasicAuth

* BidderRequester not waiting for request with deals that already came (#157)

* first try

* second try (without mixed, we not change logic at all, bc we need to wait TopDeal (parsed))

* findOut about httpBidderReq

* Refactor last solution

* start fixing tests

* fix after merge

* fix after merge

* Simple version.

We optimize only requests when all imps have topDealId.

* Improve simple method to use makeBids only once.

* Small fix after review

* Refactor to merge deal processing and mixed logic.

* small fix

* Remove unnecessary method

* fix bag when we can lost last response in case of topDealId finish

* fix bag when we can lost last response in case of topDealId finish.

I cant figure out why is there empty event...

* fix bag when we can lost last response in case of topDealId finish.

Add test

* Proceed deals equality for each imp.

And fix bug when not responded requests are skipped in debug.

* minor fix

* fix after review

* fix after review

* PG line items management and tracking progress refactoring (#161)

* Cosmetic changes

* Re-arrange methods to cluster public ones in the beginning of the class

* Move delivery progress tracking code to DeliveryProgress

* Unify line item status creation in DeliveryProgress

* Partially move line item tracking logic into LineItem class

* Move LineItem to more fitting package as it is not model anymore

* Separate LineItem creation and delivery tracking setting

* Separate delivery plan execution tracking from delivery schedule model

* Separate delivery plan execution tracking from delivery schedule model

* Move more logic to LineItem

* Rework logic around advancing to next delivery plan

* Remove redundant methods from LineItemManagementService

* Replace constructors with static factory methods

* Separate delivery progress tracking models from reporting models

* Re-organize delivery progress and report classes

* Got rid of unnecessary MITM methods in LineItemService

* Get rid of Lombok @Delegate

* Fix potential ConcurrentModificationException

* Fix LineItemManagementServiceTest

* Fix DealsProcessorTest

* Fix UserServiceTest

* Fix NotificationEventHandlerTest

* Fix PlannerServiceTest

* Fix DeliveryStatsServiceTest

* Fix DeliveryProgressReportFactoryTest

* Fix LineItemServiceTest

* Fix DealsTest

* Fix checkstyle errors

* Remove unnecessary delivery schedule field setting

* Merge LineItemManagementService into LineItemService

* Replace constructor with static factory method

* Extract DeliveryProgressService out of LineItemService

* Fix PR review issues

* Health monitor increment failed requests (#166)

* Targeting attributes overhaul for adunit.size (#168)

*  Extract premature return decision logic into separate component (#169)

* Take out premature return decision logic into separate class

* Extract premature return decision logic into separate component

* Simulation aware line item service (#171)

* Simulation components implementation

* Removed current date from auctionContext

* Pull request comments were fixed

* Integrate simulation aware components into application configuration (#172)

* Add SimulationAware components and wire them instead of normal ones when in simulation mode

* Remove SimulationAwareDealsProcessor

* Add SimulationAwareHttpBidderRequester

* Add DealsSimulationAdminHandler

* Introduce deals.simulation in default configuration

* Fix simulation configuration

* Add integration test for simulation mode

* Fix formatting

* Wrap numerous configuration properties in data classes (#173)

* Extract components' properties into corresponding data classes

* Move several properties from deals.delivery-stats to deals.delivery-progress and deals.delivery-progress-report

* Add missing @NonNull

* Deals simulation admin handler and integration test implemented (#177)

* Deals simulation admin handler and integration test implemented

* Fixed pull request comments

* Set report start time from first auction request

* Decouple events sending (auction event, win event) from their processing through Vert.x Event Bus (#180)

* Decouple events sending (auction event, win event) from their processing through Vert.x Event Bus

* Unit tests for EventService

* Rename EventService to ApplicationEventService to better reflect its purpose

* Add javadoc to ApplicationEventService

* Adjust delivery progress closing and report sending taking into account resolved race condition

* Extract Initializable interface for components requiring initialization on Vert.x event loop thread

* Rename TransactionLogRecorder to ApplicationEventProcessor (and its methods too) to better reflect its purpose

* Start parameter removed from initial planner request (#181)

*  Minor cleanup, rearrangement and simplification in simulation components (#182)

* Minor cleanup, rearrangement and simplification

* Tweak DealsConfiguration to correctly dispatch SimulationAwareHttpBidderRequester wherever needed without explicit casts

* Simplify method call

* Targeting attributes overhaul for adunit, site and app (#183)

* Fixed deals integration tests (#184)

* Update deals simulation test with targeting (#185)

* Multiple top match from one bider override fix (#179)

* Multiple top match from one bid override fix

```
       "sent_to_bidder_as_top_match": {
          "rubicon": "lineItem1"
        }
```
fixed to
```
        "sent_to_bidder_as_top_match": {
          "rubicon": [
            "lineItem1",
            "lineItem2"
          ]
        }
```

* fix small overhead

* fix deals simulation test json models (#187)

* Fix Deals integration tests (#188)

* Report deliverySchedule date fields name were updated (#189)

* Move endpoints from admin handler to default 8080 (#190)

* Rearrange geo attributes in device and geo extensions to denote a vendor they are retrieved from (#191)

* Fixed date format and content type header in deals-status (#195)

* Targeting attributes overhaul for device.geo.ext (#194)

* Rearrange device attributes in device extension to denote a vendor they are retrieved from (#196)

* Rearrange device attributes in device extension to denote a vendor they are retrieved from

* Use raw device type instead of computed standardized

* Fix NPE while /deals-status in DeliveryProgressReportFactory for dataWindowEndTimeStamp (#197)

* Refactor DeliveryProgressReportFactory for date/time formatting (#199)

* Make matching against targeting attribute values case insensitive (#198)

* LineItem  cpm conversion (#174)

* fix config

* add new fields and update tests

* add new fields and update tests

* LineItem now contains the normalized price.

* small naming fix

* fix after review

* remove toBuilder

* Targeting attributes overhaul for device.ext (#201)

* Hide imp[].pmp.deal[].ext.line.bidder (#202)

* Targeting attributes overhaul for user.ext.time (#203)

* Updated report date format and fixed deals simulation tests (#204)

* Updated report date format and fixed deals simulation tests

* Skip adding device.ext.[vendor] if empty data returned (#206)

* Deals tests were fixed

* Imp will not removed in response if all bidders dealsonly without deals (#193)

* Imp will not removed from response when all bidders dealsonly withoud deals

* Remove DealsProcessor from ExchangeService

* Remove deviceInfo extension from response

* Removed deviceInfo service initialization in DealsTest (#207)

* Make Rubicon bidder to send imp.ext.rp.target.line_item instead of line (#208)

* Added region to win event notification. Renamed frequency cap period to periods. (#209)

* Implement setting user.ext.time.user* fields based on user location and current datetime (#210)

* Implement setting user.ext.time.user* fields based on user location and current datetime

* Fix userdow and userhour targeting attributes evaluation

* Premature response tests (#214)

* Permature tests implemented

* Premature tests responses completed with new user field time

* Insert temporal fields in Premature tests

* Implement user first party data targeting category support (#217)

* Implement user first party data targeting category support (user.ext.data based lookup)

* Implement user first party data targeting category support (user based lookup)

* Implement site first party data targeting category support (#218)

* Fix up bidder param targeting category support (#221)

* Revise user segment targeting category support (#222)

* Delete obsolete targeting categories and TODOs (#223)

* Delete obsolete targeting categories

* Delete obsolete TODOs

* Fixed missing aliases line items (#225)

* Fix LineItemStatus expiration condition (#226)

* Fix LineItemStatus expiration condition

* Log tracer implemented (#220)

* Log tracer implemented

* Fixed pull request comments

* Fixed pull request comment

* Fix getting account from Rubicon alias imp.ext (#227)

* Remove periodType enum as not used (#232)

* Reduce response bids having the same impId per seatbid by top deal (#231)

* Updated trace log message with bidResponse and bidRequest in logs (#230)

* Limit tracer time to five minutes (#235)

* Fix update schedule bugs (#234)

* Fixed delivery schedule udpates

* deals stats fixes

* Deals status and plan update flow bugs are fixed

* Clean up

* Central admin service implemented (#237)

* Filtered line items without relevant plans (#238)

* Add unit tests for not targeting expression (#239)

* Add unit tests for not targeting expression

* Not tests updated to read from json

* Update report with active lineItems without statistic (#244)

* Update report with active lineItems without statistic

* Fix overriding in simulation planner service

* Updated simulation date format sent to services (#246)

* Delivery stats report dates format was changed (#247)

* Update according to sonar report (#240)

* Fixed according to sonar report

* Added more tests

* Update sharethrough with safe date formatter

* Fixed pull request comments

* Check if ready at equal to now (#263)

* Compare readyAt for equals with current time

* Add tests

* Move /tracelog to /pbs-admin/tracelog (#265)

* Fixed unsupported operation exception (#266)

* Pg simulation (#267)

* Dont run plan update timer on simulation mode

* Fix aliases

* Add all types of exceptions to failed response body

* Fixed error message in test

* Catch and log all errors in SimulationAdminHandler

* Add todo to not forget revert workarounds

* Remove not related test from LineItemServiceTest

* Add admin endpoint for pacing information (#268)

* Add admin endpoint for pacing information

* Add configuration switch for /tracelog endpoint (#270)

* Include plans to report when no tokens were spent (#271)

* Added totalSpent field to delivery progress report and fixed overall statistic (#274)

* Update totalSpent field to camel case (#275)

* Add support for pacingDeferred metric (#276)

* Fix sent to client metrics (#278)

* Fixed sentToClient and sentToClientAsTopMatch metrics

* Remove date from test json file

* Removed System.out.println

* Sent Delivery Token in report event if total is zero (#280)

* Added fcap message to trace log (#283)

* Alert proxy service when planner is not available or response is invalid (#282)

* Alert http service initial commit

* Add Unit tests

* Remove deals properties from application.yaml

* Fixed url in test

* Revert checkstyle

* Dont advance plan with fetching from planner in simulation mode (#285)

* Allow to call real bidders while PG simulation (#287)

* Add customized admin enpoints (#286)

* Measure pacing in milliseconds (#291)

* Measure pacing in milliseconds

* Fixed plan was change check

* SentToClientAsTopMatch was merged twice in overall report - fixed (#292)

* Add same restrictions for SentToBidderAsTopMatch line item (#294)

* Cover delivery progress service with tests (#295)

* Add trace log message for removed duplicating top matched line item (#296)

* Add dummy alert config options: system, sub-system, infra, data-center,profile (#302)

* Add admin-endpoints default config properties (#303)

* Add alert-proxy default config properties (#304)

* Fixed plan advance for expired LineItem (#306)

* Add pg related metrics (#305)

* Add pg related metrics

* Update unit tests and metrics documentation

* Move PG metrics to its own group

* Update documentation

* Update metrics name in documentation

* Fix duplicated plans in report (#307)

* Fix checkstyle warnings

* Fix targeting matching for device.ext and device.geo.ext attributes (#308)

* Test for issue

* Fix targeting matching for device.ext and device.geo.ext attributes

* Fix deprecation warnings

* Move planner and delivery communication to scheduler (#312)

* Initial commit without fixed tests

* Fixed Deals test

* Fixed spring cron expressions

* Fixed checkstyle warnings

* Removed PgEventScheduler

* Change planner default fetch period to 1 minute

* Update hb_deal value with bidder code (#319)

* Update hb_deal value with bidder code

* Updated hb_deal with bidder from line item source for openrtb

* Normalize ortb fields and update default config (#318)

* Normalize ortb fields and update default config

* Move segmentids from user.ext.data to user.data

* Updated user.data to correct format

* Remove config change

* Remove credentials from config

* Fixed application tests

* Fixed pbs Critical error response when plan does not have unspent tokens (#326)

* Fixed NullPointer when UserAgent was not defined. (#329)

* Document pg endpoints and configs (#335)

* Deals status, lineItem status and tracelog documentation was added.

* Update deals.md and config-app.md documentations.

* Added CachedHttpCalls for UserService to response ext.debug (#337)

* Added CachedHttpCalls for UserService to response ext.debug

* Remove redundant exception class

* Send win event in sim mode (#342)

* Enabled User service API for sim mode

* Set planner retry number to max value

* Update application.yaml with new simulation config format

* Fixed configuration issues

* Fix NPE during ExtHttpCalls creation (#346)

* Make delivery stats report sending sequential (#345)

* Update register endpoint with admin support (#328)

* Move register logic to its own class

* Update RegisterService and AdminCentralService to support new services

* Updated with unit tests

* Revert back checkstyle.xml

* Updated from State to Status in Register request

* Updated after review

* Retry failed planner once (#349)

* Update retry flow for Planner

* make handleInitialization private

* Handle inactive lineitems (#354)

* handle inactive lineItems

* Removed lineitems without plans from reports.

* Updated with unit test

* Update log message

* Fixed style error

* Fixed tests after log update

* Update alerts frequency (#357)

* Alert service attempts period implemented

* Update logs

* Fixed review comments

* Update alertTypes to ConcurrentHashMap

* Remove failureAlertPeriod

* Update config documentations

* Add deals stats to register (#359)

* Add deals-status to register request and update admin bean creation conditions.

* Remove default values for properties

* Include all active lineitems in deals status overall report

* Advance line items plans by cron expression (#369)

* Invalidate lineitems from register response. (#372)

* Invalidate lineitems from register response.

* Merge tag 1.33.0 759 (#384)

* Fix AppnexusVideoTest to use cache-response-transformer (#632)

* Prebid Server prepare release 1.30.0

* Prebid Server prepare for next development iteration

* Store win url in cache (#631)

* Fix org.prebid.server.it.ApplicationTest.ampShouldReturnTargeting test

* Enable Brightroll bidder in configuration (#314)

* Enable Facebook bidder in configuration (#317)

* Brightroll - Initial changes to add  businessinsider (#639)

* Revert enabling Facebook bidder in configuration (#321)

* Revert "Brightroll - Initial changes to add  businessinsider (#639)" (#641)

This reverts commit 00bdd50afd1e3e19c978fe03d926979775ae276d.

* Merging tcf-v2 branch into master-rubicon-tcf with basic P1 enforcement

* Prebid Server prepare release 1.31.0

* Prebid Server prepare for next development iteration

* Merging tcf-v2 branch into master-rubicon-tcf with basic P1 enforcement (#327)

* Set version to 1.32.0-SNAPSHOT with TCF 2.0 Phase 1 support

* Fix TcfDefinerService to use Purpose 1 for validation

* Fix Tcf string parsing error to don't fail the entire request (#333)

# Conflicts:
#	src/test/java/org/prebid/server/privacy/gdpr/TcfDefinerServiceTest.java

* Change `vendorException` now exclude vendors for all checks (#336)

* Fix getting allowed vendors from consent string in TCF 2.0 (#338)

* Fix after clarification of PRD (#339)

* Fix after clarification

* Added new type which used when purpose is not enforced.

Change behavior to.
```
purpose-enforce = true (!no) -> check for Purpose or PurposeLI
vendor-enforce = true -> check for Vendor or VendorLI

```

* fix

* Add mapping of names by vendorIds (#343)

* Add mapping of names by vendorIds

Tcf2 use mapping to find bidderNames to exclude this in the future.

* fix map

* fix comment

* fix comment

* Add Tcf V2 phase one (#651)

* First pass on TCFv2 service external interface and models

* Switch to using TCF service instead of GDPR service in CookieSyncHandler (#310)

* Switch to using TCF service instead of GDPR service in CookieSyncHandler

* Rename method

* Add base architecture for TCF 2.0 (#311)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* fix

* Remove JsonFormat

* Add TCF2 impl to setuid (#316)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add SetuidHandler impl

* Fix

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* small change

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* refactor required values

* fix

* Remove JsonFormat

* Fix after merge

* Refactor CookieSyncHandler and SetuidHandler for TCF 2.0 (#323)

* Fix TcfDefinerService to use Purpose 1 for validation (#330)

* Fix Tcf string parsing error to don't fail the entire request (#333)

* Change `vendorException` now exclude vendors for all checks (#336)

* Fix getting allowed vendors from consent string in TCF 2.0 (#338)

* Fix after clarification

* Revert "Fix after clarification"

This reverts commit 1620c970

* Fix after clarification of PRD (#339)

* Fix after clarification

* Added new type which used when purpose is not enforced.

Change behavior to.
```
purpose-enforce = true (!no) -> check for Purpose or PurposeLI
vendor-enforce = true -> check for Vendor or VendorLI

```

* fix

* Add mapping of names by vendorIds (#343)

* Add mapping of names by vendorIds

Tcf2 use mapping to find bidderNames to exclude this in the future.

* fix map

* fix comment

* fix comment

Co-authored-by: Sergii Chernysh <[email protected]>
Co-authored-by: Dmytro <[email protected]>
Co-authored-by: Ruslan Panchyk <[email protected]>

* Prebid Server prepare release 1.32.0

* Prebid Server prepare for next development iteration

* Apply checkstyle rules to test code (#624)

* Add JsonIgnore annotation to AuctionContext fields (#617)

* Change app-media-types in visx.yaml of Visx bidder (#625)

* Test timeout increase to 5000ms (#633)

* Add timestamp and biddercode to event URL (#635)

* Change currency rates location (#620)

* Update usersync url of Conversant Bidder (#646)

* Add siteId and zoneId to Event (#347)

* AdminManager class added (#636)

* Add Brightroll adapter config (#649)

* Logging Event body on debug enabled (#332)

* Logging event body added on debug enabled

* change request

* change request

* Enable CCPA in configuration (#659)

* Fix auctiontimestamp placement (#662)

* Expand account resolution logic (#356)

* Update emit event logic (#654)

* Code cleanup for core services (#665)

* Enable Consumable, Yieldmo and TripleLift bidders (#362)

*  Add protected endpoint to invalidate account from cache (#661)

* Update config parameter of IxBidder (#666)

* Add account support for CookieSync and Setuid (#658)

* Update IX bidder endpoint to use HTTP instead of HTTPS (#667)

* Update analytics adapter to pass timestamp (#368)

* Update analytics adapter with timestamp parameter

* Refactoring code and tests

* Move setting default value

* Add EventsContext to CacheService instead of auctionTimeout (#669)

* Add integration type to event tracking strings (#371)

* Fix getting integration type for events in BidResponseCreator (#373)

* Update Ix bidder endpoint (#672)

* Add integration to cache bid wurl (#374)

* Add retrieving of stored request id from url param tag_id for AMP (#375)

* Add retrieving of stored request id from url param tag_id for AMP

* Fix codeStyle

* Brightroll bidder- BusinessInsider changes (#660)

* Brightroll - business insider changes

* addressed review comments

* updated testcase

* updated testcase

* fixing alignment

* updated badv list for business insider

* Ignore App/Site publisher ID if not a valid number (#378)

* apply deals changes

* Update win url with line item id (#386)

* Group trace debug by lineItemId in response (#387)

* Remove line item id from cache win url (#388)

* Always update response PG bid response with events when account events enabled (#390)

* Update plan advancing with logs (#391)

* Set readyAt to current time when active plan replaced or updated (#394)

* Set readyAt to current time when active plan replaced or updated

* Fix typo in log

* Compare plans by token set instead updatedTimeStamp (#396)

* Revert pacing changes (#399)

* Revert "Set readyAt to current time when active plan replaced or updated (#394)"

This reverts commit e103dbb0b671af0a238802e450c49299a5bd38cd.

* Update logs and remove deals/lineitems from response trace when empty

* Log deferred metrics for plans with no tokens left (#400)

* Fix code smells (#403)

* Update report with dealId (#405)

* Update geo and DE with latency metrics (#406)

* Update geo and DE with latency metrics

* Move and rename geolocation request time metric

* Randomize LineItem Sorting when other fields are equal (#409)

* Update user service with metrics (#408)

* Update user service with metrics

* Fix typo

* Update register response with invalidate account cache command (#410)

* Update register response with invalidate account cache command

* Pull request comments fixed

* Fix random distribution (#416)

* Update pg logs (#417)

* Merge 1.35.1 840 (#425)

* Fix AppnexusVideoTest to use cache-response-transformer (#632)

* Prebid Server prepare release 1.30.0

* Prebid Server prepare for next development iteration

* Store win url in cache (#631)

* Fix org.prebid.server.it.ApplicationTest.ampShouldReturnTargeting test

* Enable Brightroll bidder in configuration (#314)

* Enable Facebook bidder in configuration (#317)

* Brightroll - Initial changes to add  businessinsider (#639)

* Revert enabling Facebook bidder in configuration (#321)

* Revert "Brightroll - Initial changes to add  businessinsider (#639)" (#641)

This reverts commit 00bdd50afd1e3e19c978fe03d926979775ae276d.

* Merging tcf-v2 branch into master-rubicon-tcf with basic P1 enforcement

* Prebid Server prepare release 1.31.0

* Prebid Server prepare for next development iteration

* Merging tcf-v2 branch into master-rubicon-tcf with basic P1 enforcement (#327)

* Set version to 1.32.0-SNAPSHOT with TCF 2.0 Phase 1 support

* Fix TcfDefinerService to use Purpose 1 for validation

* Fix Tcf string parsing error to don't fail the entire request (#333)

# Conflicts:
#	src/test/java/org/prebid/server/privacy/gdpr/TcfDefinerServiceTest.java

* Change `vendorException` now exclude vendors for all checks (#336)

* Fix getting allowed vendors from consent string in TCF 2.0 (#338)

* Fix after clarification of PRD (#339)

* Fix after clarification

* Added new type which used when purpose is not enforced.

Change behavior to.
```
purpose-enforce = true (!no) -> check for Purpose or PurposeLI
vendor-enforce = true -> check for Vendor or VendorLI

```

* fix

* Add mapping of names by vendorIds (#343)

* Add mapping of names by vendorIds

Tcf2 use mapping to find bidderNames to exclude this in the future.

* fix map

* fix comment

* fix comment

* Add Tcf V2 phase one (#651)

* First pass on TCFv2 service external interface and models

* Switch to using TCF service instead of GDPR service in CookieSyncHandler (#310)

* Switch to using TCF service instead of GDPR service in CookieSyncHandler

* Rename method

* Add base architecture for TCF 2.0 (#311)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* fix

* Remove JsonFormat

* Add TCF2 impl to setuid (#316)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add SetuidHandler impl

* Fix

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* small change

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* refactor required values

* fix

* Remove JsonFormat

* Fix after merge

* Refactor CookieSyncHandler and SetuidHandler for TCF 2.0 (#323)

* Fix TcfDefinerService to use Purpose 1 for validation (#330)

* Fix Tcf string parsing error to don't fail the entire request (#333)

* Change `vendorException` now exclude vendors for all checks (#336)

* Fix getting allowed vendors from consent string in TCF 2.0 (#338)

* Fix after clarification

* Revert "Fix after clarification"

This reverts commit 1620c970

* Fix after clarification of PRD (#339)

* Fix after clarification

* Added new type which used when purpose is not enforced.

Change behavior to.
```
purpose-enforce = true (!no) -> check for Purpose or PurposeLI
vendor-enforce = true -> check for Vendor or VendorLI

```

* fix

* Add mapping of names by vendorIds (#343)

* Add mapping of names by vendorIds

Tcf2 use mapping to find bidderNames to exclude this in the future.

* fix map

* fix comment

* fix comment

Co-authored-by: Sergii Chernysh <[email protected]>
Co-authored-by: Dmytro <[email protected]>
Co-authored-by: Ruslan Panchyk <[email protected]>

* Prebid Server prepare release 1.32.0

* Prebid Server prepare for next development iteration

* Apply checkstyle rules to test code (#624)

* Add JsonIgnore annotation to AuctionContext fields (#617)

* Change app-media-types in visx.yaml of Visx bidder (#625)

* Test timeout increase to 5000ms (#633)

* Add timestamp and biddercode to event URL (#635)

* Change currency rates location (#620)

* Update usersync url of Conversant Bidder (#646)

* Add siteId and zoneId to Event (#347)

* AdminManager class added (#636)

* Add Brightroll adapter config (#649)

* Logging Event body on debug enabled (#332)

* Logging event body added on debug enabled

* change request

* change request

* Enable CCPA in configuration (#659)

* Fix auctiontimestamp placement (#662)

* Expand account resolution logic (#356)

* Update emit event logic (#654)

* Code cleanup for core services (#665)

* Enable Consumable, Yieldmo and TripleLift bidders (#362)

*  Add protected endpoint to invalidate account from cache (#661)

* Update config parameter of IxBidder (#666)

* Add account support for CookieSync and Setuid (#658)

* Update IX bidder endpoint to use HTTP instead of HTTPS (#667)

* Update analytics adapter to pass timestamp (#368)

* Update analytics adapter with timestamp parameter

* Refactoring code and tests

* Move setting default value

* Add EventsContext to CacheService instead of auctionTimeout (#669)

* Add integration type to event tracking strings (#371)

* Fix getting integration type for events in BidResponseCreator (#373)

* Update Ix bidder endpoint (#672)

* Add integration to cache bid wurl (#374)

* Add retrieving of stored request id from url param tag_id for AMP (#375)

* Add retrieving of stored request id from url param tag_id for AMP

* Fix codeStyle

* Brightroll bidder- BusinessInsider changes (#660)

* Brightroll - business insider changes

* addressed review comments

* updated testcase

* updated testcase

* fixing alignment

* updated badv list for business insider

* Ignore App/Site publisher ID if not a valid number (#378)

* Prebid Server prepare release 1.33.0

* Prebid Server prepare for next development iteration

* Add TCF v2 Phase 2 (#681)

* First pass on TCFv2 service external interface and models

* Switch to using TCF service instead of GDPR service in CookieSyncHandler (#310)

* Switch to using TCF service instead of GDPR service in CookieSyncHandler

* Rename method

* Add base architecture for TCF 2.0 (#311)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* fix

* Remove JsonFormat

* Add TCF2 impl to setuid (#316)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add SetuidHandler impl

* Fix

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* small change

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* refactor required values

* fix

* Remove JsonFormat

* Fix after merge

* Refactor CookieSyncHandler and SetuidHandler for TCF 2.0 (#323)

* Pass vendorsIds for aliases from bidRequest to PrivacyEnforcementService (#324)

* Fix TcfDefinerService to use Purpose 1 for validation (#330)

* Fix Tcf string parsing error to don't fail the entire request (#333)

* Change `vendorException` now exclude vendors for all checks (#336)

* Fix getting allowed vendors from consent string in TCF 2.0 (#338)

* Tcf v2 auction (#331)

* Pass vendorsIds for aliases from bidRequest to PrivacyEnforcementService

* Add Tcf v2 for auction

* Fix tests.

Does Gdpr 1.0 need to skip bidder request or analytics ?

* Small fixes

* Fix after review

Add `resultFor` method.
Remove comments.
Rename variables.
Fix tests

* rename variables in test

* Add vendorListService impl for TCF 2.0 (#322)

* Add vendorListService impl for TCF 2.0

I tried to abstract from previous solution.
They changed List<Vendor> to Map<integer, Vendor> =(

* fix after review.

* fix after review.

* fix after review.

* Fix after clarification

* Revert "Fix after clarification"

This reverts commit 1620c970

* Fix after clarification of PRD (#339)

* Fix after clarification

* Added new type which used when purpose is not enforced.

Change behavior to.
```
purpose-enforce = true (!no) -> check for Purpose or PurposeLI
vendor-enforce = true -> check for Vendor or VendorLI

```

* fix

* Add files from gdpr-vendorlist

* Add files from gdpr-vendorlist

* Implement auction, cookie sync and user sync TCF metrics submission (#340)

* Update cookie sync TCF metric

* Update user sync TCF metric

* Extract TcfMetrics class

* Blank for adapter TCF metrics

* Blank for adapter TCF metrics

* Implement auction TCF metrics submission

* Add support for account TCF configuration in DB (#341)

* Test case for account TCF configuration support in FileApplicationSettings

* Add support for account TCF configuration in JdbcApplicationSettings

* Adjust to change TCF config model

* Change vendorExceptions type to list of strings

* Create BidderAliases object to bear configured aliases information (#344)

* Add 2,4,7 purpose

* Revert "Add 2,4,7 purpose"

This reverts commit b4ebfa6f

* Tcf v2 p2 renaming (#348)

* Renamed packages and PurposeTypeStrategy to EnforcePurposeStrategy

package org.prebid.server.privacy.gdpr.tcf2stratgies; -> tcfstratgies
package org.prebid.server.privacy.gdpr.tcfstratgies.typeStrategies; -> typestrategies
PurposeTypeStrategy to EnforcePurposeStrategy

* Merge branch 'tcf-v2-phase2-merged' into tcf-v2-phase2

# Conflicts:
#	src/test/java/org/prebid/server/privacy/gdpr/tcf2stratgies/PurposeOneStrategyTest.java

* Rename base to basic for enforcement strategy (#350)

* Rename base to basic for enforcement strategy

* Remove necessary field

* Add 3,4,7 purpose (#351)

* Add 3,4,7 purpose

* Primitive

* Get rid of unused or duplicated functionality from GdprService (#352)

* Get rid of unused or duplicated functionality from GdprService

* Add VendorListServiceV2 to Tcf2Service (#353)

* Add VendorListServiceV2 to Tcf2Service

* Remove vendorV2 from PrivacyEnforcement

* Implement purpose one treatment interpretation option handling (#355)

* Implement purposed one treatment interpretation option handling

* Fix condition on applying purpose 1 treatment interpretation

* Introduce specialized FutureAssertion and use it in TCF tests (#358)

* Introduce specialized FutureAssertion and use it in TCF tests

* Make FutureAssertion interface neater

* Add Full Enforcement type (#360)

* Add Full Enforcement type

Basic description:
```
RestrictionType = NOT_ALLOWED  - return empty list (even for excluded)

 /**
     * Purpose is flexible when {@link VendorV2} flexiblePurposes contains it id.
     * When it is not flexible:
     * We check purposeConsent and vendorConsent when it is contained in GVL purposes;
     * We check purposesLITransparency and vendorLegitimateInterest when it is contained in GVL LegIntPurposes.
     * <p>
     * If it flexible and it is contained in GVL purposes we check by {@link RestrictionType}:
     * For REQUIRE_CONSENT we check by purposeConsent and vendorConsent or
     * purposesLITransparency and vendorLegitimateInterest
     * For REQUIRE_LEGITIMATE_INTEREST we check by purposesLITransparency and vendorLegitimateInterest
     * <p>
     * If it flexible and it is contained in GVL purposes we check by {@link RestrictionType}:
     * For REQUIRE_CONSENT we check by purposeConsent and vendorConsent
     * For REQUIRE_LEGITIMATE_INTEREST we check by purposeConsent and vendorConsent or
     * purposesLITransparency and vendorLegitimateInterest
     */
```

```
NOT_ALLOVED = 0

        not_flexible_GVL_P = P_C + V_C
        not_flexible_GVL_LIP = P_LI + V_LI

        flexible_GVL_LIP =
                UNDEFINED = P_C + V_C || P_LI + V_LI
                REQUIRE_LEGITIMATE_INTEREST  =  P_C + V_C || P_LI + V_LI
                REQUIRE_CONSENT =  P_C + V_C

        flexible_GVL_P =
               REQUIRE_LEGITIMATE_INTEREST =  P_LI + V_LI
               UNDEFINED = P_C + V_C || P_LI + V_LI
               REQUIRE_CONSENT = P_C + V_C || P_LI + V_LI
```

* Update logic according to new spec.

* Add empty list to Empty model

* Tcf v2 dataflow refactoring (#379)

* Tidy up

* WIP refactoring

* WIP refactoring

* WIP refactoring

* WIP refactoring

* WIP refactoring

* Fix tests

* Add all purpoces to workflow for Tcf v2  (#380)

* Add processing of all supported purposes

Co-authored-by: Sergii Chernysh <[email protected]>
Co-authored-by: Dmytro <[email protected]>
Co-authored-by: Ruslan Panchyk <[email protected]>

* Add Yieldone bidder (#680)

* Add Special Features for TCF 2 (#683)

Tcf2Service encapsulates strategies.
PBS implements only 1 Special feature which restricts geo and ip.

* Brightroll - cookie sync config update (#682)

* Brightroll - cookie sync config update (#682)

(cherry picked from commit 9c05e0d6086475b26ef94928b3bb370add635395)

* Update version to 1.33.1

* Rubicon Rewarded Video boolean bug fix (#622)

* Update auction docs with Rewarded video section (#675)

* Rubicon Rewarded Video boolean bug fix (#622)

(cherry picked from commit e0f4de80a8ec0bd44f6a67e8e172d7c067d9ce49)

* Update auction docs with Rewarded video section (#675)

(cherry picked from commit c8ad498a5e4da417b3fef4938259528b4fffc281)

* Fix Rubicon Analytics to log bidPriceUSD correctly (#385)

* Add downgrading to basic when GVL list failed (#684)

* Add downgrading to basic when GVL list failed

When GVL call failed we downgrade type to basic.

I tried several variants for simplification of `permissionsForInternal`
* You can't create straight forward workflow (
```
.map (p)
.otherwise/recover (downgraded p)
.map` bc in this case you'll be handling exception that can be occurred in map (p).

`setHandle` also bad option, bc it will rethrow failed future.

Finally, I discovered that compose have overloaded method that can handle both use cases.

Also replace lambda with pre creation of VendorPermission. (But now we create more objects)

* fix style

* Add masking of device attributes for Purpose 4 (#689)

* Additional privacy metrics - COPPA, CCPA, TCF (#691)

* Merge TCF 2.0 Phase 2 from master into master-rubicon (#382)

* Prebid Server prepare release 1.33.0

* Prebid Server prepare for next development iteration

* Add TCF v2 Phase 2 (#681)

* First pass on TCFv2 service external interface and models

* Switch to using TCF service instead of GDPR service in CookieSyncHandler (#310)

* Switch to using TCF service instead of GDPR service in CookieSyncHandler

* Rename method

* Add base architecture for TCF 2.0 (#311)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* fix

* Remove JsonFormat

* Add TCF2 impl to setuid (#316)

* Add tcf configuration

We have same model for account (.yaml) and default config (.properties).
Additional beans was added to create ConfigurationProperties class with multiple classes. (does not work without it)

account .yaml example:
```
accounts:
  - id: 1001
    gdpr:
      host-vendor-id: asd
      enabled: true
      default-value: false
      purpose-one-treatment-interpretation: false
      purposes:
        p1:
          enforce-purpose: no
          enforce-vendors: true
          vendor-exceptions:
            - 1231
        p2:
          enforce-purpose: full
          enforce-vendors: false
          vendor-exceptions:
            - 101
      special-features:
        sf2:
          enforce: true
          vendor-exceptions:
            - 1001
```

.properties example:
```
gdpr.purposes.p1.enforce-purpose= no
gdpr.purposes.p1.enforce-vendors= true
gdpr.purposes.p1.vendor-exceptions[0]= 123
gdpr.purposes.p1.vendor-exceptions[1]= 23
gdpr.purposes.p2.enforce-purpose= base
gdpr.purposes.p2.enforce-vendors= false
gdpr.purposes.p2.vendor-exceptions[0]= 222
gdpr.special-features.sf1.enforce= true
gdpr.special-features.sf1.vendor-exceptions=
```

* small fix

* small fix

* Add basic implementation

* Change interfce

* Finish tcf first try

* fix allowance

* fix

* Fix types and change method name

* Fix types and change method name

* Add SetuidHandler impl

* Fix

* Add TODOs
Fix bug with purpose + vendor allow
Add Gdpr fix for processing bidderNames

* Fix Tests

* Add tests.
Remove eqAndHashCode exclusion from VendorPermission
Fix bugs

* small change

* Fix after review

* Change default configuration for tcf2

* small fix

* refactor required values

* refactor required values

* fix

* Remove JsonFormat

* Fix after merge

* Refactor CookieSyncHandler and SetuidHandler for TCF 2.0 (#323)

* Pass vendorsIds for aliases from bidRequest to PrivacyEnforcementService (#324)

* Fix TcfDefinerService to use Purpose 1 for validation (#330)

* Fix Tcf string parsing error to don't fail the entire request (#333)

* Change `vendorException` now exclude vendors for all checks (#336)

* Fix getting allowed vendors from consent string in TCF 2.0 (#338)

* Tcf v2 auction (#331)

* Pass vendorsIds for aliases from bidRequest to PrivacyEnforcementService

* Add Tcf v2 for auction

* Fix tests.

Does Gdpr 1.0 need to skip bidder request or analytics ?

* Small fixes

* Fix after review

Add `resultFor` method.
Remove comments.
Rename variables.
Fix tests

* rename variables in test

* Add vendorListService impl for TCF 2.0 (#322)

* Add vendorListService impl for TCF 2.0

I tried to abstract from previous solution.
They changed List<Vendor> to Map<integer, Vendor> =(

* fix after review.

* fix after review.

* fix after review.

* Fix after clarification

* Revert "Fix after clarification"

This reverts commit 1620c970

* Fix after clarification of PRD (#339)

* Fix after clarification

* Added new type which used when purpose is not enforced.

Change behavior to.
```
purpose-enforce = true (!no) -> check for Purpose or PurposeLI
vendor-enforce = true -> check for Vendor or VendorLI

```

* fix

* Add files from gdpr-vendorlist

* Add files from gdpr-vendorlist

* Implement auction, cookie sync and user sync TCF metrics submission (#340)

* Update cookie sync TCF metric

* Update user sync TCF metric

* Extract TcfMetrics class

* Blank for adapter TCF metrics

* Blank for adapter TCF metrics

* Implement auction TCF metrics submission

* Add support for account TCF configuration in DB (#341)

* Test case for account TCF configuration support in FileApplicationSettings

* Add support for account TCF configuration in JdbcApplicationSettings

* Adjust to change TCF config model

* Change vendorExceptions type to list of strings

* Create BidderAliases object to bear configured aliases information (#344)

* Add 2,4,7 purpose

* Revert "Add 2,4,7 purpose"

This reverts commit b4ebfa6f

* Tcf v2 p2 renaming (#348)

* Renamed packages and PurposeTypeStrategy to EnforcePurposeStrategy

package org.prebid.server.privacy.gdpr.tcf2stratgies; -> tcfstratgies
package org.prebid.server.privacy.gdpr.tcfstratgies.typeStrategies; -> typestrategies
PurposeTypeStrategy to EnforcePurposeStrategy

* Merge branch 'tcf-v2-phase2-merged' into tcf-v2-phase2

# Conflicts:
#	src/test/java/org/prebid/server/privacy/gdpr/tcf2stratgies/PurposeOneStrategyTest.java

* Rename base to basic for enforcement strategy (#350)

* Rename base to basic for enforcement strategy

* Remove necessary field

* Add 3,4,7 purpose (#351)

* Add 3,4,7 purpose

* Primitive

* Get rid of unused or duplicated functionality from GdprService (#352)

* Get rid of unused or duplicated functionality from GdprService

* Add VendorListServiceV2 to Tcf2Service (#353)

* Add VendorListServiceV2 to Tcf2Service

* Remove vendorV2 from PrivacyEnforcement

* Implement purpose one treatment interpretation option handling (#355)

* Implement purposed one treatment interpretation option handling

* Fix condition on applying purpose 1 treatment interpretation

* Introduce specialized FutureAssertion and use it in TCF tests (#358)

* Introduce specialized FutureAssertion and use it in TCF tests

* Make FutureAssertion interface neater

* Add Full Enforcement type (#360)

* Add Full Enforcement type

Basic description:
```
RestrictionType = NOT_ALLOWED  - return empty list (even for excluded)

 /**
     * Purpose is flexible when {@link VendorV2} flexiblePurposes contains it id.
     * When it is not flexible:
     * We check purposeConsent and vendorConsent when it is contained in GVL purposes;
     * We check purposesLITransparency and vendorLegitimateInterest when it is contained in GVL LegIntPurposes.
     * <p>
     * If it flexible and it is contained in GVL purposes we check by {@link RestrictionType}:
     * For REQUIRE_CONSENT we check by purposeConsent and vendorConsent or
     * purposesLITransparency and vendorLegitimateInterest
     * For REQUIRE_LEGITIMATE_INTEREST we check by purposesLITransparency and vendorLegitimateInterest
     * <p>
     * If it flexible and it is contained in GVL purposes we check by {@link RestrictionType}:
     * For REQUIRE_CONSENT we check by purposeConsent and vendorConsent
     * For REQUIRE_LEGITIMATE_INTEREST we check by purposeConsent and vendorConsent or
     * purposesLITransparency and vendorLegitimateInterest
     */
```

```
NOT_ALLOVED = 0

        not_flexible_GVL_P = P_C + V_C
        not_flexible_GVL_LIP = P_LI + V_LI

        flexible_GVL_LIP =
                UNDEFINED = P_C + V_C || P_LI + V_LI
                REQUIRE_LEGITIMATE_INTEREST  =  P_C…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@schernysh schernysh schernysh approved these changes

@rpanchyk rpanchyk Awaiting requested review from rpanchyk

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@BraslavskiyAndrey @schernysh