GDPR consentManagement module

[jsnellbaker]

DO NOT MERGE THIS PR IMMEDIATELY. Please consult with @mkendall07 first.

Type of change

• Feature

Description of change

This change includes:

• a new module to support the fetching (via a separately implemented Consent Management Platform) and providing user consent information for the EU GDPR changes.
• changes to the appnexusBidAdapter, prebidServerBidAdapter and adaptermanager classes to utilize this new module.
• small change to the pre1api module
• sets of new unit tests to test each of the modified areas

I have organized the overall changes for the first three items listed above into their own sections.

GDPR consentManagement Module

This module works in the following high level steps:

1. Core prebid code reads the config information from the prebid config code (ie pbjs.setConfig()) and initializes the module. This initialization sets some config variables for the module and adds a hook to the $$PREBID_GLOBAL$$.requestBids function in the prebid.js file.
2. Once the prebid code loads the pbjs.requestBids function, the hook is executed first (ie prior to the auction properly starting) - which starts the rest of the module.
3. The module reaches out to the CMP via its framework and attempts to fetch the user's consent information (an encoded string value). Note - if the prebid code is nested inside an iframe, we will use an alternate approach involving postMessage to access the CMP framework that's outside the iframe. This is handled automatically by the module's code.
4. Once the string is found and verified valid, it gets added to a consent object (named gdprConsent) which gets stored into a gdprDataHandler object (imported from adaptermanager.js). After the value is stored, the hook ends and returns back to the original pbjs.requestBids function so it can run as normal.
5. This new gdprConsent object (sample show further below) eventually gets added to the bidRequest object in the adaptermanager.js file's makeBidRequest function so that it's made available to any adapter (standard or server-side) to read the consent information and pass it along to their system. It uses the gdprDataHandler to read the stored value and insert it into the bidRequest object.

A few additional notes to the above process:

• If the module is not included as part of the setConfig(), it is effectively disabled and won't create any of the additional gdprConsent data object in the adUnits or bidRequest auction objects.
• There is a list of supported CMPs that work with this module. At the time of submission, only the IAB CMP is officially supported and integrated with the module. Further documentation on this CMP can be found in a link at the bottom of this PR. The module rejects any unknown CMPs and resumes the auction as normal.
• There is an interaction between the CMP and the module that needs to be completed in order for the module to perform its task. There is currently a time cost attributed to this process; so we've implemented a timeout as part of the CMP piece of the module. If the timeout is reached, the module effectively aborts what it's trying to do and exits in one of two configured ways.
• The site developer has an option (part of the setConfig() for the module) to either: cancel the auction outright if the CMP does not provide a valid consent string in the allotted time or allow the auction to proceed without the consent information (specifically the module sets an undefined value for the user's consentString attribute in the gdprConsent object). This decision is controlled through the allowAuctionWithoutConsent option in the config; true means the auction will go through and false means the auction will be canceled entirely.
• The same exit functionality also occurs if the string returned by CMP is not valid.
• As part of the gdprConsent object that gets added to the bidRequest object, there is a second value called consentRequired. This boolean value has no default value and is only populated in one of two ways. Either the publisher would set an override value in the config part of the module, or they would leave it up to the adapters to set their appropriate value (as some of them may have the ability to derive the ideal value for the end-user).

Samples of objects used/produced by the module
Available options for the consentManagement module in the setConfig():

consentManagement: {
              cmp: 'appnexus',
              consentRequired: true,
              timeout: 1000,
              allowAuctionWithoutConsent: true
            }

The gdprConsent object in the bidRequest object(s):

gdprConsent: {
        consentString: 'BOKAVy4OKAVy4ABAB8AAAAAZ+A==',
        consentRequired: true
}

Adapter changes

As part of this PR, I've also implemented the changes for the appnexusBidAdapter and prebidServerBidAdapter files to read the gdprConsent information and pass it along to their respective systems in the proper manner. Details on these implementations is summarized below:

appnexusBidAdapter
Per the current planned implementation approach for the /ut handler, we're storing the consent data in the data field of the POST request that the adapter executes as part of it's buildRequests function. The consent information is stored under a top level object similar in structure to how it's presented in the original bidRequest object. Below is a sample of how it looks (note the syntax difference of the object keys):

gdpr_consent: {
        consent_string: 'BOKAVy4OKAVy4ABAB8AAAAAZ+A==',
        consent_required: true
}

prebidServerBidAdapter
As per the IAB OpenRTB GDPR document (link at bottom), the consent information is stored under two different ORTB2.5 objects in its bidRequest object:

• user.ext.consent for the consentString
• regs.ext.gdpr for the consentReqired boolean.

Note the ORTB spec states to pass either a 1 or 0 to represent the true or false settings (respectively). There is no backup value automatically set by the adapter when consentRequired is undefined. This undefined value is merely carried through in the ORTB request.

The prebid GDPR module does not support the earlier sub-versions of the ORTB2 spec (which would pass the consent data in different locations in the request object).

pre1api change

Also included as part of this PR is a small change to lower the priority of the shared requestBids hook used by the pre1api module.

This change was recommended by @snapwich to allow the different modules share the same hook effectively. My testing around the change seems to show the modules are working fine when they're running together. This testing also included the PubCommonId module (which also shares the same hook on the $$PREBID_GLOBAL$$.requestBids function).

• A link to a PR on the docs repo at https://github.com/prebid/prebid.github.io/
  gdpr consentManagement module docs prebid.github.io#645

Other information

Information on AppNexus CMP framework:
http://acdn.adnxs.com/cmp/docs/#/

Information on IAB CMP framework:
https://github.com/InteractiveAdvertisingBureau/GDPR-Transparency-and-Consent-Framework

IAB OpenRTB GDPR document:
https://iabtechlab.com/wp-content/uploads/2018/02/OpenRTB_Advisory_GDPR_2018-02.pdf

[mkendall07]

@snapwich
@bretg
For review

[mkendall07]

we should probably call this appnexus and not iab

[jsnellbaker]

updated references of iab to appnexus

[mkendall07]

should we be using $$PREBID_GLOBAL$$ here?

[jsnellbaker]

I removed the $$PREBID_GLOBAL$$ fall back. I found this type of setup from another module that was hooked on the requestBids function and wanted to keep consistency.

[jsnellbaker]

update I need to have this value in place to actually grab the adUnits list object.

The config attribute that's included in the requestBids has a place for adUnits, but it's dependent on it being passed specifically in the prebid config when the requestBids is invoked. Currently only the bidsBackHandler is passed along with requestBids in most of our example docs, so I expect the adUnits is going to be undefined at this stage of the auction process.

@mkendall07 Please let me know if there are any strong concerns about relying on $$PREBID_GLOBAL$$.adUnits.

[mkendall07]

would like to see JSdocs notation on exported functions.

[jsnellbaker]

Added some more descriptive information to the exported functions.

[mkendall07]

use underscore consistently please. All or none

[jsnellbaker]

Removed the underscore for this var to make it consistent with the others.

[mkendall07]

can you move this logic to a separate function and define at the top of the file. That will make it easier / cleaner for other CMPs to integrate.

[jsnellbaker]

I have reorganized the functions, please take a look on the next commit and let me know what you think.

[mkendall07]

I don't think these vars need to be exported?

[jsnellbaker]

I'm exporting these vars so I can read them in the consentManagement_spec.js test file for the setConfig unit tests.

[mkendall07]

what's the impact of this change?

[jsnellbaker]

The number (also relates to the other comment below) acts as a priority level for the hooked functions (when there are multiple hooked functions on the same hook).

The pbjs.requestBids function has 3 hooked functions from 3 different pbjs modules: pre1api, PubCommonId and consentManagement. So lowering this priority level ensures that the pre1api module's hooked function will (generally) go last in the sequence if all these modules were enabled together.

Having the pre1api module go last (specifically after consentManagement) is related to the need for the pre1api to execute right before the pbjs.requestBids so it knows which auction is currently active. The consentManagement module uses/waits on callbacks (to retrieve information from the CMP), and this buffering of the auction process could cause issues for the pre1api knowing which auction is active if there was a time gap because it ran first. So the lower priority helps avoid this scenario.

The 5 specifically is lower than the default priority that's set for any hooked function (which is 10), but still higher than the base function pbjs.requestBids (which is 0). So it should generally always the be last hook to run before the pbjs.requestBids would execute.

[mkendall07]

defaults like these should be made constant values and defined at the top of the file

[jsnellbaker]

I have created default variables and utilized them in the setConfig part of the module code in place of the actual values.

[mkendall07]

I always forget what the addHook signature looks like. What's the 50 value signify?

[jsnellbaker]

See comment above for the change in the pre1api file.

[jsnellbaker]

@mkendall07 I have just pushed in a commit with the changes for the various points you provided. Please take a look again and let me know of any additional feedback. Thanks.

[jsnellbaker]

@vzhukovsky Can you please provide a link to the 1.1 spec's documentation that you are referring to?

[vzhukovsky]

@jsnellbaker

Unfortunately this spec has not published yet. I am not allowed to share it.

But AppNexus is member of the group that defined this spec. Can you reach out to Andrew Sweeney or someone else who might have it?

[ahubertcriteo]

Hello all,
The api method used in this PR to retrieve the consents is 'getConsentData'.
As the data retrieved is encoded, each vendor will have to decode it to check whether it has the consent to bid (either in its adapter or server side).
Wouldnt it be easier if the consent data was decoded by prebid before calling the bidder? Or if the consent data was retrieved by another api method like 'getVendorConsents'?

[vzhukovsky]

@jsnellbaker Hi.

Are there still no any plans to support GDPR for Prebid legacy?

Most our users are still on 0.x.
If you decided support it we would be happy to port this module on legacy branch and contribute it back.

[jsnellbaker]

In lieu of the more recent questions and the upcoming 1.1 CMP spec I wanted to provide an update on the general direction for this module.

Below is a series of changes coming but that are not due to the 1.1 CMP spec:

• There are still currently no plans to have this module ported to the legacy branch
• In lieu of the request from @ahubertcriteo, we will change the query to the CMP to use the getVendorConsents for all vendors instead just calling for the getConsentData. This call will return an object with all the vendor related data (including the encoded consent string). This object will be passed into the auction much like before and stored under bidderRequest.gdprConsent.vendorData. The other two fields (consentString and consentRequired) will still be available under the gdprConsent object.
• Follow-up note to the above, the vendor data will very likely not be available to prebid server adapters as there is no defined place in the ortb request to store the information (as per the IAB's page on the GDPR and OpenRTB integration). They will likely need to parse the encoded string to figure out the appropriate information.

Below is a list of changes due to the 1.1 CMP spec

• We will be renaming the gdprConsent.consentRequired field to gdprConsent.gdprApplies. This new field is present/managed by CMP and is present on the response object when querying the CMP API. To avoid redundancy, we're going to remove the module's consentRequired from the config and just rely on the CMP's gdprApplies value. Adapters can still set their own back-up value in their integration code should the off-chance the gdprApplies isn't defined for some reason.
• In the case the prebid code is encased in an iframe, we will be using the 1.1 spec's stock iframe locator code to help interact with the CMP wherever it's located outside the iframe.
• As part of the changes to the query logic, the CMP will only respond to queries (ie fire the callback) once there is consent information available for the user. For a new-user that's filling out their consent choices, this means the auction will wait for the choices to be made. To support the chance for that first time user's consent choices to be used for that auction, we're bumping up the default timeout setting back to 10s (10000ms). Publishers can set their own timeout value via the setConfig if they wish to use different values.

I am planning to push these changes to the branch either today or tomorrow once I finish some additional testing.

[PubMatic-OpenWrap]

if CMP js library is not yet loaded then __cmp will still be an object, not a function, good to have a isFunction check.

[vzhukovsky]

@jsnellbaker Hi.
Thank you for the clarification regarding legacy.
But "currently" still sounds promising

Anyway, we have already done porting in our fork and will be happy to contribute back if needed.
Please just say no if it's final decision :)

[snapwich]

Few more comments

[snapwich]

I think this needs to be cleaned up somewhere. There going to be a lot of message listeners lying around if we do multiple auctions.

[jsnellbaker]

I will add in some logic to fire off removeEventListener at proper points in this code.

[snapwich]

This API is much better 👍

[snapwich]

What do we plan to do when this list grows? A module that contains every implementation doesn't scale. I'm fine with approving this pull-request if we have a solution for this going forward (for when we need scaling).

[jsnellbaker]

Hi @snapwich, I think the idea going forward (when we have several more CMPs integrated into the consentManagement) would be store their code into other smaller modules. The consentManagement module would then import the other modules and they'd be called/managed via this cmpCallMap.

[snapwich]

This naming is confusing. Functions and arrays are both objects as well. jQuery and others distinguish objects created with {} or new Object from other builtins using the name isPlainObject which I think is a more apt name.

[jsnellbaker]

I'll rename this function to isPlainObject for better clarity.

[cwbeck]

Request that CMP is always bound and referenced via top-most window instead of window context in which Prebid is loaded.

[cwbeck]

This makes the assumption that the CMP is bound to the window in which Prebid loads. For safety, we have always loaded Prebid (and our libraries too) inside of a friendly frame, so in our case, we would need this to reference the top-most window.

export function requestCMP(): void {
    let topWindow = WindowElement.getTopMostParentWindow();
    let createHandler = () => {
        let commandQueue: any = [];
        let cmp = function(command: any, parameter: any, callback: any){
            commandQueue.push({
                command: command,
                parameter: parameter,
                callback: callback
            });
        };
        (cmp as any).commandQueue = commandQueue;
        (cmp as any).receiveMessage = (event: any) => {
            let data = event && event.data && event.data.__cmp;
            if (data) {
                commandQueue.push({
                    callId: data.callId,
                    command: data.command,
                    parameter: data.parameter,
                    event: event
                });
            }
        };
        (cmp as any).config = {
            "globalConsentLocation": "https://acdn.adnxs.com/cmp/docs/portal.html",
            "storeConsentGlobally": true,
            "storePublisherData": false
        };
        return cmp;
    };
    (topWindow as any).__cmp = createHandler();
    topWindow.addEventListener('message', (event) => {
        (topWindow as any).__cmp.receiveMessage(event);
    });
    (topWindow as any).__cmp('showConsentTool');
    Async.loadJSIn(topWindow, "https://acdn.adnxs.com/cmp/docs/../cmp.bundle.js");
}

As the CMP will always be in the top-most window, can we change this?

[jsnellbaker]

Hi @cwbeck,

The part of the code you referenced is actually part of the code where we have already determined that the CMP is not in the same window as the prebid code. So it's not making one direction assumption as to where the CMP exists, as this can vary per publisher's implementation.

To provide some additional context on the function in question:
This function is part of the stock IAB code that's part of the 1.1 CMP spec to use the CMP's iframe locator (ie __cmpLocator) to detect the exact frame that the CMP is located.

The IAB's code works basically in 3 parts:

• locates the proper window/frame that's housing the CMP code (by looking for the parent window that contains the __cmpLocator iframe)
• it then writes it's own local version of the window.__cmp API to handle the natural API call and encase it in a postMessage call to bypass the iframe.
• it finally adds an addEventListener to wait for the message response from the CMP to grab the consent data and do the next steps.

Please let me know if you have any further questions on this point.

[cwbeck]

@jsnellbaker makes sense, thanks for the clarification

[mkendall07]

@jsnellbaker
Looking great! Has 2 conflicts that need resolved.

[mkendall07]

FYI this will most likely get merged by EOD.

[mkendall07]

LGTM. Please drop the iframe example

[tomeightyeight]

Just wondering when this is going to be tagged? Is this going to be rolled out as part of a larger release?

[jsnellbaker]

Hi @tomeightyeight

Now that this is merged - this is going to be included in the 1.9 release of Prebid.js, which is expected to happen later today.

[jbartek25]

@vzhukovsky has there been any further discussion wrt porting the module into legacy? If not, would you share link to your port?