Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gitlab Integration #1030

Closed
wants to merge 69 commits into from
Closed

Gitlab Integration #1030

wants to merge 69 commits into from

Conversation

shariff6
Copy link

@shariff6 shariff6 commented Sep 18, 2024

Description

What:

This integration for GitLab and Port creates a system for synchronizing GitLab groups, issues, merge requests, and projects. It also sets up webhooks to handle live events from GitLab.

Why:

To ensure that data between GitLab and Port is consistently synchronized and up-to-date. The integration enables real-time updates and event handling through webhooks, improving data accuracy and responsiveness.

How:

  • Implements synchronization for GitLab resources such as groups, projects issues, and merge requests.
  • Configures webhooks for capturing and responding to live events from GitLab.
  • Adds gitlabToken for secure API interactions.
  • Handles webhook secret generation and removes unused code.
  • Sets up YAML configuration for managing events and resource kinds.

Type of change

Please leave one option from the following and delete the rest:

  • New Integration (non-breaking change which adds a new integration)

All tests should be run against the port production environment(using a testing org).

This implementation is pending tests

Core testing checklist

  • Integration able to create all default resources from scratch
  • Resync finishes successfully
  • Resync able to create entities
  • Resync able to update entities
  • Resync able to detect and delete entities
  • Scheduled resync able to abort existing resync and start a new one
  • Tested with at least 2 integrations from scratch
  • Tested with Kafka and Polling event listeners
  • Tested deletion of entities that don't pass the selector

Integration testing checklist

  • Integration able to create all default resources from scratch
  • Resync able to create entities
  • Resync able to update entities
  • Resync able to detect and delete entities
  • Resync finishes successfully
  • If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the examples folder in the integration directory.
  • If resource kind is updated, run the integration with the example data and check if the expected result is achieved
  • If new resource kind is added or updated, validate that live-events for that resource are working as expected
  • Docs PR link here

Preflight checklist

  • Handled rate limiting
  • Handled pagination
  • Implemented the code in async
  • Support Multi account

Screenshots

Include screenshots from your environment showing how the resources of the integration will look.

Groups

Screenshot (77)
Screenshot (73)

Projects

Screenshot (76)
Screenshot (75)

Merge Requests

Screenshot (78)
Screenshot (79)

API Documentation

For this integration, the following GitLab API documentation links are used:

  • Issues API: Documentation on how to interact with issues in GitLab.
  • Projects API: Information on managing and retrieving project data.
  • Groups API: Details on working with GitLab groups.
  • Webhooks API: Guidelines for setting up and managing webhooks.
  • Merge Requests API: Instructions for handling merge requests.

Removes if else statements, wraps kinds in dictionary(resource_map)
…dual handlers for each resource type - Replaced the single 'on_resync' function with dedicated handlers for GitLab groups, projects, merge requests, and issues. - Each resource type now has its own on_resync function to manage synchronization. - Added fetch_resource helper function to collect and return items from the respective fetch methods.
… on information in reponse from gitlab headers,
…class.

Creates a mapper_factory helper class for easier adding of resource type and resource mappings
Adds setting up webhooks once Gitlab Integrations has began

Create, List and Update Webhooks
Adjust setupwebhooks accross all projects
…itional resources

Moves fetch resources logic into get_paginated_resource function
Copy link
Contributor

@PeyGis PeyGis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

solid work. please go over my comments

.pre-commit-config.yaml Outdated Show resolved Hide resolved
.vscode/launch.json Outdated Show resolved Hide resolved
integrations/gitlab_v1/CHANGELOG.md Show resolved Hide resolved
integrations/gitlab_v1/helpers/webhook_handler.py Outdated Show resolved Hide resolved
integrations/gitlab_v1/helpers/webhook_handler.py Outdated Show resolved Hide resolved
integrations/gitlab_v1/main.py Outdated Show resolved Hide resolved
integrations/gitlab_v1/main.py Outdated Show resolved Hide resolved
@shariff6
Copy link
Author

@PeyGis Thank you for the review, all comments have been addressed.

Copy link
Contributor

@PeyGis PeyGis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some comments

Comment on lines +1 to +24
initializePortResources: true
scheduledResyncInterval: 1440 # 60 minutes X 24 hours = 1 day
port:
clientId: "{{ from env PORT_CLIENT_ID }}" # Can be loaded using environment variable: PORT_CLIENT_ID
clientSecret: "{{ from env PORT_CLIENT_SECRET }}" # Can be loaded using environment variable: PORT_CLIENT_SECRET


# The event listener to use for the integration service.
eventListener:
type: POLLING


integration:
# The identifier of this integration instance.
identifier: "gitlab_v1"
# The type of the integration.
type: "Gitlab"
config:
gitlabToken:
- "{{ from env GITLAB_TOKEN }}"
- "{{ from env GITLAB_TOKEN2 }}"
gitlabUrl: "https://gitlab.com"
webhookUrl: "{{ from env WEBHOOK_URL }}"
webhookSecret: "{{ from env WEBHOOK_SECRET }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't use the config.yaml file anymore. so delete it

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the alternative to this,
This is what I was using for reference .
Develop an Integration

integrations/gitlab_v1/main.py Outdated Show resolved Hide resolved
@shariff6 shariff6 closed this Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants