GItlab Integatration #1022

Blazingkevin · 2024-09-16T19:35:19Z

Description

Integration of Gitlab to Port
What -

Why -

How -

Type of change

Please leave one option from the following and delete the rest:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
New Integration (non-breaking change which adds a new integration)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Non-breaking change (fix of existing functionality that will not change current behavior)
Documentation (added/updated documentation)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration able to create all default resources from scratch
Resync finishes successfully
Resync able to create entities
Resync able to update entities
Resync able to detect and delete entities
Scheduled resync able to abort existing resync and start a new one
Tested with at least 2 integrations from scratch
Tested with Kafka and Polling event listeners
Tested deletion of entities that don't pass the selector

Integration testing checklist

Integration able to create all default resources from scratch
Resync able to create entities
Resync able to update entities
Resync able to detect and delete entities
Resync finishes successfully
If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the examples folder in the integration directory.
If resource kind is updated, run the integration with the example data and check if the expected result is achieved
If new resource kind is added or updated, validate that live-events for that resource are working as expected
Docs PR link here

Preflight checklist

Handled rate limiting
Handled pagination
Implemented the code in async
Support Multi account

Screenshots

Include screenshots from your environment showing how the resources of the integration will look.

lordsarcastic

Good work, however the class implementation is unnecessarily complex and could be cleaner. Please take a look at the changes

lordsarcastic · 2024-09-18T12:12:45Z

.pre-commit-config.yaml

@@ -10,7 +10,7 @@ repos:
      - id: check-merge-conflict
      - id: check-executables-have-shebangs
      - id: check-symlinks
-      - id: detect-aws-credentials
+      # - id: detect-aws-credentials


Let's not touch this file

lordsarcastic · 2024-09-18T12:14:54Z

integrations/gitlab2/gitlabcore/client.py

+def is_retryable_status_code(status_code: Optional[int]) -> bool:
+    """
+    Checks if the status code is retryable.
+    Handles rate limiting (HTTP 429) and transient errors (e.g., HTTP 500, 503).
+    """
+    if status_code == 429:
+        return True
+
+    if status_code in RETRYABLE_ERROR_CODES:
+        return True
+
+    return False


Suggested change

def is_retryable_status_code(status_code: Optional[int]) -> bool:

"""

Checks if the status code is retryable.

Handles rate limiting (HTTP 429) and transient errors (e.g., HTTP 500, 503).

"""

if status_code == 429:

return True

if status_code in RETRYABLE_ERROR_CODES:

return True

return False

def is_retryable_status_code(status_code: Optional[int]) -> bool:

return status_code in [429, *RETRYABLE_ERROR_CODES]

lordsarcastic · 2024-09-18T12:17:53Z

integrations/gitlab2/main.py

+    )
+
+
+# Required


Take this out if you aren't using it

lordsarcastic · 2024-09-18T12:18:12Z

integrations/gitlab2/main.py

+async def on_start() -> None:
+    # Something to do when the integration starts
+    # For example create a client to query 3rd party services - GitHub, Jira, etc...
+    print("Starting gitlab2 integration")


I expect you should handle webhooks creation here.

lordsarcastic · 2024-09-18T12:19:05Z

integrations/gitlab2/main.py

+    await process_webhook_event(event_type, payload)
+
+
+async def create_webhook_with_semaphore(semaphore: asyncio.Semaphore, gitlab_client: GitlabClient, project_id: int, webhook_url: str ):


Utility functions should either be part of your client class or in the client file

lordsarcastic · 2024-09-18T12:19:56Z

integrations/gitlab2/gitlabcore/client.py

+        self.token = token
+        self.base_url = base_url
+        self.http_client = http_async_client
+        self.http_client.headers.update()


You aren't passing any headers to this line

lordsarcastic · 2024-09-18T12:20:30Z

integrations/gitlab2/gitlabcore/client.py

+
+
+# PaginatedData Class
+class PaginatedData:


You should be able to handle pagination using a method in the client class. While this is a clever idea, it is good to keep things as simple as possible

lordsarcastic · 2024-09-18T12:30:34Z

integrations/gitlab2/gitlabcore/client.py

+        self.app_host = app_host
+
+    @property
+    def api_auth_header(self) -> dict[str, Any]:


Pass this directly to the client

lordsarcastic · 2024-09-18T12:32:32Z

Add screenshot of the data on your dashboard

PeyGis

Looks good. Please go over my comments and let me know if you need any further info

PeyGis · 2024-09-18T13:57:16Z

integrations/gitlab2/CHANGELOG.md

+# Changelog - Ocean - gitlab2
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+<!-- towncrier release notes start -->


Add the changelog content for the feature implementation for GitLab.

something like

0.1.0 (2024-09-13)

Feature

Implemented GitLab integration using Ocean

PeyGis · 2024-09-18T14:50:16Z

integrations/gitlab2/gitlabcore/client.py

+        """Fetch paginated projects from GitLab and return an async generator of project lists."""
+        paginated_projects = PaginatedData(
+            url=f"{self.base_url}/projects",
+            query_data={"per_page": 5},  # You can adjust per_page as needed


is there a reason we are fetching only 5 items per batch?

PeyGis · 2024-09-18T14:51:49Z

integrations/gitlab2/gitlabcore/client.py

+        url = f"{self.base_url}/projects/{project_id}/hooks"
+        webhook_url = f"{self.app_host}/integration/webhook"


creating webhooks per project is going to be spammy. You can look into how to create it on a group level such as group hooks. this way, all projects under the group will inherit the hook

PeyGis · 2024-09-18T14:52:28Z

integrations/gitlab2/gitlabcore/client.py

+        data = {
+            "url": webhook_url,
+            "issues_events": True,
+            "merge_requests_events": True,
+            "push_events": True,
+        }


GitLab has way more events you can subscribe to. Update it with more events

PeyGis · 2024-09-18T14:53:09Z

integrations/gitlab2/main.py

+    return GitlabClient(
+        ocean.integration_config["gitlab_token"],
+        ocean.integration_config["base_url"],
+        ocean.integration_config["app_host"]


what if the user doesn't provide app_host? this will fail

PeyGis · 2024-09-18T14:53:33Z

integrations/gitlab2/main.py

+# Listen to the resync event of all the kinds specified in the mapping inside port.
+# Called each time with a different kind that should be returned from the source system.
+# @ocean.on_resync()
+# async def on_resync(kind: str) -> list[dict[Any, Any]]:
+#     # 1. Get all data from the source system
+#     # 2. Return a list of dictionaries with the raw data of the state to run the core logic of the framework for
+#     # Example:
+#     # if kind == "project":
+#     #     return [{"some_project_key": "someProjectValue", ...}]
+#     # if kind == "issues":
+#     #     return [{"some_issue_key": "someIssueValue", ...}]
+
+#     # Initial stub to show complete flow, replace this with your own logic
+#     if kind == "gitlab2-example-kind":
+#         return [
+#             {
+#                 "my_custom_id": f"id_{x}",
+#                 "my_custom_text": f"very long text with {x} in it",
+#                 "my_special_score": x * 32 % 3,
+#                 "my_component": f"component-{x}",
+#                 "my_service": f"service-{x %2}",
+#                 "my_enum": "VALID" if x % 2 == 0 else "FAILED",
+#             }
+#             for x in range(25)
+#         ]
+
+#     return []
+
+
+# The same sync logic can be registered for one of the kinds that are available in the mapping in port.


delete all these commented code

PeyGis · 2024-09-18T14:54:25Z

integrations/gitlab2/main.py

+    webhook_url = ""
+    semaphore = asyncio.Semaphore(10)
+
+
+    async for projects in gitlab_client.get_projects():
+        logger.info(f"Received project batch with {len(projects)}")
+        tasks = [
+            create_webhook_with_semaphore(
+                semaphore,
+                gitlab_client,
+                project['id'],
+                webhook_url
+            )
+            for project in projects
+        ]
+
+        results = await asyncio.gather(*tasks, return_exceptions=True)
+
+        for result in results:
+            if isinstance(result, Exception):
+                logger.error(f"Error occurred during webhook creation: {result}")
+
+        yield projects


the creation of webhook should be in the on_start event, instead of here

PeyGis · 2024-09-18T14:55:22Z

integrations/gitlab2/main.py

+async def process_webhook_event(event_type: str, payload: dict):
+    gitlab_client = init_client()
+
+    if event_type == "Issue Hook":
+        await handle_issue_event(payload, gitlab_client)
+    elif event_type == "Merge Request Hook":
+        await handle_merge_request_event(payload, gitlab_client)
+    elif event_type == "Push Hook":
+        await handle_push_event(payload, gitlab_client)
+    elif event_type == "Project Hook":
+        await handle_project_event(payload, gitlab_client)
+    elif event_type == "Group Hook":
+        await handle_group_event(payload, gitlab_client)
+    else:
+        logger.warning(f"Unhandled event type: {event_type}")
+
+async def handle_issue_event(payload: dict, gitlab_client: GitlabClient):
+    issue_attributes = payload.get('object_attributes', {})
+    issue_id = issue_attributes.get('id')
+    project_id = payload.get('project', {}).get('id')
+    action = issue_attributes.get('action')  # 'open', 'close', 'reopen', 'update'
+    logger.info(f"Issue {action}: {issue_id}")
+
+    if action in ['open', 'reopen', 'update']:
+        # Fetch the latest issue data
+        issue = await gitlab_client.get_single_issue(project_id, issue_attributes.get('iid'))
+        await ocean.register_raw(ObjectKind.ISSUE, [issue])


these are great but I will prefer we keep the main.py clean and move these implementation to a separate file

adjuste pre-commit-config

d0ef7a3

github-actions bot added the size/XXL label Sep 16, 2024

lordsarcastic requested changes Sep 18, 2024

View reviewed changes

PeyGis requested changes Sep 18, 2024

View reviewed changes

Blazingkevin added 2 commits September 20, 2024 14:20

modified gitlab client

b97b1bb

Merge branch 'main' into kevin-gitlab2

2be50f3

Blazingkevin requested a review from lordsarcastic September 20, 2024 13:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GItlab Integatration #1022

GItlab Integatration #1022

Blazingkevin commented Sep 16, 2024 •

edited

Loading

lordsarcastic left a comment

lordsarcastic Sep 18, 2024

lordsarcastic Sep 18, 2024

lordsarcastic Sep 18, 2024

lordsarcastic Sep 18, 2024

lordsarcastic Sep 18, 2024

lordsarcastic Sep 18, 2024

lordsarcastic Sep 18, 2024

lordsarcastic Sep 18, 2024

lordsarcastic commented Sep 18, 2024

PeyGis left a comment

PeyGis Sep 18, 2024

PeyGis Sep 18, 2024

PeyGis Sep 18, 2024

PeyGis Sep 18, 2024

PeyGis Sep 18, 2024

PeyGis Sep 18, 2024

PeyGis Sep 18, 2024

PeyGis Sep 18, 2024

		await process_webhook_event(event_type, payload)


		async def create_webhook_with_semaphore(semaphore: asyncio.Semaphore, gitlab_client: GitlabClient, project_id: int, webhook_url: str ):

		url = f"{self.base_url}/projects/{project_id}/hooks"
		webhook_url = f"{self.app_host}/integration/webhook"

GItlab Integatration #1022

Are you sure you want to change the base?

GItlab Integatration #1022

Conversation

Blazingkevin commented Sep 16, 2024 • edited Loading

Description

Type of change

All tests should be run against the port production environment(using a testing org).

Core testing checklist

Integration testing checklist

Preflight checklist

Screenshots

lordsarcastic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lordsarcastic commented Sep 18, 2024

PeyGis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0.1.0 (2024-09-13)

Feature

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Blazingkevin commented Sep 16, 2024 •

edited

Loading