Update fix/clean up

[NicholasTurner23]

Description

This PR cleans up the get devices process by removing unnecessary fields and ops. It also improves the implementation of the devices data extraction by providing a bulk approach as well as a generator approach to optimize operations for a large number of devices.

This optimizes data flow and ops for the kafka implementations which is still a work in progress.

Related Issues

• OPS-291

Summary by CodeRabbit

• New Features

  • Enhanced Kafka message handling with improved logging and load balancing.
  • Added methods for consuming messages from a Kafka topic and publishing messages with optional keys.

• Improvements

  • Updated error handling and logging across various utilities for better clarity and efficiency.
  • Refined data processing methods to utilize DataFrames for improved performance.

• Documentation

  • Updated method docstrings to align with Python conventions and clarify input/output expectations.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

The pull request introduces several enhancements to the AirQoApi, AirQoDataUtils, KccaUtils, MessageBrokerUtils, and thingspeak_api.py files. Key changes include updated method signatures with improved type hinting, enhanced logging capabilities, and refined error handling. New methods have been added to facilitate better data processing and message handling, particularly in the context of Kafka. Overall, the changes aim to improve code clarity, efficiency, and maintainability.

Changes

File Path	Change Summary
src/workflows/airqo_etl_utils/airqo_api.py	- Updated method signatures for get_thingspeak_read_keys, get_devices, and calibrate_data. - Enhanced error handling and logging in __request.
src/workflows/airqo_etl_utils/airqo_utils.py	- Updated method signatures for extract_devices_data and process_data_for_api. - Transitioned docstrings from Parameters to Args.
src/workflows/airqo_etl_utils/kcca_utils.py	- Added logging with logger.exception in add_site_and_device_details. No changes to method signatures.
src/workflows/airqo_etl_utils/message_broker_utils.py	- Added methods for Kafka message handling: __get_least_loaded_partition, _generate_chunks, publish_to_topic, and consume_from_topic. - Updated update_hourly_data_topic method signature.
src/workflows/airqo_etl_utils/thingspeak_api.py	- Added logging functionality and removed unnecessary traceback import. No changes to method signatures.

Possibly related PRs

• Update/implement celery k8s executor #3415: Updates exception handling in airqo_utils.py, directly related to changes in the calibrate_data method.
• Update fix/reduce redundant device data called[wip] #3526: Modifies the get_devices method, which is also updated in this PR, focusing on functionality improvements.
• Update fix/clean up #3539: Ensures only devices with valid device numbers are considered, aligning with changes in the get_devices method.
• Refactor Data Processing Job for Improved Efficiency and Maintainability #3606: Refactors the data processing job, potentially relating to updates in how devices are handled in this PR.
• more optimisations/refactoring for the readings job #3608: Implements optimizations for the readings job, which may relate to overall data handling improvements.

Suggested labels

ready for review

Suggested reviewers

• Baalmart
• BenjaminSsempala
• Psalmz777

🎉 In the code's dance, enhancements take flight,
With logging and types shining bright.
Methods refined, errors now logged,
In the realm of data, our progress is progged!
So here's to the changes, both big and small,
Together we rise, united we stand tall! 🚀

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.