Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix existing custom operators' templated fields that include logic checks #36484

Closed
27 tasks done
shahar1 opened this issue Dec 29, 2023 · 11 comments
Closed
27 tasks done

Comments

@shahar1
Copy link
Contributor

shahar1 commented Dec 29, 2023

Introduction

PR #33786 aims to prevent templated field logic in __init__ method of operators automatically by introducing a new pre-commit that validates each operator (see added limitations in the PR).
In short, for templated fields, there should be only an assignment operation in the constructor (i.e., self.field = field), and the name of the assigned field should be the same as the assigned parameter (i.e., the following is invalid: self.field = something_else).

Before merging this PR, we need to fix the existing operators in separate PRs that don't follow the new limitations.
The following list includes currently invalid operators by file paths that need to be fixed (plus details of what should be fixed).
It was manually scraped on Dec. 12 2023 from the PR's output of currently invalid operators' constructors.

Please feel free to create PRs to fix any of the files.

Tasks list

  • airflow/providers/apache/hive/transfers/hive_to_samba.py
HiveToSambaOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['hql']
HiveToSambaOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.hql']
  • airflow/providers/google/cloud/operators/dataproc.py
DataprocSubmitPigJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['impersonation_chain', 'dataproc_jars', 'cluster_name', 'dataproc_properties', 'region', 'job_name']
DataprocSubmitHiveJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['impersonation_chain', 'region', 'job_name', 'cluster_name', 'dataproc_jars', 'dataproc_properties']
DataprocSubmitSparkSqlJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['impersonation_chain', 'region', 'job_name', 'cluster_name', 'dataproc_jars', 'dataproc_properties']
DataprocSubmitSparkJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['impersonation_chain', 'region', 'job_name', 'cluster_name', 'dataproc_jars', 'dataproc_properties']
DataprocSubmitHadoopJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['impersonation_chain', 'region', 'job_name', 'cluster_name', 'dataproc_jars', 'dataproc_properties']
DataprocSubmitPySparkJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['impersonation_chain', 'region', 'job_name', 'cluster_name', 'dataproc_properties', 'dataproc_jars']
  • airflow/providers/apache/livy/operators/livy.py
    LivyOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or super.init(field_name=field_name,
    ...) ):
    ['spark_params']
    LivyOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
    ['self.spark_params']

  • airflow/providers/google/cloud/operators/bigquery.py

BigQueryGetDataOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['max_results']
BigQueryGetDataOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.max_results']
BigQueryCreateExternalTableOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['gcs_schema_bucket', 'table_resource', 'destination_project_dataset_table', 'source_objects', 'schema_object', 'bucket']
  • airflow/providers/google/cloud/transfers/bigquery_to_postgres.py
BigQueryToPostgresOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['table_id', 'dataset_id']
  • airflow/providers/alibaba/cloud/operators/analyticdb_spark.py
AnalyticDBSparkSQLOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['spark_params']
AnalyticDBSparkSQLOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.spark_params']
AnalyticDBSparkBatchOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['spark_params']
AnalyticDBSparkBatchOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.spark_params']
  • airflow/providers/microsoft/azure/operators/container_instances.py
AzureContainerInstancesOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['name']
AzureContainerInstancesOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.name']
  • airflow/providers/databricks/operators/databricks.py
DatabricksCreateJobsOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.json']
DatabricksRunNowOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.json']
  • airflow/providers/google/cloud/transfers/bigquery_to_mysql.py
BigQueryToMySqlOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['dataset_id', 'table_id']
airflow/providers/amazon/aws/operators/emr.py
EmrCreateJobFlowOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['waiter_max_attempts', 'waiter_delay']
EmrCreateJobFlowOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.waiter_max_attempts', 'self.waiter_delay']
  • airflow/providers/papermill/operators/papermill.py
PapermillOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['input_nb', 'output_nb']
  • airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py
CloudDataTransferServiceCreateJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['body']
CloudDataTransferServiceCreateJobOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = 
field_name):
['self.body']

CloudDataTransferServiceListOperationsOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['filter']
CloudDataTransferServiceListOperationsOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name =
field_name):
['self.filter']
  • airflow/providers/google/cloud/operators/vertex_ai/auto_ml.py
CreateAutoMLForecastingTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['region', 'impersonation_chain', 'parent_model']
CreateAutoMLImageTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['region', 'impersonation_chain', 'parent_model']
CreateAutoMLTabularTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['region', 'impersonation_chain', 'parent_model']
CreateAutoMLVideoTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['region', 'impersonation_chain', 'parent_model']

DeleteAutoMLTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['training_pipeline']
DeleteAutoMLTrainingJobOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.training_pipeline']
  • airflow/providers/google/cloud/transfers/sftp_to_gcs.py
SFTPToGCSOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['destination_path', 'destination_bucket']
SFTPToGCSOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.destination_path', 'self.destination_bucket']
  • airflow/providers/amazon/aws/transfers/base.py
AwsToAwsBaseOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['source_aws_conn_id', 'dest_aws_conn_id']
AwsToAwsBaseOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.dest_aws_conn_id']
  • airflow/providers/amazon/aws/operators/datasync.py
DataSyncOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or super.__init__(field_name=field_name,
...) ):
['create_source_location_kwargs', 'create_destination_location_kwargs']
DataSyncOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.create_source_location_kwargs', 'self.create_destination_location_kwargs']
  • airflow/providers/weaviate/operators/weaviate.py
WeaviateIngestOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['input_data']
  • airflow/providers/amazon/aws/transfers/gcs_to_s3.py
GCSToS3Operator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or super.__init__(field_name=field_name, 
...) ):
['gcs_bucket']
  • airflow/providers/amazon/aws/transfers/redshift_to_s3.py
RedshiftToS3Operator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['unload_options', 'select_query', 's3_key']
RedshiftToS3Operator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.s3_key', 'self.unload_options']
airflow/providers/google/cloud/operators/compute.py
ComputeEngineInsertInstanceOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.resource_id', 'self.resource_id']
ComputeEngineInsertInstanceFromTemplateOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = 
field_name):
['self.resource_id', 'self.resource_id']
ComputeEngineInsertInstanceTemplateOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = 
field_name):
['self.resource_id', 'self.resource_id']
ComputeEngineInstanceGroupUpdateManagerTemplateOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., 
self.field_name = field_name):
['self.zone']
ComputeEngineInsertInstanceGroupManagerOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = 
field_name):
['self.resource_id', 'self.resource_id']
  • airflow/providers/cncf/kubernetes/operators/pod.py
KubernetesPodOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['volume_mounts', 'env_vars', 'volumes']
KubernetesPodOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.env_vars', 'self.volume_mounts', 'self.volumes']
  • airflow/providers/amazon/aws/operators/eks.py
EksCreateClusterOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['wait_for_completion']
EksCreateClusterOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.wait_for_completion']
EksCreateNodegroupOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['wait_for_completion']
EksCreateNodegroupOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.wait_for_completion']
EksCreateFargateProfileOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['wait_for_completion']
EksCreateFargateProfileOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.wait_for_completion']
EksDeleteClusterOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['wait_for_completion']
EksDeleteClusterOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.wait_for_completion']
  • airflow/providers/apache/hive/operators/hive_stats.py
HiveStatsCollectionOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['dttm', 'ds']
HiveStatsCollectionOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.ds', 'self.dttm']
  • airflow/providers/google/cloud/operators/vertex_ai/custom_job.py
CreateCustomContainerTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['dataset_id', 'region', 'parent_model', 'impersonation_chain']
CreateCustomPythonPackageTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['dataset_id', 'region', 'parent_model', 'impersonation_chain']
CreateCustomTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or 
super.__init__(field_name=field_name, ...) ):
['dataset_id', 'region', 'parent_model', 'impersonation_chain']
DeleteCustomTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['training_pipeline', 'custom_job']
DeleteCustomTrainingJobOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.training_pipeline', 'self.custom_job']
  • airflow/providers/google/cloud/operators/mlengine.py
MLEngineStartBatchPredictionJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_input_paths', '_region', '_job_id', '_model_name', '_output_path', '_version_name', '_uri']
MLEngineStartBatchPredictionJobOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name =
field_name):
['self._project_id', 'self._job_id', 'self._region', 'self._input_paths', 'self._output_path', 'self._model_name', 'self._version_name', 'self._uri', 'self._impersonation_chain']
MLEngineManageModelOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_model', '_project_id', '_impersonation_chain']
MLEngineManageModelOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model', 'self._impersonation_chain']
MLEngineCreateModelOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_model', '_project_id', '_impersonation_chain']
MLEngineCreateModelOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model', 'self._impersonation_chain']
MLEngineGetModelOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_model_name']
MLEngineGetModelOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model_name', 'self._impersonation_chain']
MLEngineDeleteModelOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_model_name']
MLEngineDeleteModelOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model_name', 'self._impersonation_chain']
MLEngineManageVersionOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_model_name', '_version', '_version_name']
MLEngineManageVersionOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model_name', 'self._version_name', 'self._version', 'self._impersonation_chain']
MLEngineCreateVersionOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_model_name', '_version']
MLEngineCreateVersionOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model_name', 'self._version', 'self._impersonation_chain']
MLEngineSetDefaultVersionOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_version_name', '_model_name']
MLEngineSetDefaultVersionOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model_name', 'self._version_name', 'self._impersonation_chain']
MLEngineListVersionsOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_model_name']
MLEngineListVersionsOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model_name', 'self._impersonation_chain']
MLEngineDeleteVersionOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_version_name', '_model_name']
MLEngineDeleteVersionOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._model_name', 'self._version_name', 'self._impersonation_chain']
MLEngineStartTrainingJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_python_version', '_project_id', '_impersonation_chain', '_package_uris', '_training_args', '_master_type', '_hyperparameters', '_training_python_module', '_job_id', '_scale_tier', 
'_runtime_version', '_job_dir', '_service_account', '_region', '_master_config']
MLEngineStartTrainingJobOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._job_id', 'self._region', 'self._package_uris', 'self._training_python_module', 'self._training_args', 'self._scale_tier', 'self._master_type', 'self._master_config', 
'self._runtime_version', 'self._python_version', 'self._job_dir', 'self._service_account', 'self._hyperparameters', 'self._impersonation_chain']
MLEngineTrainingCancelJobOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_project_id', '_impersonation_chain', '_job_id']
MLEngineTrainingCancelJobOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._project_id', 'self._job_id', 'self._impersonation_chain']
  • airflow/providers/apache/spark/operators/spark_submit.py
SparkSubmitOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_keytab', '_application', '_env_vars', '_properties_file', '_proxy_user', '_name', '_py_files', '_jars', '_driver_class_path', '_files', '_principal', '_conf', '_application_args', '_packages', 
'_exclude_packages']
SparkSubmitOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._application', 'self._conf', 'self._files', 'self._py_files', 'self._driver_class_path', 'self._jars', 'self._packages', 'self._exclude_packages', 'self._keytab', 'self._principal', 
'self._proxy_user', 'self._name', 'self._application_args', 'self._env_vars', 'self._properties_file']
  • airflow/providers/apache/spark/operators/spark_sql.py
SparkSqlOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or super.__init__(field_name=field_name,      
...) ):
['_sql']
  • airflow/providers/databricks/operators/databricks_sql.py
DatabricksCopyIntoOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or
super.__init__(field_name=field_name, ...) ):
['_file_location', '_table_name', '_files']
DatabricksCopyIntoOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self._files', 'self._table_name', 'self._file_location']

Code of Conduct

@potiuk
Copy link
Member

potiuk commented Dec 29, 2023

I removed some of the template boilerplate from the description for clarity :)

@eladkal
Copy link
Contributor

eladkal commented Mar 4, 2024

I believe all entries are fixed now?

@shahar1
Copy link
Contributor Author

shahar1 commented Mar 9, 2024

I believe all entries are fixed now?

Almost, we still have #37519 open.
Thanks everyone for helping out :)

@eladkal
Copy link
Contributor

eladkal commented Mar 9, 2024

I believe all entries are fixed now?

Almost, we still have #37519 open. Thanks everyone for helping out :)

I think it was fixed in another PR?

@eladkal eladkal removed the type:improvement Changelog: Improvements label Mar 9, 2024
@eladkal
Copy link
Contributor

eladkal commented Mar 9, 2024

@shahar1 #36909 removed the file from the exclusion test so either something is wrong with the exclusion or the issue should be resolved

@shahar1
Copy link
Contributor Author

shahar1 commented Mar 9, 2024

@shahar1 #36909 removed the file from the exclusion test so either something is wrong with the exclusion or the issue should be resolved

I checked, and there's a small bug in the pre-commit logic that checks that the field's name is the same as the parameter's. It currently affects 6 files (including this one), I'll create a PR for that.

I updated the list in the issue accordingly.

@shahar1
Copy link
Contributor Author

shahar1 commented Mar 16, 2024

It took a while, but the pre-commit is now operational with no exceptions :)
There's still some work to do to improve it, but hopefully, now it will become a bit easier to detect issues related to templated fields.
Thank you, everyone, for your joint efforts!

@shahar1 shahar1 closed this as completed Mar 16, 2024
@potiuk
Copy link
Member

potiuk commented Mar 16, 2024

🎉 🎉 🎉 🎉 🎉 🎉

@potiuk
Copy link
Member

potiuk commented Mar 16, 2024

Good job @shahar1 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants