You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Q: Does nuke have wrapper for simple Linux utilities like sort?
Sample command: sort -t'|' -k2,2n -k 4,5,6?
Background
I'll be running in k8s jobs (docker or "OCI image"). I'm looking at taking a daily multi-step ETL process, currently in Python, and breaking that down into multiple steps to run in a k8s job, but break down each step of the download and ETL via nuke build. The last step will just be my custom C# code to efficiently load batches into SQL Server (parallel foreach).
I'm a long-time .net developer. I like slicing and dicing data. I've been using python and pandas for those tasks for the last year, but now that I've tried it I am not loving the python ecosystem. For example, in the pip ecosystem I have to rely on python wheels for very basic things like connecting to MSSQL database via pyodbc. I think .NET 6 base images are more complete vs base python image, and better enterprise licensing terms vs. Anaconda for things that aren't strictly "machine learning". There's no "learning" in my jobs, just updating data in MSSQL.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Q: Does nuke have wrapper for simple Linux utilities like
sort
?Sample command:
sort -t'|' -k2,2n -k 4,5,6
?Background
I'll be running in k8s jobs (docker or "OCI image"). I'm looking at taking a daily multi-step ETL process, currently in Python, and breaking that down into multiple steps to run in a k8s job, but break down each step of the download and ETL via nuke build. The last step will just be my custom C# code to efficiently load batches into SQL Server (parallel foreach).
I'm a long-time .net developer. I like slicing and dicing data. I've been using python and pandas for those tasks for the last year, but now that I've tried it I am not loving the python ecosystem. For example, in the pip ecosystem I have to rely on python wheels for very basic things like connecting to MSSQL database via pyodbc. I think .NET 6 base images are more complete vs base python image, and better enterprise licensing terms vs. Anaconda for things that aren't strictly "machine learning". There's no "learning" in my jobs, just updating data in MSSQL.
Beta Was this translation helpful? Give feedback.
All reactions