Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline show for all a project #3661

Closed
AmineBarrak opened this issue Apr 22, 2020 · 5 comments · Fixed by #3905
Closed

Pipeline show for all a project #3661

AmineBarrak opened this issue Apr 22, 2020 · 5 comments · Fixed by #3905
Labels
awaiting response we are waiting for your reply, please respond! :)

Comments

@AmineBarrak
Copy link

I was wondering if there is a way to print all the pipeline for a project?
I found that we can find only a pipeline for a DVC file.

@triage-new-issues triage-new-issues bot added the triage Needs to be triaged label Apr 22, 2020
@AmineBarrak AmineBarrak changed the title Pipeline show for all the project Pipeline show for all a project Apr 22, 2020
@efiop
Copy link
Contributor

efiop commented Apr 22, 2020

@AmineBarrak There is dvc pipeline list that will show all pipelines when ran without arguments. Though, there is no way to prettify it like with show --ascii or something 🙁

@efiop efiop added the awaiting response we are waiting for your reply, please respond! :) label Apr 22, 2020
@triage-new-issues triage-new-issues bot removed the triage Needs to be triaged label Apr 22, 2020
@shcheklein
Copy link
Member

For the record (a context from the conversation). --dot or --ascii do not show a full pipeline (only pipeline up to a give stage), so there is no ways to visualize a pipeline that has multiple leaf stages. As far as I understood, pipeline list format is not enough - a DAG visualization was required.

@AmineBarrak
Copy link
Author

AmineBarrak commented Apr 23, 2020

There is a need for printing all the pipeline and print all interactions in one graph.
As a solution for me I did the following:

def get_nodes(dot_file, file):
	list_remove = []
	(graph,) = pydot.graph_from_dot_file(dot_file)
		# print (graph.get_node_list())
	for n in graph.get_node_list():
		list_remove.append(n.get_name().strip('"'))

	if file in list_remove:
		list_remove.remove(file)

	return list_remove
def remove_subtree(l1, l2):
	l3 = [x for x in l1 if x not in l2]
        return l3

@AmineBarrak
Copy link
Author

for file in list_dvc_files:
		print (file)
		# dvc pipeline show file --dot 
		p1 = subprocess.Popen(['dvc', 'pipeline', 'show', file, '--dot'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
		p_status = p1.wait()
		match_f = p1.communicate()[0].decode("utf-8").strip()
		# print (match_f)
		with open(output_folder_dot, "w") as text_file:
			text_file.write(match_f)
		
		list_nodes_remove = get_nodes(output_folder_dot, file)
		
		list_dvc_files = remove_subtree(list_dvc_files, list_nodes_remove)

@antonkulaga
Copy link

dvc pipeline list is somewhat misleading to me as it shows both stages and regular dvc files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting response we are waiting for your reply, please respond! :)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants