Org Browser

[ntotten]

This proposal is for a solution to deploying and retrieving metadata to non-scratch orgs that limits the required use of the package.xml file. The goal of this proposal is to simplify the dev lifecycle for developers that work directly against orgs, but don't want the complexity of crafting package.xml manifests every time they want to do some work.

This proposal introduces the concept of a Working Set (Name TBD). A Working Set is a group of metadata types or items that can be deployed or retrieved together. A working set can be exported to a package.xml file. A working set is only a concept in VS Code - it is not something that is implemented on the actual Metadata API.

Working Sets are purely temporary groups of metadata that are used to perform various tasks in VS Code - while they will persist across sessions of VS Code (mostly to avoid losing work if you accidentally close VS Code) they cannot be checked into source, nor are they designed to do production or scriptable deployments like changesets or package deployments are. The most basic case for using working sets is when developer using VS Code wants to work on a few different files at the same time and quickly deploy them to their development org to see the changes. Fundamentally, working sets are primarily just UI to expose the CLI command sfdx force:source:deploy --metadata <metata types/instance> and sfdx force:source:retrieve --metadata <metadata types/instances>. Working sets are also the way you can build apackage.xml file in VS Code. You can add items to the in-memory working set and then export them to a package.xml manifest file. In the future, this UI could be used to edit/import existing package.xml files or we could even imagine this UI being extended to help build changesets, but those features are not in the scope of this proposal now.

It is also worth noting that this UI only supports non-source tracked orgs. It does not work against scratch orgs as source tracking deployments do not currently support fine-grained push or pull. If in the future, source tracked deployments support the ability to push/pull a specific set of source to/from an org this UI will be adapted to support that functionality. This would be analogous to the hypothetical sfdx force:source:push --metadata <metadata types/instances> and sfdx force:source:pull --metadata <metadata types/instances> commands.

The UI would introduce a new tab in VS Code called the "Org Browser". The org browser would show the currently selected org and the metadata. Switching orgs would be done via the Org Picker UI (see #623).

The Org Browser would show all metadata in the org as well as a button to refresh the view with the current state of the org and a search filter. Note, the search filter is currently not possible as a result of VS Code's extensibility, but will be provided for all tree views in a release very soon.

Metadata types and instances could be right-clicked to bring up an action menu. This action menu would have four functions:

• Copy Name - Simply copies the name of the metadata class or item.
• Retrieve Source from Org - Retrieves the metadata from the org to the default directory. If the metadata already exists locally it would be overwritten.
• Deploy Source to Org - Deploys the local copy of the metadata to the org, if their is no local copy of this metadata this item would be disabled.
• Add/Remove to Working Set - Adds (or removes if already in the working set) the metadata or metadata type to the current working set.

IMPLIMENTATION NOTE: Not all metadata types can be included in a package.xml using the wildcard * notation so not every metadata type will be able to be added to a working set. In those cases only the individual metadata of that particular type will be able to be added to the working set. See: https://developer.salesforce.com/docs/atlas.en-us.api_meta.meta/api_meta/meta_types_list.htm

The pane below the metadata tree shows the items that are currently in the working set. These items can be removed by right-clicking and selecting "Remove from Working Set".

The working set pane contains three buttons:

• Save/Export - This button exports the current working set to a package.xml file. The user will be prompted for a name and location to save this file.
• Deploy Working Set - Deploys all the items in this working set to the currently selected org.
• Retrieve Working Set - Retrieves all the items in this working set from the currently selected org.

In addition to the menus, there would also be two commands in the VS Code command palette.

• SFDX: Deploy Current Working Set to Org - Deploys the items in the current working set to the org.
• SFDX: Retrieve Current Working Set from Org - Retrieves the items in the current working set from the org.

Lastly, files in the regular explorer tab could be added to the working set as well via the right click menu.

[ChuckJonas]

This would be awesome! Besides solving some problems around development (refactoring, etc) this would make for a really nice general package.xml generation tool (something which a lot of people have been looking for).

Some other potential enhancements that could be built off this:

1: Ability to right-click and add files from local workspace (either on an open text editor or from file explorer, with multi-selection support)

2: A deployment window. Would let you deploy a working set to a different ORG (EG production). Would allow for specifying things like check only, test-level, specified test, etc. Mavensmate had something similar:

3: Compare with org. Would let you select two orgs and compare the working set between them

[gbreavin]

Note I'm asking this without having used VS Code + Extensions with a non-scratch org...

How would this feature work with adding and working with metadata against an org? Let's say I create a class via SFDX in VS Code - is it automatically added to the Working Set? Do I need to add it?

Also, are the package.xml and working set completely decoupled? If I add something to the package.xml, is it part of the working set? Or is the working set just a way of building a package.xml at a point in time?

[JodieM]

OK, so I like the concept, but not sure about a few specifics. For me the almost ideal package.xml builder is what was in MavensMate, even though it was really slow and cumbersome, but that was more to do with the Metadata API being slow on large orgs.

Refresh the Org Browser is great, thanks.

Search Filter would be great, but I understand that it's not something you can do now.

Add files from Explorer is an nice option, thanks.

Can I have multiple Working Sets on the org at the same time? It seems so from your mockup, but I'm not sure. But that would be cool - eg I'm working on code, but I need to just quickly deploy some field and layout changes - it would be good to have those separated.

I would want the option to Add All Metadata to Working Set, so I can do a backup of the Org.

I would want to be able to Import a Working Set - eg I have a standard list of things I work on from one org to the next - for me that's all Object stuff, all workflows, flows, process builders, all Email Templates, etc - all the main declarative stuff. Similar to the "all the code" default package.xml you include now but for declarative side of things.

Or Default Working Sets - Illuminated Cloud has two sets of metadata to choose from, just to make the metadata list be not so overwhelming. The Welkin Suite has an Admin mode and a Developer Mode. Something like that would be a good starting point, especially for Admins who don't want to look at the code. But I do understand your priority is making VS Code work for developers first.

With Retrieve Source from Org, please make it not overwrite if you have not deployed yet - MavensMate did that and it could be quite dangerous if you havent committed or deployed (but hopefully you'd have the changes in your local git anyway).

Absolutely need the ability to retreive metadata within a Working Set then deploy that Working Set to another Org, as @ChuckJonas said.

Absolutely need the ability to create a Metadata file and add that to a selected Working Set. @gbreavin mentioned Code, but I would often create brand new Reports, or brand new Layouts or brand new Email Templates via MavensMate. You can NOT do this at all in The Welkin Suite and it's one of it's huge limitations.

Can you still Deploy single Metadata files at a time. I think this is now possible with Deploy on Save? That might be good enough to work around the adding new Metadata if that's not possible.

Yes another +1 for Compare with Org as mentioned by @ChuckJonas but not entirely sure how that would work in practice. It's probably something that could be done with a diff extension anyway.

It's really imperative that anything you do handles Managed Packages really well. MavensMate had an option to include or exclude Managed Packages. Unfortunately the way The Welkin Suite handles managed packages is not OK at all, and they don't seem to retrieve customisations to Managed Packages at all (eg if I build a Process Builder on a Managed Package object) (but hopefully that has changed since I last looked). If you are mainly dealing with wildcards then it seems though it would include Managed Package items also.

Whilst Wildcards are OK, I would still like to have the option to Select Individual Metadata Items from the org, just like MavensMate allowed. But that could be an enhancement for later. I've just dealt with large orgs - say that have FinancialForce installed, and just downloading the Reports metadata fails completely becuase there are too many reports. I really don't want to download any FF reports.

Will the Org Browser work with large orgs - yep, I've had MavensMate fail to retrieve just the fulll Metadata list from a large org, let alone the Metadata itself.

If I can have Working Sets in any way shape or form mean that I don't have to deal with Change Sets anymore, I will be ALL over it! Thank you!

[FabienTaillon]

So, basically a working set will translate to a package.xml, right ? It would just be a UI for package.xml, or am I missing something ?

[snugsfbay]

Since I already have the ability to retrieve, update and deploy metadata and code in a package.xml file (which may be a change set or a package defined in any of my orgs) I have already broken our org into major components via setup. Admins and Devs have the same access to these predefined packages to see how things are related. We also use change sets to define parts of these packages that are currently being worked on in each sandbox. Again, easily retrieved and deployed. So I am not clear on how this would be more beneficial than what we are currently doing? Doesn't this reduce in-org visibility for admins since the working set is not defined in the connected sandbox.

I have to be able to coordinate better with admins who are making changes to the org meta-data and so we all need the same visibility into what is being changed, visibility in the orgs. Also, our change set data is already connected to our work item management so that metadata related to specific work items is captured as part of record keeping for our change management.

+1 the idea by @ChuckJonas for an org compare option that would include the ability to compare profile/permission sets/sharing rules for those times when config changes get made in the wrong sandbox.

[ntotten]

@FabienTaillon So they are very similar, but what we want to actually do is eliminate the need for most people to use package.xml files directly. In conversations with developers - many people seem to either not fully understand them or dislike the complexity. In my opinion, the idea of editing a package.xml file (even with UI) and then doing a deploy and retrieve is sort of redundant or unnecessary in many cases. There are two main cases I am aiming to solve with working sets:

1. Developers that work on not so huge orgs that just want EVERYTHING. Why bother building a package.xml that have everything? This UI should make that case easier by skipping the step of building the package.xml. Just retrieve everything and then deploy what you need at any time - no need to create a file.
2. Developers that want to easily pull a few things from the org and do some work and quickly push them back. Again, it's skipping the intermediary file.

The idea would be that you can export a "working set" to a package.xml file if you want to do that. But I think most people won't need to do that - they can just work directly in the UI. For many users, the package.xml file works fine and it will continue to work. This is really just another option for people that want more simplicity.

[ntotten]

@snugsfbay See above comment, but nothing in this proposal reduces your ability to use package.xml files. For your cases, I think package.xml makes complete sense. But not all developers have such clearly defined and created packages.

[msrivastav13]

@ntotten Dev orgs or sandboxes today provide ability to create unmanaged container and users can add metadata and delete metadata as needed .How is this feature different than change sets and unmanaged containers ?

Also retrieving contents from packages automatically gives package.xml as needed and manages dependencies automatically .

[ntotten]

@JodieM

Can I have multiple Working Sets on the org at the same time? It seems so from your mockup, but I'm not sure. But that would be cool - eg I'm working on code, but I need to just quickly deploy some field and layout changes - it would be good to have those separated.

I think we will start with a single one, but quickly add multiple. A single one is just for the first release. I think multiples make sense.

I would want the option to Add All Metadata to Working Set, so I can do a backup of the Org.

Makes sense. I think there are many scenarios where you want to pull everything.

I would want to be able to Import a Working Set - eg I have a standard list of things I work on from one org to the next - for me that's all Object stuff, all workflows, flows, process builders, all Email Templates, etc - all the main declarative stuff. Similar to the "all the code" default package.xml you include now but for declarative side of things.

I think this is where you would still use package.xml. You could use the "working set" idea to build a package.xml and export it, but for well-defined groups package.xml files are the right option.

Or Default Working Sets - Illuminated Cloud has two sets of metadata to choose from, just to make the metadata list be not so overwhelming. The Welkin Suite has an Admin mode and a Developer Mode. Something like that would be a good starting point, especially for Admins who don't want to look at the code. But I do understand your priority is making VS Code work for developers first.
Good feedback. I think we could add something like this over time.

With Retrieve Source from Org, please make it not overwrite if you have not deployed yet - MavensMate did that and it could be quite dangerous if you havent committed or deployed (but hopefully you'd have the changes in your local git anyway).

Yes, see #633

Absolutely need the ability to retrieve metadata within a Working Set then deploy that Working Set to another Org, as @ChuckJonas said.
Yup, this would be as simple as switching your default org and doing a deploy/retrieve

Can you still Deploy single Metadata files at a time. I think this is now possible with Deploy on Save? That might be good enough to work around the adding new Metadata if that's not possible.

Yes, you can deploy a single metadata file - you can do this today actually by right clicking on the file and clicking deploy. Deploy on save also supports this.

Yes another +1 for Compare with Org as mentioned by @ChuckJonas but not entirely sure how that would work in practice. It's probably something that could be done with a diff extension anyway.

Yup, its something we will do but not right away. This UI will serve as the starting point though.

It's really imperative that anything you do handles Managed Packages really well. MavensMate had an option to include or exclude Managed Packages. Unfortunately the way The Welkin Suite handles managed packages is not OK at all, and they don't seem to retrieve customisations to Managed Packages at all (eg if I build a Process Builder on a Managed Package object) (but hopefully that has changed since I last looked). If you are mainly dealing with wildcards then it seems though it would include Managed Package items also.

Good feedback.

Whilst Wildcards are OK, I would still like to have the option to Select Individual Metadata Items from the org, just like MavensMate allowed. But that could be an enhancement for later. I've just dealt with large orgs - say that have FinancialForce installed, and just downloading the Reports metadata fails completely becuase there are too many reports. I really don't want to download any FF reports.

Yes, absolutely. You can choose wildcards or individual metadata.

Will the Org Browser work with large orgs - yep, I've had MavensMate fail to retrieve just the fulll Metadata list from a large org, let alone the Metadata itself.

This is something we will support. There are questions on how we do this, but its something we will test. We may put in some limit in the first release (i.e. we will only work with orgs with less than 5k objects or something). But we will do it for sure for GA - it will work on all sizes of orgs.

If I can have Working Sets in any way shape or form mean that I don't have to deal with Change Sets anymore, I will be ALL over it! Thank you!

One thing to be clear here is that this idea is only for development time. Change sets are still useful for deploying between orgs in a safe way. It might be worth considering how we can make that easier or maybe these ideas can merge but that isn't really the initial goal with this idea. This is to make development easier.

[ntotten]

@msrivastav13 The key thing is that "working sets" are ephemeral. They are something you use only while developing on your local machine - they aren't designed to be shared nor are they deployment artifacts. They are designed to make it easy for developers to deploy groups of files while they are doing development.

[snugsfbay]

@ntotten Thank you. Is this just meant for the devs who never want to login to a sandbox and who don't know how to create a change set? Because the change set in their dev org will keep track of metadata for their specific work items and those related changes (as a working set seems to do). This would be useful to us if we could directly relate the working set to the change set in-org so we don't lose the record-keeping that change sets offer in the org for our admins.

For example, when we have had developers leave, the admins can go into that person's dev sandbox and see their pending work items organized in change sets. Depending on whether that dev is in-house, vendor, remote, etc. the tech team usually does not have access to the dev's computer.

[ntotten]

@snugsfbay It is meant for development time - they are temporary and not meant to be artifacts that are shared between developers or use to deploy between orgs for production type deployments. It is a way to deploy groups of files together when doing dev. Change sets are for moving metadata between orgs and would still be the best way to do that. This isn't a replacement.

[snugsfbay]

Right, that's why I would find them more useful if they were tied directly to change sets. We have devs who come and go so keeping an individual's dev sandbox where all the admins and devs have access to pending work items organized in change sets has been beneficial to us.

[msrivastav13]

Thanks @ntotten for clarifying .I see your point , sounds like it is similar to change set builder locally on the vscode editor and to me personally as an admin salesforce UI feels more native and user friendly.However will be watching this closely as well .For dev seems like this is very good for retrieving package.xml which seems to be what most of us from mavens world miss .

[ntotten]

@snugsfbay Makes sense. I think there is an opportunity to evolve into that or provide separate UI for making change sets easier to work with. I am reluctant to try to solve too many problems though with one feature. Right now I want to focus on the dev time experience. We absolutely need to address the changeset experience and we do have a lot of ideas/plans for that too. I will share more details on those areas at some point in the future.

[ntotten]

@msrivastav13 Exactly, as I mentioned in the above comment, there is certainly more we can and will do around changesets, but I don't want to try to solve too many problems at once with one feature. We will be sharing more with changesets later.

[snugsfbay]

@ntotten just to be clear, I was not requesting a change to change sets as much as I was requesting making work sets visible to admins in ways they already have at their finger tips because they also are responsible for change management and org audits and the work set concept could potentially obfuscate those processes for them.

[ntotten]

@snugsfbay I see. I don't feel that the working set concept should interfere with what admins are doing. It is really a dev time convenience only. And is an extension of what is already available today - partial deploy/retrieve. The difference is today you have to write a package.xml then run sfdx force:source:deploy --manifest ./manifests/package.xml

This is fundamentally just a convenience layer to do the above with the UI and without having to explicitly write a file.

[ntotten]

As an alternative UI, it might be nice to do something like the Git ui has for selecting files to add to staged changes. I am not sure if that type of UI can be built though with the treeview. But it may be nice to have both the metadata and selected items in the same VS Code pane. Something to consider when we are investigating UI.

[snugsfbay]

@ntotten Here's a post describing my process for Enterprise and how easy it is to create and edit package.xml via Salesforce UI with Setup options for both Packages and Change Sets in the org (where admins and devs collaborate): http://www.snugsfbay.com/2019/01/easy-sfdx-for-enterprise-and-others.html

[JodieM]

@ntotten where would you like feedback / suggestions for improvement on the Org Browser functionality - comments on here, or new Ideas? What has happened to the Working Sets functionality, is that disappeared now or will that be split into a separate item? What is the planned new features for the Org Browser, it looks from the merged items that it is planned to be able to create or update the package.xml, but what is the rest of the roadmap?

[ntotten]

Since we have shipped the org browser, I am closing this. We will continue the discussion for package.xml creation/tools on #1710

[bazidev]

it seems that vs-code will not support search and filtering as API for tree view any soon, but they've added some cool capabilities for limited search/filter, you can find it here:
new-tree-widget-adopted-in-tree-view