Many Salesforce developers and system administrators use the Metadata API to manage the change and release process for complex orgs. I was one of the first people to start using this API over 10 years ago when our Snapshot product was originally published. I really like the Metadata API, but anything this powerful is bound to have a few eccentric quirks and special use cases that need to be considered.
This blog covers some of the most common technical issues that a Salesforce developer or system administrator working with the Metadata API is likely to encounter. In particular, we are going to discuss the Retrieve API, which pulls assets from an org, the Deploy API, which pushes assets to an org, and the List API, which tells you what assets are in the org.
Sounds simple enough, right? Please fasten your safety belts.
Deploying Managed and Unmanaged Packages
Contrary to popular belief, both managed and unmanaged packages can be deployed with the Metadata API. But the process is quite a bit different than working with unpackaged data. For unmanaged packages, you cannot use the wildcard asterisk * in the package.xml file when you call the Deploy API. You have to list every asset and child asset type explicitly by name. You also need to specify the “fullName” parameter. If you follow these guidelines, then unmanaged packages can be moved between orgs with ease.
By the way, even when deploying unpackaged assets, don’t mix asset types and child types with asterisks. For example, if you deploy Custom Objects and Custom Fields together, then the Custom Object asterisk will reference the partial Custom Field data and return an error. Rule of thumb: asterisks work fine for retrieving assets, but don’t use them for deployments.
Deploying a managed package is also possible, but the method is really unusual. There is a metadata type called Installed Packages. These assets are named after the namespaces of the currently installed packages. If you deploy one of them, and only one, then that managed package will be installed in the destination org. You will actually receive an email, just like somebody clicked “Get It Now” on the AppExchange. This is the only metadata type that sends out emails, and the only asset that must be deployed alone.
Strange List Metadata Results
One of the most mysterious calls in the Metadata API is List Metadata. Basically, this routine will return the available asset names for any metadata type. For example, if you ask for Custom Objects, then you will get the names of all custom objects, all packaged objects, and some standard objects. This is useful because an asterisk will only retrieve custom objects. The packaged and standard objects must be explicitly named.
The List Metadata API will return Custom Tab names, but not the tabs with a “standard-” prefix. However, all Custom Applications will be returned, including those with a “standard_” prefix. You can get the names of the Profiles, but they are not the same names used in the Salesforce HTML interface. Some things are not included. Custom Permissions are listed but User Permissions are not. The List Metadata API returns partially incorrect information for Page Layouts. This issue is discussed next.
Page Layout Names and Managed Packages
A Page Layout name in the Metadata API will be prefixed by the object name and a dash. But packages can also contain layouts for standard, custom, and packaged objects. Here is a field guide to Page Layout names:
Std Object and Unpkg Layout: Account-mylayout
Cust Object and Unpkg Layout: myobj__c-mylayout
Pkg Object and Unpkg Layout: myspace__myobj__c-mylayoutStd Object and Pkg Layout: Account-myspace__mylayout
Cust Object and Pkg Layout: myobj__c-myspace__mylayout
Pkg Object and Pkg Layout: myspace__myobj__c-myspace__mylayout
The problem is that List Metadata will return an unpackaged layout name for any packaged layout. If you want to work with packaged layouts, you must synthesize the correct name for the Retrieve API call. By the way, you cannot tell if a layout belongs to a managed packaged by looking for a namespace prefix. The actual namespace will be found after the dash. Layouts are the only metadata type that work this way.
Standard Assets Complicate Deployment
Dependencies between custom assets have always complicated deployment. Related assets need to be grouped together for deployment unless they already exist on the destination. Profiles and Permission Sets are often tangled up with many other asset types. Sometimes admins must clean up XML files by hand in order to get complex deployments to work.
But over the last few years, Salesforce has introduced new types of orgs with different standard fields, tabs, applications, and user preferences. For example, Data.com adds the CleanStatus field to the Account object. Just like custom assets, these new standard assets are also referenced by Profiles, Permission Sets, Page Layouts, and other metadata types.
Standard assets have made Metadata deployments between different types of orgs more challenging. Unlike custom assets, you can’t add a standard asset to a deployment. You can’t create a standard asset on the destination, either. Any reference to a standard asset on the source will break the deployment because of missing dependencies.
To make matters worse, standard assets are also difficult to discover. For example, there is no way to get a comprehensive list of User Preferences with any Salesforce API. In some cases, the only way to discover a bad reference is when the Metadata API returns an error. Snapshot has an option to automatically remove bad references that stop deployment, but missing references to standard assets must often be added to this list by hand.
Canonical Sorting and Metadata Differences
Many of the XML documents returned by the Metadata API are in Canonical Order. In other words, the elements have been sorted alphabetically. For example, the child types in a Profile are all carefully sorted. Other XML documents rely on the order of the elements. You can see this in pick lists like Standard and Global Value Sets. Other XML documents will be scrambled randomly every time you retrieve them, such as Clean Data Services and Site Dot Coms. This causes confusion when looking at difference comparisons between orgs or over time. Snapshot addresses this problem by using canonical sorting for comparison and then using the original unsorted documents for deployment.
Sparse Matrix Considerations in Profiles
Some of the child types in Profiles are sparse. Tab Visibility, Object Permissions, User Permissions, and Custom Permissions only contain enabled permissions. Disabled permissions are not included in the document. During deployment, Profiles are merged with the destination metadata. So how do you turn off a permission for one of these child types? The answer is that you have to explicitly include an element that disables the permission.
When Snapshot does bulk editing of Profiles, you can “force” permissions to be disabled. In the picture below, some of the “Disabled” User Permissions are grey in color, and others are red. In this manner, the user interface can display the difference between missing permissions and forcibly disabled ones.
All of the child types in Permission Sets are sparse. They only contain enabled permissions. But this never causes problems during deployment, because Permission Sets always replace the destination metadata. You can disable a permission simply by not including it. By the way, Layout Assignments in Profiles also work like Permission Sets. They are sparse, but they always replace the destination metadata.
Vanishing Translation Comments
When you retrieve the metadata types for Translations, Standard Value Set Translations, Global Value Set Translations, and Object Translations the XML will have comments that highlight values that need to be translated. Here is an example in bold, below:
<translation><!-— Five --></translation>
If you replace the comments with translated values and deploy the file, then your translations will work perfectly. But if you retrieve a translation file and then deploy the same file without making any changes, then Salesforce will replace each XML comment with an empty string. This makes all of the commented words magically disappear after translation. You must strip out the parent element of every translation that has a commented value before deploying the file.
Special Formats for Retrieved Metadata
Most metadata types are returned by the Retrieve API in the same way. There is a folder with a specific name containing files that have a special file extension. Each file is an XML document with a base element name and perhaps some child types. But some metadata types are different. There are four asset types with sub-folders: Reports, Dashboards, Email Templates, and Documents. Some asset types with unstructured data have XML companions. The metadata types for Territory2Model, Territory2, and Territory2Rule have a complex folder structure. The Settings type has a different base element name for every included document. Wave Templates and Aura Definition Bundles contain a variety of different files types in folders. Work through all of these special cases and you will be well on your way to taming the wild assets of the metadata jungle.
With Great Power
The Metadata API solves some very difficult problems, but there are also some eccentric quirks and special use cases that need to be considered. I spent years figuring out some of these issues, so hopefully this blog will help other Salesforce developers and system administrators working with the Metadata API get up to speed more quickly. If you would rather focus on managing your org for end users and providing compliance and security for your company right now, then please check out Snapshot on the AppExchange.