Programmatic CRUD operations on Adobe Dynamic Media

Photo by Marvin Meyer on Unsplash

Prelude

For years I have been using Scene7 or Adobe Dynamic Media. Never deceived, always worked great, easy to configure, NextGen images, CDN caching, and all its goodies have been so tasty.

Until recent, in my DAM migration project, had to dig into a rabbit hole and explore dark side of Dynamic Media, SOAP APIs!

Image Production System API

Adobe dynamic media exposes IPS APIs to interact with DM programatically. And yes you heard right, these are SOAP APIs. I wrote SOAP web services on IBM WAS servers till 2010. After 14 years, was surprised to call SOAP services, like a dejavu.

Problem Statement

We were migrating a million assets from non-aem application into AEM. Initial import, we ran into multiple issues explained in my other article, took adobe help and managed to complete initial migration. But upon reconciling, found, many assets were

1. Created in AEM author, but NOT synced into dynamic media
2. Created in AEM, created at DM, but not PUBLISHED at DM
3. Created at AEM, created at DM, but DM counted as duplicate and appended numerals into names
4. Created in AEM, created in DM, but only the duplicate -1 image was in published state. Need to rename and republish.

Totally 13092 assets were out of sync between AEM and DM and had to straighten. We started fixing usecase one by one.

General Code format

This blogpost by Sreekanth helped to kick start. All the request response objects for SOAP service is available in uber jar under `com.scene7.ipsapi` package. There are 4 steps

1. Prepare request string
2. Prepare AuthHeader string
3. Place SOAP request
4. Unmarshall response (if required)

Prepare Request string

So we start by creating a request object instance. All the services available to call s7 are listed in documentation. After instantiating request object and populating input values, we use JAXB to marshal the request object into string. SOAP services are XML based (unlike REST thats mostly json based). So we marshal the request object into XML document string using

        Marshaller marshaller = getMarshaller(requestObj.getClass());
        StringWriter sw = new StringWriter();
        marshaller.marshal(requestObj, sw);
        return sw.toString();

Prepare AuthHeader string

This is one time code that ll look like this

        AuthHeader authHeader = new AuthHeader();
        authHeader.setUser(S7_USER);
        authHeader.setPassword(S7_PASS);
        authHeader.setAppName(STAND_ALONE_APP_NAME);
        authHeader.setAppVersion("1.0");
        authHeader.setFaultHttpStatusCode(Integer.valueOf(200));
        Marshaller marshaller = getMarshaller(AuthHeader.class);
        StringWriter sw = new StringWriter();
        marshaller.marshal(authHeader, sw);

Here the S7_USER, S7_PASS are hardcoded. Instead they can be fetched from S7Config OSGI service like this

@Reference
private S7ConfigResolver s7ConfigResolver;

@Reference
private CryptoSupport cryptoSupport;

ResourceResolver s7ConfigResourceResolver = resourceResolverFactory.getServiceResourceResolver(
    Collections.singletonMap("sling.service.subservice", (Object)"MyApp"));

S7Config s7Config = s7ConfigResolver.getS7ConfigForAssetPath(s7ConfigResourceResolver, asset.getPath());

if(s7Config == null) {
    s7Config = s7ConfigResolver.getDefaultS7Config(s7ConfigResourceResolver);
}

String password = cryptoSupport.unprotect(s7Config.getPassword());
String user = s7Config.getEmail();

For my case, it was onetime execution, so we ran from RDE outside Production. But incase this needs to run at Production, it is bad practice to hardcode credentials. Should be fetched from S7Config as shown above.

Place SOAP request

The request string and auth string are appended IPS Request tag and a HttpPost method is invoked onto IPS service. The Post method code looks like this

        String authHeaderStr = prepareAuthHeaderStr();
        CloseableHttpClient client = null;
        String responseBody;

        try {
            SocketConfig sc = SocketConfig.custom().setSoTimeout(180000).build();
            client = HttpClients.custom().setDefaultSocketConfig(sc).build();

            HttpPost post = new HttpPost(S7_NA_IPS_URL);
            StringEntity entity = new StringEntity("<Request xmlns=\"http://www.scene7.com/IpsApi/xsd/2017-10-29-beta\">" + authHeaderStr + apiMethod + "</Request>", "UTF-8");

            post.addHeader("Content-Type", "text/xml");
            post.setEntity(entity);
            HttpEntity responseEntity = client.execute(post).getEntity();
            responseBody = IOUtils.toString(responseEntity.getContent(), StandardCharsets.UTF_8);
        } finally {
            if (client != null) {
                client.close();
            }
        }

Unmarshal Response

Some of the getter operations returning DM data, requires unmarshalling response to process further. This can be very simple string regex to read interested data or may need to unmarshal using XPath and JAXB. Unmarshalling respond code into java object looks like this

  private List<S7Asset> parseGetAssetByNameResponse(String responseBody) throws Exception {
        String expression = "/getAssetsByNameReturn/assetArray/items";
        NodeList itemList = getDocumentNodeList(responseBody.getBytes(), expression);
        List<S7Asset> s7assets = new ArrayList<>();
        for (int i = 0; i < itemList.getLength(); i++) {
            Node item = itemList.item(i);
            if (item.getNodeType() != Node.ELEMENT_NODE) {
                continue;
            }
            Element eElement = (Element) item;
            String assetHandle = getTextContent(eElement, "assetHandle");
            String name = getTextContent(eElement, "name");
            s7assets.add(new S7Asset(assetHandle, name));
        }
        return s7assets;
    }

Here the responseBody is an XML document. We use xpath expression `/getAssetsByNameReturn/assetArray/items` to read the items array. The getDocumentNodeList goes like this

 private NodeList getDocumentNodeList(byte[] responseBody, String expression) throws
            ParserConfigurationException, IOException, SAXException, XPathExpressionException {
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        DocumentBuilder builder = factory.newDocumentBuilder();
        ByteArrayInputStream input =  new ByteArrayInputStream(responseBody);
        Document doc = builder.parse(input);
        XPath xPath =  XPathFactory.newInstance().newXPath();
        return (NodeList) xPath.compile(expression).evaluate(doc, XPathConstants.NODESET);
    }

returns the DOM node list based of the xpath requested `/getAssetsByNameReturn/assetArray/items`.

This ll unmarshal and map the response into a List of S7Assets objects. Now the response can be accessed as simple pojo object.

Putting all together

Now we ll put all above together. Here I am renaming an asset having -1 in its name. The new name removes -1, validates for duplicates and republishes the asset under the new name.

import com.scene7.ipsapi.*;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.StringUtils;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.config.SocketConfig;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.servlets.HttpConstants;
import org.apache.sling.api.servlets.SlingSafeMethodsServlet;
import org.osgi.service.component.annotations.Component;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;

import javax.servlet.Servlet;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.StringWriter;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

import static org.apache.sling.api.servlets.ServletResolverConstants.*;

@Component(
        service = Servlet.class,
        name = "Servlet to search an asset in DM, rename by removing -1 in its name and republish the asset with new name",
        property = {SLING_SERVLET_PATHS + "=/bin/demo/renames7", SLING_SERVLET_METHODS + "=" + HttpConstants.METHOD_GET})
public class RenameS7AssetServlet extends SlingSafeMethodsServlet {

    private static final String S7_NA_IPS_URL = "https://s7sps1apissl.scene7.com/scene7/api/IpsApiService"; //available in AEM http://localhost:4502/libs/settings/dam/scene7/endpoints.html
    private static final String S7_COMPANY_HANDLE = "c|1234"; // available in AEM under /conf/global/settings/cloudconfigs/dmscene7/jcr:content
    private static final String S7_USER = "demo";
    private static final String S7_PASS = "demo";
    private static final String STAND_ALONE_APP_NAME = "demo";

    @Override
    protected void doGet(final SlingHttpServletRequest slingRequest, final SlingHttpServletResponse slingResponse) {
        Set<String> imagesToRename = new HashSet<>();
        imagesToRename.add("helloworld-1");
        imagesToRename.add("secondasset-1");
        try {
            renameAndRepublish(imagesToRename);
        } catch (Exception e) {
            // log
        }
    }

    private void renameAndRepublish(Set<String> namesWithOne) throws Exception {
        String responseBody = makeSoapRequest(prepareGetAssetsByNameParamRequest(namesWithOne));
        List<S7Asset> s7assetsWithHyphenOne = parseGetAssetByNameResponse(responseBody);
        for (S7Asset asset : s7assetsWithHyphenOne) {
            makeSoapRequest(prepareRenameAssetRequest(asset.getAssetHandle(), StringUtils.removeEnd(asset.getAssetName(), "-1")));
            makeSoapRequest(prepareSetAssetPublishStateParamRequest(asset.getAssetHandle()));
        }
    }

    private String prepareSetAssetPublishStateParamRequest(String assetHandle) throws Exception {
        SetAssetPublishStateParam pub = new SetAssetPublishStateParam();
        pub.setAssetHandle(assetHandle);
        pub.setCompanyHandle(S7_COMPANY_HANDLE);
        pub.setPublishState("MarkedForPublish"); // to unpublish, value is NotMarkedForPublish
        Marshaller marshaller = getMarshaller(pub.getClass());
        StringWriter sw = new StringWriter();
        marshaller.marshal(pub, sw);
        return sw.toString();
    }

    private String prepareRenameAssetRequest(String assetHandle, String assetName) throws Exception {
        RenameAssetParam rename = new RenameAssetParam();
        rename.setAssetHandle(assetHandle);
        rename.setValidateName(true); // this will validate for duplicate names under the company
        rename.setCompanyHandle(S7_COMPANY_HANDLE);
        rename.setNewName(assetName);
        Marshaller marshaller = getMarshaller(rename.getClass());
        StringWriter sw = new StringWriter();
        marshaller.marshal(rename, sw);
        return sw.toString();
    }

    private String getTextContent(Element eElement, String tagName) {
        return eElement.getElementsByTagName(tagName).item(0).getTextContent();
    }

    private NodeList getDocumentNodeList(byte[] responseBody, String expression) throws
            ParserConfigurationException, IOException, SAXException, XPathExpressionException {
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        DocumentBuilder builder = factory.newDocumentBuilder();
        ByteArrayInputStream input = new ByteArrayInputStream(responseBody);
        Document doc = builder.parse(input);
        XPath xPath = XPathFactory.newInstance().newXPath();
        return (NodeList) xPath.compile(expression).evaluate(doc, XPathConstants.NODESET);
    }

    private List<S7Asset> parseGetAssetByNameResponse(String responseBody) throws
            XPathExpressionException, ParserConfigurationException, IOException, SAXException {
        String expression = "/getAssetsByNameReturn/assetArray/items";
        NodeList itemList = getDocumentNodeList(responseBody.getBytes(), expression);
        List<S7Asset> s7assets = new ArrayList<>();
        for (int i = 0; i < itemList.getLength(); i++) {
            Node item = itemList.item(i);
            if (item.getNodeType() != Node.ELEMENT_NODE) {
                continue;
            }
            Element eElement = (Element) item;
            String assetHandle = getTextContent(eElement, "assetHandle");
            String name = getTextContent(eElement, "name");
            s7assets.add(new S7Asset(assetHandle, name));
        }
        return s7assets;
    }

    private String prepareGetAssetsByNameParamRequest(Set<String> assets) throws JAXBException {
        GetAssetsByNameParam getAsset = new GetAssetsByNameParam();
        StringArray types = new StringArray();
        types.getItems().add("Image");
        getAsset.setAssetTypeArray(types);
        getAsset.setCompanyHandle(S7_COMPANY_HANDLE);
        StringArray names = new StringArray();
        assets.forEach(names.getItems()::add);
        getAsset.setNameArray(names);
        Marshaller marshaller = getMarshaller(getAsset.getClass());
        StringWriter sw = new StringWriter();
        marshaller.marshal(getAsset, sw);
        return sw.toString();
    }

    private String prepareAuthHeaderStr() throws JAXBException {
        AuthHeader authHeader = getS7AuthHeader();
        authHeader.setFaultHttpStatusCode(Integer.valueOf(200));
        Marshaller marshaller = getMarshaller(AuthHeader.class);
        StringWriter sw = new StringWriter();
        marshaller.marshal(authHeader, sw);
        return sw.toString();
    }

    private String makeSoapRequest(String apiMethod) throws IOException, JAXBException {
        String authHeaderStr = prepareAuthHeaderStr();
        CloseableHttpClient client = null;
        String responseBody;

        try {
            SocketConfig sc = SocketConfig.custom().setSoTimeout(180000).build();
            client = HttpClients.custom().setDefaultSocketConfig(sc).build();

            HttpPost post = new HttpPost(S7_NA_IPS_URL);
            StringEntity entity = new StringEntity("<Request xmlns=\"http://www.scene7.com/IpsApi/xsd/2017-10-29-beta\">" + authHeaderStr + apiMethod + "</Request>", "UTF-8");

            post.addHeader("Content-Type", "text/xml");
            post.setEntity(entity);
            HttpEntity responseEntity = client.execute(post).getEntity();
            responseBody = IOUtils.toString(responseEntity.getContent(), StandardCharsets.UTF_8);
        } finally {
            if (client != null) {
                client.close();
            }
        }
        return responseBody;
    }


    private AuthHeader getS7AuthHeader() {
        AuthHeader authHeader = new AuthHeader();
        authHeader.setUser(S7_USER);
        authHeader.setPassword(S7_PASS);
        authHeader.setAppName(STAND_ALONE_APP_NAME);
        authHeader.setAppVersion("1.0");
        authHeader.setFaultHttpStatusCode(Integer.valueOf(200));
        return authHeader;
    }

    private Marshaller getMarshaller(Class apiMethodClass) throws JAXBException {
        Marshaller marshaller = JAXBContext.newInstance(apiMethodClass).createMarshaller();
        marshaller.setProperty("jaxb.formatted.output", Boolean.TRUE);
        marshaller.setProperty("jaxb.fragment", Boolean.TRUE);
        return marshaller;
    }

    private static class S7Asset {
        private String assetHandle;
        private String assetName;

        public S7Asset(String handle, String name) {
            assetHandle = handle;
            assetName = name;
        }

        public String getAssetHandle() {
            return assetHandle;
        }

        public void setAssetHandle(String assetHandle) {
            this.assetHandle = assetHandle;
        }

        public String getAssetName() {
            return assetName;
        }

        public void setAssetName(String assetName) {
            this.assetName = assetName;
        }

        public String toString() {
            return "Name: " + assetName + "; Asset handle: " + assetHandle;
        }
    }
}

Finally just have to call the servlet and it ll rename. We validated by checking the expected asset url from browser before and after this script, and able to confirm the expected asset url works. Hope this helps to run similar script for other available IPS operations.