Understanding the new Oracle Commerce Cloud Search indexing and publishing endpoints
As part of the 19C release (also known as 19.4, available starting in August 2019 or thereabouts) a series of new REST endpoints was released for Search.
These endpoints help with certain indexing and publishing tasks. However, there is an overlap in functionality between the existing endpoints and these new ones.
This article seeks to provide an understanding of these new endpoints and compare them to the existing ones.
What is the new indexing endpoint?
Starting in 19.4, there is a new tasks endpoint under /gsadmin
First, for indexing, you can invoke POST /gsadmin/v1/cloud/tasks { “type” : “baseline” } or you can POST { “type” : “partial” }
This will perform indexing operations purely on the Search side. We’ll discuss that more in a moment.
(To understand baseline updates vs partial updates, read “Understanding Indexing in Oracle Commerce Cloud Search”)
After you submit your indexing request, in the response JSON, you will have a task identifier. For instance, I kicked off a baseline locally, and my response looks like:
{"o:rspDetails": "Accepted operation: baselineUpdate",
"callback": "http://server/gsadmin/v1/cloud/tasks/BaselineUpdateTask-k045aol0-329",
"title": "Operation request",
"type": "https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.3",
"status": "202"}
So what was created? It creates an asynchronous task to perform the baseline update. In this case, my task is called “BaselineUpdateTask-k045aol0–329”
With that, we can now execute GET /gsadmin/v1/cloud/tasks/BaselineUpdateTask-k045aol0–329 in order to get its status.
Executing GET against that task ID returns this:
{"o:rspDetails": "Task '/apps/cloud/tasks/BaselineUpdateTask-k045aol0-329' has finished processing",
"title": "Operation request",
"type": "https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.1",
"status": "200"}
There are a few possible status codes to consider:
- 200 : Finished successfully
- 202: Still going
- 500 : Failure
- 404: You are trying to get the status for a wrong ID
What is the new publishing / content promotion endpoint?
One issue that many folks run into is the fact that changes to Search configuration (either using the Search admin UI in OCC Admin or using REST) don’t show up in the list of Publishing changes in OCC Admin.
Instead, there is a separate mechanism (for better or worse) for taking Search changes from your Preview to Live storefronts. Much of this was discussed in this article: https://medium.com/oracledevs/understanding-indexing-for-search-in-oracle-cloud-commerce-e4829109c03a
Prior to 19C / 19.4, in order to get your Search configuration changes from Preview to Live storefronts, you needed to kick off indexing. You could either use /ccadmin/v1/search/index directly, or you would make some small change to a product and then Publish using the UI.
Now, there is a new Publishing endpoint that just works against publishing Search configuration changes.
After you have made Search configuration changes, you can invoke POST /gsadmin/v1/cloud/tasks { “type” : “publish” } . Much like with “type” : “baseline” or “partial”, this kicks off an asynchronous task. You can then GET the /tasks endpoint to check its status
How do these endpoints compare to the existing /ccadmin/v1/search/index endpoints?
This new set of endpoints is in addition to the existing ones. Let’s discuss how they work:
/ccadmin/v1/search/index: { “op” : “baseline-full-export” } : This re-exports the entire catalog and configuration. Performs a full re-index. Publishes search configuration and content changes
/ccadmin/v1/search/index: { “op” : “baseline” }: Exports only the changed catalog records. Performs a full re-index. Publishes search configuration and content changes.
/gsadmin/v1/cloud/tasks: {“type” : “baseline” } : Performs a full re-index of the data already in Search.
/ccadmin/v1/search/index: { “op” : “partial” }: Exports only the changed catalog records. Performs a partial index. Publishes search configuration and content changes
/gsadmin/v1/cloud/tasks: { “type” : “partial” } : Performs a partial index of data already in Search
/gsadmin/v1/cloud/tasks: { “type” : “publish” } : Only publishes search configuration and content changes
(The above are listed in the order roughly from slowest to fastest. A baseline-full-export will take the longest, and “type”:“publish” will execute very quickly)
A few things to point out: Using /ccadmin/v1/search/index exports content from the catalog to Search. However, using /gsadmin/v1/cloud/tasks, indexing only uses whatever has been previously exported. (Meaning, all of the processing happens on the Search side)
Using /ccadmin/v1/search/index (whether it’s for a “baseline”, “baseline-full-export” or “partial”) will kick off a Search content publish at the end. Now, we can use /gsadmin/v1/cloud/tasks { “op” : “publish”} instead.
That brings us to the next section.
When would I use these new endpoints?
Okay, so you’re probably confused by this new set of endpoints and when you would actually use it. That’s understandable.
First, invoking the /tasks endpoint with {“type”: “publish”} is probably the most useful of these endpoints. This allows you to make a search content change and publish it WITHOUT having to do any type of indexing. This means it will take effect very very quickly.
What do I mean by search content change?
- Thesaurus entries
- Dynamic curation
- changes to Search Interfaces
- Boost/bury
- Facet ordering
You can now invoke the /tasks endpoint to publish those changes so that they take effect on your Live storefronts very quickly and no indexing is required. This should typically take only 1 or 2 seconds.
For using /tasks with “type”: “baseline” or “partial”, this is useful only in a handful of situations.
- You are using the /gsdata endpoint to index non-catalog content. This was discussed in depth in this article: https://medium.com/oracledevs/performing-geolocation-searches-and-filtering-in-oracle-commerce-cloud-search-646646e11fd8
- You are making schema-level changes to Search configuration. This might include changes using the /attributes endpoints (perhaps creating new facets, changing search settings, etc)
The benefit here is that you are performing indexing operations without having to re-export some or all of the catalog. So in the article listed above, we performed indexing of some Geolocation data. Instead of having to invoke /ccadmin/v1/search/index, we can now only invoke /tasks and it should finish much quicker.
Help me understand what I’m seeing when I invoke /ccadmin/v1/search/index, and how that compares to these new endpoints?
Okay, when you perform a GET against /ccadmin/v1/search/index, you are looking at the status of indexing from the Oracle Commerce Cloud admin side.
This shows a number of tasks related to exporting the catalog and configuration to Search for indexing. A number of interesting things happen here that are worth pointing out. (Note that the names and tasks are subject to change in the future):
- SchemaExporter: Takes the configuration from the OCC catalog and exports it to /gsadmin/v1/cloud/attributes and /gsadmin/v1/cloud/searchIndexConfig
- ProductCatalogOutputConfig: This exports the catalog (either the whole thing or only what has changed) to search before indexing proceeds.
- EndecaScriptService: This task represents kicking off the indexing in Search. This actually calls the new /gsadmin/v1/cloud/tasks for either type=baseline or type=partial.
- PromoteContentIndexable: This task represents publishing Search configuration and content changes (such as Dynamic Curation, search interfaces, etc). This actually calls the new /gsadmin/v1/cloud/tasks endpoint using “type” : “publish”
Previously, in the on-premise Guided Search product, there was a script called “promote_content.sh”. In Cloud, that process is now renamed as publishing. So to be clear:
promote_content.sh == PromoteContentIndexable == /tasks { “type” : “publish” }
Conclusion
Hopefully, you’ve found this article helpful in understanding the new endpoints and when they would be used.
Here are links to our previous articles:
Explaining Configuration in Oracle Commerce Cloud Search
Performing Geolocation Searches
Filtering Search results based on inventory
Limiting Fields returned in the JSON response from Search
Understanding and controlling the order of search results
Controlling facet and facet value ordering in search
