Python Requests: comma-separated query parameters

This is something you will have a hard time finding anywhere else.

For some particular communication partners, it was necessary to provide a query parameter to their REST API endpoint as a comma-separated list, like so:

https://server.tld/resource?param=value1,value2,value3,value4

If you do that using the defacto requests library used with the python programming language, the requests library will separate the values and place each into their own parameter, like so:

https://server.tld/resource?param=value1&param=value2&param=value3&param=value4

That appears an acceptable implementation, because the HTTP RFC allows for both and might imply that communication partners could treat both equally.

They don’t.

This one particular communication partner (Amazon Selling-Partner API) does accept the parameter array broken out like that, but it will think you are overwriting earlier parameters of the same name with later ones, so it will only see the last one you supply. Thus, we must force the request to supply a comma-separated parameter array, as shown above.

Research across the internet has come up with the suggestion that one might need to replace the comma with its url-encoded (percent-encoded) counterpart: %2C. And though the python requests library does allow that, and will keep the parameter values together rather than breaking them out, the remote REST API handler does not like that at all. It sees the %2C and says: hey, that is a single value rather than an array of multiple values. And thus whatever you needed it to do, fails.

Other advise suggests to switch out the requests library with one’s own low-level HTTP processing.

Let’s not.

With Amazon Selling-Partner API requiring us to sign any HTTP requests with their own proprietary signature algorithm, it is necessary to get the query parameters into the request before computing the signature.

The solution, at the time of writing, is a 2-step approach.

First, we set the query parameter, joining the values with an uncommon character for their contents, like a pipe:

values = ['value1', 'value2', 'value3']
request = requests.Request(
  method='GET',
  params={
    'param': '|'.join(values)
  }
)

Second, we alter the request parameter by substituting the pipe for a comma:

request.params['param'] = request.params['param'].replace('|', ',')

That has the desired effect. Now we can calculate the request signature that Amazon’s SP-API requires.

Afterthoughts

This solution is based on how the python requests library processes the query parameters when setting up a new request. It is going to be necessary to review this solution if the library maintainers decide to change their implementation in future versions.

About the author

Veltstra is a hands-on software architect and engineer with over 30 years of experience finding solutions to overcome all kinds of problems. Some of those problems are caused by tools and services created by others, unironically aimed at making our lives easier.