Idiomatic Way Handle and Concat Multiple HTTP Requests In Clojure? - api

Question

I'm working on a program that needs to make multiple calls to various microservices, possibly do some processing on those results, and then return a combination of those processes.
A very basic example might look like:
(def urls ["http://localhost:8080" "http://localhost:8080"])
(defn handle-http-request [url]
(let [request (http-kit/get url)]
(do-some-processing (:body #request))))
(defn home-page
[request]
(let [response (pmap handle-http-request urls)]
(ring-resp/response {:buildings (first response) :characters (second response)})))
In this case i'm using pmap to handle running all my requests in parallel, and then return them as JSON that the UI making the request can handle. In the real environment there will be more URLs each fetching different data.
My question is whether this is an appropriate way to handle this problem? I've looked some core.async, and see it as a possible way to handle this but worry it might be overkill? My second concern is handling errors, it would appear that core.async might be able to more elegantly handle issues where a remote has timed out. Am I right in this assumption, or is using pmap okay in this situation?
Lastly, are there any accepted patterns or reading on how to handle microservice architectures like this? I see my problem as relatively specific, but feel the idea of a server making requests to many others and compiling those results is nothing new.

Answer 1

The HTTP-Kit documentation
provides an example for this using futures:
Combined, concurrent requests, handle results synchronously
(let [urls ["http://server.com/api/1" "http://server.com/api/2" "http://server.com/api/3"]
;; send the request concurrently (asynchronously)
futures (doall (map http/get urls))]
(doseq [resp futures]
;; wait for server response synchronously
(println (-> #resp :opts :url) " status: " (:status #resp))
)
Yet, often this is still not enough. Make sure you configure timeouts
and you have an escalation strategy once the timeouts hit. The more
other services you are going to hit, the more complex this task gets.
Maybe have a look at libraries like
Resilience4JClj, that
provide configurable ways to deal with retrying, caching, timeouts, ...
Note: pmap is best suited to CPU intensive tasks. Or as the docs state:
Only useful for
computationally intensive functions where the time of f dominates
the coordination overhead.