HttpURLConnection with manual CookieHandler not working - kotlin

Question

Java's default CookieHandler is good for one instance on one program, but i need many instance to request.
so I coded manual Cookie Handler.
here is my code. (Unirest is needed cause of Cookie thing)
import kong.unirest.Cookie
class CookieJar {
val cookies = arrayListOf<Cookie>()
fun supply(url: String): String {
return cookies.filter { url.contains(it.domain?.trim('.') ?: ".") }
.joinToString(";") { "${it.name}=${it.value}" }
}
fun consume(headers: Map<String, List<String>>) {
val dist = cookies.toMutableList()
val setCookie = headers["Set-Cookie"]
val strings = headers["Set-Cookie2"]
if (setCookie != null) dist.addAll(setCookie.map { Cookie(it) })
if (strings != null) dist.addAll(strings.map { Cookie(it) })
cookies.clear()
cookies.addAll(dist.distinctBy { "${it.domain};${it.name}" })
}
}
and here is my example. (Jackson databind is needed to read json)
import com.fasterxml.jackson.databind.ObjectMapper
import java.io.IOException
import java.net.HttpURLConnection
import java.net.URL
import java.net.URLEncoder
import kotlin.system.exitProcess
private val jar = CookieJar()
private val defaultHeader = hashMapOf(
"Accept" to "*/*",
"User-Agent" to "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
"Connection" to "keep-alive"
)
fun main() {
CookieHandler.setDefault(null)
val request = request("https://stackoverflow.com/", "GET")
?: throw RuntimeException("Request failed")
val fkeyJson = (request.body.split(Regex("\r?\n")).map { it.trim() }.find {
it.startsWith("<script type=\"application/json\" data-role=\"module-args\" data-module-name=\"Shared/options.mod\">")
}
?: throw RuntimeException("fkey not found"))
val fkeyJsonTree = ObjectMapper().readTree(
fkeyJson.drop("<script type=\"application/json\" data-role=\"module-args\" data-module-name=\"Shared/options.mod\">".length)
.dropLast("</script>".length)
)
val fkey = fkeyJsonTree["options"]?.get("user")?.get("fkey")?.asText()
?: throw RuntimeException("fkey not found")
request(
"https://stackoverflow.com/legal/consent-action",
"POST",
body = buildUrlEncoded("type" to "AcceptAllFromBanner", "fkey" to fkey, "bannerVersion" to "baseline")
)
?: throw RuntimeException("Request failed")
val thisShouldNotReturnCookie = request("https://stackoverflow.com/", "GET")
?: throw RuntimeException("Request Failed")
if (thisShouldNotReturnCookie.headers["Set-Cookie"] != null) {
println("Cookie received.") // this should never happen
exitProcess(-1)
}
}
fun buildUrlEncoded(vararg pairs: Pair<String, Comparable<*>>) =
pairs.joinToString("&") { "${URLEncoder.encode(it.first, "utf-8")}=${URLEncoder.encode("${it.second}", "utf-8")}" }
fun request(url: String, method: String, header: Map<String, String> = defaultHeader, body: String? = null): Response? {
var res: Response? = null
val connection = URL(url).openConnection() as HttpURLConnection
try {
connection.requestMethod = method
connection.doInput = true
header.forEach(connection::setRequestProperty)
connection.useCaches = true
val supply = jar.supply(url)
if (supply.isNotBlank())
connection.setRequestProperty("Cookie", URLEncoder.encode(supply, "utf-8"))
if (body != null) {
connection.doOutput = true
connection.outputStream.use {
it.write(body.toByteArray())
it.flush()
}
}
connection.connect()
val stream = if (connection.responseCode == 200) connection.inputStream else connection.errorStream
res = Response(connection.responseCode, String(stream.readBytes(), charset("UTF-8")), connection.headerFields)
jar.consume(connection.headerFields)
} catch (io: IOException) {
io.printStackTrace()
} finally {
connection.disconnect()
}
return res
}
data class Response(val status: Int, val body: String, val headers: Map<String, List<String>>)
This example is just click "Accept all cookies" to store cookies, and send main page to test cookie is stored.
This should print nothing and closes with 0 exit code. but it always printing "Cookie received." and exit with -1.
I tested something, and i found HttpURLConnection is removing my "Cookie" header.
How to keep cookies on my Reqeust?
By the way, I tested it with UnirestInstance, it works.
But i don't wanna use external library (Cookie will implemented as mine).
Example code: nothing prints and exit with 0 exit code.
Actually resulted: prints "Cookie received." and exit with -1 exit code.