Questions on recursive coroutines in Kotlin - kotlin

Question

Recently I've been trying to familiarize myself with Kotlin some more, so I decided to write a webscraper utilizing coroutines. What I want to accomplish is pull each page, harvest it for links and contents or posts, then feed the links back to the process, until there is nowhere left to go. As of now it has some obvious shortcomings, such no delay enforced between calls or saving addresses and only visiting new ones. But the questions I have are regarding coroutines, here.
Consider the following class,. I've added some toy classes to simulate how it is intended to work, which I won't detail, but you can imagine how they work.
class Scraper(
private val client: Client = ToyClient(delayMillis = 1000, alwaysFindBody = "Test body"),
private val extraction: Extraction = ToyExtraction(
alwaysFindLinks = listOf("https://google.com"),
alwaysFindPosts = listOf("Test post")
),
private val repository: Repository = ToyRepository()
) {
// I could manage my own coroutine scope's lifecycle, but how would I go about this?
// private val scope = CoroutineScope(Dispatchers.Default + SupervisorJob())
private val seed = "https://google.com"
private val log = KotlinLogging.logger {}
fun start() = runBlocking {
log.info { "Scraping started!" }
scrape(seed).join()
log.info { "Scraping finished!" }
}
private fun CoroutineScope.scrape(address: String): Job = launch(Dispatchers.Default) {
log.info { "A scraping coroutine has started" }
val page = request(address)
val contents = extract(page)
save(contents)
contents.links.forEach { scrape(it) }
// Job would not progress here after submitting new jobs, only after each children have been completed
// log.info { "A scraping coroutine has finished" }
}
private suspend fun request(address: String): Page {
log.info { "Getting page: $address" }
return client.get(address)
}
private suspend fun extract(page: Page): PageContents {
log.info { "Extracting page: ${page.address}" }
return extraction.extract(page)
}
private suspend fun save(contents: PageContents) {
log.info { "Processing contents of: $contents" }
repository.save(contents.posts)
}
}
The main recursive operation is CoroutineScope.scrape() which launches a job, which itself can launch children jobs as well and so on.
My main questions are:
If I were to manage the scope myself as a property, how could I do that and achieve the same behavior? That is, I would wait for all dynamically spawned jobs to complete as well, return when all are finished.
I wrote my webclient's function using a 3rd party library as such:
fun suspend get(address: String): Page { ... }
Am I fine just marking this method as suspend to get all benefits from this in terms of coroutines?
Thanks in advance!

Answer 1

You don't even need a scope for that, launch a top-level job and use job.join() to await until it and all its children are done. If you want to block while waiting for that to happen, then you are already doing it right by using runBlocking.
No, marking a function as suspend doesn't affect its blocking behavior. It only allows the function to suspend itself, which must be explicit either in your code or the code you're calling into.