How to shake off duplicate links while parsing web-data? - vba

Question

I've written some script in vba to parse the links leading to the next page from a torrent site. My script is able to scrape them. However, the issue I'm facing is that couple of duplicate links coming along in the result. My question is whether there is any technique with which I can parse only the unique links?
Sub TorrentData()
Dim http As New XMLHTTP60, html As New HTMLDocument, post As Object
With http
.Open "GET", "https://yts.ag/browse-movies", False
.send
html.body.innerHTML = .responseText
End With
For Each post In html.getElementsByClassName("tsc_pagination")(0).getElementsByTagName("a")
If InStr(post, "page") > 0 Then
x = x + 1: Cells(x, 1) = post.href
End If
Next post
End Sub
Partial picture of the scraped links:
Be sure to check the link before proceeding:
"https://www.dropbox.com/s/647x3m65u90a1bu/Description1.txt?dl=0"

Answer 1

I couldn't make the site work. Anyways, the proper way to use dictionary to eliminate duplicates and write to cells inside the same loop should look something like this:
For Each Post In html.getElementsByClassName("tsc_pagination")(0).getElementsByTagName("a")
If InStr(Post.href, "page") > 0 Then
If Not dict.Exists(Post.href) Then
dict.Add Post.href, "whatever information you would like to store"
x = x + 1
Cells(x, 1) = Post.href
End If
End If
Next Post