How to properly use the UnProject method in OpenTK and get world coordinates from mouse hover? - vb.net

Question

I am trying to get world coordinates by hovering my mouse over the GLControl control where the world is being rendered in. I've found dozens of examples and solutions but I can't work with them, I am still learning and haven't got to their point. But I kept digging and found some examples and picked one that is approved by its original author which works with a similar render construction but still doesn't seem to work for me.
I've set up a window to output the converted coordinates. I am getting x, y = 0 at the center of the control, which means that I'm possibly going the right way. When I move the mouse around the x and y axis it outputs proper values (eg. the higher the mouse from the center of the control, the higher y is, etc.), but only to like -1.65/1.65 max from the center. It does not seem like it gets my Modelview matrix, because It should differ depending on the eyeZ, which it doesn't. Changing the eyeZ has no effect at all. It still acts as if I didn't "zoom in/out".
I've set up my Projection, Modelview matrices and Viewport like this:
GL.MatrixMode(MatrixMode.Projection)
GL.LoadIdentity()
GL.LoadMatrix(Matrix4.CreatePerspectiveFieldOfView(Math.PI / 4, glControl.Width / glControl.Height, 1.0F, 128.0F))
GL.MatrixMode(MatrixMode.Modelview)
GL.LoadIdentity()
GL.LoadMatrix(Matrix4.LookAt(0F, 0F, zoom, 0F, 0F, 0F, 0F, 1.0F, 0F))
GL.Viewport(0, 0, glControl.Width, glControl.Height)
Then I have these two functions that get called when my mouse moves over the GLControl and pass in the mouse's coordinates.
Public Function screenToWorld(ByVal x As Integer, ByVal y As Integer) As Vector4
Dim modelViewMatrix As Matrix4
Dim projectionMatrix As Matrix4
Dim viewport(4) As Integer
GL.GetFloat(GetPName.ModelviewMatrix, modelViewMatrix)
GL.GetFloat(GetPName.ProjectionMatrix, projectionMatrix)
GL.GetInteger(GetPName.Viewport, viewport)
Return UnProject(projectionMatrix, modelViewMatrix, New Size(viewport(2), viewport(3)), New Vector2(x, y))
End Function
Public Function UnProject(ByRef projection As Matrix4, ByVal view As Matrix4, ByVal viewport As Size, ByVal mouse As Vector2) As Vector4
Dim vec As Vector4
vec.X = 2.0F * mouse.X / viewport.Width - 1
vec.Y = -(2.0F * mouse.Y / viewport.Height - 1)
vec.Z = 0F
vec.W = 1.0F
Dim viewInv As Matrix4 = Matrix4.Invert(view)
Dim projInv As Matrix4 = Matrix4.Invert(projection)
Vector4.Transform(vec, projInv, vec)
Vector4.Transform(vec, viewInv, vec)
If vec.W > Single.Epsilon OrElse vec.W < Single.Epsilon Then
vec.X /= vec.W
vec.Y /= vec.W
vec.Z /= vec.W
End If
Return vec
End Function
Also I refrain from using orthrographic projection.