Replacing an object by a deserialized version of it, and preserving references - vb.net

Question

Say I have an object of my custom class, called AppSettings, which has various properties that hold both value types (integers, doubles, strings, etc.) and reference types (arrays, other custom objects, etc.). Some of these custom objects have their own custom objects, so the path down to some of the value type properties can go very deep.
For example:
<Serializable()>
Public Class AppSettings
Public Property windowHeight As Integer = 600
Public Property windowWidth As Integer = 800
Public Property defaultLengthUnit As Unit = Units.meters
Public Property defaultAngleUnit As Unit = Units.degrees
End Class
Where Unit class is defined as:
<Serializable()>
Public Class Unit
Public Property Name As String
Public Property Abbreviation As String
Public Property Scale As Double
End Class
And Units module is defined as:
Public Module Units
Public meters As New Unit With {
.Name = "Meters",
.Abbreviation = "m.",
.Scale = 1
}
Public degrees As New Unit With {
.Name = "Degrees",
.Abbreviation = "°",
.Scale = 1
}
End Module
Some other code might refer or bind to some of the reference type properties, or their internal properties. Now, let's say I provide a way for the user to save current state of AppSettings by serializing it into XML:
Public Sub SerializeAppSettings(ByVal filename As String)
Using sw As StreamWriter = New StreamWriter(filename)
Dim xmls As XmlSerializer = New XmlSerializer(GetType(AppSettings))
xmls.Serialize(sw, appSettings)
End Using
End Sub
and then load them back (by deserializing) at any time while running the application:
Public Function DeserializeAppSettings(ByVal filename As String) As AppSettings
If Not File.Exists(filename) Then Return Nothing
Using sr As StreamReader = New StreamReader(filename)
Dim xmls As XmlSerializer = New XmlSerializer(GetType(AppSettings))
Return TryCast(xmls.Deserialize(sr), AppSettings)
End Using
End Function
It is called like so:
AppSettings = DeserializeAppSettings(settingsFilePath)
The problem here is that all the references to AppSettings that other objects and bindings have, are now broken, because deserialization replaces the old instance of AppSettings with a completely new instance, and the references are not transferred to it.
It appears that this doesn't break references to value-type properties (like windowHeight, which is Integer), but it definitely breaks references to reference-type properties, like defaultLengthUnit. So for example, if some other object or WPF control is referring/binding to, say, AppSettings.defaultLengthUnit.scaleToBaseUnit, it doesn't work anymore.
I wonder, how can I fix this, so that deserialization would replace the old instance of AppSettings and transfer all the references from it to the new instance that it generated?
As I understand it, there are three ways to go about it:
Replace the old instance with an new one in the exact same memory allocation, with the same internal ID, which would probably be too hacky, and I'm not sure if at all possible.
Another way would be for the DeserializeAppSettings function to overwrite each property value of the current AppSettings instance, one by one, by the deserialized values. However, since some properties of AppSettings are objects, which have their own objects, which have their own objects (and so on), I would basically need to type out all the hierarchy tree in that DeserializeAppSettings function to get down to the value type properties. And every time I would need to add or remove any property in the AppSettings class (or in any class that is used in it's properties), I would also need to manually update the parsing code in DeserializeAppSettings function. This is seriously unmaintainable.
Lastly, it would probably be possible to automate this value replacement through reflection, but reflection is very slow, and generally discouraged if there is any other option.
I hope I am missing something obvious here. Any suggestions on how to transfer all the references to AppSettings when the old instance of it is replaced with a new one through deserialization?
EDIT: Updated the code to include all the relevant classes.