Sounds like a heavy story... And it seems like his mother connected them(Mao combines both with Xu and Wei) at the same time closing them and again falling in Wei, so very complex relationship with him.
Then Wu <> Mao destruction happens, along with Geng clashing Jia in the other representation of mother, people around will make it difficult for his mother, even more so considering all wood falls in Wei with the way I view it...
Without Mao, Wei and Xu have only relation of punishment, so he is pulling away...
Interesting that Mao in the hour pillar(a kid maybe) may also connect them a little in similar way, but that could be late in life and someone else may be taking Xu representation by then.
Xu is in year pillar, I wonder...
Is she older then him(the girlfriend) or was living originally in far away place, maybe?
Are there many people connected to her for some reason(like working a job where she meets many people maybe or something like that)?