When the Display value is set to Custom Field, the value on the reference field shows correct value i.e custom field value but on the tree picker, it shows the OOB name field.
This is the expected behavior. We cannot customize the names in the Tree Picker. It defaults from the Name field.
Below is the documentation stating the same:
You cannot customize the label names used in the tree picker. The label names are taken from the values in the table. For example, the Assignment group choices come from the group names in the Name column of the Groups table.