Slowly but surely both the tree and the building will be sorry they couldn't move to get out of the way of English ivy. Ivy won't pull off tree bark unless you try to remove it by brute force, but as it spreads upward and throughout the canopy, it shades out the tree's inner leaves. None of those leaves are there just for decoration-the tree needs all the food they can make. A trunk covered by ivy can also make it hard to see structural damage from other sources. It isn't that ivy attacks in the way that twining vines can strangle a tree, ivy does its damage inadvertently.
We may think ivy is beautiful on buildings, but the ivy League grounds and buildings managers at Harvard and Yale agree it's a problem. Ivy grows small rootlets, called holdfasts, which make a glue that dissolves some of the mortar between the bricks. Worse, the ivy traps moisture, dust, and debris next to the building. Between acid rain and the decomposition of the debris, the acidity next to the building increases. That causes further damage to mortar eventually need repointing (replacing worn mortar). Ivy makes it happen sooner.
Boston ivy because it was used to cover Harvard's brick buildings, is a least deciduous. In the winter, snow and ice are able to drop off and the walls are able to dry out. But because English ivy is evergreen, in never takes a break.