In case you weren't aware, a loved gift increases your relationship with them by 80 points (at minimum), and talking with them improves it by 20 rather than it decreasing by 20 that day. So if you're away for an extended period, you can calculate how long it'll take to raise it back up to max. Either way it won't ever permanently damage the relationship.
(Example: If you spend a month away, it'll drop 20x28 = 560 points, which is 2 full hearts. Talking to them every day and giving 2 loved gifts per week will mean it'll improve by 7x20=140, 2x80 = 160, 160+140 = 300 per week. There are other ways to improve it and those two loved gifts will make you eligible for one of them but regardless, just doing that means at worst it'll take 2 weeks to get them back up to full.)