"Despite their efforts to resolve their differences, things didn't improve," said a source close to the couple