We’ve already told you as much as we can without getting more information from you. (Please see our original answer pasted below in the additional comments section.) If you didn’t receive our answer, please check your spam folder; we want to be sure you’re getting our answers. Now that we know the leaves are yellowing and dropping off in addition to the fruit aborting, it’s more important than ever to get answers to our questions below. Without them, we can’t help you diagnose. Since things are getting worse, you might consider consulting an arborist who can make an in-person visit. One more thing we just thought to ask: is this tree growing in a container? This is not a tree that is well suited to container growing.