I always hesitate to come between someone and their pastor. So let me just deal with what I see as the main doctrinal issue here. I have known people, actually quite dedicated Christians, who refused to take of the Lord's Supper for years because they felt that they were unworthy and they felt that the warning against eating or drinking unworthily applied to them. My guess is that this scripture may be the source of the problem to which you refer.
Let us look at 1 Corinthians 11:26-32: For as often as ye eat this bread, and drink this cup, ye do shew the Lord's death till he come. Therefore whosoever shall eat this bread, and drink this cup of the Lord, unworthily, shall be guilty of the body and blood of the Lord. But let a man examine himself, and so let him eat of that bread, and drink of that cup. For he that eateth and drinketh unworthily, eateth and drinketh damnation to himself, not discerning the Lord's body. For this cause many are weak and sickly among you, and many sleep. For if we would judge ourselves, we should not be judged. But when we are judged, we are chastened of the Lord, that we should not be condemned with the world.
Paul here warns against taking of the bread and cup of the Lord's Supper "unworthily" (v.27, 29). Some who have done so are weak and sickly and some even sleep (they have died). God's judgment had fallen heavily upon the saints of Corinth because of this sin. But what actually was the sin? What had they done? They had eaten and drunk of the Lord's Supper "unworthily." But what is this? Before giving an interpretation, we must look at the word itself. The word, unworthily, is an adverb. That means it describes a verb or defines the way an action is done. In this passage, it describes how the Corinthians had taken of the Lord's Supper. But most people read this wrong. They think it describes the character of the person. But it does not describe the condition of the people taking of the supper. That would require an adjective. God did not say that unworthy Corinthians took of the Lord's Supper. He says that they took of the supper unworthily. They did it in the wrong way. The Lord's Supper is a picture of the sacrifice of Christ on the cross. No one is worthy to partake of this sacrifice. That is not the point. However, they could take of it unworthily by making a mockery of this solemn event and that is exactly what the Corinthians did.
Look at 1 Corinthians 11:20-22: When ye come together therefore into one place, this is not to eat the Lord's supper. For in eating every one taketh before other his own supper: and one is hungry, and another is drunken. What? have ye not houses to eat and to drink in? or despise ye the church of God, and shame them that have not? What shall I say to you? shall I praise you in this? I praise you not.
The Corinthians had made a drunken party out of the Lord's Supper. They brought food for a feast and fought over who got to eat what. Some of them got drunken on the beverages they brought to the "party." This made God angry and He judged a number of them: some with sickness and some with an early grave. Paul is telling them to treat the Lord's Supper with proper respect--to take of it in a worthy manner. The Lord's Supper is designed to be one of the highest forms of spiritual fellowship a body of believers can have on this earth. However, we are not to exclude people because everything is not right in their lives. We are to examine our hearts when we come and give proper honor and respect to the Lord Jesus who offered Himself to us. This way, we experience the true fellowship of the Father and the Son. What a blessed time this can be for a church! We can look to the One who gave Himself to us and praise Him for His unspeakable gift. May we not keep this blessed time from our churches.