I believe that the type of the offering ("of the fruit of the ground" - Genesis 4:3) was wrong because God expected a blood sacrifice. But I also believe that Cain brought the wrong kind of offering because his heart was wrong. As a result, he brought the product of his hard work and not the offering that God had revealed He expected.
My proof, so to speak, comes from Hebrews 11:4, which states, "By faith Abel offered unto God a more excellent sacrifice than Cain, by which he obtained witness that he was righteous, God testifying of his gifts: and by it he being dead yet speaketh." Notice first that Abel offered a more excellent sacrifice, so the kind of sacrifice he offered was better than that of Cain. The offering of the herd was a picture of the coming sacrifice of Christ. The offering of the fruit of the ground was a picture of the works of man. Notice that it came from the ground that God had just cursed (Genesis 3:17).
However, Abel offered this better sacrifice by faith. Romans 10:17 teaches us that, "faith cometh by hearing, and hearing by the word of God." Faith acts upon the word of God. Clearly, God had revealed to Adam and his family the kind of sacrifice He expected of them. We see God speaking to Adam and Eve when they sinned and we see Him speaking to Cain before he killed his brother (Genesis 4:6-7). Certainly, He spoke to them at other times also. The fact that Abel offered his sacrifice by faith means that he did it by revelation from God. Cain directly disobeyed God in bringing a vegetarian sacrifice.
But why did Cain bring the wrong sacrifice? Hebrews 11:4 also tells us that the sacrifice of Abel bore witness to the fact that he was righteous. Simple logic tells us that the sacrifice of Cain bore witness to the fact that he was wicked. But we do not have to rest solely on logic. 1 John 3:12 states, "Not as Cain, who was of that wicked one, and slew his brother. And wherefore slew he him? Because his own works were evil, and his brother's righteous." Though this refers to the slaying of Abel and not the sacrifice of Cain, it shows us the root of the problem--the wicked heart of Cain. Why did Cain have this wicked heart? Evidently, because he did not have the faith that Abel had in God.
In conclusion, God rejected the offering of Cain because it was the wrong offering. Cain brought the wrong offering because he rejected the revealed word of God. He rejected the word of God because his heart was wicked. His heart was wicked because he refused to have faith in God.
