entropy

a random variable with only one value--a coin that always comes up heads--has no uncertainty and thus its entropy is defined as zero; thus, we gain no information by observing its value. a flip of a fair coin is equally likely to come up heads or tails, 0 or 1, this counts as “1 bit” of entropy. the roll of a fair four-sided die has 2 bits of entropy, because it takes two bits to describe one of four equally probable choices. now consider an unfair coin that comes up heads 99% of the time. intuitively, this coin has less uncertainty than the fair coin--if we guess heads we’ll be wrong only 1% of the time--so we would like it to have an entropy measure that is close to zero, but positive.