Calculating the entropy of a discrete memoryless source?

How could I calculate the entropy of a discrete memoryless source S={a,b,c} in unit of bits? What would be the matlab script for calculating this?

Answer 1

[Vinod Sudheesh]

Hello Daniel,

You could write the script for the same as follows

1) Create a function, say "esingle" that computes the "entropy" of a symbol in the source alphabet. The definition of the function "esingle" could be as follows

function Y=esingle(p)
if(p==0)
Y=0;
return;
else
    Y=p*log2(1/p);    
end
end

2) Now invoke this function "esingle" using the syntax below to calculate the "entropy" of the source alphabet "a".

a=[0 1 1];
E=sum(arrayfun(@esingle,a/sum(a)));

Hope this helps!

Thanks Vinod

[Daniel_DG]

Thank you very much Vinod. It helped me a lot.

Answer 2

[Walter Roberson]

A source S={a,b,c} is a source with three alphabetic symbols of no particular numeric value. If we know the generating equations for S then we analyze it analytically to determine the entropy. Otherwise the best we can do is estimate the entropy from a stream of the generated symbols. If we have assigned definite and distinct numeric values to "a", "b", and "c" and if "s" is a finite stream of those exact values, then

counts = histc(s, sort([a,b,c]));
probs = counts ./ length(s);
nzprobs = probs(probs>0);
entropy = sum(nzprobs .* log2(1./nzprobs));

Note: this code will work even if the assigned values are characters and s is a string.

Answer 3

[rakesh ranjan]

function Y=esingle(p)

if(p==0)

Y=0;

return;

else

Y=p*log2(1/p);

end

end