NEVERMIND figured it out!
need to convert mask to integer and then multiply by the original green band
>> I = imread(fullfile('D:\MatLab_Practice', 'RGB.tif'));
>> green = I(:,:,2);
>> imhist(green)
>> mask = green > 127; %(I have also tried the roicolor(green,128,256) to create a mask with same results)
>> mask = uint8(mask); %to match the same data type as image
>> G2 = green .* mask;