Mapping the emotional face. How individual face parts contribute to successful emotion recognition.
4a. Plotting Results by Action Units¶
This notebook aggregates the tiles in accordance to the Ekman Action Units (AUs) they fall into.
The AUs have been manually defined on each individual face and each AU has a color code that identifies it. Now, the algorithm assigns each tile one or a number of AUs that are contained in it, so each tile has either an AU label, or it represents no AU.
importing the basic modules and files¶
In [1]:
from myBasics import *
%matplotlib inline
In [2]:
logList = getFile('../rawTables/','pand*.csv')
Visualize the hand-drawn Action Units (AUs)¶
Example:¶
Get all the AUs for one picture. This is a list of single images, one per AU, where each AU is drawn in a different color.
In [3]:
myPicList = getFile('../auLabels/','f_ang*.png')
Now we have an array with all AU's that belong to the female angry face (f_ang):
In [4]:
myPicList
Out[4]:
For illustration purposes, here we plot all the hand-drawn AU's of that face
In [5]:
fig = plt.figure(figsize=(12,4))
for i,pic in enumerate(myPicList):
im = Image.open(pic)
ax = plt.subplot(1,len(myPicList),i+1)
ax.set_title(pic[pic.rfind('/')+1:],fontsize=12)
ax.set_xticks([]);ax.set_yticks([])
ax.imshow(im);
Simple visualisation of all AUs on the faces¶
In [6]:
myPaintList = []
for fGender in ['f','m']:
thisList = []
for fEmo in myLabels.values()[:-1]:
thisFace = getFile('../auLabels/auVisualisation/',fGender+'_'+fEmo+'*.png')[-1]
thisList.append(thisFace)
myPaintList.append(thisList)
In [7]:
myPaintList
Out[7]:
In [8]:
def makePicturePlot(myPaintList):
fig = plt.figure( figsize=(16,8) )
i = 1 # counter for subplots
for ident in range(2):
for emo in range(6):
im=Image.open(myPaintList[ident][emo],'r')
ax = plt.subplot(2,6,i)
ax.imshow(im)
ax.set_yticks([]); ax.set_xticks([])
i+=1
plt.savefig('../figures/auIllustration.png',dpi=300)
plt.show()
In [9]:
makePicturePlot(myPaintList)
Assign AUs to tiles¶
Get a list of all possible tile coordinates
In [10]:
def makeCoordinates(xNum,yNum,squareSize):
myArray = []
xDim=xNum*squareSize
yDim=yNum*squareSize
for x in np.arange(0,xDim,squareSize):
for y in np.arange(0,yDim,squareSize):
myArray.append( (x,y) )
return myArray
In [11]:
thisCoord = makeCoordinates(6,8,50)
print thisCoord
print "\nNumber of Coordinates:", len(makeCoordinates(6,8,50))
Small script to get the rgb color of each mask¶
In [12]:
def getColor(imName):
im = Image.open(imName)
# we loop through all pixels
for y in xrange(im.size[1]):
for x in xrange(im.size[0]):
# we get the RGB+alpha values of each pixel
r,g,b,a = im.load()[x, y]
# we are only intereset in colored pixels
if (r,g,b) != (0,0,0):
# store the present color for later re-use
thisColor = (r,g,b)
# and stop here
break
return thisColor
In [13]:
getColor( myPicList[-1] )
Out[13]:
Determining for each tile, whether it contains some colored pixels¶
We loop through the image, tile by tile. For this, we use For each tile, count the occurence of each color on a pixel-by-pixel basis. We use the thisCoord list of all tile coordinates (where each coordinate tuple denotes the upper left corner of the 50x50 tile).
The first output is a dictionary, with tile coordinates as keys and number of colored pixels as values.
The second output is the RGB value of the color used for that AU. Since each image contains only one AU and should have only one color, this works by taking the color of any non-black (0,0,0) pixel.
In [14]:
def getTiles(im,thisCoord=thisCoord,squareSize=50):
# resize the image, if it is not exactly 300x400 (as it was presented in the experiment),
# this should not be necessary in general
im = im.resize((300,400))
# the dictionary where we write the occurence of colored pixels for each tile
thisCut = {}
# we loop through the coordinate list
for coord in thisCoord:
# we get the (h)orizontal and (v)ertical coordinates from the tuple
h,v = coord
# we cut out the tile
cut = im.crop((h,v,h+squareSize,v+squareSize))
pixdata=cut.load()
# we loop through all pixels of that tile
for y in xrange(cut.size[1]):
for x in xrange(cut.size[0]):
# we get the RGB+alpha values of each pixel
r,g,b,a = pixdata[x, y]
# we are only intereset in colored pixels
if (r,g,b) != (0,0,0):
# store the present color for later re-use
thisColor = (r,g,b)
# count the occurence of colored pixels for that tile/coordinate pair
try:
thisCut[coord]+=1
except:
thisCut[coord]=1
# if the tile remained empty, assign zero
if coord not in thisCut:
thisCut[coord] = 0
# return the dict and a tuple with RGB values of that Action Unit
return thisCut,thisColor
Example:
In [15]:
print getTiles(im)
get tiles for all AUs¶
In [16]:
def makeAUTable(myPicList):
# get number of colored pixels in each tile for all AUs of a face
d = {}
for pic in myPicList:
d[pic] = getTiles( Image.open(pic) )
# transform this into a dataframe
bigDf = pd.DataFrame()
for key in d:
thisDf = pd.DataFrame(d[key][0],index=[key]).T
bigDf = pd.concat([bigDf,thisDf],axis=1)
return bigDf
In [17]:
makeAUTable(myPicList).head()
Out[17]:
Binarize in a way that multiple AUs can belong to a tile, but only if they pass a defined cutoff
In [18]:
def makeBinAUs(bigDf,cutOff=25):
return (bigDf>cutOff).astype(int)
In [19]:
makeBinAUs( makeAUTable(myPicList) ).head()
Out[19]:
In [20]:
d = makeBinAUs( makeAUTable(myPicList) ).to_dict()
visualize the result¶
In [21]:
def showTiles(bgIm,d,squareSize=50):
imDict = {}
# loop through the action units
for au in d:
# get the face image as a background to visualize on
imOut = Image.open(bgIm)
imOut = imOut.resize((300,400))
# loop through the tile coordinates
for coords in d[au]:
h,v = coords
if d[au][coords] == 1:
thisColor = getColor(au)
else:
thisColor = 0
faceTile = imOut.crop((h,v,h+squareSize,v+squareSize))
if thisColor == 0:
imOut.paste(faceTile, (h,v))
else:
pixdata=faceTile.load()
# change its color by looping through all the pixels of the cutout
for y in xrange(faceTile.size[1]):
for x in xrange(faceTile.size[0]):
pixdata[x, y] = thisColor
# paste the fully colored tile onto the background image
imOut.paste(faceTile, (h,v))
imDict[au] = imOut
return imDict
In [22]:
d = makeBinAUs( makeAUTable(myPicList) ).to_dict()
imDict = showTiles(picList[0][2],d)
In [23]:
fig = plt.figure(figsize=(16,6))
for i,im in enumerate(imDict):
ax = plt.subplot( 1, len(imDict.keys()), i+1 )
ax.set_title(im[im.rfind('/')+1:],fontsize=12)
ax = plt.imshow(imDict[im])
plt.xticks([]); plt.yticks([])
plt.show()
Get all the pictures and their respective AU lists¶
Here we define a dictionary picDict, wich is used to store the filenames of the original image and the AU images for each face.
In [24]:
def makePicDict(picList):
picDict = {}
for i,ident in enumerate(picList):
for e,emo in enumerate(picList[i]):
thisPic = picList[i][e]
thisName = thisPic[thisPic.find('_')-1:thisPic.rfind('_')]
myPicList = getFile('../auLabels/',thisName+'*.png')
picDict[thisName] = {'auList':myPicList,
'picFile':thisPic}
return picDict
In [25]:
picDict = makePicDict(picList)
In [26]:
picDict
Out[26]:
make a list with face names, which is sorted in a nice way, so we can use it for looping
In [27]:
picSort = []
for entry in picList:
for subentry in entry:
picSort.append(subentry[subentry.rfind('/')+1:subentry.rfind('_')])
print picSort
show all the tile-assignments of action units¶
In [28]:
picDict[picSort[0]]
Out[28]:
In [29]:
for entry in picSort:
d = makeBinAUs( makeAUTable(picDict[entry]['auList'])).to_dict()
imDict = showTiles(picDict[entry]['picFile'],d)
fig = plt.figure(figsize=(16,6))
for i,im in enumerate(imDict):
ax = plt.subplot( 1, 4, i+1 )
imName = im[im.rfind('_')+1:im.rfind('.')]
ax.set_title(imName)
ax.set_xticks([]);ax.set_yticks([])
ax = plt.imshow(imDict[im])
saveName = '../figures/auAssignments/'+str(im[im.rfind('/')+1:im.rfind('_')])
plt.savefig(saveName,dpi=300)
plt.show()
Derive behavioral metric¶
These are the acutal behavioral data, which we now want to apply not to single tiles, but to groups of tiles which all belong to the same action unit. The algorithms to derive the values are esentially the same as in the previous notebooks.
In [30]:
metricDf = pd.read_csv('../outputs/weightDf.csv',index_col=[0,1,2])
metricDf.index.names = ['p','ident','emo']
metricDf = metricDf.sortlevel()
In [31]:
metricDf.tail()
Out[31]:
In [32]:
def getWeight(metricDf,ident,emo,identDict=identDict,emoDict=emoDict):
outDf = pd.DataFrame()
for entry in metricDf.index.levels[0]:
thisDf = pd.DataFrame( metricDf.ix[entry].ix[identDict[ident]].ix[emoDict[emo]] ).T
thisDf.index = [entry]
outDf = pd.concat([outDf,thisDf])
outDf = outDf.sortlevel()
return outDf
Table of that condition, with all participants
In [33]:
getWeight(metricDf,0,0).head()
Out[33]:
Combining The AU assigments and the tile metrics¶
In [34]:
def writeMetric(metricDf,facePic,p,picDict=picDict,thisCoord=thisCoord):
# names of action units
auList = picDict[facePic]['auList']
# transforming the coordinates to indices from 0 to 47
cCodes = {}
for c,coord in enumerate(thisCoord):
cCodes[ c ] = coord
# dict to write to
auDict = {float(np.nan):[]}
# tracker for tiles belonging to no au
notNan = []
for au in auList:
auDict[au] = []
# mapping of coordinates to AUs
d = makeBinAUs( makeAUTable(auList) ).to_dict()
# looping through the metrics
for key in metricDf.ix[p].to_dict():
thisCoord = d[au][cCodes[int(key)]]
# get the metric
thisMetric = metricDf.ix[p].to_dict()[key]
if d[au][tuple(cCodes[int(key)])] == 1:
# adding value to
auDict[au].append( thisMetric)
# keep track that this is not a nan
notNan.append(key)
# do that for the remaining nans
for key in metricDf.ix[p].to_dict():
if key not in notNan:
thisCoord = d[au][cCodes[int(key)]]
# get the metric
thisMetric = metricDf.ix[p].to_dict()[key]
# adding value to
auDict[float(np.nan)].append( thisMetric)
# transform to df
auDf = pd.DataFrame(index=[p])
for entry in auDict:
auDf[entry] = np.mean(auDict[entry])
# cleaning up columns names
cleanCols = []
for e in auDf.columns:
if type(e) == str:
cleanCols.append(e[e.rfind('_')+1:e.rfind('.')] )
else:
cleanCols.append(e)
auDf.columns = cleanCols
return auDf
Example:
In [35]:
writeMetric( getWeight(metricDf,0,3),
'f_fea',
'p001'
)
Out[35]:
Do this for all participants¶
In [36]:
def writeAllMetrics(metricDf,ident,emo,picList=picList,picDict=picDict):
facePic = picList[ident][emo]
faceName = facePic[facePic.rfind('/')+1:facePic.rfind('_')]
thisMetric = getWeight(metricDf,ident,emo)
bigDf = pd.DataFrame()
for p in thisMetric.index:
thisDf = writeMetric(thisMetric,faceName,p,picDict=picDict)
bigDf = pd.concat([bigDf,thisDf])
return bigDf,faceName
Example: fearful female face (0,3)
In [37]:
fFeaDf,dummy = writeAllMetrics(metricDf,0,3)
In [38]:
fFeaDf.head()
Out[38]:
Do this for all faces¶
In [39]:
def writeAllFaces(metricDf,picList=picList,picDict=picDict):
bigDf = pd.DataFrame()
for ident in range(2):
for emo in range(7):
thisDf,faceName = writeAllMetrics(metricDf,ident,emo,picList=picList,picDict=picDict)
thisDf.columns = [ [faceName]*len(thisDf.columns),thisDf.columns]
bigDf = pd.concat([bigDf,thisDf],axis=1)
return bigDf
In [40]:
bigAU = writeAllFaces(metricDf)
In [41]:
bigAU.head()
Out[41]:
Taking the nan (no Action Unit) as baseline¶
In [42]:
def baselineCorrection(df,cond):
# select the condition
thisCorr = df[cond]
diffDf = pd.DataFrame()
# for all action units
for actionUnit in thisCorr.columns:
# we subtract the baseline
thisDiff = thisCorr[actionUnit]-thisCorr[np.nan]
diffDf[actionUnit] = thisDiff
diffDf = diffDf.drop(np.nan,axis=1)
# restore the original structure of the multicolumns
diffDf.columns = [[cond]*len(diffDf.columns),diffDf.columns]
return diffDf
Example:
In [43]:
bigAU['f_fea'].head()
Out[43]:
In [44]:
baselineCorrection(bigAU,'f_fea').head()
Out[44]:
Do this for the whole big table¶
In [45]:
def makeBaseline(bigAU):
baselineDf = pd.DataFrame()
for face in bigAU.columns.levels[0]:
thisDf = baselineCorrection(bigAU,face)
baselineDf = pd.concat([baselineDf,thisDf],axis=1)
return baselineDf
In [46]:
baselineDf = makeBaseline(bigAU)
Example:
In [47]:
baselineDf['f_fea'].head()
Out[47]:
In [48]:
baselineDf.to_csv('../outputs/actionUnitTable.csv')
Example of re-loading¶
In [49]:
baselineDf = pd.read_csv('../outputs/actionUnitTable.csv',index_col=[0],header=[0,1])
In [50]:
baselineDf.head()
Out[50]:
Basic plotting¶
In [51]:
def makeActionUnitPlot(baselineDf,face):
# select the face condition
thisDf = baselineDf[face]
colNames = thisDf.columns
# get number of participants (with values)
n = int(thisDf.describe().ix['count'][-1])
# get all metrics into a df which we sort descending
plotDf = pd.DataFrame()
plotDf['mean'] = thisDf.mean()
#print float( thisDf.mean() ), float( thisDf.describe().ix['mean'] )
plotDf['ci'] = thisDf.std()/np.sqrt(n)*1.96
plotDf['color'] = [rgb2hex(getColor('../auLabels/'+face+'_'+colName+'.png')) for colName in colNames]
plotDf = plotDf.sort_values(by="mean",ascending=False)
# plot this bar at the correct position and using the correct color
# this is done in a
plt.xticks(np.arange(len(plotDf.index))+0.45,plotDf.index,rotation=45)
plt.xlim(-0.05,4.05)
plt.yticks(range(0,101,20),['0%','20%','40%','60%','80%','100%'])
plt.ylim(-10,100)
plt.axhline(0,color='k')
plt.title(face,fontsize=20)
im = plt.bar(range(len(plotDf.index)) ,
plotDf['mean'],
yerr=plotDf['ci'],
color= plotDf['color'],
ecolor='k')
return im
In [52]:
makeActionUnitPlot(baselineDf,'f_sad');
In [53]:
fig = plt.figure(figsize=(14,28))
count = 1
for pic in picSort:
if 'ntr' not in pic:
ax = plt.subplot(4,3,count)
ax = makeActionUnitPlot(baselineDf,pic)
plt.ylabel('Percent Signal Change')
count+=1
sns.despine()
plt.tight_layout()
plt.savefig('../figures/allAUs.png',dpi=300)
plt.show()
4b. Plotting Results by Face Half¶
This basically re-uses the scripts above, but there are only two regions and together they make up the whole face. Therfore, a difference score can be computed which summarizes for each face whether the upper face half (positive values) or the lower face half (negative values) is more important.
get the up/low mask images¶
In [54]:
myPicList = getFile('../auLabels/','*er.png')
In [55]:
myPicList
Out[55]:
assign tiles to upper/lower half¶
In [56]:
makeBinAUs( makeAUTable(myPicList) ).head()
Out[56]:
In [57]:
d = makeBinAUs( makeAUTable(myPicList) ).to_dict()
here, each face has the same two masks¶
In [58]:
def makePicDict(picList,myPicList):
picDict = {}
for i,ident in enumerate(picList):
for e,emo in enumerate(picList[i]):
thisPic = picList[i][e]
thisName = thisPic[thisPic.find('_')-1:thisPic.rfind('_')]
picDict[thisName] = {'auList':myPicList,
'picFile':thisPic}
return picDict
In [59]:
picDict = makePicDict(picList,myPicList)
In [60]:
picDict
Out[60]:
illustrate the masking¶
In [61]:
for entry in picSort:
d = makeBinAUs( makeAUTable(picDict[entry]['auList'])).to_dict()
imDict = showTiles(picDict[entry]['picFile'],d)
fig = plt.figure(figsize=(12,6))
for i,im in enumerate(imDict):
ax = plt.subplot( 1, 2, i+1 )
imName = im[im.rfind('_')+1:im.rfind('.')]
ax.set_title(imName[imName.rfind('/')+1:])
ax.set_xticks([]);ax.set_yticks([])
ax = plt.imshow(imDict[im])
plt.savefig('../figures/auAssignments/upDownExample.png',dpi=300)
plt.show()
break # do this only for the first image
get the metric for each half¶
Here, there is no nan, because each tile must belong to one of the two masks
Example:
In [62]:
writeMetric( getWeight(metricDf,0,3),
'f_fea',
'p001',
picDict=picDict
)
Out[62]:
get the metric for all faces and all participants¶
In [63]:
bigUpperLower = writeAllFaces(metricDf,picDict=picDict)
In [64]:
bigUpperLower.head()
Out[64]:
compute a difference score up minus down¶
In [65]:
def makeDiffDf(bigUpperLower):
bigDiff = pd.DataFrame()
for entry in bigUpperLower.columns.levels[0]:
thisDiff = pd.DataFrame( bigUpperLower[entry]['../auLabels/upper']- bigUpperLower[entry]['../auLabels/lower'] )
thisDiff.columns = [ [entry[0]] , [entry[2:]] ]
bigDiff = pd.concat([bigDiff,thisDiff],axis=1)
return bigDiff
In [66]:
bigDiff = makeDiffDf(bigUpperLower)
Example:
In [67]:
bigDiff.head()
Out[67]:
Make a figure¶
In [68]:
sns.set_style('white')
sns.palplot(stackColors)
In [69]:
def makeUpLowPlot(bigDiff,emoReverse=emoReverse,stackColors=stackColors):
fig = plt.figure(figsize=(16,6))
ax = plt.subplot(1,2,1)
# hard-code the order to please us
myOrder=['dis','sup','hap','ang','ntr','fea','sad']
for e,entry in enumerate(myOrder):
thisCond = bigDiff['f'][entry]
ax.bar([e],
thisCond.mean(),
color=stackColors[emoReverse[entry]],
yerr=thisCond.std()/np.sqrt(len(thisCond))*1.96,
ecolor='k'
)
ax.set_xlim(-0.1,7.1)
ax.set_xticks(np.arange(len(myOrder))+0.4)
ax.set_xticklabels(myOrder );
ax.axhline(0,color='k')
ax.set_ylabel('Upper-Lower % Difference')
ax.set_xlabel('emotion expression')
ax.set_title('Female Face')
ax.set_ylim(-33,30)
sns.despine()
ax = plt.subplot(1,2,2)
myOrder=['dis','hap','ntr','sup','fea','sad','ang']
for e,entry in enumerate(myOrder):
thisCond = bigDiff['m'][entry]
ax.bar([e],
thisCond.mean(),
color=stackColors[emoReverse[entry]],
yerr=thisCond.std()/np.sqrt(len(thisCond))*1.96,
ecolor='k'
)
ax.set_xlim(-0.1,7.1)
ax.set_xticks(np.arange(len(myOrder))+0.4)
ax.set_xticklabels(myOrder );
ax.axhline(0,color='k')
ax.set_ylabel('Upper-Lower % Difference')
ax.set_xlabel('emotion expression')
ax.set_title('Male Face')
ax.set_ylim(-33,30)
sns.despine()
plt.savefig('../figures/upDownPlot.png',dpi=300)
plt.show()
In [70]:
makeUpLowPlot(bigDiff)
In [ ]: