=iQc@sddlTddlmZddlTddlZddlZddlZdefdYZdefdYZdefd YZ dS( i(t*(tReinforcementAgentNtQLearningAgentcBsVeZdZdZdZdZdZdZdZdZ dZ RS( s Q-Learning Agent Functions you should fill in: - computeValueFromQValues - computeActionFromQValues - getQValue - getAction - update Instance variables you have access to - self.epsilon (exploration prob) - self.alpha (learning rate) - self.discount (discount rate) Functions you should use - self.getLegalActions(state) which returns legal actions for a state cKs#tj||tj|_dS(s#You can initialize Q-values here...N(Rt__init__tutiltCountertQState(tselftargs((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyR%scCs|j||fS(s Returns Q(state,action) Should return 0.0 if we have never seen a state or the Q node value otherwise (R(Rtstatetaction((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyt getQValue-scCsed}xD|j|D]3}|j|||kr|j||}qqW|dkr]dS|SdS(s Returns max_action Q(state,action) where the max is over legal actions. Note that if there are no legal actions, which is the case at the terminal state, you should return a value of 0.0. igN(tgetLegalActionsR (RR t optimalQValueR ((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pytcomputeValueFromQValues7s cCsg}d}xu|j|D]d}|j|||krX|j||}|g}n|j|||kr|j|qqWt|dkrtj|St|dkr|dSt|dkrdSdS(s Compute the best action to take in a state. Note that if there are no legal actions, which is the case at the terminal state, you should return None. iiiN(R R tappendtlentrandomtchoicetNone(RR t optimalActionR R ((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pytcomputeActionFromQValuesPs   cCsE|j|}d}tj|jr4tj|S|j|S|S(s Compute the action to take in the current state. With probability self.epsilon, we should take a random action and take the best policy action otherwise. Note that if there are no legal actions, which is the case at the terminal state, you should choose None as the action. HINT: You might want to use util.flipCoin(prob) HINT: To pick randomly from a list, use random.choice(list) N(R RRtflipCointepsilonRRR(RR t legalActionsR ((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyt getActionzs   cCsSd}d|j|j|||j||j|j||j||f nextState and reward transition. You should do your Q-Value update here NOTE: You should never call this function, it will be called on your behalf iiN(talphaR tdiscountRR(RR R t nextStatetrewardR ((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pytupdates cCs |j|S(N(R(RR ((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyt getPolicyscCs |j|S(N(R(RR ((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pytgetValues( t__name__t __module__t__doc__RR RRRRRR (((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyRs   *  t PacmanQAgentcBs,eZdZdddddZdZRS(sIExactly the same as QLearningAgent, but with different default parametersg?g?g?icKsE||d<||d<||d<||deZdZddZdZdZdZdZRS(s ApproximateQLearningAgent You should only have to overwrite getQValue and update. All other QLearningAgent functions should work as is. tIdentityExtractorcKs>tj|t|_tj||tj|_dS(N(Rtlookuptglobalst featExtractorR$RRtweights(Rt extractorR((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyRscCs|jS(N(R.(R((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyt getWeightsscCs,|jj||}|j}||}|S(sr Should return Q(state,action) = w * featureVector where * is the dotProduct operator (R-t getFeaturesR.(RR R tfeaturestwR((sVC:\MyDocuments\Studies\Berkeley CS188X\05 Projects\03 Reinforcement\qlearningAgents.pyR s  cCs||j|j||j||}|jj||}x>|jD]0}|j||j||||j| s  $