Pratik Chetan Shah

Guiding Professor

Prof. Vivek Borkar

Project Title

Reinforcement Learning in Non Markovian Environments

Field

Reinforcement Learning

Problem Statement Description

The main objective was to design an RL agent to perform well in Non-Markovian environments like Random Walks and Partially Observable Environments where standard RL agents like DQN fail due to incomplete state information

What was your motivation for choosing this field?

I was interested in robotics due to my time in AUV and ML was something I had started to explore thorugh projects and internships. RL is the field of ML which is heavily used in robotics and hence wanted to explore that too

What factors did you consider while choosing professors?

I made a list of Professors working in RL after searching online and asking seniors. Prof. Vivek Borkar is one of the most well reputed professors in this field and was also strongly recommended by seniors so I approached him.

Duration of the project

It was a non credited project. I worked on the project initially for 7 months and after revisions again for 1 to 2 months.

Overall experience

My contribution to the project was mainly writing code for the proposed RL agent and testing it out on Non Markovian Environments against standard DQN agent. The project for me was fairly difficult as the RL agent design was quite complicated and required a very good grasp on ML coding which I did not have at that point but learned on the go. The entire coding part was assigned to me so I had to find Non Markovian Environments or go through code of existing ones and modify them to become Non Markovian. The theoretical proof for the proposed design was also very complicated and although I wasn’t required to understand it completely, a higher level intuitive understanding was required for coding.

I enjoyed the project thoroughly. Prof. Borkar and the PhD student I was working with were very supportive and there wasn’t any pressure put on me, plus they were always available to clear my doubts. It was a great learning experience for me and made me more interested in ML research.

Key learnings

I attained a high level of proficiency in machine learning coding due to my involvement in this project. Plus my knowledge about RL also significantly increased. Due to this experience I got into ML research (Specifically RL).

What course material had you gone through prior to the project?

I had completed CS747 and was doing CS748 along with this project.

Review of the Professor

Prof. Vivek Borkar, as I mentioned before, is one of the best and highly reputed professors in our Institute and in his field. He wouldn’t put pressure on you or give you deadlines but at the same time if you are dilligent and regularly keep working on the project there is a lot you can learn from him and also accomplish in your project as he will always guide you with what can be done further in the project.

Future plans following this project

After this project I am currently pursuing my BTP under Prof. Borkar too in the field of RL.