Developed an image-based vision system for a racing simulator, SuperTuxKart by leveraging deep learning using Python. The initial phase involved implementing a CNN classifier using PyTorch. Employing convolutional layers, ensured effective feature extraction for classification tasks. Through hyperparameter tuning and employing various techniques such as input normalization, residual blocks, dropout, and data augmentation, the model achieved an accuracy of 94% on validation data. Transitioning from image classification to dense prediction, the CNN was converted to a Fully Convoluional Network (FCN) for semantic segmentation. This FCN is designed to handle arbitrary input resolutions and outputs predictions per pixel. Employing skip and residual connections ensured accurate object detection and segmentation. Expanding the segmentation network, it was repurposed into point-based object detection. Predicting dense heatmaps of object centers enabled efficient object localization. Implementing peak extraction algorithms facilitated the identification of relevant objects such as projectiles, karts, bombs, nitro within the scene, further enhancing the system's understanding of the environment. Beyond vision processing, a low-level controller responsible for steering the racing vehicle towards a specified aim point was integrated with the image-based predictions along with control actions, to ensure smooth and efficient navigation across diverse racing tracks. Lastly, the planner model was trained to predict aim points directly from input images. Overall, the sequential culmination of these components resulted in a high-performance vision-based system for racing simulation. This project helped me learn lot about systems that showcase the superiority of deep learning in enhancing virtual game environments.
Owing to UT Austin Honor Code, the model files will not be made public.