The final work period is done and the application is up and running at https://snd96.pythonanywhere.com/. The github repository for my work can be found here: https://github.com/SND96/twine-stories. These are the tasks that the application can so far perform. Parse through an entire Twine story and then extract the relevant data like questions and choice statements. The entire story can then be navigated using Flask. Generate the phonemes for each choice statement and then loading it into Pocketsphinx.

This week was spent mostly on loading the pronunciation phonemes for each word in every option into Pocketsphinx.js. The first step was getting all the phonemes which was already accomplished earlier and made into jsgf files. Instead of using the jsgf files, the pronunciations will be loaded directly into the html page in two steps. The first step will be to make an entire dictionary for all the word present on that particular page.

From the previous week my main aim was to implement a basic version of the recognizer from syl22-00/pocketsphinx.js. I have built a previous version of it earlier which I used as a basic from SND96/pocketsphinx-scores. I originally built this to to get the alignment scores for words in the phrase “What’s your name?”. I won’t need this for now so I removed that completely but may include it at a later time when the application is more stable.

This entire week has been devoted to adding Pocketsphinx.js recognizer to the application. The first thing we had to decide was whether to put a separate recognizer for the entire application. Most of this week has been spent trying to implement the application that I made in March into the Twine HTML files. This hasn’t working and has been incredibly buggy resulting in the microphone not working at all and has delayed my schedule for the week.

Using Flask has been a bit slow as I’ve had to learn an entirely new framework. Flask applications work on connecting different HTML pages through code which acts as a wrapper. The first thing that I had to do was make a parser for the which separates all the Twine pages into HTML files. Separating the files was important as it would make the routing between pages much more streamlined using the Flask application.

I’ve decided to combine these 2 weeks as it seems to make more sense as not much was accomplished during week 4. Right now I have to make a flask application which wraps around the Twine Stories that I’ve already used. This is the first step towards making the final Computer Aided Learning Interface. Right now the Twine stories are purely navigated through by just clicking on the correct option. The whole point of this project is to be able to do this by the means of speech recognition and to give feedback on this.

This third week has been devoted mainly to processing the necessary prompt files which will be used later for the the education system. There were two different types of recordings. One which had purely a single word. These proved challenging in the beginning but we soon managed to process the files using the correct commands in sphinx. The first step was converting all the files into .wav(16,000 samples per second, 16 signed bits per sample, little endian) format from .

Hi everyone! My name’s Sahith Dambekodi and I’m a second year undergrad student at BITS Pilani K.K. Birla Goa Campus. I’m pursuing a major in Electrical and Electronics engineering. My initial foray into open source began mainly with general purpose machine learning libraries like scikit-learn and tensorflow. I had heard about Google Summer Of Code through my peers and I was intrigued by the possibilities of coding an entire summer about something I was genuinely interested in.

Twee/Twine is an open source software that is used to make branching stories which means that there will be multiple possible paths through any story. Twine stories are an intuitive and easy way to make assignments and we thought it would be a good idea to use that to supplement the voice recognition software. Twine Stories create stories with multiple possible branching scenarios through them. These stories can be used as an engaging tutorial to help students with new technologies.