Study Tool Utilizing Advanced AI Models

https://www.youtube.com/watch?v=yWcEMUfu_zY

1. Research Phase

The research phase involved exploring the latest advancements in AI technologies that could be integrated into the study tool, ensuring that the project leveraged state-of-the-art solutions.

• AI Models and Tools: A thorough review of existing AI models was conducted. This included examining Large Language Models (LLMs) like OpenAI's GPT-4 for natural language understanding and generation, Vision Models like Google's Vision Transformer (ViT) and OpenAI's CLIP for slide content recognition, and Text-to-Speech (TTS) and Speech-to-Text (STT) models such as OpenAI’s TTS and Deepgram’s Nova 2 for enhancing user interaction through voice and audio capabilities.
• Feasibility Studies: The team compared different tools based on performance, cost, and ease of integration. For instance, while Eleven Labs and Play.ht offered highly realistic TTS solutions, they were excluded due to cost concerns. Local TTS models like Coqui TTS and Tortoise TTS were also explored but faced implementation challenges.
• Prototype Decisions: Initial decisions on the tech stack were made based on the need for rapid prototyping. Python was selected for the MVP (V0) development due to its rich ecosystem of libraries and strong community support. As the application scaled, further research led to the adoption of Golang for its performance advantages.

This phase laid the groundwork by ensuring that the right models and technologies were selected, balancing the trade-offs between performance, cost, and ease of use.

2. Design Phase

The design phase focused on architecting the application, ensuring that it would efficiently handle data processing, AI model integration, and user interaction.

• System Architecture: The application architecture was designed with a focus on modularity and scalability. The core components were organized around a "Space" model, with each Space representing a different course or study area. Slides from PDFs were treated as individual units, converted into images, and processed independently to generate notes and quizzes. The backend was designed to handle the conversion of PDFs to images and integrate with OpenAI for generating textual content.
• User Experience and Interface: The user interface was inspired by DLVU lecture notes, focusing on providing a clean and intuitive layout where slide images were displayed on the left and AI-generated notes on the right. The frontend was developed using Vercel's AI frontend tools, allowing for rapid iterations on the user interface to ensure a smooth and responsive experience for users. This design aimed to simplify the study process, making it easy for students to access detailed explanations, quizzes, and even audio playback of the generated content.
• Component Design: Each component, from slide text generation to quiz creation, was designed with efficiency in mind. For instance, the quiz generation was structured to create questions based on the content of every five slides, ensuring relevant and concise assessments without overwhelming the user.

This phase established a clear blueprint for the application, focusing on both the backend architecture and frontend user experience.

https://youtu.be/5unXwAog63Y

3. Implementation Phase

The implementation phase involved turning the design into a working application, integrating the various AI models, and ensuring the system functioned as intended.

• MVP Development: The MVP (V0) was initially developed in Python, allowing the team to quickly prototype and test key features. Python's flexibility and library support made it an ideal choice for early-stage development. The MVP included basic functionalities like slide text generation and simple user interaction via text queries.
• Transition to Production: As the application matured, the backend was transitioned to Golang to optimize performance. Golang's advantages in concurrency and execution speed made it suitable for handling the server-side operations, including PDF-to-image processing and integration with OpenAI's API for generating content. GitHub Copilot was utilized to accelerate code development, particularly for the Golang server code, improving both efficiency and accuracy.
• AI Model Integration: Multiple AI models were integrated to deliver a seamless experience. OpenAI's GPT-4 was used for generating detailed explanations of slides, while Deepgram’s Nova 2 was implemented for voice transcription. TTS models were employed to convert the generated notes into audio, enhancing accessibility. This phase also saw the development of the voice assistant, combining various models to allow users to interact with the application via voice commands.