diff --git a/README.md b/README.md index 0ac28333c50b16b0317b95744484b7a30e98a72b..87778d752eaf53532b6f060c0ad269721ccdc1c4 100644 --- a/README.md +++ b/README.md @@ -2,35 +2,63 @@ Khan, Asif, 22300224 Netflix Content Analysis -https://mygit.th-deg.de/assistance_systems +link to the MyGit Repository : https://mygit.th-deg.de/assistance_systems/gitlab-profile -https://mygit.th-deg.de/assistance_systems/gitlab-profile/-/wikis/home +link to the MyGit Wiki : https://mygit.th-deg.de/assistance_systems/gitlab-profile/-/wikis/home # Project Description Netflix Dataset Analysis and Prediction is a system designed to explore, analyze, and build predictive models on the Netflix catalog dataset. The application provides insights into content trends and implements machine learning models to make recommendations or predictions. # Installation +Follow the steps below to set up the project locally: -A list of installed packages can be found with `pip freeze`. +Clone the Repository: + git clone https://mygit.th-deg.de/assistance_systems/gitlab-profile.git + cd gitlab-profile -Versions: +Create a Virtual Environment: +It's recommended to use a virtual environment to manage dependencies. -- Python 3.9.6 -- streamlit 1.41.1 -- scikit-learn 1.1.3 -- rasa 3.6.20 + python3 -m venv venv + + source venv/bin/activate # On Windows: venv\Scripts\activate + +Install Dependencies: +Install the required Python packages using pip. + + pip install -r requirements.txt + +Verify Installation: +Ensure all packages are installed correctly. + + pip freeze # Data The Netflix dataset was sourced from Kaggle. It contains details about movies and TV shows available on Netflix, including: -Title -Genre -Release Year -Rating -Duration -Cast and Directors +show_id - Unique ID of Each content +type - Movie or Tv Show +title - Title/ Name of the Movie or TV Show +director - Director +cast - Cast in the movie or tv show +country - Country of content +date_added - Content added on Netflix +release_year - Year of Release +rating - Age rating +duration - Duration of the movie or tv show +listed_in - Genre +description - Description of the movie or tv show + +# Dependencies + +Python: 3.9.6 +Streamlit: 1.41.1 +Scikit-learn: 1.1.3 +Rasa: 3.6.20 +Flask: Latest version +Seaborn: Latest version # Basic Usage @@ -44,51 +72,99 @@ Prerequisite: The rasa model is trained! (with `rasa train`) Then call the streamlit URL, e.g. localhost:8051, in the browser. -The webpage shows the main pages in the navigation bar on the left. Typically the entries are used from top (main) to bottom (chatbot). +The webpage shows the main pages in the navigation bar on the left. All the entries are : +about_me +add_and_apply_model +algorithm_selection +augmentation +chatbot +data_metrics +feature_engineering +model_application +model_training +preprocessing +viszualization +chatbot # Implementation of the Requests -1. The pages are in the pages folder of the streamlit project +# Data Handling -2. See Wiki +1. Data Loading: +The Netflix dataset is loaded in main.py using pandas for efficient data manipulation and exploration. -3. Wiki chapter user persona +2. Data Preprocessing: +Implemented in preprocessing.py to clean the dataset by handling missing values, encoding categorical variables (e.g., genres, directors), and scaling numerical features (e.g., duration, release year). -4. Wiki chapter use cases +3. Data Augmentation: +Managed in augmentation.py, adding synthetic data to enhance the dataset for better model performance and robustness. -5.-7. The files are stored in the project root directory. +# Machine Learning Models -8. See data section +1. Logistic Regression: +Used in add_and_apply_model.py to predict the primary genre of Netflix titles based on features like director, duration, and release year. This model is lightweight and interpretable, making it ideal for quick predictions. -9. Data is loaded in the main.py +2. Random Forest Classifier: +Utilized in model_training.py for more complex, non-linear predictions. It handles categorical and numerical data effectively, improving genre prediction accuracy. -10. see pages/01_Data_*.py files +Model Evaluation: +Model performance is evaluated using metrics such as accuracy, precision, recall, and F1-score in model_training.py and add_and_apply_model.py. Confusion matrices and classification reports provide insights into performance. -11 - 13. See data chapter in the Wiki +# User Interface -14. Data_augmentation.py +# Streamlit Pages: -15. Input widgets are mainly in 05_Model_training.py +1. about_me.py: Displays project information and developer details. +2. add_and_apply_model.py: Implements logistic regression and Random Forest for genre prediction and custom predictions. +3. algorithm_selection.py: Allows users to choose between different machine learning models. +4. augmentation.py: Adds synthetic data for enhanced model performance. +5. chatbot.py: Integrates the Rasa chatbot for conversational interactions. +6. data_metrics.py: Provides data insights and key metrics. +7. feature_engineering.py: Handles feature extraction and transformations. +8. model_application.py: Applies the trained models to the dataset. +9. model_training.py: Trains logistic regression and Random Forest models. +10. preprocessing.py: Cleans and preprocesses the dataset. +11. viszualization.py: Visualizes trends, distributions, and model predictions. -16. Scikit-Learn Linear regression and Lasso are used -17. See 'right-fit' chapter in Wiki +# Input Widgets: +Integrated into pages like algorithm_selection.py and add_and_apply_model.py to capture user inputs, such as director name, duration, and release year, for custom predictions or visualizations. -18. See 'system persona' chapter in Wiki +# Chatbot Integration -19. See 'sample dialogs' chapter in Wiki +Rasa Implementation: +Built using domain.yml, data/nlu.yml, data/stories.yml, and actions/actions.py. The chatbot provides conversational recommendations, explains model predictions, and guides users through the platform. -20. See 'dialog flow' chapter in Wiki +REST API Communication: +Handled in chatbot.py, which connects the Streamlit interface to the Rasa server for real-time interactions. -21. rasa implementation with files: +Dialog Management: +Structured in Rasa's story files, allowing users to ask for recommendations, clarify predictions, or navigate functionalities in a conversational manner. -domain.yml, data/nlu.yml, data/stories.yml, actions/actions.py +# Visualizations -22. tbd +Data Insights: +Implemented in data_metrics.py and visualization.py, showcasing trends like genre distributions, release year patterns, and duration analysis through bar charts, line plots, and heatmaps. -# Work done +Model Predictions: +Plotted in add_and_apply_model.py and model_training.py using bar charts, confusion matrices, and scatter plots to display predicted vs. actual genres. -All work by Asif Khan. +Custom Predictions: +Highlighted in add_and_apply_model.py using interactive graphs, including legends to distinguish between predicted and actual genres. +This comprehensive implementation integrates both Logistic Regression for interpretability and Random Forest for enhanced predictive performance, with a user-friendly Streamlit interface and an interactive chatbot powered by Rasa for seamless user experience. +# Use Cases + +Explore the Use Cases Video : + +# Wiki Reference + +The Wiki serves as a detailed documentation repository, containing project-specific information. +User Persona: Detailed in the "User Persona" chapter of the Wiki. +Use Cases: Defined in the "Use Cases" chapter, specifying user interactions and functional expectations. + + +# Work done +All work by Asif Khan. \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 411f5c3c584a6bef8c324311028f75b59f26f2b7..81fcca38f745cafa7f5c62559c0667ab9849df0a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,9 @@ streamlit streamlit-chat scikit-learn -rasa \ No newline at end of file +rasa +flask +seaborn +pandas +numpy +matplotlib \ No newline at end of file