New language model for spaCy from scratch

I'm Marwane, a Python expert with extensive experience in Deep Learning, Machine Learning (ML) and data analysis. I understand what it takes to build a language model from scratch and how important it is to get this job done right. That's why I'm here to offer my services for your New Language Model project. I have the necessary coding tools and experience to tackle this project. My skills include Deep Learning, Machine Learning (ML), data analysis, data visualization and cleaning. Plus I have extensive experience in parsing and text normalization which will be needed for this project. I believe that my combination of knowledge and skills make me the perfect fit for this project. If you choose me for the job, you can expect high quality results delivered on time with minimal fuss.

Hi. Thanks for your posting. I have just read your proposal and I am sure I can complete the project on time. I am an expert in ML/DL who has many years of experiences. Please contact me to discuss the project in more details. Waiting for your contact now... Thanks. Best Regards.

Hello, my name is Shahawar and I am an artificial intelligence expert with more than 12 years of company work experience. As you may know, spaCy does not currently have a language model for Azerbaijani and univeraldependencies does not have a treebank for this language. I understand the importance of this project and that it needs to be done quickly without any errors. With my extensive experience in language module processing, deep learning, image processing (OpenCV), I am confident that I can develop a language model for spaCy from scratch that will find answers to given questions in documents with which it will be pre-trained. Additionally, I have extensive experience in parsing and text normalization which would be necessary for this project. I would be delighted if you considered offering me the opportunity to complete your project as I have the required abilities and expertise to do so in a timely manner. Please feel free to contact me if you have any further questions or would like to discuss further regarding this project.

I understand your need to develop a custom Azerbaijani language model for spaCy based on your extensive text corpus. This is a complex task that requires expertise in natural language processing and machine learning. I have experience in training custom language models and can guide you through the process. We'll need to preprocess the text, create linguistic annotations, and train a model. I'll assist in selecting appropriate architectures and hyperparameters. Let's discuss the specifics of your project and the resources available to proceed with model training. Your dedication to collecting data is a valuable asset for this undertaking.

Hello there, really exciting one you have here, since you are new to the NLP domain, let me give you a skeleton approach so that the problem can be better understood. Approach: Data Collection: Utilize the provided text corpus, including news articles, books, Wikipedia articles, and social media comments, totaling approximately 10 million lines of text data. Annotated Data: Manually annotate a subset of the corpus for training data, including question-answer pairs. Develop a specific annotation schema for Azerbaijani language questions and answers. Implement Named Entity Recognition (NER) tagging for key entities in the text. Model Architecture: Build a custom language model using spaCy's infrastructure, incorporating transfer learning from a base model (e.g., a similar language, if available). Train the model on the annotated data using deep learning techniques, using architectures like LSTM, Transformer, or BERT. Evaluation: Implement rigorous evaluation metrics to assess the model's performance, such as F1-score, BLEU, and ROUGE. Fine-tune the model iteratively based on evaluation results. Integration with spaCy: Develop a spaCy pipeline component for Azerbaijani language support. Enable the model for various NLP tasks, such as Named Entity Recognition (NER), part-of-speech tagging, and question-answering. Ensure compatibility with spaCy's existing features and libraries.

Hello! I have over 4 years of experience in commercial development and hold a degree in Artificial Intelligence. I also have experience integrating ChatGPT and other neural networks into bots and applications through APIs. I would be delighted to discuss project details with you. Best regards, Ilya.