![\"Harald](\"https:\/\/anexia.com\/blog\/wp-content\/uploads\/2023\/11\/IMG_1669-1-scaled.jpg\")
<\/a><\/p>\nApplication<\/h2>\n
Whisper from OpenAI is used for speech recognition. This tool needs to be supplemented in some places, for example to identify speakers and recognize pauses in conversations.<\/p>\n
The process is structured in different steps:<\/p>\n
SPEAKER stands for the speaker defined in the RTTM file and ASR_TEXT for the transcribed text.<\/p>\n JSON format It can therefore be said that artificial intelligence can make a significant contribution to summarizing video conferences. With the help of ASR and speaker diarization, it is possible to present the content of a video conference in text form and shorten it to a compact and content-relevant form using the BART model.<\/p>\n Since the work was written some time ago, it is quite possible that the model described above no longer works optimally due to changes in the open source applications.<\/p>\n The project is available on Github<\/a>.<\/p>\n What is Artificial Intelligence? <\/a>\u00a0\u2192<\/a><\/p>\n Anexia Artificial Intelligence Development \u2192<\/a><\/p>\n<\/a><\/p>\n\n
\nAn audio file is extracted from a video. The image material is no longer required for further processing.<\/li>\n
\nIn this step, all speakers in the video are recognized and an RTTM file<\/a> is created using the pyannote.audio <\/a>tool.<\/li>\n
\nThe generated RTTM file is divided into different audio blocks in this step.<\/li>\n
\nWith Whisper, the audio blocks are automatically recognized as text, transcribed and stored as TXT files.<\/li>\n
\nThe RTTM file from step 2 and the TXT file from step 4 are merged using a Python function. The following formats are generated:
\n
\nTXT format
\n<\/strong>The text is displayed as a dialog.<\/p>\n{SPEAKER}: {ASR_TEXT}<\/pre>\n
\nThe JSON output contains more details and can be used later for troubleshooting. The schema here is as follows:<\/p>\n{\r\n\u00a0 \"$schema\": \"https:\/\/json-schema.org\/draft\/2020-12\/schema\",\r\n\u00a0 \"type\": \"array\",\r\n\u00a0 \"items\": {\r\n\u00a0\u00a0\u00a0 \"type\": \"object\",\r\n\u00a0\u00a0\u00a0 \"properties\": {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"start\": {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"type\": \"number\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 },\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"duration\": {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"type\": \"number\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 },\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"speaker\": {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"type\": \"string\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 },\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"text\": {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"type\": \"string\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 }\r\n\u00a0\u00a0\u00a0 },\r\n\u00a0\u00a0\u00a0 \"required\": [\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"start\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"duration\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"speaker\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 \"text\"\r\n\u00a0\u00a0\u00a0 ]\r\n\u00a0 }\r\n}\r\n<\/pre>\n<\/li>\n
\n<\/strong>The BART model<\/a> is used to combine the TXT and JSON format and create a summary of the entire conversation. It has also been shown that the SAMSum data set can be used efficiently for fine-tuning BART. Due to the existing models and data, the summary is currently only possible in English.<\/li>\n<\/ol>\nConclusion<\/h2>\n
RELATED TOPICS<\/h2>\n