• Most new users don't bother reading our rules. Here's the one that is ignored almost immediately upon signup: DO NOT ASK FOR FANEDIT LINKS PUBLICLY. First, read the FAQ. Seriously. What you want is there. You can also send a message to the editor. If that doesn't work THEN post in the Trade & Request forum. Anywhere else and it will be deleted and an infraction will be issued.
  • If this is your first time here please read our FAQ and Rules pages. They have some useful information that will get us all off on the right foot, especially our Own the Source rule. If you do not understand any of these rules send a private message to one of our staff for further details.
  • Please read our Rules & Guidelines

AI Speech to Text may revolutionize creating subtitles for FanEdits!

Baenwort

Well-known member
Messages
49
Reaction score
38
Trophy Points
18

So there is a new OpenAI model (from the people who have created the AI stuff that is making all the waves in visual still art) that you can download and run locally to create some pretty good (to excellent) subtitle files for video and audio files. It will sync up the subtitles to the file you provide automatically so not only does it generate the text but it aligns it for you. From some of the testing people on that site did it is already better than what Google uses!

This may help more faneditors to produce subtitles for their creations and this can expand the community of people who can enjoy some of the great works found here.
 

DonkeyKonga

Well-known member
Faneditor
Messages
1,190
Reaction score
1,158
Trophy Points
133
You are my absolute hero. I have downloaded so many fanedits I have to watch alone because my fiance isn't good enough at english. I just realised songs with text in them will probably also get translated. xD
 

Baenwort

Well-known member
Messages
49
Reaction score
38
Trophy Points
18
It will technically work on any audio file you feed it of the 99 languages it currently supports. Some languages seem to do better than others but all of them are beating the Youtube auto results!

There has even been some work and tests using it to translate from one language to another but I've only used it to make same language subtitles myself. If this works out well for you please report back!
 

SIUse

Well-known member
Faneditor
Messages
161
Reaction score
71
Trophy Points
43
How well does it compared to adobes speech to text generator in premiere?
 

Baenwort

Well-known member
Messages
49
Reaction score
38
Trophy Points
18
How well does it compared to adobes speech to text generator in premiere?
I don't have that to compare. Hopefully someone here can try it. I've compared it to Youtube, MS Office/Teams, and Google's speech to text via Assistant. I'd been better than all those.

From the fact that people are seeking to add WhisperAI to Adobe products I suspect it is better?
 

Baenwort

Well-known member
Messages
49
Reaction score
38
Trophy Points
18
If you are using Davinci Resolve in your workflow here is now a plug-in to help you generate subtitles. Using the OpenAI Whisper mentioned in the OP it is still a work in progress but is a great advance and should make it a LOT easiler to add subs to your fanedit!

 

Baenwort

Well-known member
Messages
49
Reaction score
38
Trophy Points
18
For those who currently use SubtitleEdit to create their subs and would like to have a generated base to start editing from there is now a SubtitleEdit plug in:

 

NOTFLIX

Well-known member
Faneditor
Messages
128
Reaction score
72
Trophy Points
38
How interesting!
In my edit of MARVEL The TV Series I have added two tracks of subtitles BY HAND, and this type of technology can greatly help those of us who try, as a matter of principle, to include subtitles in all our edits.
Thanks for the tip!
 
Messages
5
Reaction score
2
Trophy Points
8
Oh my goodness, this could be huge. I can finally make subtitles for my edits without the need to painstakenly type them in via YouTube.
 

Baenwort

Well-known member
Messages
49
Reaction score
38
Trophy Points
18
For those with powerful GPUs (8GB of dGPU RAM) there is now GPU accelerated audio to text translation in SubtitleEdit via WhisperAI:

https://github.com/SubtitleEdit/subtitleedit (Release 3.6.11 and later contain it). I would suggest using the nightly beta as the GPU acceleration is a new feature that is still getting patches.
217328867-cf398a1f-bfba-4ccc-9acd-69bcc8b7d466.png
to change to GPU acceleration (a order of magnitude faster)
 

WilliamRedRobin

Well-known member
Faneditor
Messages
364
Reaction score
522
Trophy Points
113
I've done a bit of experimentation with Wisper.ai alongside Subtitle edit.
When you run the program you can choose from five sizes of program, "tiny", "base", "small", "medium" and "large".
The larger the file you choose, the longer it takes to transcribe, and the more accurate it is.
I found that when I made subtitles for part 1 of my miniseries, which is just under an hour long, the tiny file wrote the program very quickly, but was way off. When I used the large file, while the result was much better, it still was far from perfect and I had to go through the whole thing to make corrections. Which is a lot quicker and easier than making subtitles from nothing, but it isn't at all a one click and done affair.
Overall, most lines were transcribed correctly, but the timing was off, and the sound effects were almost entirely wrong.
The AI makes an admirable effort to figure out who's making what sounds and what their names are, but as you can imagine, that's a nigh impossible feat. And the first thing I did was go through the file to delete all instances of "(wind blowing)".
All in all, I was able to complete the subtitles in a single day, while multitasking, but the time taken for the subtitles to auto-generate seems to grow exponentially with the video length.
When I tried making a file for the full series, which comes in at just over nine hours, I found that the large file took impractically long. I found that the largest program worth using on a video of that length was "small" which turned out to be a "set it going before bed and come back after work" kind of deal.

Overall, making subtitles this way actually takes longer from start to finish, but the time you actually have to spend sat at your computer is obviously less, though I'm not sure exactly how much.
The real upside is that it's nicer to edit a file than to work from scratch (and, let's face it, this is a fanedit forum, nobody here likes working from scratch).
This will probably be the main way I create subtitles for my edits, but if someone said they preferred going totally manual I wouldn't be surprised.
 

WilliamRedRobin

Well-known member
Faneditor
Messages
364
Reaction score
522
Trophy Points
113
I just thought I'd update this based on recent experience.
I've had some trouble using the larger program files to generate subtitles for videos longer than an hour, where the program stops making progress after a few hours and in some cases freezes my computer.
This is undoubtedly a hardware issue, with my laptop's cooling system not being sufficient to prevent overheating.
If others have this problem, I've found that a cooling rack + desk fan combo works wonders.
Otherwise, it might be a good idea to break the video up into half-hour segments and do them individually before combinining them for the full file.
Unfortunately, SubtitleEdit doesn't seem to have an option to "pick up where you left off" with WhisperAI. I'll see if there's a way I can suggest this to the developers, because it would definetly come in handy.
 

cafevincent

Well-known member
Messages
61
Reaction score
4
Trophy Points
13
I can't believe this is so heavy on the system. I guess it translates the movie as a whole, learning names and stuff on the way like a human would. I would love to use one that doesn't do any of that. Where you could just use the audio track, and the AI would not try to learn names or describe sounds, or even translate. All I need is English for an English language movie. If the timing would be accurate, this really would revolutionize building up subs for fanedits no matter how accurate the automatic text would be since it would obviously have to be checked 100%. Is there a way to do it like this?
 
Top Bottom