Separating Sound Effects, Dialogue and Music?

[addiesin]

Adobe Audition, which I haven't opened or looked at in years since CS5 or 6, appears to have a sound remover effect that works like noise removal, but with parameters that allow better tweaking and specifically an option to avoid dialogue frequencies. I'll test out on a dirty center channel and see what kind of results I can get. It's in the Adobe CC 2019 release.

https://helpx.adobe.com/audition/using/noise-reduction-restoration-effects.html#sound_remover_effect


Edit: it also has a spectral editor so no need for Spectra Layers.

https://helpx.adobe.com/audition/ho...set=audition--key-techniques--noise-reduction

 

[Ryantology]

Adobe Audition, which I haven't opened or looked at in years since CS5 or 6, appears to have a sound remover effect that works like noise removal, but with parameters that allow better tweaking and specifically an option to avoid dialogue frequencies. I'll test out on a dirty center channel and see what kind of results I can get. It's in the Adobe CC 2019 release.

https://helpx.adobe.com/audition/using/noise-reduction-restoration-effects.html#sound_remover_effect


Edit: it also has a spectral editor so no need for Spectra Layers.

https://helpx.adobe.com/audition/ho...set=audition--key-techniques--noise-reduction

Very curious to see how well that works out!

 

[addiesin]

Adobe Audition, which I haven't opened or looked at in years since CS5 or 6, appears to have a sound remover effect that works like noise removal, but with parameters that allow better tweaking and specifically an option to avoid dialogue frequencies. I'll test out on a dirty center channel and see what kind of results I can get. It's in the Adobe CC 2019 release.

https://helpx.adobe.com/audition/using/noise-reduction-restoration-effects.html#sound_remover_effect


Edit: it also has a spectral editor so no need for Spectra Layers.

https://helpx.adobe.com/audition/ho...set=audition--key-techniques--noise-reduction

Very curious to see how well that works out!

I messed with it last night and it certainly isn't a magic bullet.

Here's what I tried:

Noise reduction as a control group. This works about as well as you'd expect. Complete music removal but vocal frequencies are damaged and have a "washy" bitcrush tone/timbre. This is fine if you're just doing something quick and slapdash. It's also fine in most cases that you'll be adding louder music. It's you want to have a quiet scene it's not the best option. Overall quality of the plug-in is better than the free equivalent in Audacity, but not so much better that it's worth getting an Adobe subscription just for this.

Spectrum editing, manually and visually selecting portions of the frequency spectrum and deleting them. Results: good for specific sound removal, bad for noise reduction. Deleting too many frequencies turns voices into Star Wars Stormtroopers. Without being an audio expert and knowing which frequencies are most common to human voices by heart, this is not a viable way to clean music out of a track but works for sound effects and infrequently occurring/sudden noises.

Sound reduction using my own voice and a baby noise machine "heartbeat" sound in the same recorded track. Used the portion of the track without my voice for the sound model. Was able to effectively remove the heartbeat sound and retain my voice. Minor bitcrush/washiness that could probably be reduced with settings tweaks.

Sound reduction using a portion of a movie's center channel (single mono wav WAV file) with music and without dialogue as the sound model. Applied effect to portion of track that used the same music track but with dialogue. This works surprisingly well, with some caveats. I tried default "complexity" settings and high settings and got some different results, though which is better depends situationally. I didn't test each individual setting to see what each did, just tried default then max for all. The portion I used for the sound model was a second or two long.

Sound reduction with the audio CD soundtrack used as a sound model. Applied to movie center channel. First, stereo track had to be converted to two mono tracks. Used entire track as sound model. Results: Didn't work at all. Reduced all audio to weird noises that sounded like shuffling paper. I probably will experiment more with this until I crack it. Maybe the recording was different, or the mastering, or the volumes, or maybe there's too much info in the stereo soundtrack compared to what's in the center channel of the surround track from the film.

Will report back with more findings as I go. Maybe with clips to share. Last night was rather off the cuff.

 

[TM2YC]

Here's what I tried:

Noise reduction as a control group. This works about as well as you'd expect. Complete music removal but vocal frequencies are damaged

What about non-complete music removal? Is it possible to significantly reduce (but not eliminate) music and retain (mostly) undamaged dialogue?

 

[addiesin]

Here's what I tried:

Noise reduction as a control group. This works about as well as you'd expect. Complete music removal but vocal frequencies are damaged

What about non-complete music removal? Is it possible to significantly reduce (but not eliminate) music and retain (mostly) undamaged dialogue?

Yes, if your plug-in allows you to adjust the amount of reduction.

If you don't have control over little settings and percentage of the effect applied, you could still duplicate your original track and align the two, apply the effect to one copy, and adjust volume between them. Maybe use a medium to heavy application of equalization to the original to reduce the louder frequencies already present in the noise-removed track.

But you shouldn't have to do that, you can absolutely use less aggressive settings and get a lighter result. 

The biggest problem in my opinion is if you're using in-film audio as the sound model (using sound reduction) or as the noise profile (using noise reduction) you're going to lose the room noise. That's why I wanted the CD soundtrack method to work. It might be worth trying to take the damaged audio and lay it over some home-recorded room noise to see if that fills the gaps. Maybe apply a light reverb to the dialogue.

In any case, the Adobe Audition training video for noise reduction suggests doing several light passes instead of one aggressive pass. When I have some time I'll try this with CD audio, with short instrument sounds to see if I can maybe pull the score out piece by piece.

Another thing I just thought of, there's a karaoke style vocal remove effect. In theory if that worked well and removed dialogue perfectly, you could do the track duplication trick, use vocal remove on one track, then invert it. This would, in theory, make a negative out of that track and effectively preserve ONLY what was originally removed.

 

[addiesin]

Some cool new technology and methods are surfacing. There's a free AI audio processor, and more info on Adobe Audition's capabilities, both of which were posted around the forum on specific project threads. Rather than paraphrase, here are the words from the sources in quotes.
 

Just a heads up, I was having trouble cleaning up an audio track in audacity (isolating a voice) for a Moody Blues song I was creating, but found this online vocal isolation site that works PERFECTLY: https://vocalremover.org/

Thanks Keith for posting this. I tried it and while not perfect, it is incredibly fast and easy, and impressive.

Edit: clarifying what I tested. I tried two separate files, for my own use cases. I did not try using a center track from a surround mix, but I suspect that will work much better than my heavier tests. No matter what it only allows you to export mp3s from the results but I suppose if nothing else works and rescoring is important to your edit you can overlook the inherent quality drop to an extent.

Test 1
There's an online only video source for my edit with only stereo sound. I wanted to better mix into my surround timeline. I ran the audio through (wav) and was really impressed. I still got some bleed through from some weird music frequencies specific to this video, but the voice is mostly very clear and less bit crushed compared to most methods I've tried. This is a good starting point for separating stereo audio into discrete channels, but is not a magic bullet, and will require more work after processing.

Test 2
I used to be in a band, and we had a professionally recorded and mixed album. I ran one of our songs through (mp3) and the vocals are very bit crushed, unusable. However, the music with vocals removed worked out pretty well. This is possibly a good way to create a "karaoke version" of a song.

I'll try on a center channel track when I have a chance, that will probably be a more common use case.

 

Here's my "recipe" to remove music, that I first tried while restoring the original score for H20, funny enough. I'll go over each step quickly and you can search or ask for more info as needed.

1.  Extract 5.1 Surround audio from the movie

2.  Separate each audio stem from the 5.1 track into 6 individual mono WAV files.

3.  The Left & Right channels (both front and rear) can be easily edited to replace music. The center channel contains the voices and dialog, so it needs the most work to remove music while keeping dialog and foley intact.

4.  Using the music-only tracks from the soundtrack of the movie, I sync up the parts that I want cleaned-up with their corresponding music track in Adobe Premiere. In effect, you have the music and the center channel on top of each other, in sync.

5. Export both sections as 2 WAVs.

6. In Adobe Audition, import the resulting Center Channel WAV file and Soundtrack WAV file.

7. Go to "Learn Sound Model" and select your Soundtrack WAV. 

8. Finally, go to your Center Channel WAV and select "Sound Remover". The program will now consider the whole music track as a sound to be removed, and since it's synched up perfectly with your center channel track, everything that isn't music will remain.

Some trial and error may be needed and it's not the most straightforward process, but I find it does a lot of heavy lifting before having to redo sound effects.

I confirmed with thebutcher this method can sometimes work by building the Sound Profile based on the front left and front right tracks, and applied to the center track. However, if that doesn't work, you can also try using a CD soundtrack and lining up tracks to match the film's center channel, and use the CD to create the Sound Profile.



Both these methods show a lot of promise and seemed like they'd be helpful and relevant here.

 

[Declan1974]

I can confirm that if you are stuck trying to isolate dialogue in a mono channel, if you have access to the original soundtrack CD, you can do the following and get good results:

line up the stereo track
downmix to mono
use the downmixed mono track as the learning sample
apply sound removal to the original mono track

very minimal artifacts

only thing is the sample limits to 90 seconds. Not sure if results would be better to chunk into smaller chunks if your dialogue is longer than 90 seconds.

 

[Agent Sam Stanley]

Will any of this work if I only have a stereo track to work with? I've been planning for a while to do a new edit that would replace most of a film's soundtrack, but in the scenes with dialogue and loud music, I think it'd be impossible. Kind if crushed about this

Edit: sorry just saw the post about vocalremover.org. So there is some hope

 

[addiesin]

Will any of this work if I only have a stereo track to work with? I've been planning for a while to do a new edit that would replace most of a film's soundtrack, but in the scenes with dialogue and loud music, I think it'd be impossible. Kind if crushed about this

Edit: sorry just saw the post about vocalremover.org. So there is some hope

The answer is probably no, unfortunately. Try the AI separator though, even if the result sounds damaged it might be salvageable with music and sound effects layered in.



...

The other thing to try is lining up the audio with an official soundtrack CD, matching volume, and using "Invert" on the CD. If it matches up exactly prior to inversion, this will literally cancel out the signal of the music, leaving the dialog and sound effects in tact. Conceptually it's cool, it's like the music signal equals 1 and then other sounds equal some value like x and matching and inverting it is like adding a negative 1. 1+x-1=x right?

Getting it to match is the near impossible part. Usually movie soundtracks volume slides all over the place, it's called audio ducking, music volume drops when other sounds play and bounces back up between them. So it's hard, I've never done it totally successfully. So I would lean toward recommending the AI or (I don't know what you're working with) finding a different source that's 5.1, or consider replacing audio 100% including dubbing voices.

 

[Agent Sam Stanley]

The answer is probably no, unfortunately. Try the AI separator though, even if the result sounds damaged it might be salvageable with music and sound effects layered in.



...

The other thing to try is lining up the audio with an official soundtrack CD, matching volume, and using "Invert" on the CD. If it matches up exactly prior to inversion, this will literally cancel out the signal of the music, leaving the dialog and sound effects in tact. Conceptually it's cool, it's like the music signal equals 1 and then other sounds equal some value like x and matching and inverting it is like adding a negative 1. 1+x-1=x right?

Getting it to match is the near impossible part. Usually movie soundtracks volume slides all over the place, it's called audio ducking, music volume drops when other sounds play and bounces back up between them. So it's hard, I've never done it totally successfully. So I would lean toward recommending the AI or (I don't know what you're working with) finding a different source that's 5.1, or consider replacing audio 100% including dubbing voices.

I tried the vocalremover.org link and IDK, depending on the scene I'll be working on it could work, but I don't think I'll be able to pull what I'd imagined for the edit.
I also have Adobe Audition but I'm not very familiar with it. I've been checking some Youtube tutorials but they don't work for me. This one for instance:

The Center Channel Extractor doesn't separate dialogue and background music in the files I tried, it just makes everything silent.
As for finding a 5.1 source, I wish I could, but it's a pretty old film and not a very popular one to earn a 5.1 mix.

 

[addiesin]

Some have had some success using the noise removal tools of either audition or audacity, using sections of the score as the noise profile, I think that's probably worth a shot, but again the results will be much better if the music is low to begin with.

 

[lantern51]

I've used different sites for complete removal to re-score scenes. There are some sites that do this better than others. Most are paid sites at that. The ones I've used are:

Vocal Remover & Instrumental AI Splitter | LALAL.AI

Split vocal and instrumental tracks quickly and accurately with LALAL.AI. Upload any audio file and receive high-quality extracted tracks in a few seconds.

www.lalal.ai

It has a free preview that is downloadable. So if it's a short scene you can get some high quality vocals here.

DeMIXER.com | Log into your account

Log into your DeMIXER account or register an account to make online vocal and instrumental removals of your music tracks

www.demixer.com

The first one is free then you can buy an annual package that if you are doing a lot of editing that requires this, it may be worth it to you.

VOXISO - Online AI-Powered Vocal and Music Remover

VOXISO is an online tool that allows you to instantly separate vocals, music or instruments from any track. Try it now for FREE.

voxiso.com

This is a subscription service of $5.00 USD per month.

All of these will also let you keep the audio track as as separate file so if you just wanted to lower the music level you can. I think Demixer is the best at preserving the music out of the 3. The other sites I have looked at that are 100% free the quality is just not there for me personally.

 

[addiesin]

For a completely free, use as much as you want, on your own computer option, try this. https://makenweb.com/SpleeterGUI

I believe it outputs wavs too, not just mp3s.

 

[Agent Sam Stanley]

I've used different sites for complete removal to re-score scenes. There are some sites that do this better than others. Most are paid sites at that. The ones I've used are:

Vocal Remover & Instrumental AI Splitter | LALAL.AI

Split vocal and instrumental tracks quickly and accurately with LALAL.AI. Upload any audio file and receive high-quality extracted tracks in a few seconds.

www.lalal.ai

It has a free preview that is downloadable. So if it's a short scene you can get some high quality vocals here.

DeMIXER.com | Log into your account

Log into your DeMIXER account or register an account to make online vocal and instrumental removals of your music tracks

www.demixer.com

The first one is free then you can buy an annual package that if you are doing a lot of editing that requires this, it may be worth it to you.

VOXISO - Online AI-Powered Vocal and Music Remover

VOXISO is an online tool that allows you to instantly separate vocals, music or instruments from any track. Try it now for FREE.

voxiso.com

This is a subscription service of $5.00 USD per month.

All of these will also let you keep the audio track as as separate file so if you just wanted to lower the music level you can. I think Demixer is the best at preserving the music out of the 3. The other sites I have looked at that are 100% free the quality is just not there for me personally.

Thanks for the tips lantern51. I uploaded a sample to LALAL.AI and it did the trick. I think it might work for what I'm planning here. What's the limit for the free preview, do you know?

 

[Agent Sam Stanley]

For a completely free, use as much as you want, on your own computer option, try this. https://makenweb.com/SpleeterGUI

I believe it outputs wavs too, not just mp3s.

Thanks, downloading right now!

 

[Last Impressions]

For a completely free, use as much as you want, on your own computer option, try this. https://makenweb.com/SpleeterGUI

I believe it outputs wavs too, not just mp3s.

i use spleeter too...very good and easy to use

 

[lantern51]

Thanks for the tips lantern51. I uploaded a sample to LALAL.AI and it did the trick. I think it might work for what I'm planning here. What's the limit for the free preview, do you know?

10 Min or 50mb. If you scroll down it gives you the different plan options.

 

[JCDragonBall]

With a lot of AI-driven editing software out there, I was wondering what some of the best tools are for audio isolation, particularly for isolating dialogue from background music in a video file. Is this doable or still in its infancy technologically?

 

[The Scribbling Man]

You're creating a lot of threads. Try to check there aren't already existing relevant threads you can add to before creating a new one - there's definitely a few covering this topic already, and we even have one dedicated to "random thoughts".

Edit: in fact, threads merged.

 

[addiesin]

With a lot of AI-driven editing software out there, I was wondering what some of the best tools are for audio isolation, particularly for isolating dialogue from background music in a video file. Is this doable or still in its infancy technologically?

Try Spleeter, linked above.