Using AI to manipulate film score audio (dialogue removal, instrument isolation, etc)

[Andy 6,427]

Which model do you all like for all purpose separation of score from an old movie?

[The Score Cleaner 8,452]

13 minutes ago, Andy said:

Which model do you all like for all purpose separation of score from an old movie?

Bandit Plus, or MVsep DNR

 

Demucs4HT DNR is good too, but only outputs mono, so only use that with mono sources.

[Luke Skywalker 2,262]

Would the AI work for this video?

 

https://www.youtube.com/watch?v=RS8A99ISvi4

 

is the only music piece not in the game files. And the composer only has the multi track tapes with no means of transfering them 😞

 

 

 

 

[SyncMan 388]

On 06/02/2024 at 1:20 PM, Faleel said:

Audacity has Stem Seperation plugins now:

 

https://www.audacityteam.org/blog/openvino-ai-effects/

I've installed the plugins into a 2023 build of Audactiy.

I'm using a 4-minute music file that is 16-bit/44.1khz stereo file, but when I select 'OpenVINO Music Separation' , 'the 'Preview' button is grayed-out. 

 

I tried-out another desktop where I have both CPU and GPU Inference devices and the 'Preview' button is still grayed-out.

 

Is there a step that I may have missed?

[enderdrag64 1,604]

Just now, SyncMan said:

I've installed the plugins into a 2023 build of Audactiy.

I'm using a 4-minute music file that is 16-bit/44.1khz stereo file, but when I select 'OpenVINO Music Separation' , 'the 'Preview' button is grayed-out. 

 

I tried-out another desktop where I have both CPU and GPU Inference devices and the 'Preview' button is still grayed-out.

 

Is there a step that I may have missed?

the preview button isn't really necessary since there are no real settings anyway. Just click okay and wait for it to finish

[SyncMan 388]

6 minutes ago, enderdrag64 said:

the preview button isn't really necessary since there are no real settings anyway. Just click okay and wait for it to finish

I've done that and it gave me this error message.

This is the 'Show Log'

[enderdrag64 1,604]

1 hour ago, SyncMan said:

I've done that and it gave me this error message.

This is the 'Show Log'

Looks like you didn't install the openvino-models correctly

[Richard P 4,946]

9 hours ago, Luke Skywalker said:

Would the AI work for this video?

 

X-Minus can largely lose the vocals that get in the way but less luck with all the sfx. I'd doubt you're going to be able to get a properly listenable version sadly.

[Luke Skywalker 2,262]

Shame 😔

[The Score Cleaner 8,452]

46 minutes ago, Luke Skywalker said:

Shame 😔

Do you have the original audio file of the video MVsep's Bandit Plus model might have an effect.

[The Score Cleaner 8,452]

On 18/02/2024 at 8:18 AM, Luke Skywalker said:

Would the AI work for this video?

 

https://www.youtube.com/watch?v=RS8A99ISvi4

 

is the only music piece not in the game files. And the composer only has the multi track tapes with no means of transfering them 😞

 

 

 

 

 

 

[Luke Skywalker 2,262]

Thanks faleel. Too many sfx in the video unfortunately, but i may be able to do something as the last half sounds tracked so it could be reconstructed.

[The Score Cleaner 8,452]

Pro-tip:

 

If it seems the AI model isn't really having an effect, try using normalize on the input file in Audacity, it seems to help for some models.

[Trent B 354]

Has anyone tried using AI separation to get clean the remaining music from AOTC and ROTS that we need?

Thanks for your suggestion Faleel. I've started using MVSEP and I think it's cool and definitely worth it.

[Manakin Skywalker 5,729]

I've used AI very sparingly on my YouTube edits. However I prefer not to use it for the most part because it adds a shit load of distortion, and in some cases removes instruments. It's important not to overdo it.

[Groovygoth666 1,163]

Used https://www.lalal.ai/ on that Transformers The Movie cue I uploaded on the first page to remove the electric guitar an got this which sounds a lot better -

 

 

[Richard P 4,946]

On 21/03/2024 at 8:10 PM, Manakin Skywalker said:

I've used AI very sparingly on my YouTube edits. However I prefer not to use it for the most part because it adds a shit load of distortion, and in some cases removes instruments. It's important not to overdo it.

 

Yes, for music you intend to listen to, it's best for either removal of very simple elements or to make an already polluted soundscape a bit more bearable, but the more it's got to do, the more distorted the result will be. I've come across a few instances where a speciality release presents a cue with its intended mix, and these models can almost entirely remove a synth or solo element that the film removed, but those are solo or frequency distinct elements.

 

It doesn't seem to me to be at that point (yet) where a polluted rip with effects and dialogue can be magically rescued to just isolated music.

[Edmilson 10,993]

Anyone got a tutorial on how to remove the score from the movie's dialogues and sound effects using these AI tools? It's something I'd like to try one of these days but I never did this before so I'd like to now the steps.

[Jay 43,896]

There's one step. You go to the page in my main post, and drag an audio file into it.  It spits out two tracks, a dialogue-only track and an everything-but-the-dialogue track.

 

These tools remove dialogue, they don't remove sound effects, sadly.

[Holko 11,673]

But some tools that remove percussion can fortunately mistake some sound effects for percussion.

[Richard P 4,946]

It's worth trying for just about anything - you just don't get your hopes up if it's anything more complex than dialogue, percussion and high/low frequencies.

[Edmilson 10,993]

Thanks. I've been meaning to try to do some blockbuster movies whose score I wanted to hear in complete form but their sessions never leaked. But since these are huge blockbusters with lots of explosions and stuff it would probably yield some disappointing results. But who knows, I may try it one of these days.

[The Score Cleaner 8,452]

1 hour ago, Edmilson said:

Anyone got a tutorial on how to remove the score from the movie's dialogues and sound effects using these AI tools? It's something I'd like to try one of these days but I never did this before so I'd like to now the steps.

MVSEP has a model, "BandIt" that puts out three tracks:

 

Dialogue: at best, it's usually pretty clean.

 

Music: Sometimes can get rid of 50-70% of the sfx, sometimes only 20-50%, depends, increasing the volume, and reducing the ducking can help.

 

Effects: usually clean, sometimes some music can bleed through (string sustains, etc.)

 

There is still some artifacts of course.

[Edmilson 10,993]

Interesting. I'll take a look at it. Thanks!

[The Score Cleaner 8,452]

Some tests with unreleased Hobbit music:

 

 

 

[Edmilson 10,993]

Seems nice. Is it paid? And if so, how much does it cost?

[The Score Cleaner 8,452]

44 minutes ago, Edmilson said:

Seems nice. Is it paid? And if so, how much does it cost?

Free.

[Giftheck 1,141]

NGL, I have not had much luck with Bandit on MVSEP.

[The Score Cleaner 8,452]

1 minute ago, Giftheck said:

NGL, I have not had much luck with Bandit on MVSEP.

It does seem to work less well, with lossy formats (mp3) and low volumes.

[4te 26]

I've been using the MDX23C  separation from MVSEP and I'm really impressed with the results!! Even with very low quality input files (most of the audio inputs that I've used have been mono) and it does an excellent job of removing vocals. Purchasing the premium credits is worth it

[The Score Cleaner 8,452]

3 hours ago, 4te said:

I've been using the MDX23C  separation from MVSEP and I'm really impressed with the results!! Even with very low quality input files (most of the audio inputs that I've used have been mono) and it does an excellent job of removing vocals. Purchasing the premium credits is worth it

BSRoformer (newest version) is pretty good too.

[Gabriel Bezerra 666]

I've been playing around with MVSEP for a while, utilizing the 5.1 mix without channel 3 to see if the remaining effects could be removed without much loss, it worked mostly ok, I wasn't pushing it. So I decided to try a scene from an episode of The Flash (2014), It is a very SFX-heavy show and even the dialogue gets in the other channels but the rear channels were surprisingly clean, only 2 channels remained in need of cleaning.

 

I used Demucs4 HT first, and the "instrumental" option hadn't removed any SFX, the "other" on the other hand, was 70% clean, so I went to Bandit Plus to get a real SFX clean version, and it was full of gaps and oscillations and the dialogue was still there, but no SFX. So I tried putting those two alongside the rear channels and this is the result:

Wally And Barry Race.flac

 

With the scene:

Spoiler

 

 

 

[The Score Cleaner 8,452]

21 minutes ago, Gabriel Bezerra said:

I've been playing around with MVSEP for a while, utilizing the 5.1 mix without channel 3 to see if the remaining effects could be removed without much loss, it worked mostly ok, I wasn't pushing it. So I decided to try a scene from an episode of The Flash (2014), It is a very SFX-heavy show and even the dialogue gets in the other channels but the rear channels were surprisingly clean, only 2 channels remained in need of cleaning.

 

I used Demucs4 HT first, and the "instrumental" option hadn't removed any SFX, the "other" on the other hand, was 70% clean, so I went to Bandit Plus to get a real SFX clean version, and it was full of gaps and oscillations and the dialogue was still there, but no SFX. So I tried putting those two alongside the rear channels and this is the result:

Wally And Barry Race.flac 13.02 MB · 0 downloads

 

With the scene:

  Hide contents

 

 

Did you raise the volume/envelope compress the audio? sometimes if the audio is too quiet it makes it hard to seperate.

[ThePenitentMan1 1,381]

Return Of The Jedi End Title, clean opening (via lalal.ai v4 Orion vocal removal):

 

Raw (from Demaster):

 

Edited (into Manakin's version):

 

No ESB takes here!

[Jay 43,896]

Woah.  Now that's cool!

[Cameron007 101]

Can OpenVino separate cues? If so, what preset would I want to use?

[Marcus Stöhr 42]

I came across this service which let's you scan sheet music. They use AI for the scanning process - whatever that means in detail. I haven't tested it but maybe someone can give it a shot: https://www.soundslice.com/sheet-music-scanner/

[Holko 11,673]

Somebody try what the hell it can make of Beethoven's sketches!

[The Score Cleaner 8,452]

Thought I would mention that MVsep has new and updated models:

 

MVSep Guitar (guitar, other)

Now there are two versions available: the updated version of MDX23C and MelRoformer

 

MVSep Multichannel BS (vocals, instrumental)”. This model is specially prepared for extracting vocals from multi-channel audio (5.1, 7.1, etc.). After processing, it returns multi-channel audio in the same format in which it was sent to the server with the same sample rate. 

 

MVSep Multispeaker

[ThePenitentMan1 1,381]

^Anybody remember this thread about the choir toward the end of The Map Room: Dawn?

 

 

[The Score Cleaner 8,452]

@Andy Check this out!

 

   

 

 

[Andy 6,427]

I love it!  Now we can have the percussion only tracks we always wanted!

 

EDIT - So the last two are different processors?  I have to give them a closer listen.  Do you have a preference @The Great Gonzales?

[Andy 6,427]

Hmm interesting.

 

So the Melrof one is a little more aggressive in removing things like piccolo shrieks, but it leaves behind a bit of a muffled sound remaining.  The Mvsep has a more consistent air to it, but you can hear a few of the instruments sneak through.

[The Score Cleaner 8,452]

14 hours ago, Andy said:

I love it!  Now we can have the percussion only tracks we always wanted!

 

EDIT - So the last two are different processors?  I have to give them a closer listen.  Do you have a preference @The Great Gonzales?

If I understand it correctly, MVsep Drums model is basically taking two different models and "combining forces", melroformer was one of the two (htdemucs being the other)

[Cameron007 101]

@The Great Gonzales Do you have a recommendation on a tool for separating cues?

[The Score Cleaner 8,452]

Woohoo!!!

 

Quote

3) We have added new models Bandit v2 for Cinematic source separation. The models divide the track into 3 components “music”, “speech” and “effects/sfx”. The model was trained on the new multilingual dataset Divide and Remaster (DnR) v3.

 

 

[Jay 43,896]

WOAH!

[The Score Cleaner 8,452]

25 minutes ago, Jay said:

WOAH!

Unfortunately after some tests, it doesn't really seem to work well for the music part, but in combination with BSRoformer vocal model, it does work well for isolating SFX and dialogue for rescores etc.

[Andy 6,427]

That’s awesome. It’ll get to where we want it eventually. 

[The Score Cleaner 8,452]

MVsep now has a noise removal model:

 

Standard:  

 

 

Aggressive: