Jump to content

Using AI to manipulate film score audio (dialogue removal, stem separation, etc)


Jay

Recommended Posts

13 minutes ago, Andy said:

Which model do you all like for all purpose separation of score from an old movie?

Bandit Plus, or MVsep DNR

 

Demucs4HT DNR is good too, but only outputs mono, so only use that with mono sources.

Link to comment
Share on other sites

Would the AI work for this video?

 

https://www.youtube.com/watch?v=RS8A99ISvi4

 

is the only music piece not in the game files. And the composer only has the multi track tapes with no means of transfering them 😞

 

 

 

 

Link to comment
Share on other sites

On 06/02/2024 at 1:20 PM, Faleel said:

Audacity has Stem Seperation plugins now:

 

https://www.audacityteam.org/blog/openvino-ai-effects/

I've installed the plugins into a 2023 build of Audactiy.

I'm using a 4-minute music file that is 16-bit/44.1khz stereo file, but when I select 'OpenVINO Music Separation' , 'the 'Preview' button is grayed-out. 

 

I tried-out another desktop where I have both CPU and GPU Inference devices and the 'Preview' button is still grayed-out.

 

Is there a step that I may have missed?

Link to comment
Share on other sites

Just now, SyncMan said:

I've installed the plugins into a 2023 build of Audactiy.

I'm using a 4-minute music file that is 16-bit/44.1khz stereo file, but when I select 'OpenVINO Music Separation' , 'the 'Preview' button is grayed-out. 

 

I tried-out another desktop where I have both CPU and GPU Inference devices and the 'Preview' button is still grayed-out.

 

Is there a step that I may have missed?

the preview button isn't really necessary since there are no real settings anyway. Just click okay and wait for it to finish

Link to comment
Share on other sites

6 minutes ago, enderdrag64 said:

the preview button isn't really necessary since there are no real settings anyway. Just click okay and wait for it to finish

I've done that and it gave me this error message.

error.png

This is the 'Show Log'

log.png

Link to comment
Share on other sites

1 hour ago, SyncMan said:

I've done that and it gave me this error message.

error.png

This is the 'Show Log'

log.png

Looks like you didn't install the openvino-models correctly

Link to comment
Share on other sites

9 hours ago, Luke Skywalker said:

Would the AI work for this video?

 

X-Minus can largely lose the vocals that get in the way but less luck with all the sfx. I'd doubt you're going to be able to get a properly listenable version sadly.

Link to comment
Share on other sites

46 minutes ago, Luke Skywalker said:

Shame 😔

Do you have the original audio file of the video MVsep's Bandit Plus model might have an effect.

Link to comment
Share on other sites

On 18/02/2024 at 8:18 AM, Luke Skywalker said:

Would the AI work for this video?

 

https://www.youtube.com/watch?v=RS8A99ISvi4

 

is the only music piece not in the game files. And the composer only has the multi track tapes with no means of transfering them 😞

 

 

 

 

 

 

Link to comment
Share on other sites

Thanks faleel. Too many sfx in the video unfortunately, but i may be able to do something as the last half sounds tracked so it could be reconstructed.

Link to comment
Share on other sites

  • 2 weeks later...

Pro-tip:

 

If it seems the AI model isn't really having an effect, try using normalize on the input file in Audacity, it seems to help for some models.

Link to comment
Share on other sites

  • 2 weeks later...

Has anyone tried using AI separation to get clean the remaining music from AOTC and ROTS that we need?

Thanks for your suggestion Faleel. I've started using MVSEP and I think it's cool and definitely worth it.

Link to comment
Share on other sites

On 21/03/2024 at 8:10 PM, Manakin Skywalker said:

I've used AI very sparingly on my YouTube edits. However I prefer not to use it for the most part because it adds a shit load of distortion, and in some cases removes instruments. It's important not to overdo it.

 

Yes, for music you intend to listen to, it's best for either removal of very simple elements or to make an already polluted soundscape a bit more bearable, but the more it's got to do, the more distorted the result will be. I've come across a few instances where a speciality release presents a cue with its intended mix, and these models can almost entirely remove a synth or solo element that the film removed, but those are solo or frequency distinct elements.

 

It doesn't seem to me to be at that point (yet) where a polluted rip with effects and dialogue can be magically rescued to just isolated music.

Link to comment
Share on other sites

Anyone got a tutorial on how to remove the score from the movie's dialogues and sound effects using these AI tools? It's something I'd like to try one of these days but I never did this before so I'd like to now the steps.

Link to comment
Share on other sites

There's one step. You go to the page in my main post, and drag an audio file into it.  It spits out two tracks, a dialogue-only track and an everything-but-the-dialogue track.

 

These tools remove dialogue, they don't remove sound effects, sadly.

Link to comment
Share on other sites

But some tools that remove percussion can fortunately mistake some sound effects for percussion.

Link to comment
Share on other sites

Thanks. I've been meaning to try to do some blockbuster movies whose score I wanted to hear in complete form but their sessions never leaked. But since these are huge blockbusters with lots of explosions and stuff it would probably yield some disappointing results. But who knows, I may try it one of these days.

Link to comment
Share on other sites

1 hour ago, Edmilson said:

Anyone got a tutorial on how to remove the score from the movie's dialogues and sound effects using these AI tools? It's something I'd like to try one of these days but I never did this before so I'd like to now the steps.

MVSEP has a model, "BandIt" that puts out three tracks:

 

Dialogue: at best, it's usually pretty clean.

 

Music: Sometimes can get rid of 50-70% of the sfx, sometimes only 20-50%, depends, increasing the volume, and reducing the ducking can help.

 

Effects: usually clean, sometimes some music can bleed through (string sustains, etc.)

 

There is still some artifacts of course.

Link to comment
Share on other sites

  • 2 weeks later...
1 minute ago, Giftheck said:

NGL, I have not had much luck with Bandit on MVSEP.

It does seem to work less well, with lossy formats (mp3) and low volumes.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

By using this site, you agree to our Guidelines.