Using AI to manipulate film score audio (dialogue removal, stem separation, etc)

[Faleel 5,353]

Audacity has Stem Seperation plugins now:

 

https://www.audacityteam.org/blog/openvino-ai-effects/

[Jay 37,369]

Neat!

[enderdrag64 624]

2 hours ago, Faleel said:

Audacity has Stem Seperation plugins now:

 

https://www.audacityteam.org/blog/openvino-ai-effects/

Woah I definitely have to play around with these when I get home

[fenrik 36]

19 hours ago, enderdrag64 said:

Woah I definitely have to play around with these when I get home

 

I was super excited but then...Windows only... I have a Mac, lol!

[ThePenitentMan1 744]

I was previously content with removing the vocals from the film overlay and not the sound effects...  but it had never occurred to me until yesterday that I can actually isolate the electric guitar with A.I.!

 

https://www.lalal.ai/

 

Before:

 

 

After:

 

 

 

I'm kicking myself for not thinking of this in the first place!

[Andy 4,138]

So can we mimic the percussion only tracks from The Lost World?

 

Can it remove choir?  I would love to hear The Omen with orchestra only. For science. 

[Faleel 5,353]

4 minutes ago, Andy said:

Can it remove choir?  I would love to hear The Omen with orchestra only. For science. 

IIRC @Giftheck was able to do that with Duel of the Fates....

[Giftheck 916]

4 hours ago, Faleel said:

IIRC @Giftheck was able to do that with Duel of the Fates....

 

Yes, indeed it can, though the resulting instrumental isn't perfect by any means. But I expect it to jusst get better at it over time.

[Andy 4,138]

It may seem sacrilegious () to want to hear The Omen sans choir, but the orchestration is so good. 

[Faleel 5,353]

Here is a test isolating choir from a cue with a low choral mix, Gilraen's Memorial from FOTR:

 

Audacity models:

 

 

Demucs4HT model:

 

 

MDX23C Model:

 

 

 

 

@Andy @Jay

[Andy 4,138]

Not bad!!!

[Faleel 5,353]

On 09/02/2024 at 10:25 PM, Andy said:

So can we mimic the percussion only tracks from The Lost World?

 

Rescuing Sarah (2016 boxset version):

 

 

 

 

 

[jacksparrow900 32]

 

 

lalal.ai 

Works great I did the version of Sooner Or Later from Dick Tracy where I was able to isolate Madonna's Vocal from the center channel leaving all other sfx out and isolating the piano only and mixing them together. I've done a lot more some successful some still needs work. It works best to try and separate the instrument sections. Just doing instrumental carries over sfx. Instrumental works best for vocal songs. 

[Trope 527]

4 hours ago, Faleel said:

Rescuing Sarah (2016 boxset version):

 

 

 

 

This is awesome! And it's also cool to hear those percussion-heavy tracks with percussion removed and just the orchestra. I tried it myself on the website!

[Thor 7,508]

I tried to test both of the sites suggested so far, for one of my old JW TV clips that had a fair amount of sound effects, but I'm afraid neither worked for that purpose. Is there a site like that that can remove sound effects from instrumental tracks? If you know of any, and would like to give it a go, here's a test cue:

 

 

[jacksparrow900 32]

I was even able to take the unreleased Timpani from Last Crusade which was was buried under shouting voices and grab just the percussion 

 

 

[Andy 4,138]

12 hours ago, Faleel said:

Rescuing Sarah (2016 boxset version):

 

 

 

 

Which model did you use for this, @Faleel?

 

I have to attack The Omen.  I must learn this power. 

[Bespin 8,483]

I would love to have an AI that woud recut a movie according to my desires: hey, GPT movie, remove this scene and this scene.

[Jay 37,369]

13 hours ago, Faleel said:

Rescuing Sarah (2016 boxset version):

 

 

Dude, that's incredible!  I never would have suspected this was possible.  We're truly living in the future

[Faleel 5,353]

2 hours ago, Jay said:

 

Dude, that's incredible!  I never would have suspected this was possible.  We're truly living in the future

 

3 hours ago, Andy said:

Which model did you use for this, @Faleel?

 

I have to attack The Omen.  I must learn this power. 

"Demucs4HT" with "High Quality (Slow)" setting on MVSEP.org

[GerateWohl 4,370]

4 hours ago, Bespin said:

I would love to have an AI that woud recut a movie according to my desires: hey, GPT movie, remove this scene and this scene.

GPT movie, turn Kramer vs. Kramer into an R-rated splatter film.

[Faleel 5,353]

On 13/02/2024 at 12:54 PM, GerateWohl said:

GPT movie, turn Kramer vs. Kramer into an R-rated splatter film.

GPT movie, turn the Star Wars sequels into excellent movies

 

"I'm sorry Dave, But I am afraid I can't do that"

 

LALAL has Beta models for Winds and Strings, so I tested it on the original mono recording of Quo Vadis, and while the result wasn't that great, I was able still able to create a sort of rough stereo mix!

 

 

 

[Faleel 5,353]

Just found another use for AI:

 

Clean endings/openings from combined cues on releases!

 

I just used AI to seperate the vocal sea shanty source cues from the additional composer rewrite cues from the FSM release of Rozsa's All the Brothers Were Valiant!

 

I suppose it would also be useful in isolating music in those awful OST's that include dialogue

[Faleel 5,353]

 

Ever wanted to hear Battle of the Heroes without choir?

 

 

 

@Giftheck @Andy @King Mark 

[enderdrag64 624]

10 hours ago, Faleel said:

Just found another use for AI:

 

Clean endings/openings from combined cues on releases!

 

I just used AI to seperate the vocal sea shanty source cues from the additional composer rewrite cues from the FSM release of Rozsa's All the Brothers Were Valiant!

 

I suppose it would also be useful in isolating music in those awful OST's that include dialogue

Yeah it's really awesome for this

 

I've been messing around with that OpenVINO Audacity AI this past week, managed to get a clean opening/some unreleased parts for a few ESB cues out of the radio drama...

 

In particular I got the ending of 9m6-10m1 Insert Bar 57 completely clean

 

And I managed to get the 14 missing seconds of 9m6-10m1 Carbon Freeze mostly clean as well, this doesn't sound quite as good but for how terrible the source was it's pretty impressive 

[Faleel 5,353]

6 minutes ago, enderdrag64 said:

Yeah it's really awesome for this

 

I've been messing around with that OpenVINO Audacity AI this past week, managed to get a clean opening/some unreleased parts for a few ESB cues out of the radio drama...

 

In particular I got the ending of 9m6-10m1 Insert Bar 57 completely clean

 

 

Is that the sustained strings that was used in the theatrical for the "You Will Be... You Will Be" scene?

[enderdrag64 624]

12 minutes ago, Faleel said:

Is that the sustained strings that was used in the theatrical for the "You Will Be... You Will Be" scene?

I don't think so no. I'm not certain what that is off the top of my head though.

 

Here's the cleaned up ending of the insert from the radio drama:

   

 

And actually here's the unreleased 14 seconds of the main cue as well, though like I said this has some artifacts and some sfx remaining, actually I'll include a before and after in case anyone wants to try a better one:

 

 

 

 

 

[Groovygoth666 670]

1 hour ago, Faleel said:

Is that the sustained strings that was used in the theatrical for the "You Will Be... You Will Be" scene?

I think that is tracked from the ending of the Imperial Probe Droid New Start insert, which is on the RCA/Sony release but it plays over the beginning of Insert Bar 80.

[Jay 37,369]

8 hours ago, Faleel said:

Ever wanted to hear Battle of the Heroes without choir?

 

Amazing!

[Bespin 8,483]

That's horrible.

[A. A. Ron 1,742]

Could AI be used to remove the last 9 seconds of percussion from the cue “Warpaint” that overlaps the beginning of “The Kitchen” on the 3CD Intrada edition of Consn the Barbarian? I’ve tried to removed it in Audition by using a sample from an earlier part of the drum loop, but the results don’t sound great.

[Richard Penna 3,694]

x-minus pro could do this:

 

Original:

 

Instrumental:

 

 

[Jay 37,369]

Cool!

 

Time to update the thread title since the scope has expanded

[Richard Penna 3,694]

It's really good when something relatively simple gets in the way. This is from Wild Wild West and I'm fairly sure this bit in the film doesn't have the twangy guitar.

 

Original:

No guitar:

   

 

 

I was able to get something similar (not quite as refined) from Body Double from TWINE where the percussion/synths are dropped entirely so this works to an extent where you have an orchestral and electronic/percussion element and you don't want the latter. Particularly useful where an expansion has the intended cue with layers you might not want.

 

This works well too - The Hanging Tree: (doesn't really start until about 20s)

   

 

Or, a moment from the terminal where Williams annoys me with a jazzy interruption:

 

 

 

 

 

 

[Jay 37,369]

huh, when i pressed play i wasn't expecting a couple bass notes to be something you'd describe as a "jazzy interruption"

[Richard Penna 3,694]

I may have overemphasized that  I don't personally find those bass notes fit in with that track.

[Faleel 5,353]

1 hour ago, Richard Penna said:

I may have overemphasized that  I don't personally find those bass notes fit in with that track.

I don't want jazz in my jazz!

[Andy 4,138]

18 hours ago, Faleel said:

 

Ever wanted to hear Battle of the Heroes without choir?

 

 

@Giftheck @Andy @King Mark 

 

So awesome!  When I downloaded it, it auto opened in iTunes, which auto-tagged it as "Departure/ Away for Rio / Full Sail" from  "All the Brothers Were Valiant" by Milkos Rozsa.

[Faleel 5,353]

1 minute ago, Andy said:

 

So awesome!  When I downloaded it, it auto opened in iTunes, which auto-tagged it as "Departure/ Away for Rio / Full Sail" from  "All the Brothers Were Valiant" by Milkos Rozsa.

Yeah, that was the first track I opened in Audacity, and forgot to change it, when I exported this haha.

[Andy 4,138]

Well, I have always wanted to hear the Omen without the choir, and you know what?... (shocker) it's not as good!!!  But it's still a freakin fascinating experiment.  I plan on doing the whole album for fun, including The Piper Dreams.

 

Here's "Mother's Death", a short, but standard example of the Omen without the devilish choir beginning at 00:14.  An attempt to make the Black Mass a little less black and peel away the layers.

 

 

 

 

This is using MVSEP with the mdx b Separation.  I wish I knew more about the different vocal models on the dropdown menu, but I just tried the default one.  If anyone has any suggestions for fine tuning or favorite vocal models, let me know.

[Faleel 5,353]

7 minutes ago, Andy said:

Well, I have always wanted to hear the Omen without the choir, and you know what?... (shocker) it's not as good!!!  But it's still a freakin fascinating experiment.  I plan on doing the whole album for fun, including The Piper Dreams.

 

Here's "Mother's Death", a short, but standard example of the Omen without the devilish choir beginning at 00:14.  An attempt to make the Black Mass a little less black and peel away the layers.

 

 

 

This is using MVSEP with the mdx b Separation.  I wish I knew more about the different vocal models on the dropdown menu, but I just tried the default one.  If anyone has any suggestions for fine tuning or favorite vocal models, let me know.

MDX23C is the best I find. It's also the newest IIRC

[Faleel 5,353]

A little test of the reverb removal model on Jaws's "Preparing the Cage"

 

 

Doesn't sound super great, but still interesting.

 

@Andy @Holko

[Holko 9,526]

Ah, a chamber arrangement!

[Faleel 5,353]

1 minute ago, Holko said:

Ah, a chamber arrangement!

Either that, or the streamer version

[Faleel 5,353]

Testing on an unreleased cue from the BBC Narnia Silver Chair:

 

 

 

[Andy 4,138]

7 hours ago, Faleel said:

A little test of the reverb removal model on Jaws's "Preparing the Cage"

 

Doesn't sound super great, but still interesting.

 

@Andy @Holko

 

Nobody tell Bruce Botnick about this.  He will die a little inside.

 

This is really fun to play around with.

[MrJosh 892]

Wow, this stuff is so cool! I can't wait to play around with these tools more.

[Andy 4,138]

15 hours ago, Faleel said:

MDX23C is the best I find. It's also the newest IIRC

I experienced some ducking and volume dips with this where vocals were removed.  I wish I understood what all the different filters on the drop down menu do. I’m too cheap to pay for this and it takes a while to render each one.  So experimenting with results can be slow. 
 

(my comments are all related to choir removal)

[Faleel 5,353]

1 hour ago, Andy said:

I experienced some ducking and volume dips with this where vocals were removed.  I wish I understood what all the different filters on the drop down menu do.

 

1: I think that is "natural" due to the mix (choir perhaps having been eq'd to remove unneeded frequencies), and basically having frequencies "yanked" out, think of taking a candle out of a birthday cake, and how there is a hole left.

 

I believe the numbers (10.0886 etc.) tell you the quality of Isolation, from my understanding: higher number = better Isolation

 

 

[Andy 4,138]

That makes perfect sense.  Good analogy.