Using AI to manipulate film score audio (dialogue removal, stem separation, etc)

[Jay 40,274]

I know some people here have dabbled with this, and I was wondering if we could get some info shared about the best programs to use and any setting recommendations, etc etc.

 

I just started dabbling in this today and am blown away by the results

 

This is the one I happened to find and use:

 

https://x-minus.pro/ai

 

Here's an example of a before and after

 

Since the free version only outputs as MP3, I'm contemplating paying for a subscription to get high res audio and whatever else you get with paying.  But I figured I'd ask here first if there are any alternate vocal removal websites, or apps, or whatever, that people think are better or just different. 

 

Discuss!

[Holko 10,708]

I tried multiple ones for my score rips and that's the one I ended up on too! The base tier is very cheap but completely useable. You'll probably have to try out all the models to see if one works better for your goals than the others, they modify/improve them and add new ones relatively often. You can try a model which removes percussion too, that can recognise some foley/sound effects as percussion sometimes but of course wouldn't be great in a cue like the one you shared which also has prominent percussion.

[Jay 40,274]

Yea it definitely works better for some pieces I've tried than others.

 

These two also came out great

 

 

 

It is kind of mind blowing how well this technology works.  Having spent over half my lifetime ripping unreleased music from movies and dealing with whatever dialogue and sound effects came with it, its amazing how well you can remove at least the dialogue now

[Meredith McKay 7,062]

MVSEP.com outputs FLAC.

[Thor 8,686]

What the hell are you ripping, Jay? Sounds like a 90s sitcom!

 

In seriousness, this sounds like great software. For me, things like these would be interesting not for getting music that's not on the album, but for ripping scores that have no release whatsoever. People like Sylvester Levay, for example, who's woefully underrepresented on disc. Then again, the video source material for his stuff is rather poor to begin with (VHS rips put up on YouTube and the like), so I'm guessing it would be quite dreadful in the end. Seems more fitting for films that have great, clear audio tracks.

[mstrox 6,982]

That’s Wet Hot American Summer!

 

Great movie!  Are these bits of score unreleased?

 

I use an iOS app called Moises which works pretty well, but I don’t save files.  I mostly use it for the opposite of what you do, Jay - to isolate vocals in songs which have lyrics not published elsewhere and figure out what people are saying (or use the transcription tool to do it for me).  Most of the time it’s pretty accurate, then it’s just a few tweaks and ready to paste into the metadata.  Used it on Low Below, in fact!

[Richard P 4,369]

For those who want to hear a couple of Fellowship bits without cast performances, that site 99.99% removes them. There was just a tiny, barely noticeable bit of Gandalf left in the Bag End one.

[Jay 40,274]

10 hours ago, Thor said:

What the hell are you ripping, Jay? Sounds like a 90s sitcom!

 

Wet Hot American Summer by Theodore Shapiro and Craig Wedren.  It finally got an OST album 15 years after the film came out (probably thanks to the Netflix sequel), but it was incomplete, and it's a score where I like every single cue.

 

If you like the theme played by guitar in my third clip, it's a theme that only exists on the OST album as a vocal performance

 

 

This cue is one of the first in the movie, and I love that they call back to it in a guitar arrangement in this unreleased cue from near the end of the movie!

 

It sounds like a melody that could have been intended for a song that never got finished (they wrote some original songs for the movie too)

[Thor 8,686]

Interesting. Shapiro is generally underrated and under-released, that's for sure.

[Jay 40,274]

On 19/12/2023 at 12:17 PM, Thor said:

In seriousness, this sounds like great software. For me, things like these would be interesting not for getting music that's not on the album, but for ripping scores that have no release whatsoever.

 

Oh yes!  I plan to try this technology on a completely unreleased score next, but I ran out of free uses for the day

[Jay 40,274]

2 hours ago, mstrox said:

That’s Wet Hot American Summer!

 

Great movie!

 

Brother!

 

2 hours ago, mstrox said:

Are these bits of score unreleased?

Yep!  The first is "Down The Back", which starts as Paul Rudd is flipping off Beth behind his back (lol), and features a melody at the start that only appears once on the OST album (in "Bags Are Packed"), then moves onto the score's main theme, which appears in six film cues, only two of which are on the OST album.  The second is "Grand Canyon Rapids", and the third is explained above

 

2 hours ago, mstrox said:

I use an iOS app called Moises which works pretty well, but I don’t save files.  I mostly use it for the opposite of what you do, Jay - to isolate vocals in songs

 

Well that was another thing I was blown away by actually for each of the clips I uploaded into this AI, it spit back not only an isolated music stem, but also an isolated vocal stem too.  It feels like magic using this thing!

[mstrox 6,982]

17 minutes ago, Jay said:

 

 

Well that was another thing I was blown away by actually for each of the clips I uploaded into this AI, it spit back not only an isolated music stem, but also an isolated vocal stem too.  It feels like magic using this thing!

The app I use actually separates it further - so you can knock out the percussion or goose the bass if you so choose.

 

 

And it shows what notes/chords are being played (seen here above the lyrics) - so you could drop out everything except the guitar or bass and chart the song to learn it.

 

 

Or change the tempo, or the key of the whole song, and then listen or chart that.  Really neat functionality.

[Jay 40,274]

Yes this website can do that with the paid tiers

[Jay 40,274]

One issue I've found is that if the score has choir as part of the music, the AI thinks you want it wiped to, since its vocals.

 

Also it definitely works better the further away in register the music is from the dialogue.  IE, if you have a guy with a deep voice talking as music is playing in a lower register, some of the music goes away with the vocals, but if the music is in a high register, the separation is cleaner

[Davis 3,331]

It’s called OST. 

[Andy 5,413]

So is this how we will get Story of a Woman, the complete Jane Eyre and Gilligan’s Island?

[Jay 40,274]

10 hours ago, Presto said:

MVSEP.com outputs FLAC.

 

So I'm checking this out and there's tons of different options to choose from. Any recommendations?

 

[Meredith McKay 7,062]

7 hours ago, Jay said:

 

So I'm checking this out and there's tons of different options to choose from. Any recommendations?

 

For my purposes, I mostly use:

 

Demucs4 HT (Vocals, drums, bass, other) on the high quality (slow) setting

and MDX23C (Vocals, Instrumental),

sometimes MDX B (Vocals, Instrumental)

 

And I have tried the two that say "(Dialoge, sfx, music)" and "(Music, sfx, speech)" a few times

[1977 1,746]

Not an iOS user, are there any free apps for Windows or MAC?

 

I don't know if uploading copyrighted audio to a web based solution is a good idea, especially as it presumably means some DVD or Blu-ray copy protection was defeated.

[GerateWohl 5,341]

22 hours ago, Thor said:

What the hell are you ripping, Jay? Sounds like a 90s sitcom!

 

In seriousness, this sounds like great software. For me, things like these would be interesting not for getting music that's not on the album, but for ripping scores that have no release whatsoever. People like Sylvester Levay, for example, who's woefully underrepresented on disc. Then again, the video source material for his stuff is rather poor to begin with (VHS rips put up on YouTube and the like), so I'm guessing it would be quite dreadful in the end. Seems more fitting for films that have great, clear audio tracks.

Do you have to rework your John Williams Television Walkthrough with this again?

[Meredith McKay 7,062]

3 hours ago, JTWfan77 said:

Not an iOS user, are there any free apps for Windows or MAC?

 

I don't know if uploading copyrighted audio to a web based solution is a good idea, especially as it presumably means some DVD or Blu-ray copy protection was defeated.

These sites primary purpose it to be used by people using copyrighted material...

[Thor 8,686]

2 hours ago, GerateWohl said:

Do you have to rework your John Williams Television Walkthrough with this again?

 

Oh God, no. Would have been convient to have, and know how to operate, at the time, though.

[1977 1,746]

2 hours ago, Presto said:

These sites primary purpose it to be used by people using copyrighted material...

 

Thanks. I'll rather not take the risk.

[Yavar Moradi 3,129]

1 hour ago, Thor said:

Oh God, no. Would have been convient to have, and know how to operate, at the time, though.

 

Could use it on your upcoming Checkmate walkthrough!

 

Yavar

[Cameron007 77]

This sounds like it could be groundbreaking. Now, what about using AI to separate cues? 🤔

[Jay 40,274]

That could already be possible with some of these tools, if the instrument(s) playing at the end of cue 1 are different from the one(s) playing at the start of cue 2

[Andy 5,413]

What I am curious about is how and if the professionals who work for our labels could begin making their work even better. 

[Jay 40,274]

The specialty label releases are sourced from internal studio elements, not rips of the movies.  They wouldn't really need to use this tech for a score release.

[Andy 5,413]

True, but I can imagine we might be rebuying older recordings again in which AI is used to make it sound like it was recorded yesterday. 

[Jay 40,274]

How would AI do that?

[enderdrag64 991]

34 minutes ago, Jay said:

The specialty label releases are sourced from internal studio elements, not rips of the movies.  They wouldn't really need to use this tech for a score release.

Could be useful for scores where the original elements are lost

[Andy 5,413]

51 minutes ago, Jay said:

How would AI do that?

I know nothing of how it works but I can imagine it enhancing the sound of a recording by filling in the deficient bits with sampled sounds.  The way you use the clone stamp tool in photoshop, but with audio.  I bet it’ll be able to achieve miraculous restoration that was previously too labor intensive to do manually. 

[Meredith McKay 7,062]

You could also potentially use it to lower artificial reverb that was baked in.

[Andy 5,413]

I mean, none of us really knows, but I think if it’s that easy for AI to remove sound effects, we are just seeing the tip of the iceberg.  I imagine it’ll be able to refresh old recordings from the pre Hi Fi era, clarify vocals on classic songs, and yeah add polish to existing recordings that are too dry, too wet, too old, or just badly recorded. 

[Meredith McKay 7,062]

2 hours ago, Andy said:

I mean, none of us really knows, but I think if it’s that easy for AI to remove sound effects, we are just seeing the tip of the iceberg.  I imagine it’ll be able to refresh old recordings from the pre Hi Fi era, clarify vocals on classic songs, and yeah add polish to existing recordings that are too dry, too wet, too old, or just badly recorded. 

IIRC, something AI was used for some L2P concerts, to get the dialogue etc. seperate from the music.

[Manakin Skywalker 5,507]

45 minutes ago, Presto said:

IIRC, something AI was used for some L2P concerts, to get the dialogue etc. seperate from the music.

 

Not any that I've heard of. All the Star Wars ones for example use the raw stems, any bleedthrough is a tape-related issue and not the result of AI artifacting.

[Holko 10,708]

8 hours ago, Andy said:

I know nothing of how it works but I can imagine it enhancing the sound of a recording by filling in the deficient bits with sampled sounds.  The way you use the clone stamp tool in photoshop, but with audio.  I bet it’ll be able to achieve miraculous restoration that was previously too labor intensive to do manually. 

You could train a model on JW's LSO recordings and then feed it the ST to give them the same sound, keeping strictly close to the recording so JW's instructions and performance choices remain in there? And then proceed to pay the LSO members for this usage and JW for his "re-conducting". Nah this just sounds wrong, I prefer the film stem cleanup usage for lost masters like Jane Eyre or Fiddler, which already used some voice removal for the disc 3 film cues.

[Meredith McKay 7,062]

3 hours ago, Manakin Skywalker said:

 

Not any that I've heard of. All the Star Wars ones for example use the raw stems, any bleedthrough is a tape-related issue and not the result of AI artifacting.

I was referring to really old stuff like Wizard of Oz.

59 minutes ago, Holko said:

You could train a model on JW's LSO recordings and then feed it the ST to give them the same sound, keeping strictly close to the recording so JW's instructions and performance choices remain in there? And then proceed to pay the LSO members for this usage and JW for his "re-conducting". Nah this just sounds wrong, I prefer the film stem cleanup usage for lost masters like Jane Eyre or Fiddler, which already used some voice removal for the disc 3 film cues.

Abbey Road/Watford Colloseum Hobbitses.....

 

(Also theoretically, that's not too far off what an impulse response does....)

[Manakin Skywalker 5,507]

12 hours ago, Presto said:

I was referring to really old stuff like Wizard of Oz.

 

Huh, interesting.

[4te 24]

19 hours ago, Presto said:

I was referring to really old stuff like Wizard of Oz.

 

On that note, I'm curious if anyone has suggestions for AI software which can restore/enhance older recordings (from the 1930's and 1940's)?

[Jay 40,274]

Working on a completely unreleased sci-fi score today

 

 

 

[Groovygoth666 1,057]

I've used vocalremover.org for some things, it only lets you save a couple of files a day for free, but you can download them as wav.

 

 

One example is the opening of Hook, I tried the English version but it didn't turn out so good, the Italian dub worked much better though -

 

Original Film Audio -  

 

And the result -  

 

 

I also used it for a track from Transformers The Movie (1986) to remove the electric guitar that had been dialed out in the film -

 

Original -  

 

  

Without the guitar -  

 

  

 

 

 

[Andy 5,413]

Your Transformers example @Groovygoth666reminded me there’s a lot that could be done with GI Joe, Superfriends,… the cartoon possibilities are pretty endless 

[Meredith McKay 7,062]

8 minutes ago, Andy said:

Your Transformers example @Groovygoth666reminded me there’s a lot that could be done with GI Joe, Superfriends,… the cartoon possibilities are pretty endless 

I'm gonna try it on the Rankin/Bass ROTK at some point.

[Groovygoth666 1,057]

11 hours ago, Andy said:

Your Transformers example @Groovygoth666reminded me there’s a lot that could be done with GI Joe, Superfriends,… the cartoon possibilities are pretty endless 

Transformers and G.I. Joe shared a lot of music, this YouTube channel has uploaded a fan edit that was done for Transformers and so has plenty of Joe tracks aswell. It was made using Izatope RX though and sounds great. 

 

Hasbro has created its own record label so hopefully a lot of this music will get official releases someday (that aren't vinyl exclusive). 

 

But yeah you're right the possibilities for other cartoons to finally get a release using these A.I tools is endless 

[Meredith McKay 7,062]

On 24/12/2023 at 10:55 AM, Presto said:

I'm gonna try it on the Rankin/Bass ROTK at some point.

Well, I tried it on a few cues:

     

And from another score:
     

@Andy

[Andy 5,413]

Pretty damn impressive!

[CatastrophicJones 62]

9 hours ago, Presto said:

And from another score:
     

@Andy

 Is that Westworld by chance? I swear this sounds familiar...

[Meredith McKay 7,062]

4 minutes ago, CatastrophicJones said:

 Is that Westworld by chance? I swear this sounds familiar...

Chuck, by Tim Jones.

[CatastrophicJones 62]

23 minutes ago, Presto said:

Chuck, by Tim Jones.

What's funny is that I almost guessed that second but for some reason felt it was more recent since it's been years since I've seen that show. Fantastic score, glad they eventually released some of the music. Now it makes sense, thank you