Replace lyrics of an existing song

[firepol]

Hi, my goal is to generate a song from an existing song, copying as much as possible its style, basically same music and same singer voice, just, new lyrics (lyrics that resemble a lot the original ones, same length, same structure so that they could match the existing melody, chorus etc).

Note: I used a song from a famous Italian pop/rock singer, wit lyrics in Italain.

My hardware: a linux machine with 40+ GB available RAM, GPU RTX 3070 with 8 GB VRAM

I run acestep as follows:

conda activate ace_step # to activate the conda environment I created for acestep
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True # this is handy as 8 GB VRAM are often used at their limit
acestep --server_name 0.0.0.0 --port 7865 --torch_compile true --cpu_offload true

I tried the Audio2Audio feature in ACE-Step, without success. Here what I did:

• found the lyrics of the song and asked chatgpt to make new lyrics using the exact same structure, making it e.g. a parody song of [some topic] and I obtained lyrics with [verse], [chorus] etc.
• in ACE-Step I checked the "Audio2Audio" checkbox and uploaded the mp3 of the original song (of a famous author)
• tags: I left it empty
• in the lyrics section I pasted the lyrics generated via chatgpt
• Refer audio strength: I left as is: 0.5
• Basic Settings > all left as is, changed "Infer Steps" to 27 (which was a good value to generate music from scratch)

I obtained bad quality results and bad singing barely noticeable singing over loud music, or no singing at all.

I tried to change "Refer audio strength" to 0.8 and to 0.3, same issues.
I tried to change infer steps to 40 and to 20, same issues.

I also tried to describe the original song as asked in #263 via https://chat.qwen.ai/ (uploaded 3 minutes of the song as there is a limit of 3 minutes, and asked "Describe this audio, I need the style description so I can produce a similar music with an AI generator"). I tried to use Audio2Audio and instead of leaving tags empty, I tried to insert the description obtained via qwen, but the issues were the same (bad quality music, singing barely noticeable over loud music or so...)

So I come here asking some help from who tried the same experiment. If it's even possible at all.

I first learnt about ACE-Step in this tutorial https://www.youtube.com/watch?v=PO_u7xEaL5I&t=1797s and at 26:51 the youtuber explains that "Refer audio strength" closer to 1 would reproduce exactly the same song, so he suggested a value lower than 0.5 but it's not really clear how to proceed in my case, if I really want to get exactly the same song, but only with different lyrics...

Thanks for your answers and hopefully this could serve as tutorial for who wants to create covers of existing songs.

Note, I tried to do this also on suno, but on suno, upoading copyrighted music is not allowed, it gives of course the following error:

Uploaded audio matches existing work of art.
Please upload a different audio file.

Else, in theory suno would be perfect for this, upload a song or remix/make a cover of an existing one, depending on the model and parameters it should be possible to get a song really close to the original one (but with different lyrics).

Please let's not debate about copyight infringment etc. my goal is educational, thus no copyright infringment is done if I have some fun locally as ACE-Step runs locally. Thank you.

[rhythmmedia]

Why this keeps failing (not just a settings problem):

This is a structurally hard ask for any generative resynthesis model — not a tuning issue on your end. No model has training data of "song A" paired with "song A, different words, everything else identical," because that pairing doesn't exist. There's nothing for a model to learn "same voice and instrumental, just re-sung on the same timing" from.

In practice, people report "Refer audio strength" behaving like a tradeoff dial: push it high and timing/production stay close to the original but new lyrics don't line up (mumbled or buried vocal); push it low and you get a cleaner new performance that drifts from the reference entirely. There's no control that isolates "keep instrumental," "keep voice," and "re-time to new lyrics" as separate axes — which is what this task actually needs. Consistent with what others report in the ACE-Step community (a HF thread has someone getting Audio2Audio outputs that just mirror the input) — this is a known pain point, not something specific to your setup.

If you actually want to give this to someone rather than keep experimenting — the way this gets solved today isn't generative resynthesis at all. It's a real singer performing the new lyrics (a human naturally handles timing to fit an existing melody in a way a reference-strength slider can't), with AI voice matching applied after so it sounds like the original artist. Full disclosure, I run a service built around exactly this, SongRetold, if that's ever useful.