`tiup cluster deploy` with TLS may unexpectedly `scp` zero-sized `ca.crt` cert file to target directories

[amet13x]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

• Write a yaml file describing a TiDB cluster for deployment with enable_tls: true.
• Deploy it using TiUP with default concurrent settings.

2. What did you expect to see?
   The cluster is correctly deployed with all components having the same ca.key file.

3. What did you see instead?
   Some distributed ca.key files are zero-sized. As a result, components with these abnormal cert files fail to run normally.

4. What version of TiUP are you using (tiup --version)?

1.16.5 tiup
Go Version: go1.25.0
Git Ref: v1.16.5
GitHash: 9bdae04

[ti-chi-bot]

Welcome @amet13x! It looks like this is your first issue to pingcap/tiup 🎉

[amet13x]

Probable Root Cause Analysis

Currently, this is how a task distributes a file to its target:

func (c *TLSCert) Execute(ctx context.Context) error {
    the_file := prepare_the_file()
    utils.SaveFileWithBackup(the_file.path, the_file.data, backupDir)
    transfer(the_file)
}

Take a look into utils.SaveFileWithBackup():

func SaveFileWithBackup(path string, data []byte, backupDir string) error {
    fileLock(path).Lock()
    defer fileLock(path).Unlock()
    //...
    os.WriteFile(path,data,permission)
}

The problems are:

• There can be more than one tasks running in parallel pointing to the same path calling utils.SaveFileWithBackup(), in this case of ca.crt file.
• os.WriteFile() truncates the file to zero if it exists.
• The combinate operation of utils.SaveFileWithBackup AND the file transfer is not atomic. This lead to a race condition.

It is possible that a task can unexpectedly truncate the ca.crt file just waiting for another task to be transferred. In that case, the zero-sized truncated file can be copied.