Hello all,
I am trying to use csvlink across two input files as normal. However, one of the csv files I'm using is several GB in size (how the file is given to me, I have no control over this).
If I attempt to run csvlink with this file, I get the following
error: Could not find the file <file>
Upon investigating the issue, the file is of course there. I took time to break it out into smaller files as I assumed perhaps this issue was being caused by the file size. This appears to be the case. If I attempt to run csvlink with a smaller (still large, 750MB) file, it is running. Hasn't finished yet, but it's running ;)
I'm reporting this issue to let someone at the team know about it. If there is a designed upper limit to the file size that csvdedupe/csvlink can use, let me know here and I can open a PR with changes to the README.
Thanks!
Hello all,
I am trying to use csvlink across two input files as normal. However, one of the csv files I'm using is several GB in size (how the file is given to me, I have no control over this).
If I attempt to run csvlink with this file, I get the following
Upon investigating the issue, the file is of course there. I took time to break it out into smaller files as I assumed perhaps this issue was being caused by the file size. This appears to be the case. If I attempt to run csvlink with a smaller (still large, 750MB) file, it is running. Hasn't finished yet, but it's running ;)
I'm reporting this issue to let someone at the team know about it. If there is a designed upper limit to the file size that csvdedupe/csvlink can use, let me know here and I can open a PR with changes to the README.
Thanks!