https://www.transtats.bts.gov/DL_SelectFields.asp?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
https://www.transtats.bts.gov/tables.asp?gnoyr_VQ=FGJ&flf_gnoyr_anzr=g_bagVZR_eRcbegVaT
https://developer.ibm.com/exchanges/data/all/airline/
https://www.transtats.bts.gov/DL_SelectFields.aspx?gnoyr_VQ=FGJ&QO_fu146_anzr=b0-gvzr
copy 2020-01.csv+2020-02.csv+2020-03.csv+2020-04.csv+2020-05.csv+2020-06.csv+2020-07.csv+2020-08.csv+2020-09.csv+2020-10.csv+2020-11.csv+2020-12.csv 2020.csv

sed 1d test.sql > test2.sql

# remove first line
sed -i 1d 2021-02.csv
sed -i 1d 2021-03.csv
sed -i 1d 2021-04.csv
sed -i 1d 2021-05.csv
sed -i 1d 2021-06.csv
sed -i 1d 2021-07.csv
sed -i 1d 2021-08.csv
sed -i 1d 2021-09.csv
sed -i 1d 2021-10.csv
sed -i 1d 2021-11.csv
sed -i 1d 2021-12.csv

cp 2021-01.csv 2021.csv
cat 2021-02.csv >> 2021.csv
cat 2021-03.csv >> 2021.csv
cat 2021-04.csv >> 2021.csv
cat 2021-05.csv >> 2021.csv
cat 2021-06.csv >> 2021.csv
cat 2021-07.csv >> 2021.csv
cat 2021-08.csv >> 2021.csv
cat 2021-09.csv >> 2021.csv
cat 2021-10.csv >> 2021.csv
cat 2021-11.csv >> 2021.csv
cat 2021-12.csv >> 2021.csv

cat 2021.csv >> 2015-2020.csv

cat transactions_200{607..819}.csv >>transactions.csv 2>/dev/null

Fix! DEC 2021 onwards
Convert date format to YYYY-MM-DD
Add an extra column, e.g., TEMP

sed -E 's,([0-9]{4})-([0-9]{2})-([0-9]{2}),\2/\3/\1,g' monthly/2021-01.csv

sed -E 's,([0-9]{4})-([0-9]{2})-([0-9]{2}),\3/\2/\1,g' 

sed -E '2,$ s/(("[^"]*",){3})("[^"]+")(.*)/echo \x27\1"\x27$(date -d \3 date +%m/%d/%Y)\x27"\3\x27/e'

With GNU sed

sed -E '2,$ s/(("[^"]*",){4})("[^"]+")(.*)/echo \x27\1"\x27$(date -d \3 +%s)\x27"\4\x27/e'
2,$ to skip header from getting processed
(("[^"]*",){4}) first four columns
("[^"]+") fifth column
(.*) rest of the line
echo \x27\1"\x27 and \x27"\4\x27 preserve first four columns and rest of line after fifth column, along with adding double quotes to result of date conversion
$(date -d \3 +%s) calling shell command with fifth column value
Note that this command will fail if input can contain single quotes. That can be worked around by using s/\x27/\x27\\&\x27/g.


You can see the command that gets executed by using -n option and pe flags

sed -nE '2,$ s/(("[^"]*",){4})("[^"]+")(.*)/echo \x27\1"\x27$(date -d \3 +%s)\x27"\4\x27/pe'
will give

echo '"58.75","5.33","kg","7","'$(date -d "2020-7-11 19:29:29" +%s)'","5"'

For 58.25,5.89, kg, 7,2020 / 7/12 11:23:46, "5" format, try

sed -E '2,$ s/(([^,]*,){4})([^,]+)(.*)/echo \x27\1\x27$(date -d "\3" +%s)\x27\4\x27/e'
or (adapted from https://stackoverflow.com/a/62862416)

awk 'BEGIN{FS=OFS=","} NR>1{$5=mktime(gensub(/[:\/]/, " ", "g", $5))} 1'

Note: For the sed solution, if the input can come from outside source, you'll have to take care to avoid malicious intent as mentioned in the comments. One way is to match the fifth column using [0-9: -]+ or similar.