使用熊猫提高性能

由 发布于

I am reading data from SQL server and writing into a .dat file using dataframes.

The total number of records is 31M+ or 4.87GB data. Its taking 11.40sec to read and write into the .dat file. The current VB system is completing in 6min.

无论如何，我可以改善脚本的性能吗？

def test_SQLPandas():
  conn = pyodbc.connect(sql, conn_str)
  df = pd.read_sql(sql, conn, chunksize=50000)
  i=1
  for data in df:
     data.to_csv('smaple.dat', sep='\t', mode='a')
     i+=1