使用熊猫提高性能

I am reading data from SQL server and writing into a .dat file using dataframes.

The total number of records is 31M+ or 4.87GB data. Its taking 11.40sec to read and write into the .dat file. The current VB system is completing in 6min.

无论如何,我可以改善脚本的性能吗?

def test_SQLPandas():
  conn = pyodbc.connect(sql, conn_str)
  df = pd.read_sql(sql, conn, chunksize=50000)
  i=1
  for data in df:
     data.to_csv('smaple.dat', sep='\t', mode='a')
     i+=1
评论