I converted a pandas df into a Spark df using the following and it seems to populate just fine. Notice how the df is ordered by date: https://i.imgur.com/gPfYyyc.png
df_spark.write.format('jdbc').options( url='jdbc:mysql://localhost:3306/final_project?serverTimezone=UTC', driver='com.mysql.jdbc.Driver', dbtable='yfinance', user='USER', password='PASSWORD').mode('overwrite').save()
This creates a table in my database called
yfinance but it does not order the table by date:
How do I move this Spark df into MySQL and keep it ordered by
date asc? Is there a way to assign
date as the primary key?