Replace table / schema
Sometimes you may want to replace a Delta table. For example:
You discover the data in the table is incorrect
You want to rewrite the whole table to do incompatible schema changes (such as changing column types).
But delete and re-create table is not recommended, due to:
deleting is slow
lose the data, of course, no way to recover
not atomic, so concurrent query can read half-deleted data
recommend to:
delete rows from table and re-insert, if no schema changes
overwrite schema, if schema changes. However, the "overwriteSchema" seems to still complaining about column type changes.
dataframe.write \
.mode("overwrite") \
.option("overwriteSchema", "true") \
.saveAsTable("<your-table>") # Managed table