replace table / schema

Replace table / schema

  Sometimes you may want to replace a Delta table. For example:

1. You discover the data in the table is incorrect

2. You want to rewrite the whole table to do incompatible schema changes (such as changing column types).

  But delete and re-create table is not recommended, due to:

1. deleting is slow

2. lose the data, of course, no way to recover

3. not atomic, so concurrent query can read half-deleted data

  recommend to:

1. delete rows from table and re-insert, if no schema changes

2. overwrite schema, if schema changes. However, the "overwriteSchema" seems to still complaining about column type changes.

dataframe.write \

  .mode("overwrite") \

  .option("overwriteSchema", "true") \

  .saveAsTable("<your-table>") # Managed table