S3 Dag Bundle won't remove files removed on remote S3

[ilgiznurgaliev]

Hello.

I'm using S3 Dag Bundle and it works perfectly except one thing. Files removed remotely (in S3 bucket) won't be removed in Airflow. My source S3 bucket contains some config files additionally to dag files. I use these config files to generate dags dynamically like that:

configs_dir = Path(__file__).resolve().parent / 'configs'

for file in glob(f'{configs_dir}/*.yaml'):
  with open(file, 'r') as f:
    config = yaml.load(f)

  # Generate pipeline
  @dag(
    dag_id=config['dag_id'],
    schedule=config['cron'],
    tags=config['tags'],
  )
  def yac_instance_autoshutdown_pipeline():
    ...

So number of dags have to be the same as number of yaml files in configs folder.
When one of the yaml files removed from configs folder it's expected that it will be removed locally too and appropriate dag will be deactivated.
Actual behavior: dags will appear when new config files created, but won't be deactivated when config files removed.

Is there any way to clean up files that were removed on remote S3 bucket? Any workaround would help. Would be best to allow something like --delete flag for S3 Bundle configuration, but currently it is not supported.

[cookesan]

The current source says stale local files should be deleted.

S3DagBundle.refresh() calls S3Hook.sync_to_local_dir(..., delete_stale=True), and sync_to_local_dir() has a stale-file cleanup path that removes local files not present in the current S3 object list.

So I would first check the exact Airflow and apache-airflow-providers-amazon versions in the environment. If your installed provider version predates that delete_stale=True behavior, upgrading the Amazon provider may be the fix.

If you are already on a version with that code, then this is probably worth a small reproducible bug report: bucket prefix, bundle config, provider version, whether the deleted file is under the configured prefix, and debug logs from S3Hook.sync_to_local_dir().

[ilgiznurgaliev]

Hello @cookesan.
Thank you for your response. My current version of apache-airflow-providers-amazon provider was 9.21.0. After updating it to the latest 9.25.0 the issue has been solved. Airflow now removes local files not present in the S3.
Seems it has been fixed in 9.23.0 by e491aac commit. Couldn't find it before.

Thanks for your help.