How to recursively copy all files except empty files including subdirectories in Databricks file system - apache-spark-sql

Question

I have a requirement to recursively move all files except empty files (0-byte files) to a destination folder while preserving hierarchies in a Databricks file system.
Example:
folder_path/
1/
2/
file1.json- 0 byte
file2.json- 128 kb
3/
4/
file3.json- 0 byte
file4.json- 20 kb
outputs:
folder_path/
1/
2/
file2.jpg - 128 kb
3/
4/ file4.jpg - 20 kb
I'm able to implement this using shell script and AWK but it seems Databricks does not support AWK.
Is there any way to implement this either in Python or Scala using Databricks notebook ?

Answer 1

The below implementation works for me. If the size is 0 and does not end with "json", then it is a directory. Else it is a file with 0 size.
import os
src = 'dbfs:/<src_path>'
def createSubDir(dir):
for i in range(0, len(dir)):
if(dir[i].size == 0 and not dir[i].name.endswith('json')):
dbutils.fs.mkdirs(dir[i].path.replace("<src_dir>", "<dest_dir>"))
createSubDir(dbutils.fs.ls(dir[i].path))
elif(dir[i].size != 0 and dir[i].name.endswith('json')):
dbutils.fs.mv(dir[i].path, dir[i].path.replace("<src_dir>", "<dest_dir>"))
dir = dbutils.fs.ls(src)
createSubDir(dir)