Hive LOAD DATA INPATH query delete source data (s3) - hive

Question

When i use hive in EMR, I use 'LOAD DATA INPATH' query.
That query deletes the source data. The EMR role I use doesn't have delete permission, so that query always fails. Is there any way to solve this problem?
LOAD DATA INPATH 's3://my-bucket/folder/data.csv'
OVERWRITE INTO TABLE {my-database}.{my-table};
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
s3://my-bucket/folder/data.csv to destination
hdfs://{emr_master_node}:8020/{internal_folder}/data.csv
The query throws an error, but the data loads into the table just fine.
Hive internally sends a delete request after a get request, but the get request succeeds and the delete request fails because EMR does not have permission.
Caused by:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception:
Access Denied (Service: Amazon S3; Status Code: 403; Error Code:
AccessDenied; Request ID: F7XPGZW7BVAFFGBV; S3 Extended Request ID:
9MnXACfWf/y2wr4OklowfrJAzzRMYMT7cfE/OHzldS7L/yR2dYufa5sq3OtloAnVjlRdNzYxWm8=;
Proxy: null) at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1415)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1154)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:811)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:779)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:753)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:713)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:695)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:539)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5445)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5392)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2295)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3.lite.call.DeleteObjectCall.perform(DeleteObjectCall.java:25)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3.lite.call.DeleteObjectCall.perform(DeleteObjectCall.java:11)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor$CallPerformer.call(GlobalS3Executor.java:108)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_352] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_352] at
com.amazon.ws.emr.hadoop.fs.shaded.retry.invocation.ProxyingInvocation.invoke(ProxyingInvocation.java:52)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.retry.policies.aimd.RateAdjustedInvoker$1.get(RateAdjustedInvoker.java:43)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.rateadjuster.RateAdjusted.execute(RateAdjusted.java:231)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.retry.policies.aimd.RateAdjustedInvoker.invoke(RateAdjustedInvoker.java:39)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.retry.invocation.DelegatingInvocation.invoke(DelegatingInvocation.java:47)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.retry.invocation.RetryingInvoker.invoke(RetryingInvoker.java:52)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.shaded.retry.policies.aimd.AIMDInvocationHandler.invoke(AIMDInvocationHandler.java:92)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.sun.proxy.$Proxy108.call(Unknown Source) ~[?:?] at
com.amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.java:135)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:191)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.java:186)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.deleteObject(AmazonS3LiteClient.java:122)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.delete(Jets3tNativeFileSystemStore.java:434)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.delete(S3NativeFileSystem.java:426)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.delete(EmrFileSystem.java:418)
~[emrfs-hadoop-assembly-2.48.0.jar:?] at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:423)
~[hadoop-common-3.2.1-amzn-5.jar:?] at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:387)
~[hadoop-common-3.2.1-amzn-5.jar:?] at
org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:633)
~[hive-common-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:600)
~[hive-common-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:3854)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:4318)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.exec.HiveDataCommitter.replaceFiles(HiveDataCommitter.java:69)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:2458)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:376)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:209)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
~[hive-exec-3.1.2-amzn-6.jar:3.1.2-amzn-6] at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
~[hive-service-3.1.2-amzn-6.jar:3.1.2-amzn-6] ... 11 more
When I checked the bucket access log, it found that the EMR Role makes a Get request and then a Delete request.
I wonder why Hive is requesting move instead of copy.
When I checked the error log above, I found that Hive uses moveTask instead of copyTask.
https://github.com/apache/hive/blob/e427ce0d572c9adf6f194693a1b3ba85f246f3b7/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4865
Also, when I checked the hive code in github, I found that if the source file is not located in local, hive is forced to request to delete the source.
Anyone know the reason for this?