AWS CodeBuild local cache failing to actually cache? - aws-codebuild

Question

I've been trying to get AWS CodeBuild's local cache to work and for the life of me I can't get even the most basic cache to work. My ultimate goal is to cache Gradle artifacts, as discussed here.
But because I couldn't get that to work, I tried an even simpler test, where I try to cache the directory /root/foo with a file counter.txt that I increment each build. My expectation is that if I run subsequent builds within a few minutes of each other, I would see "2", "3", etc in the logs. But the reality is that despite the symlink being established, the next build never sees the previous counter.txt file, which suggests to me something is very broken.
Can anyone confirm that their local cache is actually working in CodeBuild? I'm starting to wonder if the feature is presently broken! Or am I totally misunderstanding what it is supposed to do?
buildspec.yml:
version: 0.2
phases:
install:
runtime-versions:
java: corretto8
build:
commands:
- pwd
- ls -l /root/
- ls -l /root/foo/
- ./cache-test.sh
- ls -l /root/
- ls -l /root/foo/
cache:
paths:
- '/root/foo/*'
cache-test.sh:
#!/bin/bash
if [ -d "/root/foo" ]; then
C=$(cat /root/foo/count.txt)
C=$((C + 1))
echo "*********************************"
echo "*********************************"
echo "Incrementing counter to $C"
echo $C > /root/foo/count.txt
echo "*********************************"
echo "*********************************"
else
mkdir /root/foo
echo "*********************************"
echo "*********************************"
echo "File not found, starting count at 1"
echo "*********************************"
echo "*********************************"
echo 1 > /root/foo/count.txt
fi
CodeBuild output: (same output even when executed in quick succession)
[Container] 2019/11/10 22:35:08 Waiting for agent ping
[Container] 2019/11/10 22:35:10 Waiting for DOWNLOAD_SOURCE
[Container] 2019/11/10 22:35:10 Phase is DOWNLOAD_SOURCE
[Container] 2019/11/10 22:35:10 CODEBUILD_SRC_DIR=/codebuild/output/src905503483/src
[Container] 2019/11/10 22:35:10 YAML location is /codebuild/output/src905503483/src/buildspec.yml
[Container] 2019/11/10 22:35:10 No commands found for phase name: INSTALL
[Container] 2019/11/10 22:35:10 Processing environment variables
[Container] 2019/11/10 22:35:10 Moving to directory /codebuild/output/src905503483/src
[Container] 2019/11/10 22:35:10 MkdirAll: /codebuild/local-cache/custom/de68c9f22ae028d4e4dfb0d11bbb481053d28b1373db0d6a56ebee0416bf13b2/root/foo
[Container] 2019/11/10 22:35:10 Symlinking: /root/foo => /codebuild/local-cache/custom/de68c9f22ae028d4e4dfb0d11bbb481053d28b1373db0d6a56ebee0416bf13b2/root/foo
[Container] 2019/11/10 22:35:10 Registering with agent
[Container] 2019/11/10 22:35:10 Phases found in YAML: 2
[Container] 2019/11/10 22:35:10 BUILD: 6 commands
[Container] 2019/11/10 22:35:10 INSTALL: 0 commands
[Container] 2019/11/10 22:35:10 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2019/11/10 22:35:10 Phase context status code: Message:
[Container] 2019/11/10 22:35:11 Entering phase INSTALL
[Container] 2019/11/10 22:35:11 Running command echo "Installing corretto(OpenJDK) version 8 ..."
Installing corretto(OpenJDK) version 8 ...
[Container] 2019/11/10 22:35:11 Running command export JAVA_HOME="$JAVA_8_HOME"
[Container] 2019/11/10 22:35:11 Running command export JRE_HOME="$JRE_8_HOME"
[Container] 2019/11/10 22:35:11 Running command export JDK_HOME="$JDK_8_HOME"
[Container] 2019/11/10 22:35:11 Running command for tool_path in "$JAVA_8_HOME"/bin/* "$JRE_8_HOME"/bin/*;
do tool=`basename "$tool_path"`;
if [ $tool != 'java-rmi.cgi' ];
then
rm -f /usr/bin/$tool /var/lib/alternatives/$tool \
&& update-alternatives --install /usr/bin/$tool $tool $tool_path 20000;
fi;
done
[Container] 2019/11/10 22:35:11 Phase complete: INSTALL State: SUCCEEDED
[Container] 2019/11/10 22:35:11 Phase context status code: Message:
[Container] 2019/11/10 22:35:11 Entering phase PRE_BUILD
[Container] 2019/11/10 22:35:11 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2019/11/10 22:35:11 Phase context status code: Message:
[Container] 2019/11/10 22:35:11 Entering phase BUILD
[Container] 2019/11/10 22:35:11 Running command pwd
/codebuild/output/src905503483/src
[Container] 2019/11/10 22:35:11 Running command ls -l /root/
total 4
lrwxrwxrwx 1 root root 103 Nov 10 22:35 foo -> /codebuild/local-cache/custom/de68c9f22ae028d4e4dfb0d11bbb481053d28b1373db0d6a56ebee0416bf13b2/root/foo
[Container] 2019/11/10 22:35:11 Running command ls -l /root/foo/
total 0
[Container] 2019/11/10 22:35:11 Running command ./cache-test.sh
cat: /root/foo/count.txt: No such file or directory
*********************************
*********************************
Incrementing counter to 1
*********************************
*********************************
[Container] 2019/11/10 22:35:11 Running command ls -l /root/
total 4
lrwxrwxrwx 1 root root 103 Nov 10 22:35 foo -> /codebuild/local-cache/custom/de68c9f22ae028d4e4dfb0d11bbb481053d28b1373db0d6a56ebee0416bf13b2/root/foo
[Container] 2019/11/10 22:35:11 Running command ls -l /root/foo/
total 4
-rw-r--r-- 1 root root 2 Nov 10 22:35 count.txt
[Container] 2019/11/10 22:35:11 Phase complete: BUILD State: SUCCEEDED
[Container] 2019/11/10 22:35:11 Phase context status code: Message:
[Container] 2019/11/10 22:35:11 Entering phase POST_BUILD
[Container] 2019/11/10 22:35:11 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2019/11/10 22:35:11 Phase context status code: Message:
CodeBuild project JSON:
{
"projects": [
{
"name": "test-project",
"arn": "arn:aws:codebuild:us-east-2:xxx:project/xxx",
"source": {
"type": "CODEPIPELINE",
"insecureSsl": false
},
"secondarySourceVersions": [],
"artifacts": {
"type": "CODEPIPELINE",
"name": "test-project",
"packaging": "NONE",
"encryptionDisabled": false
},
"secondaryArtifacts": [],
"cache": {
"type": "LOCAL",
"modes": [
"LOCAL_SOURCE_CACHE",
"LOCAL_CUSTOM_CACHE"
]
},
"environment": {
"type": "LINUX_CONTAINER",
"image": "aws/codebuild/amazonlinux2-x86_64-standard:1.0",
"computeType": "BUILD_GENERAL1_SMALL",
"environmentVariables": [],
"privilegedMode": false,
"imagePullCredentialsType": "CODEBUILD"
},
"serviceRole": "arn:aws:iam::xxx:role/service-role/xxx",
"timeoutInMinutes": 60,
"queuedTimeoutInMinutes": 480,
"encryptionKey": "arn:aws:kms:us-east-2:xxx:alias/aws/s3",
"tags": [],
"created": 1573364156.631,
"lastModified": 1573423155.674,
"badge": {
"badgeEnabled": false
},
"logsConfig": {
"cloudWatchLogs": {
"status": "ENABLED",
"groupName": "xxx",
"streamName": "xxx"
},
"s3Logs": {
"status": "DISABLED",
"encryptionDisabled": false
}
}
}
],
"projectsNotFound": []
}

Answer 1

I have been trying to make the cache work myself with limited success.
Not from any public source, but these are some observations:
The cache will only be avilable if the build time is more than 5 minutes.
Cache can be used if the new build is successfully placed on the same build host.
Cache can be used if the new build is being run within 5-15 minutes of the last build. Cache may stay available based on the last build time with a max of 15 minutes.
Despite builds exceeding 5 minutes, the cache may not always work probably due to build being placed on different build host.
Additionally, in the case where the cache speeds up the new build to below 5 minutes, that build will not be cached resulting in subsequent miss.
While I trust CodeBuild engineers had good reasons to design it this way, the above limitations renders this local cache functionality of limited use in my opinion.

Answer 2

The documentation isn't exactly clear, but AWS CodeBuild Local cache can only cache directories (as of time of writing). This is slightly confusing because the AWS CodeBuild buildspec specification allows the path to be individual files or a wildcard, but in reality specifying a file will yield an error.
Unable to initialize cache download: only directories can be cached locally: ...
In your example you use specify the cache as
cache:
paths:
- '/root/foo/*'
Where * would refer to all individual files and folders inside foo but only folders would be cached.
Specifying the entire directory should work
cache:
paths:
- /root/foo/