I'm trying to build Superset locally using docker-compose.
After cloning the repository, I modify docker-compose.yml so that it builds images from local source code instead of pulling from Docker Hub. My modifications include:
In service db, change Postgres image version from image: postgres:14 to image: postgres:10 since the service cannot be built properly with Postgres 14.
In services superset, superset-init, superset-worker, superset-worker-beat and superset-tests-worker, change image: *superset-image to build: . so that Docker builds the application from local source code.
However, after running docker-compose build and then docker-compose up, I got the blank screen like this. I checked out the logs and found out that a lot of asset files are missing, for example /static/assets/images/loading.gif is missing which results in that blank screen.
What am I wrong or missing from my configuration steps? Please help me.
I finally figured it out, it's because the webpackage of the superset frontend is installed inside the container superset_node instead of while building it. That's why although the superset_node is built, we have to wait (in my case for about 15-20 more minutes) for the webpackage to be completely installed. Another point to note is that this installation takes up a lot of memory, so make sure you allocate enough RAM to it (in my case I allocate 16GB to Docker).
Related
I'm using GitLab.com's managed CI runners, and I'd like to run my CI jobs using the if-not-present pull policy to avoid the extra minutes it takes to pull the image for each job. Trying to set that value in the .gitlab-ci.yml file gives me this error:
pull_policy ([if-not-present]) defined in GitLab pipeline config is not one of the allowed_pull_policies ([always])
This led me to the config.toml settings for restricting Docker pull policies, so I created a config.toml file at the root of my repository and tried that. However, I still get the same error.
Is config.toml only available for manual/self-hosted runners? Is there any other way to get past this?
Context
Image selection in .gitlab-ci.yml:
default:
image:
name: registry.gitlab.com/myorg/myrepo/ci/builder:latest
pull_policy: if-not-present
Contents of config.toml:
[[runners]]
executor = "docker"
[runners.docker]
pull_policy = ["if-not-present"]
allowed_pull_policies = ["always", "if-not-present"]
First of all, the config.toml file is not meant to be in your repo but on the runner machine (or container).
But anyways, the always pull policy should not cause image pulls to last minutes if the layers are already cached locally: it just ensures you have the latest version by checking the metadata. If the pulls take minutes, it means that either the layers are not available locally, or the image was actually updated (or that the connection to your container registry is so incredibly slow that just checking the metadata takes minutes, but that is unlikely).
It is very possible that Gitlab's managed runners do not have a way to locally cache layers, and thus there would be no practical difference between the always and if-not-present policies. For instance if you use Gitlab Saas:
A dedicated temporary runner VM hosts and runs each CI job.
(see https://docs.gitlab.com/ee/ci/runners/index.html)
Thus the downloaded layers are discarded as soon as the job finishes.
On AzureML Batchendpoint, I'm recently hitting the following error:
Unable to get image details : Environment version Autosave_(date)T(time)Z_******** provided in request doesn't match environ.
when I setup the batch-endpoint with a yml config:
environment: azureml:env-name:env-version
So, AzureML creates and builds the environment with the version I specify env-version, which is just a number (in my case = 3).
and then for some weird reason, AzureML creates an extra environment version called Autosave_(date)T(time)Z_********, which is not built, but based on the previous one just created, and then it becomes the latest version of that environment.
In summary, AzureML instead of looking for the version that I specified as env-name:3 it seems to be looking for env-name:Autosave_(date)T(time)Z_******** and then throws the error message mentioned above.
I found the problem was that when creating an environment from a YAML specification file, one of my conda dependencies was cmake, which I needed to allow installation of another python module. The docker image is exactly the same as a previously created environment.
Removing the cmake dependency from the YAML file, eliminated the issue. So the workaround is to install it using a Dockerfile.
The error message was very misleading to start with, but got there in the end after understanding that AzureML reuses a cached image, based on the hash value, from the environment definition accordingly to this
So for that reason, the automatically created Autosave docker image references to that same build, which only happens once when the first job is sent.
Steps to reproduce are very easy.
Create a Dockerfile.
My Dockerfile has many more lines, but I have trimmed them so we can focus in the source of the problem.
Said that, these two lines alone (without anything more) show the problem.
FROM microsoft/iis
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; $VerbosePreference = 'Continue'; "]
Run docker build . and you get hcsshim::PrepareLayer - failed failed in Win32: Función incorrecta. (0x1).
Windows 10 Pro 1909 (but it happened too in 1903)
Docker version: 2.1.0.5
Engine: 19.03.5
Machine: 0.16.2
I have found the solution to the problem.
Reading all the https://github.com/docker/for-win/issues/3884 bug, some have found a simple solution: rename C:\windows\system32\driver\cbfsconnect2017.sys so it isn't loaded the next boot.
Disabling that driver enables me to do a docker build for the first time in windows containers in almost a year.
In my case Box Sync was the one using that driver.
EDIT: #GustavoTM have found that pCloud raises the same problem.
EDIT2: #VonC have noticed that some people in the issue in GitHub has solved it deleting this other file: C:\Windows\System32\drivers\cbfs6.sys. I haven't tried that, but i put it if it helps others.
The good thing is that I don't need to uninstall Box, but only rename that file.
This is still an issue (still open) with Win10.
Looks like uninstalling cloud storage providers with file system filters like Dropbox, Box, etc. as a workaround is an option for some users.
Deinstall cloud storage providers or virus scanners; if you identify which one is not working please share in https://github.com/docker/for-win/issues/3884
In my case was the problem similar but the file cbfs6.sys was placed somewhere in the rest of uninstalled application Jungle disk, somewhere in the folder c:\Program files\Jungle disk .... It's part of Callback File System signed by EldoS Corporation.
The folder could be rename only and not delete directly. So I could delete its immediately after the PC restart, before running the Docker. So it could be delete during the Docker service restart too.
This is part of a major issue i've been fighting to get resolve in a span of 2 or even 3 weeks, first of all, i'm not a docker expert, in fact, i don't even know a thing about docker, all i know is that i need to use it in order to make a connection between an api in localhost and my app in react native, the thing is, i manage to make it work on another two projects i created to test docker, but not in the one i actually need to. This is a dockerfile for an api in .net core 2.2
my dockerfile is a combination of the code i found in stackoverflow and the example in docker documentation to create a docker in .net core, this specific file worked for me on another two api, one as a blank project, and the other one with a class library.
The code below shows the dockerfile, when i run the command line and create the image, it shows no errors, but i know there is something wrong, because when i run docker image ls, the docker image is around 200-300mb size, which seems way too small, and when i run that image with docker run... and check the list of docker containers runnning, it shows nothing
FROM mcr.microsoft.com/dotnet/core/sdk:2.2 AS build-env
WORKDIR /app
# Copy csproj and restore as distinct layers
WORKDIR /src
COPY ISARRHH.sln ./
COPY ISARRHH.BusinessGraph/*.csproj ./ISARRHH.BusinessGraph/
COPY ISARRHH.APIWeb/*.csproj ./ISARRHH.APIWeb/
RUN dotnet restore
# Copy everything else and build
COPY . ./
WORKDIR /src/ISARRHH.BusinessGraph
RUN dotnet publish -c Release -o /app
WORKDIR /src/ISARRHH.APIWeb
RUN dotnet publish -c Release -o /app
# Build runtime image
FROM mcr.microsoft.com/dotnet/core/aspnet:2.2
WORKDIR /app
COPY --from=build-env /app .
ENTRYPOINT ["dotnet", "isarrhh.dll"]
#######################################################
I want this bloody docker to work, this was the plan b on one of the modules i'm working on, and is giving me a headache, i managed to make it work on another project, i want it to work on this api which works with office 365 and sharepoint
EDIT: this is the project structure
ISARRHH (Solution)
|
|--ISARRHH.APIWeb (API)
| |_Dependencies
| |_Controllers
| |_Models
| |_Properties
| |_appsettings.json
| |_appsettings.Development.json
| |_Authentication.cs
| |_Configuration.cs
| |_Program.cs
| |_ProtectedApiCallHelper.cs
| |_PublicAppUsingUsernamePassword.cs
| |_SiteInformation.cs
| |_Startup.cs
| |_SiteInformation.cs
|
|--ISARRHH.BusinessGraph (Class Library)
| |_Dependencies
| |_UserGraph.cs
|
|--Solution Items
|_Dockerfile
|_.dockerignore
EDIT2: More information
REPOSITORY TAG IMAGE ID CREATED SIZE
isarrhh latest 67fc0628c921 13 minutes ago 268MB
according to this, the image was created succesfully apparently, but when i run it with
docker run -d -p 3001:80 ...
then i check with
docker container ls
i see no container running, also, when i check with the command you provided here
docker logs -t isachile
i get this:
MacBook: ISARRHH$ docker logs -t isachile
2019-07-31T18:49:22.553317346Z Did you mean to run dotnet SDK commands? Please install dotnet SDK from:
2019-07-31T18:49:22.553390430Z https://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409
EDIT 3: SOLVED IT -- SORT OF...
i manage to run my docker by manually copy and pasting ever file on a different project, each file individually copy and paste in this second project, and each time creating the docker image, yes, a seriously horrible and tedious process, but it worked, although, we're not considering this solution anymore, since the process is too slow for our scrum project, we need to connect react native to our localhost api, i still need an answer for this
So there's two things here, and neither necessarily indicates a problem with Docker or your Dockerfile.
Size is only 200-300MB
That's about right. You haven't indicated whether you're using Windows or Linux containers, but in either case, most of the weight comes simply from the .NET Core runtime. The whole point of containers is that the host OS is shared (unlike a VM where every VM gets its own separate OS installation). The only things coming from the base OS image are user-specific files and directories. The main system components are proxied to the host operating system. Long and short, I don't know what you're expecting here in terms of size, but honestly 200-300MB is a bit on the large size for an image. It's possible in many cases to package ASP.NET Core app images down to as little as 25MB-30MB, though if you include the full runtime, it's generally going to be closer to your 200-300MB.
The container isn't running.
All the means is that it exited. When the container is run, the entrypoint line will be called, which just starts up the ASP.NET Core app running in Kestrel. That of course runs Program.Main, since it's just a console app, after all. That in turn builds the web host and calls Run, which listens for TCP socket connections, keeping the app running, which therefore keeps the container running.
If the container isn't running, then the app exited. That could happen for different reasons, but the most likely cause is that a runtime exception was thrown during the web host build phase (i.e. something in Program or Startup is throwing an exception). Try running something like:
docker logs -t {container name}
And you'll probably see a stacktrace and exception there. Fix the issue accordingly.
I have a question regarding best practice.
Lets say I have a webfrontend developed in angularjs and a api to get data from. I put those two in separate repositories.
Now I want to dockerize the whole thing. My idea was to put a Dockerfile in each project which specifies their environment. Until now its fine but what when I also have a docker-compose file which starts those two services simultanously? First of in which repo should I put this file and secondly how can I assure, that the images are always uptodate?
You can use a structure like this:
project/
- project/docker-compose.yaml
- project/frontend/ (contains Dockerfile + necessary files)
- project/api/ (contains Dockerfile + necessary files)
In your docker-compose.yaml you can write something like this for each image:
frontend:
build: ./frontend #folder in which Dockerfile of frontend is saved (to build the image)
image: my-frontend:1.0 #give a name to your image
container_name: frontend-container #containername when a container instance of your image is running
To start docker-compose you can run docker-compose up --build. The --build tag will recreate your images when there are changes so you can keep your compose up to date when there are changes in the dockerfile (image).
Docker-compose is 'reading' from up to bottom. So the service you describe first in your docker-compose.yaml will be created first. I would think it's your API because that can probably exist on its own and your frontend needs to connect with it. Sometimes your first service is started too slow, which means the second service is already up but can not find the first service (to connect with) and it crashes. This can be solved by using a wait-for-it.sh script. Your second service will use the script and check when the first service is up. When it's up, it will start its own service.