ACA Blog

ACA Blog


September 2017
M T W T F S S
« Jul    
 123
45678910
11121314151617
18192021222324
252627282930  

Categories


Speeding up local Docker builds

Jan EerdekensJan Eerdekens

Lately I’ve being diving into Docker (and Docker Compose) to see how it can be used to create a containerised version of Liferay. During this experiment I ran into a small issue: the Docker builds were pretty slow. The reason for this is that we don’t want to have the different installation files, needed during the build, in our Docker project (and consequently our GIT repo). This means that we’re actually downloading them on the fly during the build using wget

As you can see we try to put most of these installation files in our own Nexus Maven repository. The reason for this is that this way your build server isn’t dependent on having an internet connection and it also allows other developers the re-use the same artifact in their own builds. Because we need a lot of ‘stuff’ to build a functioning Liferay instance we need to download a lot, in our case almost 600 Mb:

So you can see that if your Docker build is going to try to download all these files every time you try a build after changing your Dockerfile it might be a tad on the slow side.

the downloads are too damn slow

Putting all these files inside of our project and using Docker’s ADD command to add them to our image would definitely speed up the build, but it would also mean that we’d need to check in those files into our GIT repo… which is something we definitely don’t want to do. So how can we solve this problem, preferably without needing to do too much trickery in the Dockerfile itself? The answer, for once, is Python! I knew that Python course I did on Coursera would come in handy one day.

Python, which is by default installed on my Macbook, comes with a nice little build-in web server called SimpleHTTPServer that you can start with a simple command

which will cause it to start serving the files from that directory on localhost:8000. When we combine this with your local Maven repository directory that we as Java developers probably already have anyway we get this:

Screenshot 2016-10-23 19.18.38

 

This is already pretty close to what we’re looking for as this will allow you to execute a wget command, including the authentication (which SimpleHTTPServer will happily ignore), by adding 1 parameter to your Docker build command to overrule the default value of the build parameter, REPO_BASE_URL, that is already in place to define the URL of the Nexus repository:

You only need to remember to use your actual Docker host IP here and not 127.0.0.1 or localhost as Docker otherwise won’t resolve it to your machine, but to the image/container itself, which won’t work.

While this works, there is one small issue with it: what if you need to download files from different repositories inside of your Nexus? While Nexus is perfectly capable to do this, your local Maven repo isn’t. So now we have wget commands in our Dockerfile with URLs that can’t all be correctly served by our little Python webserver. Luckily there is a simple solution: extend SimpleHTTPServer to ignore the repository part of the URL. The few lines of code below will accomplish this:

If you save the code above (or from this Gist) to a file named NotSoSimpleHTTPServer.py and put it in you local Maven repository directory (/Users/username/.m2/repository) you can start it with the command below and it will correctly serve artifacts locally that seem to come from different repositories in your Nexus:

With this setup and a simple override of a Docker build argument you can now speed up your local Docker build immensely. A simple test with the Dockerfile I was using shows that we go from a remote Nexus download that takes 3 minutes 19 seconds

to a local Maven repo that only takes 15 seconds

Ludicrous speed

The only thing that is left is to how to get the artifacts into your local Maven repo (if they’re not already in there from referencing them somewhere in a pom). The easiest way to do this is by using the Maven Dependency plugin and executing the command below with the correct groupId, artifactId, packaging and classifier parameters will accomplish that, e.g.:

So if you’re having a similar (or equivalent) problem with slow downloads in your Dockerfile you can use the information in this post to take your build from slow and sluggish to ludicrous speed (or maybe even to plaid)!

Belgian Java and Liferay developer with some weird interests, eternal complainer, atheist, skeptic and geocacher.

Leave a Reply

Be the First to Comment!

Notify of
avatar
wpDiscuz