oci-image-builder
v3.2.2
Published
container image builder. without docker, docker/oci registry client and image building helpers for protocol version 2, fork of container-image-builder
Downloads
1,267,656
Readme
container-image-builder
A container registry client and image builder with no dependency on docker. This supports protocol version 2.
This library has a few parts:
- An
Image
class: takes care of authentication, creating new registry clients, and packaging up local files. - Registry clients: perform exactly the api calls defined in the oci spec.
You can do most things with the Image
class. It takes care of tedious tasks like copying layers missing from one registry into another so you can save your new image.
Example
Create a new image based off of the official node image with your files in it.
;(async()=>{
const image = new Image('node:lts-slim','gcr.io/my-project/my-image')
// add local files to the image.
await image.addFiles({'/directory-root-for-files-in-container':"./"})
// creates a layer in the image with files like
// /directory-root-for-files-in-container/[the files in ./]
// set the default working directory for commands
image.WorkingDir = "/directory-root-for-files-in-container"
// set the default command to run in the container
image.Cmd = ["node","index.js"]
// append environment variables.
image.Env = []
// save the new image at latest
const result = await image.save()
console.log(result)
})();
You can run the image using docker, or deploy it to a docker host:
docker run gcr.io/my-project/my-image:latest
Docker hub and gcr.io have custom auth plugins. If you already have a token or username and secret you can pass them directly when making an Image client.
const image = new Image('node:lts-slim','myspecialregistry.io/my-project/my-image',{
"auth":{
'myspecialregistry.io':{
token:"xxxxxxxxxxx"
}
}
})
node:lts-slim
is a public image so no need to provide auth for docker.io
Also if you have a docker credential helper installed on the system, and in your docker config that matches the registry we'll call it before giving up.
see auth information for more.
Install
npm install container-image-builder
Glossary
Learning about this distribution api I struggled mapping the names of some of the things to things I was familiar with.
Defined in the order that they compose into an "Image":
digest
- sha256 sum encoded as hex prefixed with "sha256:".
blob
- any file stored in a docker registry.
- B=but they are mostly 2 kinds. "config" json or "layer" tarball.
layer
- is a tarball.
- these tarballs are expanded at the root of the file system when run as part of an image as a container.
config
- a json document stored as a blob and referenced in the manifest.
- this hold details about the environment and what commands should be run when your image run as a container.
- this holds an array of the shasums of every layer in a property called diff_ids. I call this
uncompressedDigest
throughout this client. - I call it an uncompressedDigest because its the shasum of the layer before it's gzipped.
- because we need the uncompressed digest for every layer quite a few things in this api are less straight forward that you might expect.
manifest
- a json document that stores information about all of the layers in the image, and the digest of the config.
- each layer is a reference to a blob by digest along with contentLength and other optional properties.
- a manifest can have a tag which is a convenient way to name a specific version of the image.
Image
- is a combination of N layer blobs.
- one image config blob.
- and the manifest.
Container
- a process running an Image.
- docker and other container runners make images useful.
API
const {Image} = require('container-image-builder')
- or
import {Image} from 'container-image-builder'
in typescript etc.
Image builder API
image = new Image(baseImage:string,targetImage:string|ImageOptions,options:ImageOptions)
- baseImage the name of the base image. these are image specifiers just like you would pass to other container tools.
- targetImage the name of the image you're going to be saving to. calls to image.save() will replace this image.
- options
- see ImageOptions for options.
- see Detailed auth information for more about how to authenticate with any registry.
image.addFiles({[targetDirectory]:localDirectory},options): Promise<..>
tar local directories and place each at targetDirectory in a single layer.
symlinks are replaced with their files/directories as they are written to the layer tarball by default.
options
options.ignores
- optional set of globs applied at the root of the local directory processed by micromatch.
options.ignoreFiles
- optional array of file names to be parsed for ignore globs. like
[".ignore"]
.
- optional array of file names to be parsed for ignore globs. like
options.tar
- control how the tar is packed with options from node tar.
- control directory traversal with options from walkdir.
you can also call it with arguments like this but you should probably use the default.
image.addFiles(localDirectory,targetDirectory,options) :Promise<..>
image.addFiles(localDirectory,options) :Promise<..>
BREAKING CHANGE between 1x AND 2x
- positions of
targetDirectory
andlocalDirectory
were flipped in the object. - when paths are specified as an object the keys are now
targetDirectory
. - this enables copying the same files into a container to different paths and CustomFiles
- positions of
image.save(tags?: string[], options)
- save changes to the image. by default this saves the updated image as the
latest
tag tags
,string[]
- if you specify tags the manifest will be taged with each of them instead of the default.
options
- Cmd. see image.Cmd below
- Env. see image.Env below
- WorkingDir. see image.WorkingDir below
- copyRemoteLayers, default true
- set to false if you use nondistributable layers and cannot copy them to the target registry.
- layers with a urls field will be copied into the target registry by default from the first url to return a 200 in the array.
- ignoreExists, default false.
- primarily for debug/testing. this copies a blob into the target registry even though it already exists in the target
- save changes to the image. by default this saves the updated image as the
image.WorkingDir = "/custom working dir"
string
- If set this set the container's default working directory for commands.
image.Cmd = ["ls","./"]
string[]
- If set this will replace the base images default command with the one you specify.
image.Env = ["HOME=/workspace"]
string[]
- append environment variables to the base image's env vars.
image.getImageConfig() Promise<{ImageConfig}>
- returns reference to the part of the image config you're most likely to want to change.
- changes to this object are saved when you call
image.save()
image.getImageData() Promise<{manifest:ImageManifest,config:ImageConfig}>
- returns internal copies of the image manifest and config
- you need this if you are doing fancy things like replacing a specific layer
- warning its easy to make changes here that will create an image that does not work.
- changes to these objects will be saved when you call
image.save()
image.client(imageSpecifier,writePermission) Promise<RegistryClient>
- args are optional. uses a cached client if one already exists.
- return a promise to an authenticated registry client.
- auth is performed with options passed to new Image.
- imageSpecifier,
string
- just like you would pass to docker or the image constructor.
- default is the base image used in the constructor
image.sync(options)
- options. optional
- copyRemoteLayers, default true
- set to false if you use nondistributable layers and cannot copy them to the target registry.
- layers with a urls field will be copied into the target registry by default from the first url to return a 200 in the array.
- ignoreExists, default false.
- primarily for debug/testing. this copies a blob into the target registry even though it already exists in the target.
- manually trigger a sync from base image to target image registry.
- this copies blobs from one registry to another. you may want to use this for performance to start copying base image layers while you do other work.
- if the image has been synced it will not be synced in the call to
image.save()
image.addLayer(digest: string, uncompressedDigest: string, size: number,urls?: string[])
- digest required
- this is the id for the layer in the manifest
- uncompressedDigest required
- this is the id for the layer in the diff_ids array from the image config that maps to the layer.
- size required
- the number of bytes of the compressed layer tarball.
- urls, optional
- string[] of urls where we can download this layer tarball. used mostly for nondistributable layers.
- digest required
image.removeLayer(digest:string)
digest, string required
- the sha256 sum of the compressed layer tarball from a manifest.
remove the layer tagged in the manifest by digest. save it's offset in the array.
remove the uncompressedDigest from the image config that matches the offset above
const {CustomFile} = require('container-image-builder')
- you can pass CustomFile to image.addFiles as a localPath to write in memory data or a stream to the layer tarball.
image.addFiles({'/help.md':new CustomFile({data:Buffer.from('hello')})})
image.addFiles({'/google.html':new CustomFile({data:request('https://google.com'),size:**you must have size beforehand for streams**})})
- useful for creating whiteout files etc.
customFile = new CustomFile(options)
- options
- mode
- defaults to
0o644
owner read write, everyone read. - see fs.chmod
- permission bits are extracted from the mode provided via
& 0o7777
and type bits are set based on type.
- defaults to
- size
- required if stream. optional with buffer
- data
- a stream or buffer of data which will be the contents of the file.
- required if type is File
- type
- defaults to File
- supported are File, Directory, Symlink
- linkPath
- only used if Symlink
- mode
- options
customFile.uid
- default 0. set to number if you want to set uid
customFile.gid
- default 0. set to number if you want to set gid
customFile.ctime
- default new Date(). set to Date if you want to set
customFile.atime
- default new Date(). set to Date if you want to set
customFile.mtime
- default new Date(). set to Date if you want to set
ImageOptions
- ImageOptions is an object with the optional key auth. if
auth
is provided it will be used to provided authentication metadata for calls to the specified registry. If there is no authentication provider for your particular registry you can an object with the key token as it's value. {auth:{'my.registry.io':{token?:string}}}
- default support for docker.io and google container registry (gcr.io) are built in.
- if your registry has
gcr.io
in it you can pass additional options.
- ImageOptions is an object with the optional key auth. if
GCRAuthOptions
- {credentials?:{private_key:string,client_email:string},keyFilename?:string,token?:string}
- if no options are provided it falls back to using GOOGLE_APPLICATION_CREDENTIALS to provide the keyFilename
- the values for crdentials normally come from a key file like you would pass to keyFilename and download as service account credentials.
docker registry auth
const {auth} = require('container-image-builder')
Authenticate to docker registries. This library has built in support for Google Container Registery (gcr.io
) and docker hub.
for all other cases it'll fall back to using docker credential helpers already installed on your system.
auth(imageSpecifier,scope,options) Promise<{Secret?:string,Username?:string,token?:string}>
- imageSpecifier, string
- image location like you pass to docker.
- scope, string "push" or "push,pull"
- passed to the auth api if requesting docker hub or
gcr.io
.
- passed to the auth api if requesting docker hub or
- options
options['any.gcr.io/google-cloud-project']
options['any.gcr.io']
options['gcr.io'] = {...}
- see GCRAuthOptions for valid properties.
- you can also set the environment variable
GOOGLE_APPLICATION_CREDENTIALS=path to key file.json
and it will work as expected like other google client libraries. - if you need to authenticate to multiple GCR projects you can provide multiple sets of crdentials directly as auth options.
options['docker.io'] = {...}
- accepts these options. all are strings.
- either
token
is required orUsername
,Secret
is required. we'll either try Basic or Bearer auth depending on credentials provided. options.Secret
options.Username
options.token
options[ any other registry ] = {Username:string,Secret:string} or {token:string}
- if username and secret is provided the client will attempt to use http basic auth
- if token is provided it will use Bearer header auth.
- imageSpecifier, string
credential helpers
if docker is configured to look for credentials in credentials helpers authentication will be fetched from them automatically https://docs.docker.com/engine/reference/commandline/login/#credential-helpers
this library does not attempt to read base64 or credentials from a secret service/keychain on the host.
registry client API
As you get more creative you'll find you have to combine work done in the image builder with lower level api calls. like adding a new blob directly to the target registry before you call addLayer.
const RegistryClient = require('container-image-builder')
or
import {RegistryClient} from 'container-image-builder
client = new RegistryClient(registry, repository, auth)
- registry, string required
- repository, string required
- this is the image name including any namespace portion. so something like
my-google-project/my-app
- this is the image name including any namespace portion. so something like
- auth required
- auth.Username
- auth.Secret or
- auth.token
client.manifest(tag: string) Promise<Manifest>
- note: get the image manifest
- tag, string required
- this can be a tag name or a digest for manifests without tags.
client.tags() Promise<TagResult>
- note: list all tags associated with the image (repository)
- no arguments.
- TagResult
- child: any[];
- manifest: {};
- name: string;
- tags: string[];
client.manifestUpload(tag: string|false, manifest: Buffer|{}) Promise<{status: number, digest: string, body: Buffer}>
- tag, string| false
- if ! tag manifest will be uploaded untagged.
- manifest
- if manifest is a buffer it'll be uploaded as is.
- if manifest is an object it'll be json.stringified and uploaded.
- this does no verification at all. the registry api will fail if anything is wrong with it.
- tag, string| false
client.blobExists(digest) Promise<boolean>
- digest, string required
- the sha256sum of the blob.
- both layers and image configs are stored as blobs in the registry api
- digest, string required
client.blob(digest, stream) Promise<Buffer|Readable>
- note: download a blob as a buffer or stream
- digest, string required
- the sha256 sum of the blob you want to download
- stream, boolean
- default false
- if you'd like to download to a buffer or resolve to a readable stream.
client.upload(blob, contentLength, digest) Promise<{contentLength: number, digest: string}>
- note: upload a blob to the registry. you do not need to know the content length and digest before hand. if they're not provided they'll be calculated on the fly and a 2 step upload will be performed. It's more efficient if you know the contentLength and digest beforehand, but if you're streaming it can be more efficient to calculate on the fly.
- blob, Readable|Buffer required
- a readable stream or buffer.
- contentLength, number optional
- the size in bytes of the blob.
- digest, string optional
- sha256sum of the blob.
client.mount(digest: string, fromRepository: string)
- note: likely incomplete. this is much faster than alternatives if it works =)
- mount api has a neat feature where if the mount fails it'll give you a redirect header with the download url but this doesnt have the fallback bit implemented.
- this also may need work around auth. in Image make a client to download from one and pipe to and upload on two.
- digest, string required
- the sha256 sum of the target blob
- the repository to mount from.
Common actions
Authenticating to multiple GCR projects
this example copies 'gcr.io/project1/image' to 'us.gcr.io/project2/image' . The credentials value is the parsed JSON object from the file in GOOGLE_APPLICATION_CREDENTIALS
const creds1 = JSON.parse(fs.readFileSync(process.env.OTHER_GOOGLE_APPLICATION_CREDENTIALS))
const creds2 = JSON.parse(fs.readFileSync(process.env.OTHER_GOOGLE_APPLICATION_CREDENTIALS2))
const image = new Image('gcr.io/project1/image','us.gcr.io/project2/image',{
"auth":{
'us.gcr.io/project2':{credentials:creds1},
'gcr.io/project1':{credentials:creds2},
}
})
console.log(await image.save())
update env vars or exec path of a container
add a layer to an image
copy an image to another docker registry
Notes
This is not an official google product.