API guide - Reference
Access Control HTTP Handers
Caching un API Calls
HTTP Request Methods
REST API
API guide - Reference
Access Control HTTP Handers
Ansible Automation
Ansible is simply an open-source IT engine that automates application deployment, intra service
orchestration, cloud provisioning, and many other IT tools.
Ansible is easy to deploy because it does not use any agents or custom security infrastructure
Ansible uses playbook to describe automation jobs, and playbook uses very simple language i.e. YAML (It’s a human-readable data serialization language & is commonly used for configuration files, but could be used in many applications where data is being stored) which is very easy for humans to understand, read and write. Hence the advantage is that even the IT infrastructure support guys can read and understand the playbook and debug if needed (YAML – It is in human-readable form)
Ansible is designed for multi-tier deployment. Ansible does not manage one system at a time, it models IT infrastructure by describing all of your systems are interrelated. Ansible is completely agentless which means Ansible works by connecting your nodes through ssh(by default). But if you want another method for connection like Kerberos, Ansible gives that option to you
After connecting to your nodes, Ansible pushes small programs called “Ansible Modules”. Ansible runs those modules on your nodes and removes them when finished. Ansible manages your inventory in simple text files (These are the hosts file). Ansible uses the hosts file where one can group the hosts and can control the actions on a specific group in the playbooks
Advantages of Ansible
Free: Ansible is an open-source tool.
Very simple to set up and use: No special coding skills are necessary to use Ansible’s
playbooks (more on playbooks later).
Powerful: Ansible lets you model even highly complex IT workflows
Flexible: You can orchestrate the entire application environment no matter where it’s deployed.
You can also customize it based on your needs.
Agentless: You don’t need to install any other software or firewall ports on the client systems
you want to automate. You also don’t have to set up a separate management structure. Efficient: Because you don’t need to install any extra software, there’s more room for
application resources on your server.
What is Configuration Management? Configuration management in terms of Ansible means that it maintains the configuration of the product performance by keeping a record and updating detailed information that describes an enterprise’s hardware and software
How Ansible Works? The picture given below shows the working of Ansible. Ansible works by connecting to your nodes and pushing out small programs, called "Ansible Modules" to them. Ansible then executes these modules (over SSH by default) and removes them when finished. Your library of modules can reside on any machine, and there are no servers, daemons, or databases required
Understanding YAML
In this section, we will learn the different ways in which the YAML data is represented.
key-value pair
YAML uses simple key-value pairs to represent the data. The dictionary is represented in key: value pair.
Note − There should be space between: and value
Abbreviation You can also use abbreviation to represent dictionaries.
Example James: {name: ram shing, rollNo: 34, div: C, sex: male}
Representing List We can also represent List in YAML. Every element(member) of the list should be written in a new line with the same indentation starting with “- “ (- and space)
Example --- countries:
- America
- China
- Canada
- Iceland
Abbreviation You can also use the abbreviation to represent lists.
Example Countries: [‘America’, ‘China’, ‘Canada’, ‘Iceland’]
List of Dictionaries We can also make a list of dictionaries
YAML uses “|” to include newlines while showing multiple lines and “>” to suppress newlines while showing multiple lines. Due to this we can read and edit large lines. In both cases,
the indentation will be ignored. We can also represent Boolean (True/false) values in YAML. where boolean values can be case insensitive.
Some common words related to Ansible.
Service/Server − A process on the machine that provides the service.
Machine − A physical server, VM (virtual machine), or a container.
Target machine − A machine we are about to configure with Ansible.
Task − An action (run this, delete that), etc. managed by Ansible.
Playbook − The YML file where Ansible commands are written and YML is executed on a machine. Ansible.cfg – ansible configuration file
Ansible - Ad hoc Commands
Ad hoc commands are commands which can be run individually to perform quick functions. These
commands need not be performed later.
For example, you have to reboot all your company servers.
For this, you will run the Adhoc commands from ‘/usr/bin/ansible’.
These ad-hoc commands are not used for configuration management and deployment,
because these commands are of one-time usage. ansible-playbook is used for configuration management and deployment.
Parallelism and Shell Commands
Reboot your company server in 12 parallel forks at a time.
For this, we need to set up SSHagent for connection.
$ ssh-agent bash $ ssh-add ~/.ssh/id_rsa
To run reboot for all your company servers in a group, 'abc', in 12 parallel forks −
$ Ansible abc -a "/sbin/reboot" -f 12
By default, Ansible will run the above Ad-hoc commands from the current user account.
If you want to change this behavior, you will have to pass the username in Ad-hoc commands as follows − $ Ansible abc -a "/sbin/reboot" -f 12 -u username
File Transfer
You can use the Ad-hoc commands for doing SCP (Secure Copy Protocol) lots of files in parallel on multiple machines. Transferring file to many servers/machines
$ Ansible abc -m copy -a "src = /etc/yum.conf dest = /tmp/yum.conf"
File Transfer 9 You can use the Ad-hoc commands for doing SCP (Secure Copy Protocol) lots of files in parallel on multiple machines.
Transferring file to many servers/machines
$ Ansible abc -m copy -a "src = /etc/yum.conf dest = /tmp/yum.conf"
Creating a new directory
$ Ansible abc -m file -a "dest = /path/user1/new mode = 777 owner = user1 group =
user1 state = directory"
Deleting whole directory and files
$ Ansible abc -m file -a "dest = /path/user1/new state = absent" Managing Packages
Managing Packages
The Ad-hoc commands are available for yum and apt. Following are some Ad-hoc commands using yum. The following command checks if yum package is installed or not, but does not update it.
$ Ansible abc -m yum -a "name = demo-tomcat-1 state = present"
The following command checks the package is not installed.
$ Ansible abc -m yum -a "name = demo-tomcat-1 state = absent"
The following command checks the latest version of the package is installed.
$ Ansible abc -m yum -a "name = demo-tomcat-1 state = latest"
Gathering Facts
Facts can be used for implementing conditional statements in playbook.
You can find adhoc information of all your facts through the following Ad-hoc command −
$ Ansible all -m setup
Ansible – Playbooks
Ansible – Role
Roles provide a framework for fully independent, or interdependent collections of variables, tasks, files,
templates, and modules.
In Ansible, the role is the primary mechanism for breaking a playbook into multiple files. This simplifies
writing complex playbooks, and it makes them easier to reuse. The breaking of playbook allows you to
logically break the playbook into reusable components.
Creating a New Role
The directory structure for roles is essential to create a new role.
Role Structure
Roles have a structured layout on the file system. The default structure can be changed but for now let
us stick to defaults.
Each role is a directory tree in itself. The role name is the directory name within the /roles directory.
$ ansible-galaxy -h
Usage
ansible-galaxy [delete|import|info|init|install|list|login|remove|search|setup] [--
help] [options] ...
Options
-h, --help − Show this help message and exit.
-v, --verbose − Verbose mode (-vvv for more, -vvvv to enable connection debugging)
--version − Show program's version number and exit.
Creating a Role Directory The above command has created the role directories.
$ ansible-galaxy init Emamrole
ERROR! The API server (https://galaxy.ansible.com/api/) is not responding, please
try again later.
$ ansible-galaxy init --force --offline Emamrole
- Emamrole was created successfully
$ tree Emamrole/
Emamrole/
├── defaults │ └── main.yml
├── files ├── handlers
│ └── main.yml
├── meta │ └── main.yml
├── README.md ├── tasks │
└── main.yml ├── templates ├── tests │ ├── inventory
│ └── test.yml
└── vars
└── main.yml
8 directories, 8 files
Not all the directories will be used in the example and we will show the use of some of them in the example
Utilizing Roles in Playbook
This is the code of the playbook we have written for demo purpose.
This code is of the playbook Emam_orchestrate.yml.
We have defined the hosts: tomcat-node and called the two roles – installtomcat and start-tomcat.
The problem statement is that we have a war which we need to deploy on a machine via Ansible.
---
- hosts: tomcat-node
roles:
- {role: install-tomcat}
- {role: start-tomcat}
Contents of our directory structure from where we are running the playbook.
$ ls
ansible.cfg hosts roles Emam_orchestrate.retry Emam_orchestrate.yml
There is a tasks directory under each directory and it contains a main.yml.
The main.yml contents of install-tomcat are −
--- #Install Emam artifacts
- block:
- name: Install Tomcat artifacts
action: >
yum name = "demo-tomcat-1" state = present
register: Output
always: - debug: msg: - "Install Tomcat artifacts task ended with message: {{Output}}"
- "Installed Tomcat artifacts - {{Output.changed}}"
The contents of main.yml of the start tomcat are
− #Start Tomcat
- block:
- name: Start Tomcat
command:
The deployed war just has a servlet which displays “Hello World”.
The detailed output shows the time
taken by each and every task because of the entry added in ansible.cfg file
Ansible - Variables
Variable in playbooks are very similar to using variables in any programming language. It helps you to use and assign a value to a variable and use that anywhere in the playbook. One can put conditions around the value of the variables and accordingly use them in the playbook
Exception Handling in Playbooks
Exception handling in Ansible is similar to exception handling in any programming language. An
example of the exception handling in playbook is shown below.
tasks:
- name: Name of the task to be executed
block:
- debug: msg = 'Just a debug message , relevant for logging'
- command: <the command to execute>
rescue:
- debug: msg = 'There was an exception.. '
- command: <Rescue mechanism for the above exception occurred)
always:
- debug: msg = "this will execute in all scenarios. Always will get
logged"
Following is the syntax for exception handling.
Loops in Ansible.
Below is the example to demonstrate the usage of Loops in Ansible. The tasks is to copy the set of all the war files from one directory to tomcat webapps folder. Most of the commands used in the example below are already covered before. Here, we will concentrate on the usage of loops.
To loop, the 'with_items' syntax is being used. with_items: "{{output.stdout_lines}}" --> output.stdout_lines gives us the line by line output and then we loop on the output with the with_items command of Ansible. Attaching the example output just to make one understand how we used the stdout_lines in the with_items command.
--- #Testing
- hosts: tomcat-node
tasks: - name: Install Apache
shell: "ls *.war"
register: output
args: chdir: /opt/ansible/tomcat/demo/webapps
- file: src: '/opt/ansible/tomcat/demo/webapps/{{ item }}'
dest: '/users/demo/Emam/{{ item }}'
state: link
with_items: "{{output.stdout_lines}}"
Ansible - Advanced Execution
we will learn what is advanced execution with Ansible.
How to Limit Execution by Tasks?
This is a very important execution strategy where one needs to execute only one execution and not the entire playbook. For example, suppose you only want to stop a server
(in case a production issue comes) and then post applying a patch you would like to only start the server. Here in original playbook stop and start were a part of different roles in the same playbook but this can be handled with the usage of tags. We can provide different tags to different roles
(which in turn will have tasks) and hence based on the tags provided by the executor only that specified role/task gets executed. So for the above example provided, we can add tags like the following
− - {role: start-tomcat, tags: ['install']}}
The following command helps in using tags
−
ansible-playbook -i hosts
Common Playbook Issues
we will learn about the a few common playbook issues.
The issues are −
Quoting
Indentation Playbook is written in yaml format and the above two are the most common issues in yaml/playbook.
Yaml does not support tab based indentation and supports space based indentation,
so one needs to be careful about the same.
Note − once you are done with writing the yaml ,
goto this site(https://editor.swagger.io/) and copy
paste your yaml on the left hand side to ensure that the yaml compiles properly.
This is just a tip.
Swagger qualifies errors in warning as well as error & issue if any.................
Resources
https://www.ansible.com/
https://docs.ansible.com/ansible/latest/index.html
https://en.wikipedia.org/wiki/Ansible_(software)
https://github.com/ansible/ansible https://opensource.com/resources/what-ansible https://www.simplilearn.com/tutorials/ansible-tutorial/what-is-ansible
https://www.guru99.com/ansible-tutorial.html
https://hackr.io/tutorials/learn-ansible
Docker All in one essentials
Content cover points:
Prerequisites
Linux
MacOS
Installation
Linux
Mac OS X
Containers
Lifecycle
Starting and Stopping
Information on Docker Containers, Processes and Performance
Import / Export (Backup / Restore)
Executing Commands
Images
Lifecycle of Containers (Create, Run, Build, Commit)
Info
Cleaning up
Images Created by Redirection
Import/Export Container
Private and Public Registries/Repositories
Running a Local Registry
Dockerfile
Layers
Links Between Containers
Volumes
Docker Volume Lifecycle
Volume Information
Exposing Ports
Security Considerations
Security Tips
Prerequisites
Linux • The 3.10.x kernel or newer
MacOS • 10.8 “Mountain Lion” or newer
Installation
Linux Install script provided by Docker:
curl -sSL https•//get.docker.com/ | sh
If unwilling to run a random shell script, please see the installation instructions for your distribution
Mac OS X
Download and install Docker Toolbox. If this fails, see the installation instructions.
If you have an existing Docker Toolbox, you may think you can upgrade Docker Machine binaries directly (either from URL or docker-machine upgrade default).
This does not work. Your Docker Machine will be 1.10.3, while Docker itself remains at its previous version
Once you’ve installed Docker Toolbox, install a VM with Docker Machine using the VirtualBox provider:
docker-machine create --driver=virtualbox default
docker-machine ls
eval "$(docker-machine env default)"
Then start up a container: docker run hello-world That’s it; you have a running Docker container
Containers
Your basic isolated Docker process. Containers are to virtual machines, as threads are to processes. Or you can think of them as larger-than-life chroot environments
Lifecycle
• docker create • Creates a container but does not start it
• docker rename • Allows the container to be renamed
• docker run • Creates and starts a container in one operation
• docker rm • Deletes a container
• docker update • Updates a container’s resource limits
• docker run --rm • Removes container when stopped
• docker run -v $HOSTDIR•$DOCKERDIR • Maps a directory on the host to the Docker container; see also
• Volumes
• docker rm -v • Removes volumes associated with container
• docker run --log-driver=syslog • Runs Docker with custom log driver Starting and Stopping
• docker start • Starts a container, so it is running
• docker stop • Stops a running container
• docker restart • Stops and starts a container
• docker pause • Pauses a running container, “freezing” it in place
• docker unpause • Unpauses a running container
• docker wait • Blocks until running container stops
• docker kill • Sends a SIGKILL to a running container
• docker attach • Connects to a running container
If you want to integrate a container with a host process manager, start the daemon with -r=false then use docker start -a. If you want to expose container ports through the host, see the exposing ports section.
• docker ps • Shows running containers
• docker logs • Gets logs from container; you can use a custom log driver, but logs are only available for json-file and journald in 1.10
• docker inspect • Looks at all the info on a container (including IP address) • docker events • Gets events from container
• docker port • Shows public facing port of container
• docker top • Shows running processes in container
• docker stats • Shows containers’ resource usage statistics
• docker diff • Shows changed files in the container’s filesystem
• docker ps -a • Shows running and stopped containers
• docker stats --all • Shows a running list of containers Import / Export (Backup / Restore)
• docker cp • Copies files or folders between a container and the local filesystem
• docker export • Turns container filesystem into tarball archive stream to STDOUT Executing Commands
• docker exec • Executes a command in container
To enter a running container, attach a new shell process to a running container called foo, use:
docker exec -it foo /bin/bash.
Images
Images are templates that Docker containers are based on.
They are the foundational layer from which your container is launched, and your changes then become independent from it (as another layer).
Lifecycle of Containers (Create, Run, Build, Commit)
• docker images • Shows all images
• docker import • Creates an image from a tarball
• docker build • Creates image from Dockerfile
• docker commit • Creates image from a container, pausing it temporarily if it is running
• docker rmi • Removes an image
• docker load • Loads an image from a tar archive as STDIN, including images and tags (as of 0.7) • docker save • Saves an image to a tar archive stream to STDOUT with all parent layers, tags and versions (as of 0.7)
• docker history • Shows history of image
• docker tag • Tags an image to a name (local or registry)
While you can use the docker rmi command to remove specific images, there’s a tool called docker[1]gci that will clean up images that are no longer used by any containers in a safe manner.
Images Created by Redirection Load an image from file:
docker load < my_image.tar.gz
Save an existing image:
docker save my_image•my_tag > my_image.tar.gz
Import/Export Container
Import a container as an image from file:
cat my_container.tar.gz | docker import - my_image•my_tag
Export an existing container:
docker export my_container > my_container.tar.gz
Differences between loading a saved image and importing an exported container as an image:
• Loading an image using the load command creates a new image, including its history.
• Importing a container as an image using the import command creates a new image, excluding the history which results in a smaller image size compared to loading an image
Private and Public Registries/Repositories
A repository is a hosted collection of tagged images that, together, create the file system for a container.
A registry is a host -- a server that stores repositories and provides an HTTP API for managing the uploading and downloading of repositories.
Docker.com hosts its own index to a central registry (the Docker Hub) which contains a large number of repositories.
• docker login • Logs into a registry
• docker logout • Logs out from a registry
• docker search • Searches registry for image
• docker pull • Pulls an image from registry to local machine
• docker push • Pushes an image to the registry from local machine.
Running a Local Registry
You can run a local registry by using the docker distribution project and looking at the local deployment instructions.
The configuration file. Sets up a Docker container when you run docker build on it
• Sections/Directives in a Dockerfile:
» .dockerignore • Files and directories to be ignored during the build -t of the Dockerfile
» FROM • Sets the base image for subsequent instructions
» MAINTAINER • Sets the Author field of the generated images
» RUN • Executes any commands in a new layer on top of the current image and commits the results
» CMD • Provides defaults for an executing container
» EXPOSE • Informs Docker that the container listens on the specified network ports at runtime; does not make ports accessible
» ENV • Sets environment variables
» ADD • Copies new files, directories or remote file to container; invalidates caches; avoid ADD and use COPY instead
» COPY • Copies new files or directories to container
» ENTRYPOINT • Configures a container that will run as an executable
» VOLUME • Creates a mount point for externally-mounted volumes or other containers
» USER • Sets the username for following RUN/CMD/ENTRYPOINT commands
» WORKDIR • Sets the working directory
» ARG • Defines a build-time variable
» ONBUILD • Adds a trigger instruction when the image is used as the base for another build
» STOPSIGNAL • Sets the system call signal that will be sent to the container to exit
» LABEL • Apply key/value metadata to your images, containers, or daemons
Layers
The versioned filesystem in Docker is based on layers. They’re like Git commits or changesets for filesystems.
Links Between Containers
Links are how Docker containers talk to each other through TCP/IP ports. As of 0.11, you can resolve links by hostname. If you want containers only to communicate with each other through links, start the docker daemon with -icc=false to disable interprocess communication.
If you have a container with the name CONTAINER (specified by docker run --name CONTAINER) and in the Dockerfile, it has an exposed port: EXPOSE 8080
Then if we create another container called LINKED:
docker run -d --link CONTAINER•ALIAS --name LINKED user/example
The exposed ports and aliases of CONTAINER will show up in LINKED with the following environment variables
$ALIAS_PORT_8080_TCP_ADDR
And you can connect to it that way. To delete links, use docker rm –link
Volumes
Docker volumes are free-floating filesystems.
They don’t have to be connected to a particular container. You could use volumes mounted from data-only containers for portability.
Docker Volume Lifecycle
• docker volume create
• docker volume rm Volume Information
• docker volume ls
• docker volume inspect
Volumes are useful in situations where you can’t use links (which are TCP/IP only). For instance, if you need to have two Docker instances communicate by leaving items on the filesystem
You can mount them in several Docker containers at once, using docker run --volumes-from.
Because volumes are isolated filesystems, they are often used to store states from computations between transient containers.
That is, you can have a stateless and transient container run from a recipe/playbook/ manifest, blow it away, and then have a second instance of the transient container pick up from where the last one left off
Exposing Ports
This is done by mapping the container port to the host port (only using localhost interface, for example) using -p
• docker run -p 127.0.0.1•$HOSTPORT•$CONTAINERPORT --name CONTAINER -t someimage
You can tell Docker that the container listens on the specified network ports at runtime by using
EXPOSE:
EXPOSE <CONTAINERPORT
EXPOSE does not expose the port itself, only -p will do that. To expose the container’s port on your localhost’s port, see above, and add the appropriate ports to your firewall ruleset, as needed. If you forget what you mapped the port to on the host container, use docker port to show it
• docker port CONTAINER $CONTAINERPOR
Security Considerations
Docker runs as root. If you are in the Docker group, you effectively have root access. If you expose the Docker Unix socket to a container, you are giving the container root access to the host.
Docker should not be your only defense. You should secure and harden it
Security Tips
For the greatest security, you want to run Docker inside a virtual machine. Then, run with AppArmor/ seccomp/SELinux/grsec, etc. to limit the container permissions. See the Docker 1.10 security features for more details.
Docker image IDs are sensitive information, and should not be exposed to the outside world. Treat them like passwords. Since Docker 1.11 you can easily limit the number of active processes running inside a container to prevent fork bombs. This requires Linux kernel 4.3 or higher with CGROUP_PIDS=y in the kernel configuration.
docker run --pids-limit=64
Also available since Docker 1.11 is the ability to prevent processes to gain new privileges. This feature is in the Linux kernel since version 3.5.
docker run --security-opt=no-new-privileges
Turn off interprocess communication:
docker -d --icc=false --iptables
Set the container to be read-only: docker run --read-only
docker pull debian@sha256•a25306f3850e1bd44541976aa7b5fd0a29
be Set volumes to be read only:
docker run -v $(pwd)/secrets•/secrets•ro debian
Set memory and CPU sharing:
docker -c 512 -mem 512m
Define and run a user in your Dockerfile,
so you don’t run as root inside the container:
RUN groupadd -r user && useradd -r -g user user
REF : Docker Documentation | Docker Documentation
https://docs.docker.com/
Comparison of GCP Anthos, AWS Outpost and Azure Arc/Azure Stack
How to Install Kubernetes How to Install Kubernetes? What is a Pod? previously , I discussed on what is Kubernetes, and its architecture. He...