-
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathDocker-Container.html
More file actions
334 lines (296 loc) · 21.2 KB
/
Docker-Container.html
File metadata and controls
334 lines (296 loc) · 21.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta content="width=device-width, initial-scale=1.0" name="viewport">
<title>Portfolio Details</title>
<meta content="" name="description">
<meta content="" name="keywords">
<!-- Favicons -->
<link href="assets/img/Favicon-1.png" rel="icon">
<link href="assets/img/Favicon-1.png" rel="apple-touch-icon">
<!-- Google Fonts -->
<link href="https://fonts.googleapis.com/css?family=Open+Sans:300,300i,400,400i,600,600i,700,700i|Raleway:300,300i,400,400i,500,500i,600,600i,700,700i|Poppins:300,300i,400,400i,500,500i,600,600i,700,700i" rel="stylesheet">
<!-- Vendor CSS Files -->
<link href="assets/vendor/aos/aos.css" rel="stylesheet">
<link href="assets/vendor/bootstrap/css/bootstrap.min.css" rel="stylesheet">
<link href="assets/vendor/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
<link href="assets/vendor/boxicons/css/boxicons.min.css" rel="stylesheet">
<link href="assets/vendor/glightbox/css/glightbox.min.css" rel="stylesheet">
<link href="assets/vendor/swiper/swiper-bundle.min.css" rel="stylesheet">
<!-- Creating a python code section-->
<link rel="stylesheet" href="assets/css/prism.css">
<script src="assets/js/prism.js"></script>
<!-- Template Main CSS File -->
<link href="assets/css/style.css" rel="stylesheet">
<!-- To set the icon, visit https://fontawesome.com/account-->
<script src="https://kit.fontawesome.com/5d25c1efd3.js" crossorigin="anonymous"></script>
<!-- end of icon-->
<!-- =======================================================
* Template Name: iPortfolio
* Updated: Sep 18 2023 with Bootstrap v5.3.2
* Template URL: https://bootstrapmade.com/iportfolio-bootstrap-portfolio-websites-template/
* Author: BootstrapMade.com
* License: https://bootstrapmade.com/license/
======================================================== -->
</head>
<body>
<!-- ======= Mobile nav toggle button ======= -->
<i class="bi bi-list mobile-nav-toggle d-xl-none"></i>
<!-- ======= Header ======= -->
<header id="header">
<div class="d-flex flex-column">
<div class="profile">
<img src="assets/img/myphoto.jpeg" alt="" class="img-fluid rounded-circle">
<h1 class="text-light"><a href="index.html">Arun</a></h1>
<div class="social-links mt-3 text-center">
<a href="https://www.linkedin.com/in/arunp77/" target="_blank" class="linkedin"><i class="bx bxl-linkedin"></i></a>
<a href="https://github.com/arunp77" target="_blank" class="github"><i class="bx bxl-github"></i></a>
<a href="https://twitter.com/arunp77_" target="_blank" class="twitter"><i class="bx bxl-twitter"></i></a>
<a href="https://www.instagram.com/arunp77/" target="_blank" class="instagram"><i class="bx bxl-instagram"></i></a>
<a href="https://arunp77.medium.com/" target="_blank" class="medium"><i class="bx bxl-medium"></i></a>
</div>
</div>
<nav id="navbar" class="nav-menu navbar">
<ul>
<li><a href="index.html#hero" class="nav-link scrollto active"><i class="bx bx-home"></i> <span>Home</span></a></li>
<li><a href="index.html#about" class="nav-link scrollto"><i class="bx bx-user"></i> <span>About</span></a></li>
<li><a href="index.html#resume" class="nav-link scrollto"><i class="bx bx-file-blank"></i> <span>Resume</span></a></li>
<li><a href="index.html#portfolio" class="nav-link scrollto"><i class="bx bx-book-content"></i> <span>Portfolio</span></a></li>
<li><a href="index.html#skills-and-tools" class="nav-link scrollto"><i class="bx bx-wrench"></i> <span>Skills and Tools</span></a></li>
<li><a href="index.html#language" class="nav-link scrollto"><i class="bi bi-menu-up"></i> <span>Languages</span></a></li>
<li><a href="index.html#awards" class="nav-link scrollto"><i class="bi bi-award-fill"></i> <span>Awards</span></a></li>
<li><a href="index.html#professionalcourses" class="nav-link scrollto"><i class="bx bx-book-alt"></i> <span>Professional Certification</span></a></li>
<li><a href="index.html#publications" class="nav-link scrollto"><i class="bx bx-news"></i> <span>Publications</span></a></li>
<li><a href="index.html#extra-curricular" class="nav-link scrollto"><i class="bx bx-rocket"></i> <span>Extra-Curricular Activities</span></a></li>
<!-- <li><a href="#contact" class="nav-link scrollto"><i class="bx bx-envelope"></i> <span>Contact</span></a></li> -->
</ul>
</nav><!-- .nav-menu -->
</div>
</header><!-- End Header -->
<main id="main">
<!-- ======= Breadcrumbs ======= -->
<section id="breadcrumbs" class="breadcrumbs">
<div class="container">
<div class="d-flex justify-content-between align-items-center">
<h2>Portfolio Details</h2>
<ol>
<li><a href="portfolio-details-8.html" class="clickable-box">Go to Project page <i class="fas fa-arrow-right"></i></a></li>
<li><a href="index.html#portfolio" class="clickable-box">Go to portfolio</a></li>
</ol>
</div>
</div>
</section><!-- End Breadcrumbs -->
<!------ right dropdown menue ------->
<div class="right-side-list">
<div class="dropdown">
<button class="dropbtn"><strong>Shortcuts:</strong></button>
<div class="dropdown-content">
<ul>
<li><a href="cloud-compute.html"><i class="fas fa-cloud"></i> Cloud</a></li>
<li><a href="AWS-GCP.html"><i class="fas fa-cloud"></i> AWS-GCP</a></li>
<li><a href="amazon-s3.html"><i class="fas fa-cloud"></i> AWS S3</a></li>
<li><a href="ec2-confi.html"><i class="fas fa-server"></i> EC2</a></li>
<li><a href="Docker-Container.html"><i class="fab fa-docker" style="color: rgb(8, 8, 8);"></i> Docker</a></li>
<li><a href="Jupyter-nifi.html"><i class="fab fa-python" style="color: rgb(8, 8, 8);"></i> Jupyter-nifi</a></li>
<li><a href="snowflake-task-stream.html"><i class="fas fa-snowflake"></i> Snowflake</a></li>
<li><a href="data-model.html"><i class="fas fa-database"></i> Data modeling</a></li>
<li><a href="sql-basics.html"><i class="fas fa-table"></i> QL</a></li>
<li><a href="sql-basic-details.html"><i class="fas fa-database"></i> SQL</a></li>
<li><a href="Bigquerry-sql.html"><i class="fas fa-database"></i> Bigquerry</a></li>
<li><a href="scd.html"><i class="fas fa-archive"></i> SCD</a></li>
<li><a href="sql-project.html"><i class="fas fa-database"></i> SQL project</a></li>
<!-- Add more subsections as needed -->
</ul>
</div>
</div>
</div>
<!-- ======= Portfolio Details Section ======= -->
<section id="portfolio-details" class="portfolio-details">
<div class="container">
<div class="row gy-4">
<h1>Installing Docker and Creating a Container with Python, Apache NiFi, and Apache ZooKeeper</h1>
<div class="image">
<img src="assets/img/portfolio/docker-logo.png" alt="Image Description" width="300" height="200">
</div>
<h3>Introduction</h3>
<ul>
<p>In today's fast-paced world of software development and deployment, efficiency and consistency are paramount. Docker, a leading containerization platform,
has revolutionized the way we build, package, and distribute applications. It offers a lightweight, portable, and efficient solution for managing and running
applications in isolated environments known as containers.</p>
<p>Whether you're a developer, system administrator, or IT professional, understanding Docker and how to use it can greatly enhance your workflow. This
comprehensive guide will take you through the process of installing Docker on a Linux operating system and creating a Docker container that includes
Python, Apache NiFi, and Apache ZooKeeper. By the end, you'll have the knowledge and skills to harness the power of containerization for your projects.</p>
<h3>What is Docker?</h3>
<p>Before we dive into the installation and setup, let's briefly explore what Docker is and why it has become a cornerstone of modern software development and deployment.</p>
<p>Docker is an open-source platform designed to automate the deployment, scaling, and management of applications within lightweight, portable, and self-sufficient containers.
These containers encapsulate everything an application needs to run, including code, libraries, dependencies, and runtime environments. Unlike traditional
virtual machines, Docker containers share the host OS kernel, which makes them extremely efficient and fast to start.</p>
<p>Docker provides an environment that enables developers to build, test, deploy, and run applications by using containerization technology.</p>
<p>The key benefits of Docker include:</p>
<ol style="list-style-type: disc; margin-left: 30px;">
<li><b>Isolation:</b> Containers run in isolated environments, ensuring that applications don't interfere with each other.</li>
<li><b>Consistency:</b> Docker ensures that an application behaves the same way in development, testing, and production environments.</li>
<li><b>Portability:</b> Containers can run on any system that supports Docker, making it easy to move applications between different environments.</li>
<li><b>Scalability:</b> Docker makes it simple to scale applications up or down as needed, adapting to varying workloads.</li>
</ol>
<p>In essence, Docker simplifies the development and deployment process, making it easier to package, distribute, and run applications consistently,
whether you're working on a small-scale project or managing complex microservices architectures.</p>
</ul>
<h3>Prerequisites</h3>
<ul style="list-style-type: disc; margin-left: 30px;">
<li>You have installed the latest version of <a href="https://docs.docker.com/get-docker/" target="_blank">Docker Desktop</a></li>
<li>You have installed a <a href="https://git-scm.com/downloads" target="_blank">Git client</a></li>
<li>You have an IDE or a text editor to edit files. I normally use VS studio code. You can use nano or vi editor. In ubuntu systems, you can you 'gedit'.</li>
<li>On windows systems, you may need to install <a href="https://learn.microsoft.com/en-us/windows/wsl/install" target="_blank">Windows Subsystem for Linux (WSL)</a> to run the
linux like commands and access docker on windows systems.</li>
<li>For this, you can just find the linux-distributio (of your choice) in search bar, and click it. A new command prompt will open, where you may have
to provide the username and password (set at the time of installation as discussed in above link for WSL). </li>
</ul>
<h3>Docket installation</h3>
<ul style="list-style-type: disc; margin-left: 30px;">
<li>Visit the <a href="https://docs.docker.com/engine/install/" target="_blank">Docker download page</a> and choose the version compatible to your operating system.</li>
<li>After the installation file is downloaded on your local system, install it as per the suggestions given on the above link.</li>
<li>Once Docker is installed, launch Docker Desktop. You will see the Docker icon in the system tray when it's running. Next create a account and login into it at the
docker desktop.</li>
<li>Create a directory on your system where you'll work on your Docker project. You can use File Explorer or the command prompt for this.</li>
<li>Inside your project directory, create a file named "Dockerfile" (without a file extension) using a text editor or the command prompt.</li>
<li>Open the Dockerfile in your preferred text editor.</li>
<li><b>Create a Dockerfile:</b></li>
<ul style="list-style-type: disc; margin-left: 30px;">
<li>Create a directory on your local system where you'll work on your Docker project. You can use the command prompt for this.</li>
<li>Inside your project directory, create a file named 'Dockerfile' (without a file extension) using a text editor or the command prompt.</li>
<li>Open the 'Dockerfile' in your preferred text editor.</li>
<li>In the 'Dockerfile', define the instructions to build your container. Save the Dockerfile. For example:</li>
<pre><code>
FROM node:18-alpine
WORKDIR /app
COPY . .
RUN yarn install --production
CMD ["node", "src/index.js"]
EXPOSE 3000
</code></pre>
<li>Build the image using the following commands:</li>
<pre><code>docker build -t my-image .</code></pre>
<li>Start an app container:</li>
<pre><code>docker run -d -p 3000:3000 my-image</code></pre>
<li>After a few seconds, open your web browser to http://localhost:3000.</li>
</ul>
<li><strong>Installation via a command line on linux operating system (UBUNTU):</strong></li>
<ul style="list-style-type: disc; margin-left: 30px;">
<li></li>
</ul>
<h3>Installation of Jupyerlab, apache nifi, zookeeper on EC2 machine</h3>
<p>We can do this in following steps:</p>
<p><b>Step-1:</b> Creating a customized Docketfile.</p>
<p>A docker-compose.yml file is a configuration file used with Docker Compose, a tool that simplifies the management of multi-container Docker applications.
This file defines the services, networks, and volumes for your application, allowing you to specify how different Docker containers interact and share resources</p>
<ul style="list-style-type: disc; margin-left: 30px;">
<li>To create a Docker container that includes Jupyter Lab with the specified ports, ZooKeeper, and NiFi with the specified ports,
you can use a "docker-compose.yml" file to define and run these services together.</li>
<li> You can create a docket-compose.yml and save the docker-compose.yml file to your working directory: </li>
<pre><code>
version: "3.6"
volumes:
shared-workspace:
name: "hadoop-distributed-file-system"
driver: local
services:
jupyterlab:
image: jupyter/datascience-notebook # Other options:jupyter/base-notebook, or scipy-notebook, or all-spark-notebook or scipy-notebook
container_name: jupyterlab
ports:
- 4888:4888
- 4040:4040
- 8050:8050
volumes:
- /path/on/host/jupyterlab:/path/in/container
zookeeper:
image: bitnami/zookeeper
container_name: zookeeper
ports:
- 2181:2181
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
volumes:
- /path/on/host/zookeeper:/path/in/container
nifi:
image: apache/nifi:1.14.0 # Use an appropriate NiFi image
container_name: nifi
ports:
- 2080:2080
environment:
- NIFI_WEB_HTTP_PORT=2080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=2084
- NIFI_ZK_CONNECT_STRING=zookeeper:2181
- NIFI_ELECTION_MAX_WAIT=1 min
- NIFI_SENSITIVE_PROPS_KEY=vvvvvvvvvvvv
volumes:
- /path/on/host/nifi:/path/in/container
</code></pre>
<ul style="list-style-type: circle;">
<li>The version specifies the version of the Docker Compose file.</li>
<li>Under the services section, we define three services: jupyterlab, zookeeper, and nifi.</li>
<li>Replace `/path/on/host` with the path on your host (EC2 instance) where you want to create the volumes</li>
<li>For each service: image specifies the Docker image to use for that service. </li>
<li>`container_name` sets the name for the container.</li>
<li>`ports` map the host machine's ports to the container's ports. The format is `host_port:container_port`.</li>
<li>For the zookeeper service, we set the environment variable ALLOW_ANONYMOUS_LOGIN to "yes" to allow anonymous logins. Adjust environment
variables as needed for your specific ZooKeeper and NiFi configurations.</li>
</ul>
</ul>
<p><b>Step-2: </b> Transfer the docker-compose.yml File</p>
<ul style="list-style-type: disc; margin-left: 30px;">
<li>Open a terminal and navigate to the directory where the docker-compose.yml is located.</li>
<li>Transfer the `docker-compose.yml` file to your EC2 instance using `SCP` or a similar method. Replace [`your-key.pem`] with the path to your
private key and [`your-ec2-instance-ip`] with the public IP or DNS name of your instance.</li>
<li>Run the following command to start the services:</li>
<pre><code>scp -i [your-key.pem] docker-compose.yml ec2-user@[your-ec2-instance-ip]:/path/to/destination</code>
</pre>
<ul style="list-style-type: circle;">
<p>It is to be noted here that the `.pem` file is obtained, when you set the IAM user at yout AWS account. You need to save this file in a secret place
and never shared it with anyone. </p>
<p>It has normally two values: {"Key": "ExampleKey", "Value": "ExampleValue"}'</p>
</ul>
<li>To start the container, you just need to run following command in the ec2 instance:</li>
<pre><code>docker-compose up</code></pre>
<li>The Jupyterlab and the APache nifi can be access in your browser using:</li>
<pre><code>Jupyter Lab at: http://ip_address:4888/lab? </code></pre>
and
<pre><code>NiFi at: http://ip_address:2080/nifi/</code></pre>
<li>Here `ip_address` is the ec2 machine IP.</li>
</ul>
<a href="portfolio-details-8.html" class="clickable-box">Go back</a>
</div>
</div>
</section><!-- End Portfolio Details Section -->
</main><!-- End #main -->
<!-- ======= Footer ======= -->
<footer id="footer">
<div class="container">
<div class="copyright">
© Copyright <strong><span>Arun</span></strong>
</div>
</div>
</footer><!-- End Footer -->
<a href="#" class="back-to-top d-flex align-items-center justify-content-center"><i class="bi bi-arrow-up-short"></i></a>
<!-- Vendor JS Files -->
<script src="assets/vendor/purecounter/purecounter_vanilla.js"></script>
<script src="assets/vendor/aos/aos.js"></script>
<script src="assets/vendor/bootstrap/js/bootstrap.bundle.min.js"></script>
<script src="assets/vendor/glightbox/js/glightbox.min.js"></script>
<script src="assets/vendor/isotope-layout/isotope.pkgd.min.js"></script>
<script src="assets/vendor/swiper/swiper-bundle.min.js"></script>
<script src="assets/vendor/typed.js/typed.umd.js"></script>
<script src="assets/vendor/waypoints/noframework.waypoints.js"></script>
<script src="assets/vendor/php-email-form/validate.js"></script>
<!-- Template Main JS File -->
<script src="assets/js/main.js"></script>
<script>
document.addEventListener("DOMContentLoaded", function () {
hljs.initHighlightingOnLoad();
});
</script>
</body>
</html>