Amazon Web Service

1. Introduction:-

We will require designation such as AWS Certified Solution Architect, Cloud Practioner, DevOps.

What are the things we will learn in this course:-

How we can use Services in Real life?

We can learn how to design the system.

At which time, which service you need to use

AWS -> NoSQL Databse is DynamoDB

MongoDB AWS -> use EFS otherwise use NFS Global Infra Region, Availability Zone

2. Traditional VS Cloud Computing

Traditional:- If you are creating website or a application for your company and you want that anyone can able to access the website. First you need to buy a server which is very costly, then you need to store the server in a garage, you need to take garage as a rent. The server needs cold environment, you need to buy AC. The server needs to be online in 24*7, then you need to hire a employee who will do the maintenance. Sometimes the hard disk, RAM of the server gets corrupted. Also I have an expect of 10k users will access my website but now only 10 users have an access for the website. Consider if there are 10 crore users have an access for my website then the CPU gets stuck (give late response).

There is another friend (AWS) of mine. He said to me that just you need to create a website while I manage your infra. I have told him what service you will give to me. He said that I will create a machine for you which handle 10k users. If your user get increases, then I will create other server. He said that I will maintain your server in 24*7. He said that I will take charges from you for per sec or per hour or you can pay to me as per your time utilization to access server (pay as you go).

Using web browser of AWS, you can select resources on demand basis. This things happened in data center. You can learn how to access your resource like server, database. Where is my server is located. You can use Cloud on daily basis like files storing on google drive, access gmail using email service, seeing movies on netflix

3. Cloud Service Computing Models:-

You can identify which service and what it does?

a)Infrastructure as a Service: Hardware is been purchased by cloud, cloud gives OS to you on that hardware. I need to choose like which OS I want, what you need to do on that OS. For eg, EC2

b)Platform as a Service: You are linux admin and you have purchased some server, then you need to configure database on server so that your data will not get loss. AWS said that I'll manage your database but your charge will increase. For eg RDS

c)Software as a Service: You can create the email ID of your employees who are working in your company. AWS will manage your data.

4. Type of Cloud:-

   a)Public Cloud: you can access over the internet. anyone create his/her resources

   b)Private Cloud: We have 4 server and install open stack on it. If a person require virtual machine then we will use OpenStack (open source cloud platform that manages, build private and public clouds) and create a VM. Then we will give key, username, internal IP to that person. user can use resources at company network. I will create virtual private network in my aws via console but that thing is private for our account.

   c)Hybrid Cloud: how organizes uses private cloud (data center) or a public cloud.

5. AWS Pricing Model:-

   How AWS takes money from user a)Compute: how you will utilize the cpu. For eg you have purchase 8 core cpu, 16 core cpu. then it will charge it

   b)Storage: 30 gb storage (volume)

   c)Data Transfer: how many time you use for accessing data site or a server. how much time you can access your website from your server, it will utilize your network. how much data it sent outside of AWS network, it will charge that thing. It won't charge if you transfer the data internally on AWS. It will charge how much data uses on AWS.

6. AWS Global Infrastructure:-

   How it handles Understand in which region you have created a VM You can use this link https://aws.amazon.com/about-aws/global-infrastructure/ to identify how many regions and availability zones are available. Each AWS region have three availability zone, min: 2 and max: 6 There are multiple data centers inside availability zone.

7. Why AWS created region in different places -> Consider I have created a website and my audience is from India. I've created servers and located at India so that user can able to access easily and data travelling will take less time. Therefore AWS is expanding regions. Also your application performance will be better because AWS expanded your application server in different continent. It will stop Disaster Recovery which means that there are lots of data present in your database at Mumbai. If flood occurs at Mumbai and if it affects your server inside data center, then your data will get lost. So will create a same replica and locate it at Hyderabad. All regions are connected to each other via wired connection. Their transfer speed is 100 gb/ps (super high network speed) Each region have its own code name. For eg Mumbai has a code name 'ap-south-1'. Refer this link https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html. Also services are different for each regions.

8. Availability Zones:-

   Consider you have put 1 availability zone in each region OR you have out all data centers inside one region (no availability zone). There are multiple data centers and you have located the server at one place. Consider electricity got disconnected or someone has cut a network wire in our data center and hence forth no electricity was been there for 25 days. so your machine, replica will be down. In one rack, multiple electricity sources are available. If light is not coming from one place then it will come from another place. I am at Mumbai at zyz.com. So is there 3 availability zones have been created from zyz.com. They are connected with one other at 100 km or 60 miles. Each availability zone is connected with different power supply so that if there is any issue in one availability zone, then your application will be accessible from another availability zone. Data will be transfer to each other availability zone in an encrypted form. Each availability zone have its own code. For eg Mumbai has its own code 'ap-south-1-a'. 3 availability zones are located at a distance of 100km in Mumbai -> reality.

9. Local Zone:-

   I want to specify that my server will be locate at ABC city in India. You can refer the link https://aws.amazon.com/about-aws/global-infrastructure/localzones/locations/ In this link, blue color dot defines an availability zone For eg I created one server in Delhi so for delhi people if they click my website to access the page, the time should be less than 10 milli sec (single digit latency) Services -> Resources You can find the services for local zones in link https://aws.amazon.com/about-aws/global-infrastructure/localzones/locations/

10. Create AWS Free Tier Account:-

    Type 'aws free tier' in google. Go to aws.amazon.com/free Click on "Sign in to Console". Enter your email address, enter account name, verify it. Copy verification code on your email and put on browser. Set password. Enter personal details, billing details. Then it will debit Rs.2 from your account and it will again credit to you in sometime. This can be done for verification purpose to check whether the account belongs to you or not. Enter mobile no, captcha. Select Free plan. To check which things are free -> Go to your profile icon (located at top right) > Billing Dashboard > Free Tier -> Whatever services you will be using free tier, it will be shown there. Additionally you will get 750 hrs for 12 months for Amazon EC2, for Amazon S3 you will get 5 gb storage free for 12 months, for database (Amazon RDS) you will get 750 hrs for 12 months

11. Terminate AWS Account

Click your profile icon (located at top right corner). Click on Account and close it.

11. Terminate your VM:-

    Search EC2 on search bar. Click on EC2, then click on Instances. Your machine should be in stopped condition. Click on Actions > Terminate Remove the Storage like Volumes and Snapshots under Elastic Block Store. Remove the Elastic IP which is under Network & Security

12. Create EC2 instance EC2 -> Elastic Compute Cloud We need to rent a server. I have 2 gb ram and my fried have 64 gb ram , 5 tb hard sik, 6 core cpu. I will tell my friend to create a VM and give me access to that VM through ssh. That VM is known as cloud for me. When you create a VM, firstly we need to select the type of OS, select how much compute power you will require (no of CPU), ram, storage, uploading and network speed (network), which traffic in and out for that machine (firewall), apply bootstrap script (when machine launches, this script will get execute. once you create machine, you need to add the machine into DNS, allocate the ip address, launch application on machine) Launch instance name of the server search ubuntu and select Long term support select free tier eligible public key -> goes to instance, we will have private key No change on network security Allow ssh from anywhere or you can use your own public ip so that only you can use your machine. Allow http protocols if you want to use web server on instance By default how much storage should C drive have OR Configure Storage -> 8 gb Launch it. Go to instances instance ID used to verify your machine by AWS Security > Traffic sent to server so server listen on which port -> Inbound rule Server can access to web server -> outbound rule

13. Access EC2 instance from Windows machine:-

    ssh is used on Linux, whereas it is enabled from Windows 10 Open cmd at Downloads folder, type ssh -i filename.pem username@public_DNS lscpu -> how much cpu core do you have free -m -> see total column (1 gb ram) ip a -> inet (private ip address) df -h -> /dev/root (8 gb) -> config storage cat /etc/os-release -> os version

14. Access EC2 instance from Linux machine:-

    Go downloads folder and right click it > Open terminal pwd: to know the current location, where you are ls: see files in this location ls -l: see the permission for owner/user, group and other user. type ssh -i filename.pem username@public_DNS OR username@public-ip you will get permission denied error because all user can read this pem file but linux won't allow this. We have to give only read permission to the user Use chmod 400 filename.pem then run ssh command

15. Install Nginx in EC2 instance:-

    sudo -i -> login as a root user apt-get update apt-get install nginx -> install nginx nginx -t -> check the configuration of nginx service nginx status -> check the status curl localhost -> access from local host (you will get a output) The output comes from root@ip-172-31-0-25:/var/www/html# ls index.nginx-debian.html Create a file here -> echo "Hello World" > index.html root@ip-172-31-0-25:/var/www/html# curl localhost Hello, Welcome to DNEG When you terminate the instance, it's root volumes (ELB - elastic block storage) will also get deleted.

16. Bootstrap script in EC2 instance:-

    Create an instance, Select the option Allow HTTP traffic from internet -> traffic will come from internet to server While creating instance, click on Advanced and enter shell script in UserData section Script: #!/bin/bash apt-get update apt-get install nginx echo "welcome to DNEG" > /var/www/html/index.html Launch it copy public ip and paste it in URL.

Note: You can use Ansible playtool to launch an application inside UserData section You need to create new build for application, create a build using jenkin, create instance using jenkin, use shell script. jenkin is like a server where it create the instance or infra using terraform, then using jenkins you can run the playbook or shell script on that created infra. Make sure create a build before launching instance. This is our pipeline.

Also build -> Docker image -> deploy on server

17. AWS Security Group:-

    Which traffic comes under instances (inbound rules) and which traffic will go to outside OR instances is talking to whom at outside (outbound rules). create a rule and passes the traffic on port -> Allowed ports and ip address, same for denied you need to create a rule for allow traffic only All traffic is by default selected at outbound rule, whereas we will allow everyone on ssh by creating new security group You can create new security group where you can use http protocol to send the traffic on port 80. If your application works on another port no then use custom TCP in inbound rule. You can attach multiple security group to one server. If you want to add any security when your instance is online then follow Actions > Security > change security group

18. EC2 Instance type:-

    t2.micro Here t is a class of instance, 2 is a generation, micro is a instance size Which instance type do you need to use for which purpose -> see the url You can choose the instance type depends on application You need to run only specific web server -> General Purpose. When you use 1 cpu then we can sue 2 gb ram, if you use 2 cpu then use 4 gb ram with same network bandwith. If application generates lot of pdf, using more compute power, machine learning, gaming server -> compute optimized if more memory require then sue Memory Optimize. eg, if table has 3k fields then when you retrieve the data using query it will take 3 sec gpu based instance -> accelerated computing, we can use large floating point application such as NASA, ISO, Air traffic control system storage optimized -> high performance (data read & write) eg facebook

19. Access Windows machine from linux:-

    sudo apt-get update; sudo apt-get install remmina

20. Instance MetaData with UserData:-

    We can find the private IP using ip a command, private IP, amazon ID image of our instance. We can find it through magic ip address curl 169.254.169.254 -> it shows api versions curl ________/latest curl ________/latest/meta-data curl ________/latest/meta-data/ami-id OR hostname OR instance-id

OR You can put in UserData section while creating an instance #!/bin/bash instance_id=$(command)

Dynamic Data: You can find the details of instance ID curl ________/latest/meta-data/dynamic/instance-identity/document OR signature curl ________/latest/meta-data/public-ipv4 -> public ip address OR local-ipv4

21. EBS (Elastic Block Storage):-

    Data store in form of block in hard drive Computer can take 100 gb hard sisk from storage machine which is located at storage area network over the network. like it's inert a pendrive. The disk which you create and attach is known as EBS. Create a block storage and throughout the network you can connect it to EC2 instance and use. We can access / (root volume) or C drive through storage area network Login to AWS console console.aws.amazon.com The OS which is installed on AWS is virtually attached at hard drive. This is known as EBS. The CPU is located at client and hard disk which is present in SAN so I want to know if the processor can read, write the data through SAN so is my application have a latency: yes. But in order to avoid this latency issue, we can create a storage at our instance. If you terminate machine, then data will get lost. Go to Images - AMI - type ubuntu for AMI. Make sure you need to search "Public Images". Root device type will be EBS or instance-store. If you select instance-store, storage will create on machine. You can use this type, if you need your speed will be fast and you are working on temporary data. We can attach multiple EBS volume to one machine. If your machine is terminated then you can store the data in root volume In short EBS volume is a hard disk which it attached to our machine. Can one EBS volume attach to multiple EC2 instances: ans is yes or no There are 5 types of EBS volumes. Out of which, 2 types will be attach to multiple EC2 instance whereas remaining type will be attach to one EC2 instance. If you create EBS volume in ap-south-1-a (AZ), we can attach to that instance only who is present in ap-south-1-a Consider I have an application where I attach 10 gb hard disk (ebs volume). Tomorrow I have filled 8 gb out of 10 gb but I knew 150 gb data will require, we can increase the size of EBS volume. We can also switch EBS volume to another AZ with the help of snapshot. While creating instance go to Confure Storage section. When you add new volume, you will see io1 and io2 (storage types). You can attach these storage types to multiple instances. Consider you instance is running on t2.micro and it is located at rack or running on another server and AWS wants to replace the server or physical infra then AWS will stop our instance, then you can restart instance. It will switch to another server but if you create storage on that machine, then your data will get lost. But before stopping instance by AWS, AWS will confirm whether you need to keep your instance somewhere or we can stop the instance. AWS will create its same replica in same AZ if you create EBS volume.

22. Create EBS volume and attach to our instance:-

    Go to AWS console - Elastic Block Store - Volumes - Create Volume Select gp2 (General P type, 5 gb hard disk size - create volume. Check in which AZ it is created (ap-south-1-a) Create another volmue in ap-south-1-b Create an instance in ap-south-1-a. Go to launch instance - network setting. Select the subnet which is ap-south-1-a. Go to configure storage - Add new volume, select 6 gb - launch instance Got to EBS - volumes (see 4 volumes have been attach) It's like a pendrive attach to machine. Connect to instance, see whether it is 8 gb or 6 gb lsblk - list block storage device (xvda - root volume, xvdb - 6 gb) attach 5 gb volume to our instance which is ap-south-1-b. Go to EBS - volumes - select particular volume - Action - attach volume - you can see there is no instance. It's because instance are not created in ap-south-1-b. Now select ap-south-1-a volume which is of 5 gb - actions -.... Attach volume (status : In Use which is shown in EBS - volume) run lsblk - see xvdf You cna detach volume by selecting volume in EBS section - Action - detach volume - status : available You can store data in xvdf (5 gb volume). Format the volume. mkfs.ext4 (using extension ext we will format) /dev/xvdf - permission denied sudo !! -it will run that command with sudo permission sudo mkdir /test login with sudo user - sudo su cd / ls - you can see test directory is created mount xvdf volume to test directory : mount /dev/xvdf /test Run mountpoint /test - Output: /test is a mountpoint Create a file inside test directory unmount the volume - cd .. umount /test/ cd /test/: no data is showing (you have eject the usb) Go to EBS - volume - select volume (xvdf) - action - detach volume (you have removed the usb from machine) run lsblk - that volume will not shown to you Make sure if you don’t want to pay the money for volume then you can delete the volume by going on Action - Delete volume

23. Detach an EBS volume from one instance and attach it into another:-

    We have deleted the volume in previous section. Now we will see whether data is persist on that volume or not. create instance, put that instance in same AZ i.e ap-south-1-a. launch instance. connect it mount the volume to that instance but don't run mkfs command because it will format the data lsblk check data is present or not -> sudo file -s /dev/xvdf (you will see ext4 or xfs file system means data is present) attach 6 gb volume also then run file -s /dev/xvdg (you will see /dev/xvdg:data means we need to format it on file system) mkdir /data mount /dev/xvdf /data/ cd /data/ ls -> you will see the data

24. Resize the EBS volume and resize the file system:-

    I want to increase xvdf volume (5 gb) to 8 gb. Go to EBS - Volume - select particular volume - actions - modify volume - change to size. make sure you cannot decrease the size - modify lsblk df -h -> xvdf size will be 5 gb how to increase in file system: run resize2fs /dev/xvdf df -h -> shows 8 gb You have assume that 100 gb data will arrive in your application. but data arrives till 1 gb but you are paying money for 99 gb which if of no use, then how we can decrease the volume: if you have 100gb drive attached in your instance, add another EBS volume, copy the 1 gb data in this EBS, unmount that 100g drive and mount this EBS volume

25. Resize the root EBS volume select instance:-

    Go down to Storage - select volume - actions - modify volume - change the size from 8 gb to 16 gb - modify Go to EBS - volumes - refresh access the insatnce using console lsblk: xvda has 16 gb (hard disk) focus on its partition (xvda1,xvda14 and xvda15) df -h: /dev/root is showing 8 gb, not 16 gb sudo file -s /dev/svda1: it is an ext4 file system sudo resize2fs /dev/xvda1 (this is for ext4 file system for resizing): Output (Nothing to do) How we can extend this sudo -i: login to root user growpart /dev/xvda 1 -> it means we need to resize the partition no 1 for block storage /dev/xvda now resize the same partition in file system -> resize2fs /dev/xvda1 df -h: /dev/root has 16 gb

Note:- To resize the xfs file system, run xfs_growfs

26. Attach one EBS volume to multiple EC2 instance create 2 instances:-

    While creating instance, put 2 in nu of instances (ap-south-1-a) Go to EBS - volumes - create volume - select "Provioned IOPS SSD (io2)" for Volume type, size: 5g, enable: multi attach - create volume (create volume in (ap-south-1-a)) select volume - actions - attach volume - select instance - attach volume - you will get error. Its because AWS has 2 types o hypervisor. First one is Xen and second one is Nitorway t2 is Xen based hypervisor where as t3isi nitro based We cannot change the instance type of running instance Stop the instance - actions - Select instance setting - change instance type - t3.micro - apply start the instances got o EBS - volume - actions - attach volume - select instance same EBS volume attach on another instance lsblk

27. types of EBS volume - which EBS volume should I use go to ebss - create volume IOPS - input output operation per second Throughput: how much data you need to transfer. In one sec, how many data you can transfer i.w MiB/s If you need more speed of IOPS, select volume type as io1 and io2. For gp2 and gp3, aws will balance for price and speed. hence when you create instance gp2 is bydefault delected. if you select gp2, size is 100 gb abd IOPS is 300/3000 then it means 1 gb, 3 IOPS if i put 10 gb size, min 100 IOPS, if i change size to 1500 gb then IOPS is 1500 if you select gp3 type then if you select 100 gb size, it will use 3000 IOPS You can achieve this high speed if you have created the instance in nitro based

Cold HDD (sc1): max iops: 250, less frequently data access, max volume size - 125 gb - 16 tb Throughput Optimized HDD (s11): max iops: 500, min size of this volume is 125 gb, throughot needs to be less but data need to access frequently like data warehouse, max volume size - 125 gb - 16 tb above 2 are non bootable ebs volumes means you cannot use this type for root volume general purpose SSD (gp2): max iops: 16000 ie . 1gbb - 3 iops, 100 gb = 300 iops, max volume size is 1 gb - 16 tb, max throughput - 7500 MiB/s general purpose SSD (gp3): starting 3000 iops, max iops is 16000, if you are using any type of database, max volume size is 1 gb - 16 tb provisioned iops ssd(io1): iops - 64000, data read write fast, amx volume size is 4 gb to 16 tb, if you are using machine critical application provisioned iops ssd(io2): iops - 64000, data read write fast, amx volume size is 4 gb to 16 tb https://aws.amazon.com/ebs/ - select volume types drop down

28. AWS / EBS Snapshot When I try new things, I am doing configuration in such a way that machine has an issue with it then I take the backup of configuration file or I can do this open Oracle VM virtualbox, select vm on left side, select take snapshot on top, put name as "Any name (snapshot 1)". means i have taken the photo of os which i can revert in previous state if any issue occur. right click on snapshot and then you can restore f you want to revert. If i have taken snapshot at 20:37 then it can take the backup the data till that date and time. After that time, it won't take any backup

In AWS you can take the snapshot of EBS volume. For eg if you run rm -rvf command in ebs volume, then data will delete in ebs. You can take snapshot for backup your ebs volume or for eg if you have an application where users order and your data got crashed. AWS will snapshot the EBS volume and store into s3 bucket (region specific). when you create ebs volume, you need to focus on regions as well as AZ. EBS volume is a locked at AZ. Snapshot is locked under region Consider your volume is lcoated at ap-south-1-a. take the snapshot and using that snapshot you can create another volume under different AZ. transfer volume from one AZ to other AZ. when you create the snapshot of volume, it will create the snapshot like incremental order. means consider you have volume which contains 100 g of data. today you have 50 g data and you have taken the snapshot which will take backup of 50 gb data. now 10 gb data is inserted in volume. then it will take the snapshot of 10 gb data only. storage cost is less but if you delete 1st snapshot i.e 50 gb then AWS moves the 1st snapshot blocks in 2nd snapshot. if you delete 3rd snapshot then aws transfer the data of 3rd snapshot to 4th snapshot

29. Create first snapshot We will create the snapshot from EBS volume create ec2 instance, attach EBS volume to it and create a snapshot of that EBS volume create instance - launch instance - t2.micro, add new volume of ebs (10 gb) - launch check whether data is available or not on 10 gb volume lsblk sudo -i file -s /dev/xvdb - output (/dev/xvdb: data) - means data is coming create its filesystem -> mkfs.ext4 /dev/xvdb file -s /dev/xvdb mkdir /data mount /dev/xvdb /data mountpoint /data -> /data is a mountpoint cd /data create a file yes "gaurav" > abc.txt- return same value in loop Go to EBS - volume - select volume - actions - create a snapshot Add description such as application name with date and time Go to EBS - Snapshot - status will be complete. Also s3 bucket is not available within aws account echo "hello world" > /data/xyz.txt check the instance AZ go to ebs - snapshot - action - create volume from snapshot (you cannot decrease the size) you can mount on same AZ because you can mount same instance - create volume go to ebs - volume - status: available - actions - attach volume - same instance lsblk -> xvdf 10 gb file -s /dev/xvdf -> filesystem is already there, don't run mkfs command cd .. mkdir /dataone mount /dev/xvdf /dataone cd /dataone ls -> only you can see abc.txt

30. Encrypt the EBS Volume When you encrypt the EBS volume, data will travel from SAN to computing machine in an encrypted form. Go to EBS - Volume - Create Volume - type: gp2, size: 5 GB, AZ: ap-south-1-a, Enable the option "Encrypt the volume", Add a tag: Name - Encrypted-Vol Create another volume without encrypting it, size: 5 GB, Add a tag: Name - Encrypted When you create a snapshot of encrypted volume then it will create the snapshot as an encrypted way. Select Volume - Actions - Create snapshot - Descriptions: Encrypted-snap Create a snapshot for another unencrypted volume - Description: Unenc-snap Go to EBS - Snapshot - Action - Select encrypted snapshot - Create volume from snapshot - It will be encrypted snapshot - Create Volume Check on Volume list Go to EBS - Snapshot - Action - Select unencrypted snapshot - Create volume from snapshot - You can encrypt the volume If you encrypt volume, you will get the data in decrypted form in your machine. Also it won't affect the performance when you do read/write the data.

31. Delete EBS Snapshot. Go to EBS - Snapshot - Select snapshot - Actions - Delete the snapshot Go to Snapshot - Recycle bin - Resources, you will see the deleted snapshot

32. AutoScaling You have a website, you have imagined 1k users will access your website. But when you launch your machine, suddenly 1 crore users occur and access the website. How you handle infra? server will response late and server will get down if the request is not handled. Sometimes you have imaged traffic will come from 1 crore user but only 1k user have come. Your infra get wasted because you have create 1150 server as per 1 crore user. Our server should increase the server or should not increase as per traffic requirement.

Types: a)Vertical Scaling - Suppose 1k users has come and you have created t2.micro, if multiple users occur then change server from t2.micro to some instance type. b)Horizontal Scaling - If traffic increases then add more number of servers after load balancer. If we add server, AWS should knew what will be the image, AMI ID, instance type, security group so that it will create

Go to Instance - Launch template - create launch template - enter template name, template version: v1, select image, instance type: t2. micro, key pair, ebs volume by default it will be encrypted when you create, advanced: userdata - create launch template

Requirement is 1 server needs to be up. If it delete then it should automatically create Go to auto scaling - auto scaling group - create group, enter group name, select template (when AWS create), select VPC and AZ, if possible override the template - next

• next maximum capacity: if load increase then how much server you will require - 10 minimum capacity: it should have min 1 server - next - next - create auto scaling group desired capacity: 1 Go to auto scaling group - select - instance management (bottom) - activity (bottom) Go to instances - you can see 1 instance get created terminate instance, t will create new instance automatically You can go to auto scaling group - edit