Setup HSP Ambari Cluster Prerequisites 1. An initialized HSP cluster running at least HSP 1.1.1. 2. Downloaded copy of the ISO containing the vm-template for Ambari from HortonWorks. This ISO is available from TISC, and may be called something like HDP-2.3.4.7_Ambari-2.2.1.1_HSP- 1.1.1.iso 3. Linux client. First, add the vm-template to the HSP cluster. 1. Extract the qcow2 image from the ISO. On the linux client, mount the ISO image as displayed below: linux# mount t iso9660 o loop /tmp/hdp-2.3.4.7_ambari-2.2.1.1_hsp-1.1.1.iso /mnt/iso 2. Mount a share from the HSP cluster on your Linux system. Copy the qcow2 image from the ISO to the share: linux# mount IP:/myshare /mnt/hsp linux# cp /mnt/iso/hdp-2.3.4.7_ambari-2.2.1.1_hsp-1.1.1.qcow2 /mnt/hsp Where IP is the VIP of the HSP cluster, and /myshare is the name of the available share. 3. On the HSP cluster, add the vm-template: admin@node003:~$ hspadm vm-template add --name ambari_template1 --image-path myshare:/hdp-2.3.4.7_ambari-2.2.1.1_hsp-1.1.1.qcow2 --use-address-pool Y If this was successful, you can verify: admin@node003:~$ hspadm vm-template list -------------------------------------------------------------------------------------------------------------------------------------------------- Name Run State Golden VM Size Memory Size CPUs Pool IP Address MAC Address --------------------------------------------------------------------------------------------------------------------------------------------------- ambari_template1 DOWN No medium 64.000 GiB 6 Yes --------------------------------------------------------------------------------------------------------------------------------------------------- 4. Now deploy the vm-template. In this example we are creating a 5 node Hadoop cluster. admin@node003:~$ hspadm vm-template deploy name ambari_template1 num-instances 5 instance-name ambaridev If this was successful, you can verify: admin@node001:~$ hspadm vm-instance list --template ambari_template1 ------------------------------------------------------------------------------------------------------------------------------- Name Run State Template IP Address MAC Address Node Name -------------------------------------------------------------------------------------------------------------------------------- ambaridev-1 UP ambari_template1 10.129.71.21/24 52:54:00:5e:f3:76 Node001 ambaridev-2 UP ambari_template1 10.129.71.22/24 52:54:00:0a:30:94 Node004 ambaridev-3 UP ambari_template1 10.129.71.23/24 52:54:00:75:06:47 Node002 ambaridev-4 UP ambari_template1 10.129.71.24/24 52:54:00:17:c1:7f Node005 ambaridev-5 UP ambari_template1 10.129.71.25/24 52:54:00:ad:d2:eb Node001 ------------------------------------------------------------------------------------------------------------------------------- MK-95HSP017-01 1
Setting up Ambari 1. Log into the first vm-instance. This instance will become the Ambari server. ssh 10.129.71.21 Username = root Password = smrace1 2. Run the install, setup and start of Ambari server. To do this run /opt/escale/bin/ambariprep.py. There are several options that can be given, these can be seen by using the h option. The vminstances are configured to inherit the domain name that is set on the HSP cluster. Note that the script creates the file system supplied via the fsname option. The script will fail if the filesystem was created prior. Example: [root@ambaridev-1 ~]# /opt/escale/bin/ambariprep.py --fsname ambaridevfs domain escale.bluearc.com --user admin pw <admin_password> Upon completion of the ambariprep.py, ambari will have been installed, configured and started. Please note that it can take Ambari server up to 3 minutes to start responding on port 8080 for the GUI. You can follow the progress on ambaridev-1 (i.e. the instance on which you ran ambariprep.py) by: a. cd /var/log/ambari-server b. tail f ambari-server.log c. Wait for words to the effect of: 04 May 2016 16:16:16,003 INFO [main] AmbariServer:548 - ********* Started Services ********** Configuring a New Cluster 1. Open up a browser on a desktop that has access to the vm-instance running ambari. This is the vm-instance on which you ran ambariprep.py. Connect to http://10.129.71.21:8080 The login will be Username = admin Password = admin MK-95HSP017-01 2
2. The wizard screen will come up next. Choose to Launch install Wizard. 3. Assign a name to the new Hadoop cluster. MK-95HSP017-01 3
4. Select HDP stack version to load on new cluster. Currently only HDP 2.3 is available with this template. 5. Install options. On this page we will need to gather information from the Ambari server. Ssh 10.129.71.21 MK-95HSP017-01 4
Username = root Password = smrace1 First we need to gather the FQDNs of each of the vm-instances that will run Ambari agents and be a part of the Hadoop cluster. The ambariprep.py script has already setup the hosts file which is where the information can be pulled from. Example: [root@ambaridev-1 ~]# grep escale /etc/hosts awk '{print $2}' ambaridev-1.escale.bluearc.com ambaridev-2.escale.bluearc.com ambaridev-3.escale.bluearc.com ambaridev-4.escale.bluearc.com ambaridev-5.escale.bluearc.com Paste the output above into the Target hosts section. Second we will need to get the private ssh key from the Ambari server. This must be pasted into the text box below the radio button indicating Provide your SSH Private Key... Please copy and paste from id_rsa. For example: [root@ambaridev-1 ~]# cat /root/.ssh/id_rsa -----BEGIN RSA PRIVATE KEY----- MIIEowIBAAKCAQEAu7WJMc2HZWO0S/iPTPXpjMkbB+m6DAAweMUwJu7aw325dfR6 5VIPWR0GRZR+zJPJho4EvvgKYmHg2t7ZA3yfwU8G3ghAwvTHqB4f4jPoaAkw9wH1 6U299ZVhdNdjwuE09o0GGidpd8Wd8Yvfuot78JjpttL7H5vtIfq6WfrMqUbjVdou IlLkuIwvMP1U8Uskzz1pvSkaokpXZ/3yuWSXQOx/cArEDlRALhAt+B7/tz54gF1a Nog/ZKUcHhM50NiazB04AJiQbP1cWPXF+XD2F/jm5OkeyYVKA0TqQfveFbD27u6r ff48/vxqvncppbh5/zqtpquardmzic1b2y6vnqidaqabaoibaarxkrdgpd2qln+f BGuXVKu5CzMHqRIV/wkMJqz/4XecNOoRmbYsS8aX8UkywiwhTwwZxOpV+oeEFBt8 BDEcdOWX2mTQdA8Ocn/lEn+9C6c8bzGHjhSzg61mozZ8tQYRB5ScdoOsXbgzjg/E EkDcsafHUYO0dlBpXiyd9dPlDgyoZoiM8awo6mbXNchmFZdJ9oJvnlH5TG3APaJh 0Qm4iIXytz4buwKI8THv6QKBgEfjjvQPaIhzbsLQaxeyv4zvrM0R2HOHNC5vBV1b dkq01ve8nejh+pmx1kpbkuddlh7gc7zdnv3anv1bovjv0qhmucmt6ft0lf8kfcbz ngpqt3alkmodvrcqmd8rlpp/g/kai1w79jecez81ulivrba1tereva5ycltsiwlp depraogbalkkwr5/hshhfpgofianptch/l7djupql7j1opin5mpjzapu1t1bktvk AhVeqO8IYq7bX4p0bnK/hDgEF0YJ9TdmHlyb7E+feOGN/3YmUR3SMHW1E9nvui41 fusi2eeqy63uhmzxo7p7grranhrbynndivwugn56c3a3c5o8ltwq -----END RSA PRIVATE KEY----- Important: Please do not copy and paste the key from this document. You must use the id_rsa private key that is on the VM on which you ran ambariprep.py. MK-95HSP017-01 5
6. Click Register and Confirm. All hosts should show success. 7. Choose Services. Install the following: Yarn + Mapreduce2 Tez MK-95HSP017-01 6
Hive HBase Pig Sqoop Zookeeper Ambari Metrics EscaleFS Spark Currently not in the local repo: Oozie Falcon Storm Flume Accumulo Atlas Kafka Knox Mahout Slider 8. Assign Masters. In this step ensure that the Ambari server is only going to be used for nodemanager client. It is also a good practice to run Hive components (HiveServer, Hive MetaStore) on the same VM. Depending on the load in your environment, you may want to consider a separate VM for HBase. MK-95HSP017-01 7
9. Assign Slaves and Clients. Assign Nodemanager for all. RegionServer for all except the ambari server. Phoenix server none. Spark Thrift server none. Client all. 10. Customize Services. There should be no changes required here. Any memory settings can be changed here on a per service basis if needed. MK-95HSP017-01 8
A memory warning may come up. This is ok. There is some memory left over and Ambari wants to try and assign 100%. 11. Review. The following information can be saved off of printed. This is a high level layout of what is going to be installed. MK-95HSP017-01 9
12. Start Install. Everything should complete and present as green. Takes ~10 minutes. MK-95HSP017-01 10
13. Summary of the Install. 14. Dashboard view of the up and running Hadoop cluster. MK-95HSP017-01 11
MK-95HSP017-01 12