Setup HSP Ambari Cluster

Transcription

1 Setup HSP Ambari Cluster Prerequisites 1. An initialized HSP cluster running at least HSP Downloaded copy of the ISO containing the vm-template for Ambari from HortonWorks. This ISO is available from TISC, and may be called something like HDP _Ambari _HSP iso 3. Linux client. First, add the vm-template to the HSP cluster. 1. Extract the qcow2 image from the ISO. On the linux client, mount the ISO image as displayed below: linux# mount t iso9660 o loop /tmp/hdp _ambari _hsp iso /mnt/iso 2. Mount a share from the HSP cluster on your Linux system. Copy the qcow2 image from the ISO to the share: linux# mount IP:/myshare /mnt/hsp linux# cp /mnt/iso/hdp _ambari _hsp qcow2 /mnt/hsp Where IP is the VIP of the HSP cluster, and /myshare is the name of the available share. 3. On the HSP cluster, add the vm-template: admin@node003:~$ hspadm vm-template add --name ambari_template1 --image-path myshare:/hdp _ambari _hsp qcow2 --use-address-pool Y If this was successful, you can verify: admin@node003:~$ hspadm vm-template list Name Run State Golden VM Size Memory Size CPUs Pool IP Address MAC Address ambari_template1 DOWN No medium GiB 6 Yes Now deploy the vm-template. In this example we are creating a 5 node Hadoop cluster. admin@node003:~$ hspadm vm-template deploy name ambari_template1 num-instances 5 instance-name ambaridev If this was successful, you can verify: admin@node001:~$ hspadm vm-instance list --template ambari_template Name Run State Template IP Address MAC Address Node Name ambaridev-1 UP ambari_template /24 52:54:00:5e:f3:76 Node001 ambaridev-2 UP ambari_template /24 52:54:00:0a:30:94 Node004 ambaridev-3 UP ambari_template /24 52:54:00:75:06:47 Node002 ambaridev-4 UP ambari_template /24 52:54:00:17:c1:7f Node005 ambaridev-5 UP ambari_template /24 52:54:00:ad:d2:eb Node MK-95HSP

2 Setting up Ambari 1. Log into the first vm-instance. This instance will become the Ambari server. ssh Username = root Password = smrace1 2. Run the install, setup and start of Ambari server. To do this run /opt/escale/bin/ambariprep.py. There are several options that can be given, these can be seen by using the h option. The vminstances are configured to inherit the domain name that is set on the HSP cluster. Note that the script creates the file system supplied via the fsname option. The script will fail if the filesystem was created prior. Example: [root@ambaridev-1 ~]# /opt/escale/bin/ambariprep.py --fsname ambaridevfs domain escale.bluearc.com --user admin pw <admin_password> Upon completion of the ambariprep.py, ambari will have been installed, configured and started. Please note that it can take Ambari server up to 3 minutes to start responding on port 8080 for the GUI. You can follow the progress on ambaridev-1 (i.e. the instance on which you ran ambariprep.py) by: a. cd /var/log/ambari-server b. tail f ambari-server.log c. Wait for words to the effect of: 04 May :16:16,003 INFO [main] AmbariServer:548 - ********* Started Services ********** Configuring a New Cluster 1. Open up a browser on a desktop that has access to the vm-instance running ambari. This is the vm-instance on which you ran ambariprep.py. Connect to The login will be Username = admin Password = admin MK-95HSP

3 2. The wizard screen will come up next. Choose to Launch install Wizard. 3. Assign a name to the new Hadoop cluster. MK-95HSP

4 4. Select HDP stack version to load on new cluster. Currently only HDP 2.3 is available with this template. 5. Install options. On this page we will need to gather information from the Ambari server. Ssh MK-95HSP

5 Username = root Password = smrace1 First we need to gather the FQDNs of each of the vm-instances that will run Ambari agents and be a part of the Hadoop cluster. The ambariprep.py script has already setup the hosts file which is where the information can be pulled from. Example: [root@ambaridev-1 ~]# grep escale /etc/hosts awk '{print $2}' ambaridev-1.escale.bluearc.com ambaridev-2.escale.bluearc.com ambaridev-3.escale.bluearc.com ambaridev-4.escale.bluearc.com ambaridev-5.escale.bluearc.com Paste the output above into the Target hosts section. Second we will need to get the private ssh key from the Ambari server. This must be pasted into the text box below the radio button indicating Provide your SSH Private Key... Please copy and paste from id_rsa. For example: [root@ambaridev-1 ~]# cat /root/.ssh/id_rsa -----BEGIN RSA PRIVATE KEY----- MIIEowIBAAKCAQEAu7WJMc2HZWO0S/iPTPXpjMkbB+m6DAAweMUwJu7aw325dfR6 5VIPWR0GRZR+zJPJho4EvvgKYmHg2t7ZA3yfwU8G3ghAwvTHqB4f4jPoaAkw9wH1 6U299ZVhdNdjwuE09o0GGidpd8Wd8Yvfuot78JjpttL7H5vtIfq6WfrMqUbjVdou IlLkuIwvMP1U8Uskzz1pvSkaokpXZ/3yuWSXQOx/cArEDlRALhAt+B7/tz54gF1a Nog/ZKUcHhM50NiazB04AJiQbP1cWPXF+XD2F/jm5OkeyYVKA0TqQfveFbD27u6r ff48/vxqvncppbh5/zqtpquardmzic1b2y6vnqidaqabaoibaarxkrdgpd2qln+f BGuXVKu5CzMHqRIV/wkMJqz/4XecNOoRmbYsS8aX8UkywiwhTwwZxOpV+oeEFBt8 BDEcdOWX2mTQdA8Ocn/lEn+9C6c8bzGHjhSzg61mozZ8tQYRB5ScdoOsXbgzjg/E EkDcsafHUYO0dlBpXiyd9dPlDgyoZoiM8awo6mbXNchmFZdJ9oJvnlH5TG3APaJh 0Qm4iIXytz4buwKI8THv6QKBgEfjjvQPaIhzbsLQaxeyv4zvrM0R2HOHNC5vBV1b dkq01ve8nejh+pmx1kpbkuddlh7gc7zdnv3anv1bovjv0qhmucmt6ft0lf8kfcbz ngpqt3alkmodvrcqmd8rlpp/g/kai1w79jecez81ulivrba1tereva5ycltsiwlp depraogbalkkwr5/hshhfpgofianptch/l7djupql7j1opin5mpjzapu1t1bktvk AhVeqO8IYq7bX4p0bnK/hDgEF0YJ9TdmHlyb7E+feOGN/3YmUR3SMHW1E9nvui41 fusi2eeqy63uhmzxo7p7grranhrbynndivwugn56c3a3c5o8ltwq -----END RSA PRIVATE KEY----- Important: Please do not copy and paste the key from this document. You must use the id_rsa private key that is on the VM on which you ran ambariprep.py. MK-95HSP

6 6. Click Register and Confirm. All hosts should show success. 7. Choose Services. Install the following: Yarn + Mapreduce2 Tez MK-95HSP

7 Hive HBase Pig Sqoop Zookeeper Ambari Metrics EscaleFS Spark Currently not in the local repo: Oozie Falcon Storm Flume Accumulo Atlas Kafka Knox Mahout Slider 8. Assign Masters. In this step ensure that the Ambari server is only going to be used for nodemanager client. It is also a good practice to run Hive components (HiveServer, Hive MetaStore) on the same VM. Depending on the load in your environment, you may want to consider a separate VM for HBase. MK-95HSP

8 9. Assign Slaves and Clients. Assign Nodemanager for all. RegionServer for all except the ambari server. Phoenix server none. Spark Thrift server none. Client all. 10. Customize Services. There should be no changes required here. Any memory settings can be changed here on a per service basis if needed. MK-95HSP

9 A memory warning may come up. This is ok. There is some memory left over and Ambari wants to try and assign 100%. 11. Review. The following information can be saved off of printed. This is a high level layout of what is going to be installed. MK-95HSP

10 12. Start Install. Everything should complete and present as green. Takes ~10 minutes. MK-95HSP

11 13. Summary of the Install. 14. Dashboard view of the up and running Hadoop cluster. MK-95HSP

12 MK-95HSP