Dinh:Genome Miner Upgrades

From ZhangLabWiki
Revision as of 22:02, 3 April 2015 by >Dinh (→‎bcl2fastq installation)
Jump to navigation Jump to search

Make a list of all installed packages on R

    #Get currently installed packages
    package_df <- as.data.frame(installed.packages("/Library/Frameworks/R.framework/Versions/2.15/Resources/library"))
    package_list <- as.character(package_df$Package)
     
    #Re-install Install packages
    install.packages(package_list)

Make a list of all installed programs

  • Used dpkg...

Make sure rEFIt or rEFInd is installed on Os X.

  • This is the equivalent of BootCamp for Mac/Linux dual boot.
  • Download rEFInd by Rod Smith
sudo ./install.sh --notesp

Install GPT fdisk (gdisk)

  • gdisk should be installed on Os X. If not, download and install version 7.0, newer versions will not work on Os X 10.5.

Make a UBUNTU USB EFI bootable drive

  • Make sure USB is properly formatted:
    • In Os X, open Disk Utilities and click on the USB stick volume, and then Information to see the disk number.
    • The Partition Map Scheme should be "Master Boot Record" and the format should be FAT32.
  • If not formatted correctly, click the drive, then click Partition. Make 1 large partition with "MS-DOS (FAT)" format and click Options at the bottom to choose "Master Boot Record"
    • Name the partition "UBUNTU"
  • Next, download the ISO-2-USB EFI-Booter for Mac 0.01 beta
  • Next, download the iso file of a recent version of Ubuntu Desktop Edition 64 bit. I downloaded this: http://cdimage.ubuntu.com/releases/14.04.1/release/ubuntu-14.04.1-desktop-amd64+mac.iso
  • Create the following directories on USB drive: efi and efi/boot
  • Copy the bootX64.efi from "ISO-2-USB EFI-Booter for Mac 0.01 beta" into efi/boot on the USB drive.
  • Copy Ubuntu image into efi/boot and rename it to "boot.iso"
  • There should be two files on USB drive now: bootX64.efi and boot.iso, both are in efi/boot.
  • Reboot computer and in the rEFInd menu, we should be able to select "boot from bootX64.efi on UBUNTU".

IMPORTANT: clone boot drive!!!

  • Make sure to do the installation on a cloned Os X drive. The Linux installation will not work on first try so do not risk losing the only bootable disk we have!
  • Partition the new drive using Os X and its Disk Utility. Make the partition table a "Master Boot Record", and make the first partition only 500 GB for Os X clone.
  • Use "Restore" function of Disk Utility to make an exact & bootable copy of the MacHD.
  • It is not necessary to clone the Linux partition since we will not be making any change to the root drive after this. If cloning the Linux partition is necessary, make sure to research how to do it correctly, this step is time consuming and if done incorrectly may be a waste of time.

IMPORTANT: install the new HDD to be the only bootable drive on the Mac Pro

  • The root drive is in bay number 1. I removed the 2TB Hitachi drive in bay 1 and put in the 4TB Seagate hybrid sshd.
  • I tried installation with the new HDD mounted on the external enclosure, however, this didn't work out. The best thing to do was to remove all bootable or current boot drive and install only the new HDD (with bootable Os X) to the Mac Pro. Make sure the new HDD is not just mounted on the external enclosure.

Installing UBUNTU

  • NOTE: The following instructions are copied from Rod Smith's instructions here: http://www.rodsbooks.com/ubuntu-efi/.
  • Boot into Os X and find out using disk utility what the disk number for the new HDD is.
  • Open Terminal program and type: (replace diskN with the correct disk number, eg. disk3).
 sudo gdisk /dev/diskN
  • Type p to display partition table. Verify that it's the same one shown by Disk Utility. (Note that Disk Utility hides the EFI System Partition (ESP), which is normally the first partition on the disk.)
  • If you created a Linux partition, type d in gdisk to delete it. Enter its number when prompted. (If you created multiple partitions for Linux with the intention of using them as you created them, you can skip this step.)
  • In gdisk, type n to create a new partition. Give it a number of 99, hit the Enter key to use the default starting sector (the program may tell you it's changed the sector value, but you can ignore this message), +128M as the ending sector, and a hex code of ef02.
    • This creates a BIOS Boot Partition of 128 MiB at the start of the largest block of free space on the disk. This partition will be used by the initial Ubuntu GRUB installation, but you can delete it once you've got native EFI booting working. 128 MiB is actually much bigger than is required for this purpose, but OS X likes to see gaps between partitions when it installs or upgrades the OS, so when you eventually delete this partition, it will leave the type of gap that OS X likes to see. (If you created all your Linux partitions in OS X's Disk Utility, you can put the BIOS Boot Partition in one of the gaps that Disk Utility leaves, but you may need to accept a default value or use +1M rather than enter +128M as the end value.)
  • Type p again to see the results
  • Type w to save changes and type y to confirm this choice.

File:Gdisk setuphdd.png

  • Insert USB boot drive for Ubuntu into the computer.
  • Restart Mac Pro and wait for it to boot to rEFInd screen.
  • Select "boot from bootX64.efi on UBUNTU" or the option that you can identify as your USB boot drive.
  • After booting into Ubuntu Live, select Install Ubuntu and proceed with a normal Ubuntu installation, except as noted in the next few steps.
  • When the "Installation Type" dialog box appears, select "Something Else."
  • You will initially see a partitioning screen with your original OS X partitions, a biosgrub partition (what Ubuntu's partitioner calls the BIOS Boot Partition), and free space at the end of the disk. Select the free space and click Add to begin adding partitions.
  • Create partitions for your Ubuntu installation in the free space near the end of the disk. (If you did this in OS X's Disk Utility, you should modify each partition by selecting it and clicking Change, then clicking the Format option and setting an appropriate filesystem and mount point.)
  • Make /, swap, and /home partitions.
    • swap should be created first, should be 1-2X RAM size, should be primary, and select Linux swap or swap space for type.
    • / should be the root partition, should be ~100 GB in size, should be logical and ext4 type.
    • /home should be the rest of the drive, should be logical and ext4 type.
  • Be sure that the "Device for boot loader installation" option is set to the same drive (it will include a complete name for your hard disk)
  • Click Install now and go through the final selection steps.
  • When the installation finishes, reboot the computer.

Fixing the Installation

  • The following steps will return the disk from a hybrid MBR to a protective MBR as GPT standard requires, and to set up grub.
  • Boot into Os X and launch a terminal program
  • Type "sudo gdisk /dev/disk0" (changing the disk identifier as necessary). If at any point in the next few steps something seems wrong, type q to exit without saving your changes.
  • In gdisk, type x. The command prompt will change to read "Expert command (? for help):".
  • In gdisk, type o. This command displays the contents of the hybrid MBR, which will probably consist of four partitions, one of which is of type 0xEE. The Ubuntu installer created a hybrid MBR (if one wasn't already present) in an attempt to be helpful.
  • In gdisk, type n. The program won't seem to do anything; it will just show you another command prompt.
  • In gdisk, type o again. The MBR contents should be different from before; there should just be one partition, of type 0xEE. This is a standards-compliant protective MBR.
  • Type w to save your changes.

File:Gdisk mbr.png

  • After this step in Rod Smith's guide, rEFInd needs to be installed, however, I've already done this.
  • Load /efi/refind/refind.conf into a text editor and locate the commented-out scanfor line. (If you installed rEFInd to the ESP, you'll need to mount the ESP and load the configuration file from there.) Uncomment the scanfor line by removing the leading hash mark (#) and then add a new item, cd, to the end of the line. It should read scanfor internal,external,optical,cd. (If you're using USB flash drives, add biosexternal instead of or in addition to cd.)
    • I just used the default Mac Os X scanfor settings.
  • Reboot into the rEFInd menu.
  • At this point, Rod Smith's guide requires Super Grub 2, however, I didn't need this since rEFInd was able to find the kernel for the new Ubuntu to boot into.
    • If rEFInd can only find the grub*.efi file, something is wrong. PROCEED CAUTIOUSLY. I tried continueing with this, but ended up having to start over with the Ubuntu Installation.
  • Once booted into the new Ubuntu, open a Terminal program.
  • Check to see if the /boot/efi/EFI/ubuntu folder exists.
  • If the above folder does not exist, type "sudo mkdir /boot/efi"
  • Type "sudo blkid" to find the EFI partition on the new HDD.
  • Type "sudo mount /dev/sdX1 /boot/efi" to mount the EFI System Partition. Change "dev/sdX1" to the correct partition number.
  • Type "sudo mkdir /boot/efi/EFI/ubuntu" (if the directory didn't already exist)
  • Type "ls /boot/efi". You should see the contents of the ESP, which will probably consist of a single directory called efi. There could be other files and directories, but probably not many of them. If it appears you've mounted the wrong partition, review your partition layout and the commands you've typed to mount the ESP. Proceed only when you're sure you've mounted the ESP at /boot/efi
  • Type "sudo apt-get install grub-efi"
  • Type "sudo grub-install". Recent versions of GRUB seem to require an option. Pass it your ESP's device filename, as in sudo grub-install /dev/sda1.) You'll see a few messages appear on the screen, including a couple of "fatal" errors about EFI variables. Ignore those messages. They're caused by the fact that you're booted in BIOS mode, and they're irrelevant because the task they're intended to perform will be handled by rEFInd.
  • Type "ls -l /boot/efi/EFI/ubuntu". You should see two files, boot.efi and either grubia32.efi or grubx64.efi, depending on your platform. They should have the same file size; in fact, they're identical. Remove other *.efi files, leaving only grubX64.efi and a grub.cfg file.
  • check /etc/fstab to make sure there is the following line to mount EFI System Partition automatically whenever you boot (change /dev/sdX1 to the appropriate partition number):
 /dev/sdX1     /boot/efi   vfat     ro,fmask=133    0 0

Final configurations on Ubuntu 14.04

  • Run updates:
 sudo apt-get update
  • Install current nVidia drivers
 sudo apt-get install nvidia-current-updates
  • Install vim
 sudo apt-get install vim
  • Install ssh, the client should be already installed but run apt-get to make sure.
 sudo apt-get install openssh-client
 sudo apt-get install openssh-server
  • Install antivirus
 sudo apt-get install clamav clamav-daemon clamav-freshclam
 sudo freshclam
 sudo /etc/init.d/clamav-daemon start
  • Install gparted
 sudo apt-get install gparted
  • The RAID enclosures need gparted to be mounted. Use Athurva's tips on how to mount them.
  • Make the directories for mounting the three other Hitachi 2TB drives:
 sudo mkdir /media/TmpStore1
 sudo mkdir /media/TmpStore2
 sudo mkdir /media/FreshReads
  • Mount the three other Hitachi drives by copying these lines into /etc/fstab:
 UUID=aab32527-c79a-4c42-b5e1-6dac0d5fa104 /media/TmpStore1 ext3 user,exec 0 0
 UUID=cdd8e7e8-ae10-49f8-bb0d-4a226cf9f255 /media/TmpStore2 ext3 user,exec 0 0
 UUID=33a12e29-91f2-41dc-a441-d0d526f740e5 /media/FreshReads ext3 defaults,exec 0 0
  • Run mount all command:
 sudo mount -a
  • Mount the previous root drive on an external enclosure, and copy over all user home directories to /home.
  • The three Synology diskstation folders need to be mounted. Make sure all users and groups are set up correctly. Then use Athurva's tips on how to mount them
    • Genome-miner's local connection to the NetGear router via port number 2 needs to have an IP address set up manually to be able to connect to the Synology.
      • Set up the IPv4 IP as 192.168.137.11
      • subnet mask as 255.255.255.0
      • leave Gateway address empty
    • smbmount command is not available, used mount -t cifs (options for this command are the same).
  • Set up firewall
sudo ufw enable
sudo ufw allow ssh
sudo ufw allow www
sudo ufw allow ftp
sudo ufw allow Samba
sudo ufw reload
  • Install other services
#sudo apt-get install build-essentials
#sudo apt-get install gnuplot
#sudo apt-get install fftw-dev
#sudo apt-get install imagemagick --fix-missing
#sudo apt-get install gambas3-gb-compress-bzlib2
#sudo apt-get install nfs-server
#sudo apt-get install mysql-server #(root password is genomeclub)
#sudo apt-get install gdm #(default display set as gdm)
#sudo apt-get install lm-sensors
#sudo apt-get install irqbalance
#sudo apt-get install fancontrol
#sudo apt-get install bootlogd
#sudo apt-get install binfmt-support
#sudo apt-get install pcmciautil
#sudo apt-get install cifs-utils
#sudo apt-get install sysfsutils
#sudo apt-get install samba
#sudo apt-get install portmap #(Note, selecting rpcbind instead of portmap)
#sudo apt-get install screen
#sudo apt-get install r-base-core
  • mysql database still need to be ported over. need to figure out how to do this.
  • copy over files in /opt
#sudo cp -r /media/ddiep/79163ff1-f258-4fd7-8d1a-ce0ef71fa14e/opt/* /opt/
  • Install perl modules
 CPAN, install XML::Simple, install File::Temp, install Sort::Array
  • Install unafoldv3.8
  • Install R packages using list of packages saved from old server
  • Install mac applications:
sudo add-apt-repository ppa:mactel-support/ppa
sudo apt-get update
sudo apt-get install macfanctld
  • Edit and copy over user accounts and group information
#dir="/media/dinh/79163ff1-f258-4fd7-8d1a-ce0ef71fa14e"
#awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' $dir/etc/passwd > passwd.sync
#awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' $dir/etc/group > group.sync
#awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=35534) {print $1}' $dir/etc/passwd | tee - | sudo egrep -f - $dir/etc/shadow > shadow.sync
#awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534) {print $1}' $dir/etc/group | tee - | sudo egrep -f - $dir/etc/gshadow > gshadow.sync
  • Change ajgore uid to 1005 and delete entries for dinh
  • Port over user accounts:
#cat passwd.sync >> /etc/passwd
#cat group.sync >> /etc/group
#cat shadow.sync >> /etc/shadow
#cat gshadow.sync >> /etc/gshadow
  • Since all the uid and gid are maintained for most people, just need to run chown on Athurva's and Dinh's folders.

bcl2fastq installation

  • Install Zlib
sudo apt-get install libzzip-dev
sudo cp /usr/lib/x86_64-linux-gnu/libbz2.so /usr/lib/
sudo apt-get install libboost-all-dev
  • Install bcl2fastq
$ tar -vxf bcl2fastq_v1.8.4.tar
$ mkdir bcl2fastq_v1.8.4-build
$ cd bcl2fastq_v1.8.4-build
$ ../bcl2fastq_v1.8.4/src/configure
$ make
$ make install
  • Change configure scripts to use older version of perl: #!/usr/bin/env perl5.10.1