ROSA data transfer

The original plan for ROSA data transfer was to use LTO3 tapes, but that was made obsolete by the rapid advance in hard drive capacities, so the default now is to use hard drives connected via the fast eSATA interface. At the moment there are eSATA facilities on das1 and das2 only. This is because the normal mode of operations for ROSA has das1 and das2 running at the highest frame rates, typically generating 10x the amount of data that das3-das6 do. We therefore suggest that observers bring a number of drives with eSATA capabilities, plug these into das1/2 and then perform all data copies from those machines. Note that due to the vintage of the hardware only disks up to 2TB capacity are supported.

All drives being used with ROSA should be configured with a single partition, formatted as Linux ext2 or ext3 - this can be done on any Linux system, or on the ROSA machines if needed. Instructions are available below.

Given the number of variables involved - which cameras are in use, and at what cadence, etc - it is impossible to give a definitive guide on how to transfer data, or indeed script it in any meaningful manner. Different observers will have different needs and are likely the best judges of their own interests. So below we provide a general guide on what to do - this should be trivial for anyone with a basic knowledge of using Unix on the command line. The notes assume that you are logged onto das1 or das2 using the credentials given in the User Guide, and that your drive is partitioned and formatted as outlined below.

  1. Physically connect your drive to the computer via the eSATA port
  2. Mount the drive so it may be used:
    1. find the Linux device name for the drive by typing dmesg | tail and look for a line looking like Jan 27 16:26:24 das2 kernel: sd 14:0:0:0: Attached scsi removable disk sdc
    2. the Linux device name is therefore something like /dev/sdc
    3. mount the drive under the directory /mnt/ by typing mount /dev/sdc1 /mnt - replacing 'c' in sdc1 as appropriate. We specify the 1 as we wish to mount the first (in this case only) partition on the drive.
    4. verify the mount succeeded by running ls -l /mnt and df -h /mnt to list the disk contents and get the amount of free space on the drive respectively
  3. Make a directory on the external disk to hold the data by typing something like mkdir /mnt/20100315_das1
  4. Copy the data for each das by simply typing something like cp -av /data/das1/rosa/*.fit /mnt/20100315_das1/
  5. Repeat steps 3 and 4 for any additional data from other machines which are being stored on the same disk
  6. Unmount the drive by typing umount /mnt after which you may unplug the disk. If the computer complains that the disk is busy make sure that you do not have any terminals running in a directory under /mnt/

You may wish to use the du and df commands to check how much disk space is needed and free respectively:

  • du -sh /data/das1/rosa/ will give the amount of disk space used by the das1 data
  • df -h /mnt will give the amount of space free on the disk mounted under /mnt

Formatting

The ROSA computers run RedHat Enterprise Linux 4, and only understand the Linux ext2 or ext3 formats. Third party software is available to permit such disks to be read under Mac OS X and Windows, or you can mount them using a virtual machine. In case you need to format disks while at the telescope, here is how you can do so from the ROSA computers. This is a standard Linux partition and format routine and can be done on any modern Linux system - ask your systems manager if in doubt. Note once more that ROSA only supports disks of up to 2TB.

  1. Log onto das1 or das2 as root using the credentials in the ROSA user guide
  2. Before plugging the external drive into the eSATA interface, run the command fdisk -l which will give you a list of all known drives and partitions on the system
  3. Plug the drive into the eSATA port and turn it on
  4. Re-run the fdisk -l command, and verify that a new drive of the capacity you expect has been added to the list. It will likely have a name of the form /dev/sdX where X is a letter such as d or e.
  5. Type fdisk /dev/sdX replacing X as appropriate to start the fdisk partitioning package
  6. Within fdisk you can type:
    • p to print the current partition map
    • d to delete a numbered partition
    • n to create a new partition
    • w to quit and write the new partition map to disk
    • q to quit without making changes
  7. If any partitions exist on the drive it's best to remove them all using d, then write to disk using w and restart fdisk.
  8. Create a single new primary partition by typing n and then accepting the default values, then w to write the partition map to disk and exit fdisk.
  9. To format the partition type mkfs.ext3 /dev/sdX1 replacing X as needed.

You should now be able to mount the formatted disk and check the available capacity by typing:

  • mount /dev/sdX1 /mnt
  • df -lh /mnt

Finally you may unmount the drive (if needed) by typing umount /mnt at which point the drive may be unplugged.

public/research_areas/solar_physics/rosa_data_transfer.txt · Last modified: 2013/09/02 11:35 by Robert Ryans

Back to Top Sitemap News