Accessing data on the WebDAV service

Formatted data is available for download for selected flares. This is stored on our WebDAV server for easy download. WebDAV is an extension of HTTP, the protocol which the web runs on, allowing a server directory to be treated as a network disk, while still working as a plain web page.

The top level of our webdav is located at https://star.pst.qub.ac.uk/webdav/public/fchroma/

If you simply click on the link then it will open in your browser like a regular web page, and from there you can browse the available directories and download individual files by clicking the links. For a few files this is fine, but for directories with multiple files this does not scale.

Recursive download with wget

With a regular web server one can use a 'scraping' program to recursively download entire directory trees. There are many such programs, but a commonly used one on Unix-type operating systems (Linux, OS X) is GNU wget, which is commonly installed with Linux-based OSes and is available via MacPorts and other such services on OS X. wget has a recursive mode which will target a specific directory, download all files in it, and then repeat for each subdirectory in turn, replicating the server file stru ctures.

To demonstrate, we have a directory called example on the server. To download it with wget you would type:

wget -r -np -nH --cut-dirs=3 -R index.html https://star.pst.qub.ac.uk/webdav/public/fchroma/example/ 

where

  • -r turns on recursive mode
  • -np stops wget following the links to parent directories, which would make it copy the entire server
  • -nH suppresses the creation of a folder star.pst.qub.ac.uk
  • –cur-dirs=3 suppresses creation of the folders webdav/public/fchroma/ and only creates example and those directories underneath
  • -R index.html suppresses download of the auto-generated index.html files from the server

Direct access via WebDAV

While the recursive download approach above will work it is somewhat inelegant. WebDAV allows one to mount the server directory like a network drive, and from there simply use drag and drop, or the command line, to copy files as one would with any other disk.

On Mac OS X one simply uses the Finder; select Go/Connect to server… and enter the server address as https://star.pst.qub.ac.uk/webdav/public/fchroma/ - see the screenshot below.

When prompted for a username, opt to connect as a guest; the folder will then appear as network disk in the Finder, and you can treat it like any other drive. In this case it would even be available under /Volumes/fchroma/ from the terminal. When you have copied the files you need just right-click on the desktop icon and select Eject.

Most modern Linux desktop file managers will connect to a WebDAV service, though the exact mechanism very much depends on your specific distro; there is also the davfs2 option. If the client does not offer an explicit anonymous login option then just specify anything for the username - it should work.

With Windows one can map a drive letter to a WebDAV share, but this is 'delicate' and doesn't work in our testing. It's better to use a third-party client. Cyberduck is free software (donations requested), has been tested, and is known to work 1). It has an option for anonymous login, and you specify the directory options as shown in the screenshot below. Then you can drag folders to your local desktop as needed.

Cyberduck also works on OS X, and may be better than the Finder for large transfers. A command-line version is also available for Linux.

Connecting to the F-CHROMA WebDAV share from the Mac Finder

Above: Connecting to the F-CHROMA WebDAV share from the Mac OS X Finder.

Connecting to the F-CHROMA WebDAV share from Cyberduck

Above: Connecting to the F-CHROMA WebDAV share from Cyberduck in Windows.

1) though on 64-bit Windows one may have to install a 32-bit MS Visual C++ Runtime package to deal with an error about Missing MSVCR100.dll
public/solarflares/fchroma-webdav.txt · Last modified: 2015/06/29 15:08 by Robert Ryans

Back to Top Sitemap News