TSS development

URGENT Tasks Required

PSDB2 SSD crashed. We had to halt PS1 data ingest. In order to restart the Ingest, the following tasks are required.

  • Verify SSD filesystem is OK [COMPLETE]
  • Move small (and no longer updated) databases to SAS spinning disk. [COMPLETE]
  • Modify ingest code to stop classifying the object on ingest. [TESTING]
  • Modify web pages to be able to deal with non-classified objects. [TESTING - looks OK so far]
  • Implement post-ingest classifier immediately after objects promoted to the eyeball list
  • Write a script to download the MPC database daily.
  • Write a new script to utilise the MPC database download (either from scratch or using the pyephem library).

Immediate future improvements

  • Ingest PS1 Ubercal Stars as new crossmatch catalog [IN PROGRESS]
  • Modify ConeSearch utility to see the new PS1 catalog [COMPLETE]
  • Implement Larry's MOPS object finder.
  • Implement newly ingested PS1 Ubercal Stars catalog into the (post-ingest) classification algorithm.
  • Rewrite the Finder Maker to extract offset stars from the new PS1 Ubercal Stars catalog.

Longer term improvements

  • Create a Python binding for the C++ classifier. (Speed test with pure the python version.)

Other Tasks Required

Create a common queries library - in particular for lightcurves, and to facilitate colour plots alongside lightcurves. [COMPLETE]

Do the g-r and g-r decay queries for Suvi's TDEs. [COMPLETE]

Run the Python crossmatcher against the OLD FGSS pages - in particular we want Milliquas matches. [COMPLETE]

Upgrade the OLD FGSS database and pages to use the new web page code. [COMPLETE]

Add a date axis to the javascript lightcurve code.

Redo the Photometry comparison for the selected objects. Find out what can be done with Mark Hubers's Ubercal data. Redo the lightcurve CfA/QUB comparative plots for pre- and post-Ubercal data. [COMPLETE]

Examine Mark Huber's re-convolution data. [COMPLETE]

Download SDSS DR8 Catalogs. Current issues to be resolved include which columns in DR8 currently represent the photometry flags (e.g. TARGET_QSO_FAINT). Script prepared, but still need to resolve which columns to select.

Under Development

Web Page Light Curve External Data Incorporation

Currently looking at modifications required to read data from external files (e.g. CSV files) and superimpose this data on top of existing PS1 light curves. [DRAFT in Gnuplot done for LT data in FGSS.]

Replace lightcurves with HTML5 Canvas object (e.g. Flot). [COMPLETE]

Python Crossmatcher

Produce python script to do crossmatching. This incorporates the C++ HTM library (via SWIG) as a python shared library, which means that crossmatching is now done in python as fast as the C++ version. [COMPLETE]

Note that this code must be recompiled if we want to use it with a different python version (e.g. the Ureka version).

Do Milliquas crossmatching, utilising python cone searcher above for FGSS. [IN PROGRESS]

  • Updates can now be done manually. The script crossmatches with Milliquas, inserts the Milliquas row into the tcs_cross_matches table and updates tcs_transient_objects to set object_classification to 8 (AGN). [COMPLETE]
  • Web interface code has been modified to select only Sloan data out of the database for the time being for FGSS. This will be changed (below).
  • Automation will be done shortly [COMPLETE].
  • Major updates required to the web interface (see next section) [COMPLETE].

Other Web Page Modifications

Web pages updates are required for FGSS, but need to be generic. This has demanded a complete re-think about how the "Nearest Object" is rendered on the followup web pages. Currently this is calculated on-the-fly as the pages are displayed. However, this is not only unnecessary and CPU intensive, but places a very restrictive boundary on the type of query that is done to produce this.

The new proposal is to add the crossmatch data to the tcs_latest_object_stats table and calculate this when the object stats are produced. This gives us the flexibility of python rather than SQL. Whilst we are here, why not add RA and DEC to the stats table, which will allow us to calculate the AVERAGE.

The following is required:

  • Add 5 new columns to tcs_latest_object_stats: ra_avg, dec_avg, catalogue, nearest_object, separation. [COMPLETE]
  • Modify the generateDailyStats.py code to calculate average RA & DEC, and get all crossmatch info for each object (and use this to determine which object is selected as the nearest). This data, once calculated, needs to be inserted into the tcs_latest_object_stats table.
  • Add new Django SQL views and python views (dbviews) [MOSTLY COMPLETE]. Addition of logic to select out average RA and DEC needs to be incorporated.
  • Determine how to get the star/galaxy info for FGSS - must search nearest object in Sloan catalogue, but only if there is no Milliquas crossmatch. If there is no Milliquas crossmatch, change the Nearest Object info to star or galaxy [COMPLETE].
  • If there IS a Milliquas match, use the nearest Milliquas info. Link to Milliquas catalogue [COMPLETE].
  • Create Milliquas catalogue Django view.
  • Roll out the changes to both MD and FGSS interfaces [COMPLETE].

Upcoming Developments


  • Complete MD Algorithm modifications for new MD database. [COMPLETE]

Web Interface

  • Candidate web page to show AVERAGE RA and DEC. [COMPLETE]
  • Candidate web page to show all objects within 3 arcsec of the current one being viewed. This should help the person on eyeball duty to choose the most representative object. (Utilises new python conesearching code.). [COMPLETE]
  • Average RA and DEC to be represented on SDSS Crosshairs. [COMPLETE]
  • Remove sortability from object recurrence list and make it a custom query (like for text only).

Forced Photometry

  • Detectability server Forced Photometry requests. Fix request FITS tables (according to Bill's suggestions) and ask for results to be aggregated into a single table. [COMPLETE]
  • Test this on PS1-11ap, PS1-12in and PS1-12rg. [COMPLETE]

Other Software

  • Generic finder maker - especially for LSQ data. [STARTED - issues with postage stamp server]

Other Enhancements


  • Reset primary detection to be one nearest centre of cluster. (Still thinking about how to deal with outliers.)
  • Age the detections so that some can be discounted. Thinking about new flag for all detections, then resetting centre as above to the nearest remaining one to average RA and DEC.

Future Suggestions

  • FGSS objects near nearby galaxies. Need to consider alterations to FGSS algorithm.
  • Download SDSS DR9 data and integrate into both MD and FGSS. [COMPLETE]
  • With conesearch offer to merge points? Note that rollback of this would be very difficult.
  • Form to deprecate old points and recentre? Roll back?
  • If recentring is done, need to recalculate the catalogue crossmatch distance(s). There may also be a closer catalogued object that the one(s) listed. How do we re-run the crossmatch.
  • Take a look at replacing the Recurrence Plots with Flot. [COMPLETE]
  • Look at Flot replacement for Location Maps. Issues with speed. May need another crossmatch environment table. Data inserted into table as one-off and then used to do the location maps.
users/kwsmith/tss_development.txt · Last modified: 2015/02/06 18:45 by Ken Smith

Back to Top Sitemap News