Some code snippets and functioning programs, mostly in Python (except GLADOS which is IDL). Also includes a selection of Blender files, mostly containing Python scripts.

Python Basics (another, better resource is the Python cheat sheet)

ChDir.py : Change to directory where the script is :

BlankArray.py  Declare a blank array 

NLines.py Find the number of lines in a file  

AddTrimNaNs.py FInd and trim NaNs

FindNans.py Find indices of NaNs in a list and delete them

ListFiles.py List files with specified extension

ListRandom.py List of random numbers

ArrayFromFile.py : Read array or list from file. Alternative, possibly clearer version : ArrayFromFile_2.py

MathTime.py : Some basic maths functions and timing.

SplitString.py : Split a string by character.

FileOps.py : Write or append to a file.

ReadFile.py : Read parameters from a file, avoid reading in line breaks.

ParallelPythonMultiprocessing.py : simplest possible use of the multiprocessing module.

CopyAndRunFiles.py : copies files from a specified location to multiple subdirectories and runs them. Useful for batch processing.

LineTest.py : how to use linecache to quickly read a specific line from a file (reads in the whole file though, so uses a lot of memory).

SortArrayByColumn.py : very simple demo of how to sort a 2D array by values in a specific column.

PILCrop.py : simple demo of cropping an image using PIL. Designed to be run inside Blender but easy to modify.

AddAlphaChannel.py : takes a series on input images and uses their RGB information to generate and apply a transparency channel map.



Blender basics - Blender (2.49) specific (see also the Blender 2.49 Python API)

ChBlendDir.py : Change to .blend file directory. 

DelVerts.py : Delete verticies.

GUIExample.py : Basic GUI.

FileBrowser.py : GUI file broswer (note this is bugged - can't send name to rest of program properly).

MeshMaterials.py : Get mesh from specified object, give it materials.

GetMaterial.py : Get materials of an object.

MoveVerts.py : Move verticies.

PointInMesh.py : Check if a point is inside a mesh.

DrawModes.py : Set draw mode to solid or transparent.

ExtrudePoints.py : Extrude points.

SelectObjects.py : Select objects.

ImageTexture.py : Create a material with a texture.

BoundingBox.py : Get the scale-independent, bounding box size of an object.

AutoRender.py : schedule the time for when a render starts (only on Linux).



Blender 2.78 or simialr versions

CreateMatrixText.py : Create a grid of Matrix-style text values from an input file.

AnimateMatrixText.py : Animate a grid of Matrix-style text from a series of input files.

AnimateVisibility.py : Animate object visibility in the 3D view and/or render display based on current frame. Designed for use with the Matrix text scripts above but could very easily be adapted.

CreateMeshes.py : Create a heightield (i.e. landscape) mesh from an input text file. There's not really any point in this since Blender has in-built tools for displacement. I suppose it's useful if you want very detailed control over the scaling.

EditMeshesAnimated.py : Animates heightfield meshes using a series of input text files.

AnimateClipAlpha.py : The "clip alpha" value is one of the few parameters in Blender that can't be keyframed, so here's a way to work around that.

AnimateLines_simplelayers.py : Controls the layers an object is visible on.

X-ray.blend : Not a script but an example of an "X-ray" style material. Very old technique which is hard to find described properly elsewhere so I want to make sure it's still available.

ModifyModifiers.py : Script to set the start and end frame of a "build" modifer.



FITS processing

Cleaning programs :

FitsBaseline.pyFitsBaseline_SN.py :Fits and subtracts nth-order polynomials  with sigma clipping on spectral axis of data cubes. The SN version divides the spectra by their rms, returning a signal-to-noise cube rather than flux (new version uses the robust noise, old version uses the standard rms and gives weirdly inconsistent results). Also available in IDL versions : FitsBaseline.pro, FitsBaseline_SN.pro

FitsMaskedBaseline.py : Takes two cubes as input, one with sources/RFI masked, one with no masking. Does the polynomial fitting as above on the masked cube, then subtracts the result from the unmasked cube. This can give better results if a large portion of the spectral baseline contains signal.

FitMed.py : Measure the median of the spatial (x-axis) bandpass and subtracts it.

SimMinMed .py: Chops the spectral bandpass up into boxes and measure the median in each box. Finds the minimum of the medians and subtracts it from the whole scan.

SimMedMed.py : As above, but usies the median of the medians instead of the minimum of the medians.

UserMed.py : Takes two cubes as input, one with sources masked, one without. Measures the median of the spatial bandpass in the masked cube, then subtracts this from the unmasked cube. Since the user has direct control over what to mask, this can use the maximum possible amount of the bandpass, so may be more accurate than MinMed or MedMed.

AGESCleaning.py : Combines all of the above and more to make it simpler to generate a cleaned, science-ready AGES data cube, which is a lot easier than running half a dozen different programs to get a single output file. Allows Hanning smoothing (using miriad), baseline fitting, S/N conversion, MINMED/MEDMED spatial bandpass correction, and keyword editing (defaults to ensure output is miriad-friendly as AGES cubes sometimes aren't). 


Working with non-Carterisan coordinates :

LABCreatePPP.py : Takes a position-velocity cube in Galactic coordinates and converts it into a true distance cube of the specified pixel and physical resolution. Unlike the crude version in the next section, it operates backwards. First it creates the PPP cube, then it calculates where each pixel should be extracted from in the original data. Calibrated on the coordinate system of the LAB data.

HI4PICreatePPP.py : As above, but designed for the gargantuan HI4PI data. The user must first split the data into a series of FITS files each containing a single velocity channel. Miriad is good for this as it's excellent at working with obscenly large data sets even on small systems.

FITSSequenceToPNG.py : Modified version of the FRELLED script for converting FITS files to PNG images. Designed for batch processing a sequence of single-channel FITS files, such as required when processing enormous data cubes.

SphericalPipeline.blend : FRELLED is designed for working with data in Cartesian coordinates. While the above scripts help convert Galactic data to Cartesian format, it's also possible to display the data directly in its original format. This works by creating a series of nested spheres instead of image planes. Full instructions are given in the file. Accompanies this paper. This file is designed for all-sky data and is relatively easy to work with.

ConicalPipeline.blend : As above, but designed for conical data. Slightly more complex to work with but enables the user to provide lookup tables (this example requires this file and this file)


Generating isosurfaces :

A sort-of add-on (read : crude hack) for FRELLED to allow the creation of isosurfaces, as an alernative/supplement to volume renders and renzograms. This uses the Python scikit-image module so is written for Blender 2.79 (could probably be adapted easily to deal with later versions too). At presents this necessitates a rather annoying three-stage process :

1) Load the data as normal in FRELLED (in Blender 2.49). Create a region around the feature of interest.

2) Download this file for Blender 2.79 and import the region created in FRELLED.

3) Download this script and modify the name of the FITS file, then run the internal script in the .blend file.

More instructions can be found here. Obviously this isn't a remotely sensible setup but it does work. At present I'm re-writing FRELLED for Blender 2.79, which will incorporate this feature so it can be used directly in a much more logical and intuitive way.

EDIT : I needed to make this work for FRELLED version 5 and I encountered some problems running it under Windows. So the single script in this .blend file is a much simpler method - you still have to create the region elsewhere and import it, but now there's only a single internal Blender script to run. It's also a lot faster and doesn't create external object files.


Other FITS operations :

Fits2Text.py : Converts a FITS file into ASCII.

FITS2TextCubeSlicer.py :Converts a FITS cube into a series of ASCII files, one per channel.

Text2Fits.py : Converts ASCII to FITS. Not very robust though.

FluxInject.py : Adds flux into a data cube at the specified location.

GaussFluxInject.py : Adds flux with a 2D Gaussian profile.

Interpolate.py : Interpolates extra channels.

LatLong-XYZ.py : Crudely converts cubes of latitude-longitude format into XY format (see better versions above).

DeGuass.py : Removes a Gaussian function. User can specify input parameters or allow them to vary (chi-squared minimization), e.g. center coordinates, FWHM, peak value. Works on multiple channels in 3D data cubes.

DeGuass_2D,py : Ony works on 2D data but can also (unlike DeGauss.py) iteratively fit ellipticity and position angle.

DeGauss3.py : removes Gaussians from a data cube without doing any fitting, just using the user-specified parameters exactly.

BlankFits.py : Creates a blank FITS file with the same header information as another.

AddHeader.py : Add FITS header keyword. Can also be used to edit existing keywords.

Clip.py : set all values above a specified threshold to zero (use this for setting values in an array in general).

ChannelClip.py : set all values in all channels outside a specified range to zero.

FitsEdit.py : basic image editing (e.g. changing image values of specific pixels)

FitsMultiply.py : takes two FITS files and multiplies them together.

LogFits.py : converts flux image values to logarithmic units.

FixAxes.py : takes data with 4 axes and converts it to 3, which is much easier to process.

MeasureRMSImage.py : measure the rms of an image using the spatial bandpass. Useful comparing the effects of processing when the spectral rms is unaltered.

PNG2FITS.py : takes a sequence of (greyscale ?) PNG images and converts them to a single FITS cube.

ReplaceNan.py : replaces all NaN's in an image with zero. Very inefficient though ! 

CheckHeaderKeyword.py : check if a keyword is present in the header.

FitsExtractVolume.py : masks arbitrary-shaped regions in a FITS file by reading in coordinate data from a file. Designed to be used in conjunction with FRELLED.

GfGridder.py, GfMultiGridder.py : grids numerical SPH (or n-body) particle data into PV FITS cubes. MultiGridder is for batch processing and works on files in multiple subdirectories.


Viewing FITS files in VR with Blender 2.78 Cycles

You should probably first be familiar with FRELLED to use these files ! On the other hand, these are actually much simpler. This set of files takes image sequences (e.g. .png files) and loads them in Blender with Cycles materials, which you can then render in VR (or normally, that's fine too). This is useful for viewing 3D volumetric data by loading in slices of the data with transparency.

Cycles.blend : Main file. Loads sequences of images with transparency based on colour. Ideally use 3 sequences, one for each orthogonal projection through the data. S

ConvertXY.blend, XZ.blend, ZY.blend : Combines separate images for transparency and colour into one sequence for each projection. No script - just edit the two image textures on the plane (and the scaling and image size).

SingleImageMultiVolumeConversion.blend : Designed for multi-volume data in FRELLED, which uses four (or more) images - two pairs of transparency and colour, which could be based on the same or different data sets in order to overlay them. This file converts them into a single image sequence, which uses a lot less resources to display. Again no scripts needed, just edit the image textures on the planes.

FBXConversion.blend : Some guidelines and scripts for converting Blender files into FBX files that can be imported into Steam VR Workshop, which is vastly more powerful than Blender for creating VR content. Blender objects should have shadeless UV mapped image texture materials. This is not a simple one-shot conversion script (I'm not sure that's really possible) but it shoud make the procedure much more straightforward. 



FRELLED : 3D FITS viewer. Does ALL of the things !

GLADOS : Source-finding program for HI data cubes. Takes three cubes as input : two orthogonal polarisations plus the average. Searches the average cube for sources spanning a specified number of channels, then checks for corresponding detections in each polarisation. Afterwards, matches sources in close proximity using starlink routines.

AGES cataloguing : Takes separate files containing inputs for mbspect and runs mbspect using each of them. Saves a file containing the input parameters and mbspect measurements, the ASCII log file of the specta, and postscripts. A version for Python 3 with some bugfixes is also available.

Photometry : Programs to assist in bulk downloading and naming of SDSS FITS files, as well as automatically generating ds9 regions for aperture photometery. 

Wedge plotting : Reads in a file of RA, Dec, velocity and makes a wedge plot (aka pie slice). You'll have to make your own axes or modify the ones provided.

BatchImageConvert.py : resize/reformat multiple images using parallel processing. Useful for storing smaller copies of your photogrpahs (files may be overwritten !).

DropTime.py : super simple numerical calculation of freefall time under gravity varying with distance. Unnecessary, can be done analytically, but never mind.

GetSDSSRGBs.py : download multiple SDSS RGB images from a table of RA/Dec coordinates.

GForce.py : simple gravity simulator for Blender (non-Blender version here).

RadialProfile.py : measures the radial profile of a source in a FITS file.

IntegrateRings.py, IsoDisc py : set up discs of particles with density variations given by a user-input radial profile (can combine with RadialProfile.py). Useful when setting up simulated galaxies with radial density profiles taken from observations.

ParticleViewer_lite.blend : SPH/n-body particle viewer, adapted from the version in FRELLED to use linecache. This can be a lot faster than the standard version and makes for very small .blend files, but linecache needs a lot of memory.

SpectraPlotter.py : Plots artificial spectra to demonstrate how integration drives down random noise but doesn't help with systematic biases. Generates spectra sequentially and averages them to mimic integration time. Noise is random (optionally with some baseline function) with a simple tophat signal injected over a range of channels, set to be too weak to be visible in any individual spectrum.

rhysy.net | feedback@rhysy.net