Projects / Makeflow


Makeflow is a workflow engine for executing large complex applications on clusters, clouds, and grids. It can be used to drive several different distributed computing systems, including Condor, SGE, and the included Work Queue system. It does not require a distributed filesystem, so you can use it to harness whatever collection of machines you have available. It is typically used for scaling up data-intensive scientific applications to hundreds or thousands of cores.

Operating Systems

Recent releases

  •  29 Jul 2013 16:26

    Release Notes: Support for workers hierarchies, with a master-foremen-workers paradigm. Additionally, a worker now is able to handle more than one task at a time. Support has been added for task categories. A rule can be labeled with a category, and required computational resources (disk, memory, and CPU) can be specified per category. Makeflow then automatically communicates these requirements to the work queue or condor.

    •  18 Feb 2013 22:36

      Release Notes: Work Queue now allows you to specify chunks (pieces) of an input file to be used as input for worker tasks. Makeflow now supports a -i switch which outputs pre-execution analysis of the Makeflow DAG. Work Queue and Makeflow support submitting tasks to the PBS batch submission platform. The Makeflow makeflow_log_parser now ignores comments in Makeflow logs. Work Queue supports querying workers and tasks at run-time. Makeflow Input file (dependency) renaming is supported with a new "->" operator. Work Queue Tasks are now killed using SIGKILL. Work Queue Protocol based keep-alives were added to workers.

      •  11 Feb 2013 19:06

        Release Notes: In WorkQueue, memory errors that lead to a SEGFAULT, race conditions in signal handling for workers, and incorrect handling of the -C option where a worker would not use the same catalog server as work_queue_pool were corrected. Tasks now have a unique process group to properly kill all task children on abort. Escape codes such as \n and \t are now properly interpreted in Makeflow files.

        •  02 Nov 2012 19:57

          Release Notes: A bug in WorkQueue that resulted in a cancelled task becoming a zombie was fixed. Various corrections were made to the manual and HTML documentation. Makeflow -I and -O options now correctly the output file list to stdout. WorkQueue now correctly removes directories during cleanup.

          •  19 Sep 2012 01:25

            Release Notes: WorkQueue now has a new API for logging, improved python binding support and documentation, and various other bugfixes and minor features. Makeflow now supports garbage collection of intermediate files, lexical scoping of makeflow variables, and a new MAKEFLOW keyword for recursive makeflows.

            Recent comments

            12 Apr 2011 17:47 imipak

            Suggest changing the "Information" tag to "Website" as the front page doesn't show there's a website for the program.


            Project Spotlight


            A Fluent OpenStack client API for Java.


            Project Spotlight

            TurnKey TWiki Appliance

            A TWiki appliance that is easy to use and lightweight.