Manageability

• Posted by
• Published: 2009-02-25

In a past life I wrote a language that would serve as a DSL to a variety of solvers. These solvers would include the Simplex based solvers, Quadratic solvers and in the most generalized form Constraint solvers. So I've had more than a passing interest in Constraint Programming (CP). Recently my interest in CP was rekindled. I was studying Javafx and found it's binding mechanism to be eerily familiar. So, I looked back at my archives and to my surprise discovered a boat load of open source projects in existence that catered to Constraint Programming. Here is the list I compiled over the years.

• Choco Solver - Choco Solver is a library for constraint satisfaction problems (CSP), constraint programming (CP) and explanation-based constraint solving (e-CP). Choco is built on a event-based propagation mechanism with backtrackable structures.
• Cream - Cream is library helping for developing intelligent programs requiring constraint satisfaction or optimization on finite domains. Cream provides a natural description of constraints. It allows extensions of constraint and satisfaction algorithms. It includes simulated Annealing and Taboo Search. It provides an interface to OpenOffice Calc (Spreadsheet).
• JSL - Java Search Library (JSL) is a framework for general searching on graphs. The standard search algorithms depth-first, breadth-first and A* are provided. JSL also allows the plug in of other search algorithms.
• JHCR - The K.U.Leuven JCHR System is an integration of Java and Constraint Handling Rules (CHR) designed with three aims in mind: user-friendliness, flexibility and efficiency. User-friendliness is achieved by providing a high-level rule-based syntax that feels familiar to both Java programmers and users of other CHR embeddings, and by full compliance to the refined operational semantics. Flexibility is the result of a well thought-out design, allowing e.g. an easy integration of built-in constraint solvers and variable types. An optimized compilation to Java code and the use of a very efficient constraint store make the performance of the K.U.Leuven JCHR System competitive with that of state of the art CHR implementations in e.g. Prolog and HAL.
• jOpt jOpt is an open source implementation of the Optimization Programming Language (OPL). OPL is a modeling language for combinatorial optimization problems that combines mathematical programming and constraint programming into a single language. Job Scheduling (JS), is the first add-on to have been released. Typically, JS problems involve assigning a set of activities to a limited number of resources with consideration given to time and ordering limitations.
• Drools Solver - Drools Solver is library to help you solve planning problems using heuristic algorithms. Examples of planning problems include: employee shifts, freight routing, supply sorting, class scheduling and the traveling salesman problem. The Drools-solver combines a search algorithm with the power of the drools rule engine. Drools Solver uses the Drools rule engine for score calculation, which greatly reduces the complexity and effort to write very scalable constraints in a declarative manner. Drools-solver supports several search algorithms to efficiently wade through the incredbly large number of possible solutions. Drools-solver implements local search, including tabu search and simulated annealing.
• JCL - Java Constraints Library (JCL) was one of the first libraries to bring constraints satisfaction problem solving to Java. JCL deals with discrete, fine domains as well as with continuous domains. Constraints can be crisp (as in the classic CSP problems) or soft.
• SAT4J - The SAT4J library provides a SATisfiability problem solver. Compared to the OpenSAT project, the SAT4J library targets first users of SAT "black boxes", willing to embed SAT technologies into their application without worrying about the details. SAT4J is an implementation in of Een and Sorenson's MiniSAT specification: An extensible SAT solver. The original implementation was in C++.
• Cassowary - Cassowary is an incremental constraint solving toolkit that efficiently solves systems of linear equalities and inequalities. The library appears to be designed to support support UI applications.
• DragonBreath - The engine is built to solve search problems, i.e., problems for which you don't really know how to construct a solution but can describe what potential parts a solution can consist of and which restrictions must be satisfied by the parts / the parts' constellation. Parts can be variables as well as structural components. In addition, you can specify a preference between different solutions, i.e., solve optimization problems. Examples for such problems are scheduling, action planning and configuration.
• JaCop - JaCoP is a Java Constraint Programming solver. It provides a significant number of (global) constraints to facilitate efficient modeling.It is currently more than 90.000 lines of code, not including examples and testing code. There are more than 20.000 lines of code explaining how to model and solve diverse problems.

If you know of other projects in this area, please free to let me know.

• Posted by cperez
• Published: 2009-02-15

Here's a new list inspired by a post by a reader (i.e. Hemant). The list covers libraries for developing 2 dimensional structured graphics. Just like the previous lists, everything is open source and written in Java.

• Piccolo - Piccolo is a Java 2 toolkit that supports the development of 2D structured graphics programs in general, and Zoomable User Interfaces (ZUIs) in particular. It makes it easy for Java programmers to build their own animated graphical applications with zooming, multiple cameras, layers, images, etc. Jazz was the original toolkit, and Piccolo is Jazz's successor. Other interesting visualization kits from the same group (i.e. UMD HCIL) include TreeMap and SpaceTree for hierarchies and TimeSearcher for time series.
• JGraph - JGraph is a powerful, lightweight, feature-rich, and thoroughly documented open-source graph component available for Java. It is accompanied by JGraphpad, the first free diagram editor for Java that offers XML, Drag and Drop and more. With the JGraph zoomable component, you can display objects and relations (networks) in any Swing UI. JGraph can also be used on the server-side, for example to read a GXL graph, apply a custom layout algorithm, and return the result as a HTML image map.
• Arakhnê - A network editing environment. A Swing framework that permits to edit and show networks. It is domain-independant but it proposes a useful user interface that includes a dictionary of model's objects, a manager of critic on model, a graphic user interface for model-object's properies
• Mica - Mica has complete support for advanced features such as infinite undo/redo, zoom and pan, network graph layouts and interactive graph templates, connections and connection points, annotations and annotation points, event handling and action percolation, layers and layer tabs, arrows, shadows, cut/copy/paste to/from clipboard, multiple printable pages, postscript, jpeg, and pdf output, rulers, toolhints, status bars, tool bars, default save/load to/from ASCII files, complete working network and diagramming editors, and more.
• Diva - Diva is an architecture for visualizing and interacting with dynamic information spaces. In Diva, visualizations are built by hooking together software components which generate, supply, filter, and display information. The current release focuses on Diva's canvas and graph visualization infrastructure.
• Satin - Software support for making effective pen-based applications is currently rudimentary. To facilitate the creation of such applications, we have developed SATIN, a Java-based toolkit designed to support the creation of applications that leverage the informal nature of pens.
• Batik - With Batik, you can manipulate SVG documents anywhere Java is available. You can also use the various Batik modules to generate, manipulate, transcode and search SVG images in your applications or applets.
• GEF - The goal of the GEF project is to build a graph editing library that can be used to construct many, high-quality graph editing applications. A simple, concrete design that makes the framework easy to understand and extend. Node-Port-Edge graph model that is powerful enough for the vast majority of connected graph applications. Model-View-Controller design based on the Swing Java UI library makes GEF able to act as a UI to existing data structures, and also minimizing learning time for developers familiar with Swing. High-quality user interactions for moving, resizing, reshaping, etc. GEF also supports several novel interactions such as the broom alignment tool and selection-action-buttons. Generic properties sheet based on JavaBeans introspection. XML-based file formats based on the PGML standard.
• Eclipse GEF - The Graphical Editing Framework (GEF) allows developers to take an existing application model and easily create a rich graphical editor. GEF allows a developer to quickly map any existing model to a graphical editing environment. The graphical environment is the SWT-based drawing plugin "Draw2d". The developer can take advantage of the many common operations provided in GEF and/or extend them for the specific domain. GEF is suitable for creating a wide variety of applications, including: flow builders, GUI builders, UML diagram editors (such as work-flow and class modeling diagrams), and even WYSIWYG text editors like HTML. GEF does not assume that you must build one of these applications and is application domain neutral.
• JHotDraw - JHotDraw is a Java GUI framework for technical and structured Graphics. It has been developed as a "design exercise" but is already quite powerful. Its design relies heavily on some well-known design patterns. JHotDraw's original authors have been Erich Gamma and Thomas Eggenschwiler.
• OpenJGraph - OpenJGraph is a Java library to create and manipulate graphs. Its features include directed, undirected, directed-acylic, and weighted graphs, and simple graph algorithms such as graph traversal, minimum spanning tree, and shortest path spanning trees for weighted graphs. The library also has some support for graph drawing, including straight line and orthogonal graph drawing, and for user interaction such as creating and removing a vertex, creating and removing an edge, dragging a vertex, and changing some of the vertex and edge properties.
• Grappa - Grappa is a Java graph drawing package that simplifies the inclusion of graph display and manipulation capabilities. Methods for building, manipulating, traversing and displaying graphs of nodes, edges and subgraphs. Grappa includes methods for reading in and writing out graphs using the dot text format.
• Grace - Grace is a generator for direct manipulation graph editors in Java. It can be used for any graph-like datastructure of your application.
• Vector Visuals - Vector Visuals provides an easy-to-use, object-based API for creating and manipulating Java2D-rendered shapes and images. It features object embedding, dynamic connectors, and multithreaded task support. Vector Visuals is in use in a variety of applications. A few that we know of are a business ontology viewer, a particle simulator, and a children's e-book builder.
• GeoVISTA Studio - Studio is a programming-free environment that allows users to quickly build applications for geocomputation and geographic visualization.
• NetBeans Graph Library - Graph Library has been designed to support visualization and editing of node-edge structures using drag'n'drop style of work. The library has been used in the Visual Designer in NetBeans Mobility Pack 4.1.
• Graph Modeling Framework - The Eclipse Graphical Modeling Framework (GMF) provides a generative component and runtime infrastructure for developing graphical editors based on EMF and GEF.
• Mascopt - Mascopt (Mascotte Optimization) project provides a set of tools for network optimization problems. Examples of such problems are routing, grooming, survivability, or virtual network design. Mascopt implements graph data structures, basic algorithms for graphs and import/export functionality. Mascopt provides graphical tools to display graph results.

If you happen to know of an active project that I should include, please let me know!

• (7)

• Posted by cperez
• Published: 2009-02-03

Ever think that XSLT is too inflexible, too heavywieght or just plain too complex to use? If your answer is yes, then here's a review of open source java projects that help you do ETL (i.e. Extract, Transform and Load) in a easy and scalable manner.

• Octopus - Octopus is a simple Java-based Extraction, Transform, and Loading (ETL) tool. It may connect to any JDBC data sources and perform transformations defined in an XML file. A loadjob-generator is provided to generate Octopus loadjob skeletons from an existing database. Many different types of databases can be mixed (MSSQL, Oracle, DB2, QED, JDBC-ODBC with Excel and Access, MySQL, CSV-files, XML-files,...) Three special JDBC drivers come with Octopus to support JDBC access to CSV-files (CSV-JDBC), MS-SQL (FreeTDS) and XML. Octopus supports Ant and JUnit to create a database / tables and extract/load data during a build or test process.
• Xineo - Xineo XIL (XML Import Langage) defines an XML language for transforming various record-based data sources into XML documents, and provides a fully functional XIL processing implementation. This implementation has built-in support for relational (via JDBC) and structured text (like CSV) sources, and is extensible thanks to its public API, allowing dynamic integration of new data source implementations. It also provides an abstraction over output format, and the Xineo implementation can generate output documents into stream or as DOM document. Xineo's implementation built-in data sources include : Relational data via JDBC and Structured text via regular expressions.
• CloverETL - CloverETL Features include internally represents all characters as 16bit, converts from most common character sets (ASCII, UTF-8, ISO-8859-1,ISO-8859-2, etc), works with delimited or fix-length data records, data records (fields) are internally handled as a variable-length data structures, fields can have default values, handles NULL values, cooperates with any database with JDBC driver, transforming of the data is performed by independent components, each running as an independent thread, framework implements so called pipeline-parallelism, metadata describing structure of data files (records) can be read from XML and transformation graphs can be read from XML
• BabelDoc - BabelDoc is a Java framework for processing documents in linear stages, it tracks documents and can reintroduce documents back into into the pipelines, it is monitorable and configurable through a number of interfaces, it can be run standalone, in server processes or in application servers, it can be reconfigured dynamically by text files and database tables.
• Joost - Java implementation of the Streaming Transformations for XML (STX) language. Streaming Transformations for XML (STX) is a one-pass transformation language for XML documents. STX is intended as a high-speed, low memory consumption alternative to XSLT. Since it does not require the construction of an in-memory tree, it is suitable for use in resource constrained scenarios.
• CB2XML - CB2XML (CopyBook to XML) is a COBOL CopyBook to XML converter written in Java and based on the SableCC parser generator. This project includes utilities to convert an XML instance file into its COBOL copybook equivalent string buffer and vice versa. You can find additional information about supporting Jurasic systems here.
• mec-eagle - JAVA XML XSL B2B integration software:SWING based GUI,an EDI to XML, XML to XML and XML to EDI converter,client-server architecture.All EDI standards are supported:EDIFACT,ANSI X.12,SAP IDOC,XCBL,RosettaNet,Biztalk.Included comm:SMTP,FTP,HTTP(S),PGP/MIME
• Transmorpher - Transmorpher is an environment for processing generic transformations on XML documents. It aims at complementing XSLT in order to:
  • describe easily simple transformations (removing elements, replacing tag and attribute names, concatenating documents...);
  • allowing regular expression transformations on the content;
  • composing transformations by linking their (multiple) output to input;
  • iterating transformations, sometimes until saturation (closure operation);
  • integrating external transformations.
• XPipe - XPipe is an approach to manageable, scaleable, robust XML processing based on the assembly line principle, common in many areas of manufacturing. XPipe as being an attempt to take what was great about the original Unix pipe idea and apply it for structured information streams based on XML.
• DataSift - DataSift is a powerful java data validation and transformation framework, aimed at enterprise software development, which provides developers with an extensible architecture they can fully adapt. Almost every feature in it can be configured and extended in some way.
• Xephyrus Flume - Flume is a component pipeline engine. It allows you to chain together multiple workers into a pipeline mechanism. The intention of Flume is that each of the workers would provide access to a different type of technology. For example, a pipeline could consist of a Jython script worker followed by a BeanShell script worker followed by an XSLT worker.
• Smallx - Smallx supports streaming of XML infosets to allow processing of very large documents (500MB-1GB). Processing is specified in an XML syntax that describes an XML pipeline--which is a sequence of components that consume and produce infosets. This allows chaining of XML component standards like XSLT. Also, there is a full component API that allows developers to easily write their own components.
• Nux - Nux is a toolkit making efficient and powerful XML processing easy. It is geared towards embedded use in high-throughput XML messaging middleware such as large-scale Peer-to-Peer infrastructures, message queues, publish-subscribe and matchmaking systems for Blogs/newsfeeds, text chat, data acquisition and distribution systems, application level routers, firewalls, classifiers, etc. Nux reliably processes whatever data fits into main memory (even, say, 250 MB messages), but it is not an XML database system, and does not attempt to be one. Nux integrates best-of-breed components, containing extensions of the XOM, Saxon and Lucene open-source libraries.
• KETL - KETL is an extract, transform, and load (ETL) tool designed by Kinetic Networks. KETL includes job scheduling and alerting capabilities. The KETL Server is a Java-based data integration platform consisting of a multi-threaded server that manages various job executors. Jobs are defined using an XML definition language.
• Kettle - K.E.T.T.L.E (Kettle ETTL Environment) is a meta-data driven ETTL tool. (ETTL: Extraction, Transformation, Transportation & Loading). No code has to be written to perform complex data transformations. Environment means that it is possible to create plugins to do custom transformations or access propriatary data sources. Kettle supports most databases on the market and has native support for slowly chaning dimensions on most platforms. The complete Kettle source code is over 160,000 lines of java code.
• Netflux - Metadata based tool to allow for easier manipulations. Spring based configuration, BSF based scripting support, pluggable JDBC based data sources and sinks. A server and a GUI are planned.
• OpenDigger - OpenDigger is a java based compiler for the xETL language. xETL is a language specifically projected to read, manipulate and write data in any format and database. With OpenDigger/XETL you can build Extraction-Transformation-Loading (ETL) programs virtually from and to any database platform.
• ServingXML - ServingXML is a markup language for expressing XML pipelines, and an extensible Java framework for defining the elements of the language. It defines a vocabulary for expressing flat-XML, XML-flat, flat-flat, and XML-XML transformations in pipelines. ServingXML supports reading content as XML files, flat files, SQL queries or dynamically generated SAX events, transforming it with XSLT stylesheets and custom SAX filters, and writing it as XML, HTML, PDF or mail attachments. ServingXML is suited for converting flat file or database records to XML, with its support for namespaces, variant record types, multi-valued fields, segments and repeating groups, hierarchical grouping of records, and record-by-record validation with XML Schema.
• Talend - Talend Open Studio is full-featured Data Integration OpenSource solution (ETL). Its graphical user interface, based on Eclipse Rich Client Platform (RCP) includes numerous components for business process modelling, as well as technical implementations of extracting, transformation and mapping of data flows. Data related script and underlying programs are generated in Perl and Java code.
• Scriptella - Scriptella is an ETL and script execution tool. Its primary focus is simplicity. It doesn't require the user to learn another complex XML-based language to use it, but allows the use of SQL or another scripting language suitable for the data source to perform required transformations.
• ETL Integrator - ETL (a highly unimaginative name) consists of 3 components. An ETL service engine that is a JBI compliant service engine implementation which can be deployed in a JBI container. An ETL Editor that is a design time netbeans module which allow users to design ETL process in a graphical way. An ETL Project that is a design time netbeans module which allows users to package ETL related artifacts in a jar file which could be deployed onto the ETL service engine.
• Jitterbit - Jitterbit can act as a powerful ETL tool. Operations are defined, configured, and monitored with a GUI. The GUI can create document definitions, from simple flat file structures to complex hierarchic files structures. Jitterbit includes drag-and-drop mapping tool to transform data between your various system interfaces. Furthermore, one can set schedules, create success and failure events and track the results for your integration operations. Jitterbit supports Web Services, XML Files, HTTP/S, FTP, ODBC, Flat and Hierarchic file structures and file shares.
• Apatar - Apatar integrates databases, files and applications. Apatar includes a visual job designer for defining mapping, joins, filtering, data validation and schedules. Connectors include MySQL, PostgreSQL, Oracle, MS SQL, Sybase, FTP, HTTP, SalesForce.com, SugarCRM, Compiere ERP, Goldmine CRM, XML, flat files, Webdav, Buzzsaw, LDAP, Amazon and Flickr. No coding is required to accomplish even a complex integration. All metadata is stored in XML.
• Spring Batch - Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications. Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advance technical services and features that will enable extremely high-volume and high performance batch jobs though optimization and partitioning techniques.
• JasperETL - JasperETL was developed through a technology partnership with Talend. JasperETL includes Eclipse based user interfaces for process design, transformation mapping, debugging, process viewing. The project includes over 30 connectors like flat files, xml, databases, email, ftp and more. It includes wizards to help configure the processing of complex file formats including positional, delimited, CSV, RegExp, XML, and LDIF formatted data.
• Pentaho Data Integration - Pentaho Data Integration provides a declarative approach to ETL where you specify what to do rather than how to do it. It includes a transformation library with over 70 mapping objects. In includes data warehousing capability for slowly changing and junk Dimensions. Includes support for multiple data sources including over 25 open source and proprietary database platforms, flat files, Excel documents, and more. The architecture is extensible with a plug-in mehcanism.
• Mural - Mural is an open source community with the purpose of developing an ecosystem of products that solve the problems in Master Data Management (MDM). Projects include: Master Index Studio which provides the supports the creation of a master index through the matching, de-duplication, merging, and cleansing . Data Integrator which provides extract, transform, load capability and a wide variety of data formats. Data Quality which features matching, standardization, profiling,and cleansing capabilities. Data Mashup Data Mashup which provides data mashup capability. Data Migrator which supports the migration of database objects across database instances
• Smooks - Smooks provides a wide range of Data Transforms. Supports many different Source and Result types - XML/CSV/EDI/Java/JSON to XML/CSV/EDI/Java/JSON. It supports binding of Java Object Models from any data source. It is designed to process huge messages in the GByte range.
• Data Pipeline - Data Pipeline provides data conversion, data processing, and data transformation. The toolkit has readers and writers for common file formats (CSV, Excel, Fixed-width, JDBC) along with decorators that can be chained together to process and transform data (filter, remove duplicates, lookups, validation).

Let me know if there are anymore that I may have missed.

• (5)