Donkey

We have created some tests that demonstrate some basic use cases. You can check out our code and run these tests for yourself.  Each demonstration test consists of a properties file, some input, expected output, a Specification, a Wirer, and the test itself.

As part of our drive for our alpha release, we are going to create more and more demonstration tests illustrating what the Donkey can do.

1. CsvToSqlDemoTest.java 

This test demonstrates how to read a CSV file and convert each record in the CSV file into a SQL statement that is written to an output file. But you aren't restricted to just writing SQL statements. Depending on the template you specify in the properties file, you can write to any output format you want. 

Here's a walk through of this demo test.  

2. DirectoryWatcherDemoTest.java

This test demonstrates how to:

1. Watch an "in" directory for files.
2. Read each file that appears in the directory.
3. Convert the contents to a format specified by a template.
4. Write the result to an "out" directory.
5. Move the original input files to a "completed" directory.

Whereas the CsvToSqlDemoTest runs only once, this test demonstrates a directory watcher running in a continuous loop.  

3. JdbcToCsvDemoTest.java

This test demonstrates how to setup a JDBC inbound end point, perform a simple transformation, and write the result to a CSV file.
We use an in memory HSQLDB as the inbound end point. The test creates an EMPLOYEE table and populates it with some harness data. The Donkey then slurps data from the EMPLOYEE table (as specified by the SQL query referenced in the jdbc2csv.properties file). Then each row in the result set is transformed by JdbcToCsvTransform. The results are written out as a CSV file.

This demonstration test illustrates a very simple data migration (ETL) scenario. For ETL, it is often the case that the outbound end point is CSV and not a JDBC connection.

Here's a walk through of this demo test.  

4. FileMetadataToFixedWidthDemoTest.java

This test has a really long name. It demonstrates how to read a directory, get some metadata about the files in the directory, and write a report to an output file in fixed width format. It demonstrates the use of FixedRecordDefinitions to help define the output format of the file.

It also demonstrates how to override properties using two properties files at once. This is especially useful if you want to use config in a test that is slightly different from your production configuration.

5. JdbcToJdbcDemoTest.java

This test demonstrates how to set up a JDBC inbound end point and a JDBC outbound end point. For this test we are using two in memory database schemas. One called "input" and one called "output".

It also demonstrates how to prefix your configuration properties in order to use different configuration for the same objects in different contexts. You can see what we mean by looking in jdbc2jdbc.properties. In there you can see we prefix the input database configuration with input. and the output database configuration with output.. We then "flatten" the property keys by removing the prefix depending on whether we are slurping or spitting.

Check out the wiring in JdbcToJdbcWirer.java to see what we mean.