| Share
Group discussion > Schema Transformation with HALE

Schema Transformation with HALE

Thorsten Reitz
1979 days ago

The HUMBOLDT Alignment Editor (HALE) is an open source tool for defining, evaluating and executing conceptual schema mappings. HALE allows domain experts to create logically and semantically consistent mappings and to transform geodata based on these mappings. This is done in a highly interactive and intuitive manner, so that users can concentrate on producing high-quality harmonised data sets.

Furthermore, a major focus is put on the documentation of the schema transformation process and its impacts, e.g. in the form of lineage information attached to the resultant transformed data.

Screenshot Hale

HALE uses a high-level language for expressing the mappings. They can later be used by the Conceptual Schema Transformer processing component to execute a data transformation, e.g. from a non-harmonised data source to a INSPIRE-compliant data set.

HALE is in active use. Since the release of the 1.0 version, it has been downloaded more than 6.000 times. Both research and commercial projects employ HALE.

Download HALE at http://www.esdi-community.eu/projects/hale/files.
Find the HALE Documentation online in the HALE Wiki.
Please don't forget to give us feedback using the HALE evaluation form or here, in the INSPIRE forum!

Permalink

Thorsten Reitz
1817 days ago

Loading an INSPIRE schema in a few seconds!

 

HALE 2.5.0 RC1 is now available for download. Some of the highlights include:

  • Preconfigured INSPIRE and CityGML schemas load without any dependency issues. More preconfigured schemas to come - just request them!
  • All transformations now include instant validation, with the results being displayed in the data views
  • When working with instances, counts are displayed for each schema element
  • Batch Transformation

Find out about the updates by reading the What's new page:
http://hale.igd.fraunhofer.de/new/2.5.0.RC1/

We consider this version to be a feature complete and quality assured release candidate. However, you can provide a much wider set of test environments and thus we would like to give you the opportunity to test it with your data and schemas.

Permalink

Thorsten Reitz
1578 days ago

HALE 2.6.0 brings integrated data harmonisation workflows

In the past 6 months, we've had a set of interesting projects on HALE and have been able to add a host of new features to the software. Among the most important are:

  • Integration with FME 2014
  • Create an XSLT (eXtensible Stylesheet Language Transformation) 2.0 from an Alignment
  • Work with mapping modules to collaborate on the creation and to and reuse alignments
  • Additional tools to analyse your source data such as an Occurring Properties view and a new way to view your schema as a type hierarchy
  • A lot of additional functions for the transformation engine, such as priorities and the handling of inherited property relations

Find out what's new at
http://hale.igd.fraunhofer.de/new/2.6.0/

Download HALE 2.6.0 here:
http://www.esdi-community.eu/projects/hale/files

We look forward to present you the new HALE release at the INSPIRE conference at Firenze. Get more information on the workshop here:
http://blog.dhpanel.eu/ 

 

Permalink

Maël REBOUX
1537 days ago

Great : this tool begin to be reusable by (almost) end-users.
To test soon.

Permalink

Thorsten Reitz
1440 days ago

HALE 2.7.0 with focus on Collaboration features

A new HALE version is available - we're now at 2.7.0 (downloads, documentation). These are the things that you can now do with HALE:

Project Templates

To make your start with HALE easier, HALE now offers the possibility to use pre-configured project templates, e.g. for mapping to the INSPIRE Application Schemas. This saves you steps such as loading the schemas and setting up codelists. You can share your own projects as templates, example, or even as reference mappings online, to let others in your community profit from them:

http://hale.igd.fraunhofer.de/templates/

To select a template and to load it in HALE, use the New project from template option in the File menu or use the Project URL copied from the template website and use Open Alignment project and From URL.

The Join Retype operation

Joins were on the backlog for a long time, since they are complex to resolve in our declarative approach. Now however HALE offers attribute based joins of different feature classes – to an arbitrary depth. To create a Join select multiple source types and one target type and choose the Join function.

Project and resources view

The new Project view lists all resources like source schemas, target schemas or code lists that are associated to the project. Also, you can edit basic project information in combination with the Properties view. Some types of resources, like code lists and lookup tables, can be removed here from the project using the context menu.

Export to JSON/GeoJSON

Transformed data can now be exported to JSON or GeoJSON, independently of what kind of schema the data is associated to. Objects are generically encoded as JSON/GeoJSON according to their structure.

Improved support for INSPIRE

HALE now supports the new code list XML format introduced recently by the INSPIRE registry. These code lists are relevant for the latest versions of the Annex II and III Application Schemas. In addition, transformed INSPIRE compliant features can now be saved to GML directly as an INSPIRE SpatialDataSet instead of the deprecated GML FeatureCollection.

Permalink

Fabio Vinci
1439 days ago

I'm starting to test HALE for mapping INSPIRE datasets. Just a question, what is the reference (INSPIRE documentation and /or others) of this sentence: "an INSPIRE SpatialDataSet instead of the deprecated GML FeatureCollection"?

Thank you.

Permalink

Peter Parslow
1439 days ago

gml:FeatureCollection existed in GML3.0 through GML3.1.1; it does not appear in GML3.2 (reference OGC document 07-036, clause 9.9, or equivalent in ISO 19136) - application schemas now create their own 'feature collection' features.

For example, the INSPIRE base model defines 'SpatialDataSet' (D2.5 9.8.2.1, implemented in BaseTypes.xsd). Some (draft) theme specifications (technical guidance) define their own feature collection types; others make no mention of the issue, suggesting (to me, at least) that they'd expect a conformant data set to use the base:SpatialDataSet as its root element. Unless of course the dataset / feature collection in question is being returned from a WFS, in which case examples in INSPIRE documentation (e.g. D2.5 F.3) imply that wfs:FeatureCollection would be used.

Permalink

Fabio Vinci
1439 days ago

Ok, but there is also a recommendation in the D2.7_v3.3rc3 (Recommendation 11 at B.3.3) that states: For the exchange of spatial objects in GML, an XML document with a <FeatureCollection> root element from ISO 19142:2010 (WFS 2.0) should be used. 

Permalink

Peter Parslow
1436 days ago

Over to JRC then, for any comment on the impending contradiction between that new / draft recommendation & the existing SpatialDataSet feature (and specialisation of it in certain themes).

Perhaps D2.7 recommendation 11 will only apply to exchange by WFS - although there is no technical reason why an element of 19142 shouldn't be used in exchanging whole files via any other means.

Permalink

Thorsten Reitz
1433 days ago

All right, while I can't resolve the question how to use SpatialDataSet and featureCollection correctly for now, I can announce the next version of HALE (2.8.0):

HALE 2.8.0 brings PostGreSQL access

Out of the “box”, you can now connect to PostgreSQL/PostGIS databases as a schema and data source for a transformation with the new JDBC based database support in HALE. Each table in the database is loaded as a type; to combine information from different database tables in a mapping, use the Join function, which has also been improved. To support additional database types, HALE’s plug-in mechanisms can be used. For more information, see the Database Import documentation.

Sample Source Data

Source data is used in the live transformation to directly see the impact of changes to the mapping in the transformed data. Live transformation performance will become sluggish is the source dataset is very big, however. So, you can now configure that only a sub-set of the data is loaded into HALE, which in most cases also speeds up the import.

Groovy Transformation Scripts

While the transformations functions delivered with HALE cover a lot of ground, an occasional request by advanced users was to provide their own functions or customize existing ones. You can now can combine the regular HALE transformation functions with powerful Groovy scripts. HALE provides easy-to-use APIs for accessing and creating complex instances. To author the scripts, a new script editor is included that supports syntax highlighting and script validation. Detailed help and example code is available as well.

Improved classification mapping function

The classification mapping function now uses a tabular representation and has some new features:

  • Fill the lookup table with values encountered in the source data (Occurring values) or enumeration values defined in the schema
  • Load the lookup table from a CSV or Excel file
  • Save the lookup table as a CSV or Excel file

As usual, there has been a set of other improvements (about 15). You can go here to learn more about these.

My colleague Simon will post separate workflow descriptions at the dhpanel blog for the usage of PostGreSQL databases and on the Groovy Scripting in the next days. Enjoy your work with HALE 2.8.0!

Permalink

Simon Templer
1431 days ago

On the FeatureCollection/SpatialDataSet topic:

The SpatialDataSet is recommended to be used with pre-defined data sets, e.g. as offered through an INSPIRE Download Service that is based on an Atom feed. The SpatialDataSet in addition to the member objects may include the metadata and an INSPIRE identifier for the data set.

The FeatureCollection element ist still part of the GML 3.2.1 XML Schema, but in the module deprecatedTypes. Thus probably the recommendation to use the WFS 2.0 FeatureCollection element. As the INSPIRE GML Application Schemas usually don't import the WFS 2.0 schema, HALE will by default only find the GML FeatureCollection type for the GML export.

Permalink

Thorsten Reitz
1119 days ago

Time to announce....
HALE 2.9.0 – new look, new tools, INSPIRE codelists + Annex II/III

We have just completed testing and documentation for the new HALE 2.9.0 release. There is a lot that has changed under the hood for the 2.9 release. Apart from a lot of bug fixes and an upgrade of the underlying platform from Eclipse Indigo to Eclipse Luna, there also was a new build system introduced that allows us to again provide an installer for Windows and a version of HALE that runs on current Mac OS X operating systems.

It comes with a long list of useful new features, as you would expect after the relatively long break since the last release of HALE. Get your download here and read in detail about the new features here. Here are the headline news:

New mapping documentation format

We added a new mapping documentation format based on HTML and SVG that is interactive and optimal for web display.

Extended command line interface

For integrating schema transformation into your infrastructure an easy way is to use the HALE command line interface. Some have already used the command line interface part of the integration with the HALE FME plug-in – but this is very limited in functionality. The new command line interface is far more flexible in its configuration and allows data export to be easily configured with support through the HALE user interface. A detailed post with an example project will be posted in the next days.

INSPIRE Code List Support

Browse and search the code lists from the INSPIRE registry provided by the JRC and directly load them in HALE.

Create ATOM feed based INSPIRE download service

HALE aids you in creating an ATOM feed based INSPIRE download service by generating a dataset feed for your transformed data automatically. Just enable the creation of the feed in the INSPIRE SpatialDataSet export and provide some basic information for the feed metadata. Information on feature types and coordinate reference systems will be included automatically.

INSPIRE Annex II and III schemas now built in

HALE supports any schema, but usage of the inbuilt schemas is super-simple, so we have extended the set of inbuilt schemas to also include the final versions of all Annex II and III schemas.

Smart Retype

Do you have source and target schema types that are very similar and differ just in a few properties or only the namespace? Now the Retype transformation function has configuration options to automatically copy properties/attributes with similar names.

Reprojection/CRS Conversion

You can now include a conversion to a given coordinate reference system (CRS) as part of writing the transformed data to GML. You can identify the CRS with an EPSG code or define a custom CRS using a WKT representation.

As usual, there has been a set of other improvements (about 15). You can go here to learn more about these. We will post separate workflow descriptions for the usage of the CLI and on the INSPRIE Code Lists in the next days. Enjoy your work with HALE 2.9.0!

 

Permalink

Giuseppe Procino
887 days ago

I'm using HALE to transform my building dataset in according to the last Inspire Data Specification on  Buildings theme (v.3.0).

For each building I can use ConditionOfConstructionValue (a code list - pag. 61/323) to describe building costruction status.

Whithin Buildings schema (BuildingsCore2D.xsd) at ConditionOfCostruction complex attribute I see the following attributes:

image

What's the attribute that I must fill with ConditionOfConstructionValue value?

Is there a Inspire document that describes it (How use Data specification rules with GML encoding)? Is there a tutorial? Where I can download any examples?

NOTE:

In the OGC GML Specification 3.2.1 there is AssociationAttributeGroup (§ 7.2.3.2 pg. 25)

gmlimage

 

with similar attributes at ConditionOfConstruction within Hale (similar because ConditionOfConstruction within Hale has nilReason
owns, remoteSchema attributes)

Regards

Giuseppe Procino

Permalink

Simon Templer
886 days ago

@Giuseppe Procino

For code lists in the current versions of the INSPIRE schemas reference types are used. The intention is to point to unique URIs that represent the code list values and in an optimal case those URIs are resolvable and point to a definition of the value like in the INSPIRE registry provided by the JRC (e.g. http://inspire.ec.europa.eu/codelist/ConditionOfConstructionValue/functional).

Usually you use a simple link where it is only necessary to provide a value for the href property.

In HALE you can make use of the INSPIRE registry to be able to easily select appropriate code list values. Example steps if using the Assign relation to assign a fixed code list value for all buildings:

1. Load a code list

HALE - Load an INSPIRE code list

2. Select the href property of the corresponding code list property

HALE - Reference property for code lists

3. Create a new Assign relation

4. Assign a code list to the property (in the relation wizard)

HALE - Assign a code list to a property

5. Select the desired value

HALE - Select a code list value

Naturally you can also use that when not assigning a fixed value, e.g. in the Classification mapping function.

Regards,

Simon

Permalink

Marta
593 days ago

Hi, i did an export alignment of an app-Schema Configuration. It seemed to work because the new workspace and layers are listed in geoserver. But, in the layer preview, on the Open Layers option, i see no data.

Is there any other step that i'm missing?

I exported the transformed data to a GML (INSPIRE) and i do visualize the data in QGIS. So, i can't figure out the problem.

Can you help me?

Tks

Permalink