Posts tagged with: arc

If there was a "most productive SemWeb coder" category in Danny's "This Week's Semantic Web", this week's turn would probably be Keith Alexander's. Last week, he provided no fewer than three ARC2 Plugins:

• An RDFa Serializer
• Utilities for working with ARC structures (e.g. filter, merge, or diff)
• A SPARQL (Re-)Serializer (very handy for checking/adjusting SPARQL queries before they are passed to the store)

While at it, he also implemented a SPARQL+ wrapper for Talis Platform stores.

I think I blogged about Morten's RemoteEndpoint plugin a while back (this one should really become part of the core codebase), but did I mention Peter Krantz' File System Synchronizer? It keeps an RDF Store in sync with a file system directory which enables a really nice option to implement larger RDF editing systems on top of ARC: By using editing tools that work with small RDF files (quick response times and everything) and his plugin, it becomes possible to provide rich query functionality over the whole dataset without the store getting in the way of the publishing tools. RDF index rebuilding can be slow, de-coupling read from write operations and introducing an asynchronous update process is a nice solution.

Awesome stuff.

The latest ARC revision got two new features: SPARQL Triggers and MySQL function extensions for SPARQL.

SPARQL Triggers

Triggers in ARC were suggested by Dan Brickley, who is experimenting with dynamically populated/updated Group definitions. What you can effectively do now in ARC is associating custom trigger classes with SPARQL query types, which will then be automatically called after registered query types, for example to refresh inferred Graphs:

$config = array(
  ...
  'store_triggers' => array(
    /* register LOAD triggers */
    'load' => array('updateFriendsList', 'crawlXFNLinks'),
  ),
);
$ep = ARC2::getStoreEndpoint($config);
$ep->go();

MySQL Extension Functions

Morten Frederiksen did it again. He sent in about 10 lines of code which he suggested to add to ARC's SQL rewriter. The effect? ARC suddenly has access to dozens of MySQL functions. That's CONCAT, CURDATE, MD5, UNIX_TIMESTAMP, and many more. A namespace for MySQL function URIs is now online, and queries look like this:

PREFIX mysql: <http://web-semantics.org/ns/mysql/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT ?person WHERE {
  ?person foaf:givenname ?n1 ;
          foaf:family_name ?n2 .
  FILTER (mysql:concat(?n1, " ", ?n2) = "Alec Tronnick") .
}

I talked a little bit more about these things with Danny Ayers in a recent podcast.

OK, you're probably already wondering if Morten and I have a link exchange contract, but anyway: He just announced a plugin for ARC that provides "access to remote SPARQL endpoints as if they were local stores." Cool stuff :-)

I guess I already pushed out enough ARC spam today, so I'll keep things short: SPARQL is now a W3C Recommendation!

What I'm personally very happy about is the Implementation Survey which features two pure-PHP implementations*. This really opens the door for mainstream Web Developers to start exploring RDF and SPARQL on off-the-shelf hosted web servers. Everything I create these days (e.g. the ARC site, including the bots and archive generators there, or this blog) is powered by SPARQL. It's an amazing productivity booster as you never have to worry about complicated JOINs or evolving database schemas again. You can just code away and it's great fun to work with. Want more Testimonials? The Data Access Working Group collected quite a number of them from W3C member organizations.

* Don't let yourself be fooled by RAP's low report scores, their SPARQL engine is quite mature, they just didn't run the whole test suite.

Together with Morten Frederiksen and Dan Brickley (who is revisiting his SparqlPress idea), I've created a WordPress extension (called "RDF Tools") that adds an (ARC-based) RDF Store and SPARQL Endpoint to the blogging system. The store is kept separate from the WP tables (i.e. it's not a wrapper), but you can use WP's nice admin screens to configure it (screenshot), and given the amount of developer-friendly hooks that WP offers, I'm curious what can be done now, possibly in combination with other extensions such as those Alexandre Passant is working on. It could perhaps also be handy as a deployment accelerator for knowee.

I'm blessed with a small but first-class community around ARC that helps me with bug reports, patches, encouraging feedback, and nifty ideas. One example for the latter was Morten Frederiksen's invention to allow ARC to be extended with third party plugins. He even demonstrated the utility by enhancing the toolkit with a remote SPARQL endpoint for his named graph exchange work. ARC plugins are not bundled with the core codebase (which is meant to stay compact), but can easily be integrated in any ARC installation (Developer documentation is now online, too).

My first own plugin was triggered by Tim Berners-Lee's suggestion to write a lightweight request handler for an RDF-powered Data Wiki, as described in a recent Tech Report (PDF) and already implemented with Algae. I had to tweak the SPARQL+ spec and ARC's Query Parser to make it compatible with Eric Prud'hommeaux's SPARQL/Update flavor. This had the nice side-effect that all three SPARQL Write proposals (SPARUL, SPARQL/Update, SPARQL+) now (almost) share a common subset for basic INSERTs and DELETEs. After these updates, writing the plugin itself became almost trivial.

The code is still experimental and limited, but it's available for download, together with setup instructions. The Data Wiki plugin doesn't require a database (unlike the other SPARQL components in ARC) and supports update calls sent by RDF editors such as the Tabulator. I've set up a demo RDF wiki and will try to add remote update functionality to my own editor (to be renamed) now as well. Hmmm, would be cool to have a selection of generic tools to collaboratively read from and write to shared RDF spaces one day.

After writing a bunch of instructions for ARC2 this week, I think/hope it is now finally ready for experiments and feedback. The site will get more documentation and code snippets in the coming weeks, and some components are not even part of the release. However, I've been waiting long enough already. So, here goes:

• Mailing list announcement
• The new website (built with PHP and ARC2)
• "Introducing ARC2"
• ARC-DEV group mailing list

Shout-outs to everyone who helped with bug reports, encouraging feedback, and suggestions for the new release. Special thanks to CivicActions and Jonathan Hendler for development support and various stress tests of earlier versions.

The new ARC site is coming along quite nicely. Last week I implemented two (low-level) agents that log IRC conversations and mails to ARC-DEV. RDF and SPARQL make such things incredibly easy. Today, I started writing documentation for the preview release of ARC2, and one the core changes to ARC1 is the removal of the API class for inserts and deletes in favour of an extended SPARQL, called SPARQL+ which enables aggregates, LOAD, INSERT, and DELETE, without the need for major query engine code additions.

LOAD is compatible with the LOAD operation introduced in the SPARUL proposal:

LOAD <http://example.com/> INTO <http://example.com/archive>

INSERT and DELETE are different, though. They re-use the LOAD and CONSTRUCT handlers which simplified the implementation and will hopefully make it easier for people who just learned SPARQL's standard syntax. INSERT and DELETE in SPARQL+ each support two different forms, one for explicit triples (with simple wildcards in DELETE queries), and one for dynamically CONSTRUCTed ones, e.g.

DELETE {
 <#foo> <bar> "baz" .
 <#foo2> <bar2> ?any .
}

or

INSERT INTO <http://example.com/inferred> CONSTRUCT {
  ?s foaf:knows ?o .
}
WHERE {
  ?s xfn:contact ?o .
}

More examples and detailed information about how exactly SPARQL+ extends the SPARQL grammar are available in ARC2's SPARQL+ documentation section

I'm still working on the new website for ARC, but I managed to set up a group mailing list yesterday. It's a little (*cough*) experimental, based on ARC2 and Trice (another forthcoming semsol product). So, this is a shout-out to ARC users and developers with an invitation to subscribe and help me test that "DIY SPARQL Mailman" before I do a proper announcement for the new site and community tools (hopefully later this week).

Thanks in advance,
Benji

Yesterday, SWEO's 3rd F2F took place at the MIT, and although I couldn't afford in-person attendance, Leo Sauermann from AI research center DFKI enabled a near-equivalent in Germany. I missed the first agenda item due to the usual Deutsche "zanks for traffelink wizz us" Bahn delays, but apart from that it was a fun event and worth the 4 + 6 hour train journey. Super-modern DFKI was impressive, and the multi-screen polycom with remote-controlled cam truly rocked.


pic by Leo

The train back had power sockets, so I could hack a bit, and when I arrived in DDorf at 7am this morning, I had a working RDFa parser for ARC's next release. Almost there, now..

OK, I met this week's 2nd deadline and finished ARC2's SPARQL test suite report. Pass/Fail results as of today: 317/67 (Sept. 22nd: 352/84). That's a huge step forward compared to ARC1, so I'm quite happy.

Next actions: Making the knowee prototype public (deadline missed, boo!), and relaunching the ARC site, together with proper community tools and the new release.

This is going to change everything. Well, almost. I will continue to work on my Semantic Web solutions, but there will be a major re-branding and finally a focused roadmap. My code experiments and projects are going to be critically reviewed and consolidated. (I can't tell yet what stuff is going to be continued, but I'll keep my SWEO commitments, esp. the knowee community project which is going to start in April).
Quite some orga action coming up, but I'm looking forward to a clean bengee.reboot()

• I'll move from Essen to Düsseldorf, which is closer to Cologne, the DUS airport, and also a little away from the Web periphery here, with the Ruhr Valley still in reach, though.
• The appmosphere wordplay is going to be discontinued. No German really managed to pronounce or remember it correctly, and the *-osphere naming is rather overused these days anyway.
• The new brand will most probably be semsol.com which is going to be transformed to a Semantic Web Agency. (I've always been a frontend developer, combing this with an in-house RDF system will hopefully form a nice USP for the anticipated move towards info-driven Web apps.)
• The open source RDF framework currently named semsol will get a new name (perhaps just "semsol suite", we'll see), and there will be more product-style solutions (a browser, an editor, a schema manager, etc.).
• ARC will keep its name, but is going to be re-coded as ARC2 based on the experience and feedback obtained so far.
• Less research-y slippery slopes.
• More Germany-targeted activities.

A spontaneous invitation to DrupalCon got me driving to Brussels yesterday to finally meet the CivicActions folks I've been working for during the last months. Unfortunately, I missed Jonathan Hendler's NINA presentation about adding ARC's SPARQL API to Drupal for building a faceted browser, but we chatted quite a bit about it after lunch. I still have to learn a lot about Drupal, but one of the really interesting things is that it provides an extension called Content Construction Kit (CCK) that simplifies defining flexible forms and their elements. Drupal generates an HTML page for every resource ("node" in Drupal-speak) created via CCK. The thing that's missing is mapping the structured CCK nodes to RDF to enable optimized SPARQL querying while keeping editing simple and integrated. We discussed the potential of not only ex- but also importing RDF data into CCK. And how cool it could be to directly convert RDFS/OWL to CCK field definitions. Good news is that there are several hooks to RDF-enhance Drupal without running into synchronization issues or forcing the replacement of built-in components.

CivicActions was a gold sponsor and Dan Robinson introduced me to some of the core Drupal developers. And as it turned out, some of them are already thinking about direct RDF support for Drupal (partly triggered by TimBL using Drupal for blogging, partly because Drupal's internal structure isn't really far away from a graph-based model). I'm aware of three efforts now to add RDF to Drupal in some way, there may be more.

But it's not only the Drupal crowd which is looking at SemWeb technology. At lunch, I met Johan Janssens, lead developer of the Mambo spin-off Joomla!, who told me about a SemWeb project proposal for their 2006 Google Summer of Code. (There is another one in the ideas section.) The project took more than just this summer (welcome to RDF development ;), and the outcome is not going to be added to Joomla! anytime soon, but obviously the PHP community is getting aware of RDF's potential benefits and is starting to play with RDF, OWL, and SPARQL. And it's approaching the SemWeb from a practical point of view which just can't be bad.

Update: The current RDFa primer is *not* broken wrt to WebArch, the examples were fixed two weeks ago. I've also removed the "no developer support" rant, just received personal support ;-)
While searching for a suitable output format for a new RDF framework, I've been looking at the various semantic hypertext approaches, namely microformats, Structured Blogging, RDFa, and Embedded RDF (eRDF). Each one has its pros and cons:

Microformats:

• (+) widest deployment so far
• (+) integrate nicely with current HTML and CSS
• (-) centralized project, inventing custom microformats is discouraged
• (-) don't scale, the number of MFs will either be very limited, or sooner or later there will be class name collisions

Structured Blogging:

• (+) a large number of supporters (at least potentially, the supporters list is huge, although this doesn't represent the available tools)
• (+) not a competitor, but a superset of microformats
• (-) the metadata is embedded in a rather odd way
• (-) the metadata is repeated
• (-) the use cases are limited (e.g. reviews, events, etc)

RDFa:

• (+) follows certain microformats principles (e.g. "Don't repeat yourself")
• (+) freely extensible
• (+) All resource descriptions (e.g. for events, profiles, products, etc.) can be extracted with a single transformation script
• (+) RDF-focused
• (+) W3C-supported
• (-) Not XHMTL 1.0 compliant, it will take some time before it can be used in commercial products or picky geek circles
• (-) The default datatype of literals is rdf:XMLLiteral which is wrong for most deployed properties

eRDF:

• (+) follows the microformats principles
• (+) freely extensible
• (+) All resource descriptions (e.g. for events, profiles, products, etc.) can be extracted with a single transformation script
• (+) uses existing markup
• (+) XHTML 1.0 compliant
• (+) RDF-focused
• (-) Covers only a subset of RDF
• (-) Does not support XML literals

So, both RDFa and eRDF seem like good candidates for embedding resource descriptions in HTML. The two are not really compatible, though, it is not easily possible to create a superset which is both RDFa and eRDF. However, my publishing framework is using a Wiki-like markup language (M4SH) which is converted to HTML, so I can add support for both approaches and make the output a configuration option. Maybe it's even possible to create a merged serialization without confusing transformers.

I'll surely have another look at RDFa when there is better deployment potential. For now, I've created a M4SH-to-eRDF converter (which is going to be available as part of the forthcoming SemSol framework), and an eRDF parser that can generate RDF/XML from embedded RDF. I've also added some extensions to work around (plain) eRDF's limitations, the main one being on-the-fly rewriting of owl:sameAs assertions to allow full descriptions of remote resources, e.g.

<div id="arc">
  <a rel="owl-sameAs" href="http://example.com/r/001#001"></a>
  <a rel="doap-maintainer" href="#ben">Benjamin</a>
</div>

is automatically converted to

<http://example.com/r/001#001> doap:maintainer <#ben>

The parser can be downloaded at the ARC site (documentation).
I've also put up a little demo service if you want to test the parser.

As of today, ARC has its own website: arc.web-semantics.org. It comes with a new release that adds a convenient API to access the different components (Store, Loader, Smusher, etc.). This should make working with RDF a lot easier (easyR ™ ;)

Basic documentation is up, more to follow. I've also finally set up an SVN repository and there is going to be a mailing list soon as well.

The site is based on a new publishing framework I'm currently developing, by the way. It's SPARQL-based and uses a Wiki-like markup (M4SH, pronounced "mash") which can be converted to HTML or RDF/XML. The HTML is going to be either eRDF- or RDFa-enhanced, I haven't decided yet. At the moment it simply produces classical HTML.