[dash-dev] First conversion of 3.6.2 projects

[Aaron Digulla]

Hello,

I've converted the first batch of 3.6.2 archives from
http://www.eclipse.org/downloads/, namely:

eclipse-3.6.2-delta-pack.zip
eclipse-modeling-helios-SR2-incubation-linux-gtk.tar.gz
eclipse-rcp-helios-SR2-linux-gtk.tar.gz
eclipse-reporting-helios-SR2-linux-gtk.tar.gz
eclipse-SDK-3.6.2-win32.zip
org.eclipse.rcp-3.6.2.zip
org.eclipse.rcp.source-3.6.2.zip

That's 1074 artifacts, 283 with sources.

You can find the result in
/home/nexus/workspace/org.eclipse.dash.m4e.tools/tmp/m2repo/

Due to a bug, the repo is still polluted with non-Eclipse artifacts.

During the conversion, I had warnings like this one:

WARNING ../tmp/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
differs from
../tmp/eclipse-SDK-3.6.2-win32_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar

A quick search found four archives which contain this file:

./eclipse-reporting-helios-SR2-linux-gtk_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
./eclipse-modeling-helios-SR2-incubation-linux-gtk_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
./eclipse-rcp-helios-SR2-linux-gtk_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
./eclipse-SDK-3.6.2-win32_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar

Comparing the first two shows that JUnit 3.8.2 from
eclipse-reporting-helios-SR2-linux-gtk.tar.gz and
eclipse-modeling-helios-SR2-incubation-linux-gtk.tar.gz have differences
near the beginning.

In the file, it looks like this:

948 Defl:N 517 46% 03-18-11 17:09 7ce61aa8
META-INF/MANIFEST.MF
948 Defl:N 517 46% 03-18-11 17:15 7ce61aa8
META-INF/MANIFEST.MF

So you can see the file date is different. How can that happen? I
understand that Eclipse recreates the MANIFEST.MF when the JAR is signed
but why are many plug-ins signed several times?

Background: To make sure everything is ok, I check that all files are
identical when I merge repositories. Only, they aren't...

Regards,

[Alex Blewitt]

Post by Aaron Digulla
Hello,
I've converted the first batch of 3.6.2 archives from
eclipse-3.6.2-delta-pack.zip
eclipse-modeling-helios-SR2-incubation-linux-gtk.tar.gz
eclipse-rcp-helios-SR2-linux-gtk.tar.gz
eclipse-reporting-helios-SR2-linux-gtk.tar.gz
eclipse-SDK-3.6.2-win32.zip
org.eclipse.rcp-3.6.2.zip
org.eclipse.rcp.source-3.6.2.zip
That's 1074 artifacts, 283 with sources.
You can find the result in
/home/nexus/workspace/org.eclipse.dash.m4e.tools/tmp/m2repo/

Looks good from what I've seen briefly. A couple of thoughts:

<dependency>
<groupId>org.eclipse.jface</groupId>
<artifactId>org.eclipse.jface</artifactId>
<version>[3.2.0,4.0.0)</version>
<optional>false</optional>
</dependency>

We can probably optimise out the <optional>false</optional> if it's optional.

Secondly, how does Maven handle the version range dependency? I know technically it should support it, but I am concerned it might not.

We *should* be able to find out what the release version is based on what we know is in the repository. For example, org/eclipse/jface/org.eclipse.jface/3.6.2 exists, so we could replace [3.2.0,4.0.0) with 3.6.2. This might make it easier for direct resolutions, as well as recording what was actually compiled against.

Post by Aaron Digulla
Due to a bug, the repo is still polluted with non-Eclipse artifacts.

No problem - we can probably just miss out anything that isn't in the org/eclipse tree.

Post by Aaron Digulla
WARNING ../tmp/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
differs from
../tmp/eclipse-SDK-3.6.2-win32_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
./eclipse-reporting-helios-SR2-linux-gtk_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
./eclipse-modeling-helios-SR2-incubation-linux-gtk_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
./eclipse-rcp-helios-SR2-linux-gtk_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
./eclipse-SDK-3.6.2-win32_home/m2repo/org/junit/org.junit/3.8.2/org.junit-3.8.2.jar
Comparing the first two shows that JUnit 3.8.2 from
eclipse-reporting-helios-SR2-linux-gtk.tar.gz and
eclipse-modeling-helios-SR2-incubation-linux-gtk.tar.gz have differences
near the beginning.
948 Defl:N 517 46% 03-18-11 17:09 7ce61aa8
META-INF/MANIFEST.MF
948 Defl:N 517 46% 03-18-11 17:15 7ce61aa8
META-INF/MANIFEST.MF
So you can see the file date is different. How can that happen? I
understand that Eclipse recreates the MANIFEST.MF when the JAR is signed
but why are many plug-ins signed several times?

There may be a bug in which a signed jar is already re-signed; this wouldn't change the bits but would update the time processed.

Post by Aaron Digulla
Background: To make sure everything is ok, I check that all files are
identical when I merge repositories. Only, they aren't...

Is the junit a special case? If so, we can probably ignore it.

Alex

[Aaron Digulla]

Post by Alex Blewitt

Post by Aaron Digulla
You can find the result in
/home/nexus/workspace/org.eclipse.dash.m4e.tools/tmp/m2repo/

<dependency>
<groupId>org.eclipse.jface</groupId>
<artifactId>org.eclipse.jface</artifactId>
<version>[3.2.0,4.0.0)</version>
<optional>false</optional>
</dependency>
We can probably optimise out the <optional>false</optional> if it's optional.

Yes.

Post by Alex Blewitt
Secondly, how does Maven handle the version range dependency? I know

technically it should support it, but I am concerned it might not.

Maven creates local metadata files which contain the available versions
on a server and then tries to find a match. Unlike p2, this happens
instantly, so it's not a performance issue.

Post by Alex Blewitt
We *should* be able to find out what the release version is based on
what we know is in the repository. For example,
org/eclipse/jface/org.eclipse.jface/3.6.2 exists, so we could replace
[3.2.0,4.0.0) with 3.6.2. This might make it easier for direct
resolutions, as well as recording what was actually compiled
against.

I think there is a bit more to that. If you see a version range, then
that version range was in the manifest. Without a version range, the
conversion step selects what it can find in the plugins/ directory that
is being converted. Examples:

org.jdom:org.jdom:1.0.0 wants org.apache.xerces but no version was
specified and this dependency is not available and it's optional. Result:

<dependency>
<groupId>org.apache.xerces</groupId>
<artifactId>org.apache.xerces</artifactId>
<version>[0,)</version>
<optional>true</optional>
</dependency>

org.eclipse.pde.build-3.6.2.pom contains ranges and fixed versions. For
example the dependency for org.eclipse.equinox.p2.publisher has no
version range in the MANIFEST.MF but it is available during the
conversion, so it's fixed to 1.1.0:

<dependency>
<groupId>org.eclipse.equinox</groupId>
<artifactId>org.eclipse.equinox.p2.publisher</artifactId>
<version>1.1.0</version>
<optional>true</optional>
</dependency>

There are several goals here:

- Preserve the original version range so people can see the minimum
dependency which still satisfies the needs of an artifact
- Stable builds
- Automatic updates

I see several solutions:

- Use properties for versions so consumers can override them
- Omit all versions and create one or more parent POMs with huge
dependencyManagement elements to specify the default versions
- Create profiles with the dependency versions. One profile uses the
range, the other uses the version which is in the same conversion.

Open issues:

- How do we resolve conflicts?
- How do we merge updates?

Resolving conflicts:

Imagine there is a.b.c:3.6.0 and e.f.g:3.6.0.

a.b.c:3.6.0 wants x.y.z:3.5.0 while e.f.g:3.6.0 wants x.y.z:3.*6*.0.

Example: org.apache.commons.logging comes as 1.0.4 and 1.1.1. For
org.w3c.dom.smil, we have 1.0.0 and 1.0.1. javax.xml.bind comes as 2.0.0
and 2.2.0.

Should we:

- Notify the team behind a.b.c to update their dependencies? What if
they refuse?
- Update the dependency ourselves (probably without knowing what we're
breaking)?
- Keep two versions of x.y.z?

Merging updates:

Say we have imported 3.6.0 into Nexus. Now, SR1 is released which
includes x.y.z_3.6.1. a.b.c:3.6.0 needs it but didn't specify a version
range. There is no a.b.c:3.6.1.

Should we

- release a.b.c:3.6.0.1 and make it depend on x.y.z_3.6.1?
- Update the parent POM and release everything in 3.6.0 as 3.6.0.1?
- Release a a.b.c:3.6.*1* even though there wasn't one?
- Create a report which lists such cases to consumers can make an
educated decision themselves?
- Leave it to the consumers to figure out?
-

Post by Alex Blewitt

Post by Aaron Digulla
Due to a bug, the repo is still polluted with non-Eclipse artifacts.

No problem - we can probably just miss out anything that isn't in the org/eclipse tree.

I cleaned the repo but there is still some artifacts outside of org/eclipse:

./com/ibm/icu/com.ibm.icu/4.2.1/com.ibm.icu-4.2.1.pom
./com/jcraft/jsch/com.jcraft.jsch/0.1.41/com.jcraft.jsch-0.1.41.pom
./com/lowagie/text/com.lowagie.text/2.1.7/com.lowagie.text-2.1.7.pom
./javax/xml/stream/javax.xml.stream/1.0.1/javax.xml.stream-1.0.1.pom
./javax/xml/bind/javax.xml.bind/2.0.0/javax.xml.bind-2.0.0.pom
./javax/xml/bind/javax.xml.bind/2.2.0/javax.xml.bind-2.2.0.pom
./javax/xml/soap/javax.xml.soap/1.2.0/javax.xml.soap-1.2.0.pom
./javax/xml/javax.xml/1.3.4/javax.xml-1.3.4.pom
./javax/xml/rpc/javax.xml.rpc/1.1.0/javax.xml.rpc-1.1.0.pom
./javax/transaction/javax.transaction/1.1.1/javax.transaction-1.1.1.pom
./javax/servlet/source/javax.servlet.source/2.5.0/javax.servlet.source-2.5.0.pom
./javax/servlet/javax.servlet/2.5.0/javax.servlet-2.5.0.pom
./javax/servlet/jsp/javax.servlet.jsp/2.0.0/javax.servlet.jsp-2.0.0.pom
./javax/persistence/javax.persistence/2.0.1/javax.persistence-2.0.1.pom
./javax/mail/javax.mail/1.4.0/javax.mail-1.4.0.pom
./javax/wsdl/javax.wsdl/1.5.1/javax.wsdl-1.5.1.pom
./javax/activation/javax.activation/1.1.0/javax.activation-1.1.0.pom
./java_cup/runtime/java_cup.runtime/0.10.0/java_cup.runtime-0.10.0.pom
./org/objectweb/asm/org.objectweb.asm/3.2.0/org.objectweb.asm-3.2.0.pom
./org/apache/jasper/org.apache.jasper/5.5.17/org.apache.jasper-5.5.17.pom
./org/apache/ws/org.apache.ws.commons.util/1.0.0/org.apache.ws.commons.util-1.0.0.pom
./org/apache/batik/org.apache.batik.svggen/1.6.0/org.apache.batik.svggen-1.6.0.pom
./org/apache/batik/org.apache.batik.transcoder/1.6.0/org.apache.batik.transcoder-1.6.0.pom
./org/apache/batik/org.apache.batik.dom.svg/1.6.0/org.apache.batik.dom.svg-1.6.0.pom
./org/apache/batik/org.apache.batik.xml/1.6.0/org.apache.batik.xml-1.6.0.pom
./org/apache/batik/org.apache.batik.util.gui/1.6.0/org.apache.batik.util.gui-1.6.0.pom
./org/apache/batik/org.apache.batik.util/1.6.0/org.apache.batik.util-1.6.0.pom
./org/apache/batik/org.apache.batik.pdf/1.6.0/org.apache.batik.pdf-1.6.0.pom
./org/apache/batik/org.apache.batik.parser/1.6.0/org.apache.batik.parser-1.6.0.pom
./org/apache/batik/org.apache.batik.dom/1.6.0/org.apache.batik.dom-1.6.0.pom
./org/apache/batik/org.apache.batik.bridge/1.6.0/org.apache.batik.bridge-1.6.0.pom
./org/apache/batik/org.apache.batik.css/1.6.0/org.apache.batik.css-1.6.0.pom
./org/apache/batik/org.apache.batik.ext.awt/1.6.0/org.apache.batik.ext.awt-1.6.0.pom
./org/apache/xml/org.apache.xml.serializer/2.7.1/org.apache.xml.serializer-2.7.1.pom
./org/apache/xml/org.apache.xml.resolver/1.2.0/org.apache.xml.resolver-1.2.0.pom
./org/apache/lucene/org.apache.lucene/1.9.1/org.apache.lucene-1.9.1.pom
./org/apache/lucene/org.apache.lucene.analysis/1.9.1/org.apache.lucene.analysis-1.9.1.pom
./org/apache/axis/org.apache.axis/1.4.0/org.apache.axis-1.4.0.pom
./org/apache/wsil4j/org.apache.wsil4j/1.0.0/org.apache.wsil4j-1.0.0.pom
./org/apache/oro/org.apache.oro/2.0.8/org.apache.oro-2.0.8.pom
./org/apache/bcel/org.apache.bcel/5.2.0/org.apache.bcel-5.2.0.pom
./org/apache/commons/org.apache.commons.discovery/0.2.0/org.apache.commons.discovery-0.2.0.pom
./org/apache/commons/org.apache.commons.logging/1.0.4/org.apache.commons.logging-1.0.4.pom
./org/apache/commons/org.apache.commons.logging/1.1.1/org.apache.commons.logging-1.1.1.pom
./org/apache/commons/org.apache.commons.lang/2.3.0/org.apache.commons.lang-2.3.0.pom
./org/apache/commons/org.apache.commons.httpclient/3.1.0/org.apache.commons.httpclient-3.1.0.pom
./org/apache/commons/org.apache.commons.net/2.0.0/org.apache.commons.net-2.0.0.pom
./org/apache/commons/org.apache.commons.el/1.0.0/org.apache.commons.el-1.0.0.pom
./org/apache/commons/org.apache.commons.codec/1.3.0/org.apache.commons.codec-1.3.0.pom
./org/apache/commons/org.apache.commons.collections/3.2.0/org.apache.commons.collections-3.2.0.pom
./org/apache/log4j/org.apache.log4j/1.2.15/org.apache.log4j-1.2.15.pom
./org/apache/xalan/org.apache.xalan/2.7.1/org.apache.xalan-2.7.1.pom
./org/apache/derby/org.apache.derby/10.5.1/org.apache.derby-10.5.1.pom
./org/apache/derby/org.apache.derby.core/10.5.1/org.apache.derby.core-10.5.1.pom
./org/apache/xerces/org.apache.xerces/2.9.0/org.apache.xerces-2.9.0.pom
./org/apache/velocity/org.apache.velocity/1.5.0/org.apache.velocity-1.5.0.pom
./org/apache/xmlrpc/org.apache.xmlrpc/3.0.0/org.apache.xmlrpc-3.0.0.pom
./org/apache/ant/org.apache.ant/1.7.1/org.apache.ant-1.7.1.pom
./org/uddi4j/org.uddi4j/2.0.5/org.uddi4j-2.0.5.pom
./org/mozilla/javascript/org.mozilla.javascript/1.7.2/org.mozilla.javascript-1.7.2.pom
./org/hamcrest/core/org.hamcrest.core/1.1.0/org.hamcrest.core-1.1.0.pom
./org/w3c/dom/org.w3c.dom.smil/1.0.0/org.w3c.dom.smil-1.0.0.pom
./org/w3c/dom/org.w3c.dom.smil/1.0.1/org.w3c.dom.smil-1.0.1.pom
./org/w3c/dom/org.w3c.dom.events/3.0.0/org.w3c.dom.events-3.0.0.pom
./org/w3c/dom/org.w3c.dom.svg/1.1.0/org.w3c.dom.svg-1.1.0.pom
./org/w3c/sac/org.w3c.sac/1.3.0/org.w3c.sac-1.3.0.pom
./org/w3c/css/org.w3c.css.sac/1.3.0/org.w3c.css.sac-1.3.0.pom
./org/junit/source/org.junit.source/4.8.1/org.junit.source-4.8.1.pom
./org/junit/org.junit/3.8.2/org.junit-3.8.2.pom
./org/sat4j/pb/org.sat4j.pb/2.2.0/org.sat4j.pb-2.2.0.pom
./org/sat4j/core/org.sat4j.core/2.2.0/org.sat4j.core-2.2.0.pom
./org/mortbay/jetty/org.mortbay.jetty.util/6.1.23/org.mortbay.jetty.util-6.1.23.pom
./org/mortbay/jetty/org.mortbay.jetty.server/6.1.23/org.mortbay.jetty.server-6.1.23.pom
./org/junit4/org.junit4/4.8.1/org.junit4-4.8.1.pom
./org/jdom/org.jdom/1.0.0/org.jdom-1.0.0.pom
./org/h2/org.h2/1.1.117/org.h2-1.1.117.pom
./net/sourceforge/lpg/net.sourceforge.lpg.lpgjavaruntime/1.1.0/net.sourceforge.lpg.lpgjavaruntime-1.1.0.pom
./lpg/runtime/java/lpg.runtime.java/2.0.17/lpg.runtime.java-2.0.17.pom
./commonj/sdo/commonj.sdo/2.1.1/commonj.sdo-2.1.1.pom

Post by Alex Blewitt

Post by Aaron Digulla
948 Defl:N 517 46% 03-18-11 17:09 7ce61aa8

META-INF/MANIFEST.MF

Post by Alex Blewitt

Post by Aaron Digulla
948 Defl:N 517 46% 03-18-11 17:15 7ce61aa8

META-INF/MANIFEST.MF

Post by Alex Blewitt

Post by Aaron Digulla
So you can see the file date is different. How can that happen? I
understand that Eclipse recreates the MANIFEST.MF when the JAR is signed
but why are many plug-ins signed several times?

There may be a bug in which a signed jar is already re-signed; this

wouldn't change the bits but would update the time processed.

I agree; but why are they signed again? Isn't there a cache or something?

Post by Alex Blewitt

Post by Aaron Digulla
Background: To make sure everything is ok, I check that all files are
identical when I merge repositories. Only, they aren't...

Is the junit a special case? If so, we can probably ignore it.

No, I found 40 cases:

org.eclipse.jdt.debug-3.6.1.jar
org.eclipse.ui.intro.universal-3.2.402.jar
org.eclipse.core.runtime.compatibility.registry-3.3.0.jar
...

Regards,

[Alex Blewitt]

Post by Aaron Digulla
You can find the result in
/home/nexus/workspace/org.eclipse.dash.m4e.tools/tmp/m2repo/

I have copied these into the 'testing' repository, which is available at:

http://maven.eclipse.org/nexus/content/repositories/testing/

Alex

[Aaron Digulla]

Post by Aaron Digulla
Due to a bug, the repo is still polluted with non-Eclipse artifacts.

I've fixed the problem; the result should now be ready for deeper
inspection.

/home/nexus/workspace/org.eclipse.dash.m4e.tools/tmp/m2repo/

When the testing repo is configured, please import this. I'll then post
notices in my blog and on stackoverflow.com to get some more feedback.

Regards,