[dash-dev] Maven Nexus server

Discussion:

Alex Blewitt

2012-08-15 20:45:32 UTC

We've run out of disk space again - I'm trying to clean it up, but there's something on the filesystem which appears to be taking a lot of size. Right now the server is in the process of trying to restart and generating errors in the log files, probably due to lack of space.

Alex

Alex Blewitt

2012-08-15 21:54:58 UTC

Permalink

Post by Alex Blewitt
We've run out of disk space again - I'm trying to clean it up, but there's something on the filesystem which appears to be taking a lot of size. Right now the server is in the process of trying to restart and generating errors in the log files, probably due to lack of space.

So I've cleaned out some cruft, but I think it's likely to recur. The performance of the virtual disk is horribly slow and although the nexus jobs are set up to purge the trash and remove unused items, I think that there's a number of people hitting http://maven.eclipse.org for non-eclipse content.

There's also some data in /var which we can clean - there's a lot of data in /var/messages, for example. However, the root password (was in ~admin/pw, now gone) seems to have gone walkabout - anyone know where it went?

Finally, I'm concerned that the level of support (and management) that we (as Dash) are able to provide doesn't fit the Eclipse organisation. Given that we haven't succeeded in converting Eclipse to Maven hosting - and the m4e tools never made it to production worthy status - I'm wondering if we shouldn't shut the system down. Unfortunately, I don't know where the repository is being used from hudson slaves, so I'm not sure if that's even practical - or whether the repositories are being used by others.

Thoughts?

Alex

David Carver

2012-08-15 22:24:17 UTC

Permalink

Personally, with CBI using maven. I'd actually recommend that Nexus
server fall under that group. Especially with Nexus 2.1.x able to
support p2 repositories, and also make existing maven repositories p2
enabled. So items could technically just be deployed to it and the
basic p2 metadata is generated as needed.

Dave

Post by Alex Blewitt

So I've cleaned out some cruft, but I think it's likely to recur. The performance of the virtual disk is horribly slow and although the nexus jobs are set up to purge the trash and remove unused items, I think that there's a number of people hitting http://maven.eclipse.org for non-eclipse content.
There's also some data in /var which we can clean - there's a lot of data in /var/messages, for example. However, the root password (was in ~admin/pw, now gone) seems to have gone walkabout - anyone know where it went?
Finally, I'm concerned that the level of support (and management) that we (as Dash) are able to provide doesn't fit the Eclipse organisation. Given that we haven't succeeded in converting Eclipse to Maven hosting - and the m4e tools never made it to production worthy status - I'm wondering if we shouldn't shut the system down. Unfortunately, I don't know where the repository is being used from hudson slaves, so I'm not sure if that's even practical - or whether the repositories are being used by others.
Thoughts?
Alex
_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

Jesse McConnell

2012-08-15 22:29:13 UTC

Permalink

+1 for CBI, I know they are planning on looking into the orbit mess as
well once SR1 is done

orbit mess being how to reconcile orbit with maven central in a sane fashion

jesse

--
jesse mcconnell

Personally, with CBI using maven. I'd actually recommend that Nexus server
fall under that group. Especially with Nexus 2.1.x able to support p2
repositories, and also make existing maven repositories p2 enabled. So
items could technically just be deployed to it and the basic p2 metadata is
generated as needed.
Dave

Post by Alex Blewitt

Post by Alex Blewitt
We've run out of disk space again - I'm trying to clean it up, but
there's something on the filesystem which appears to be taking a lot of
size. Right now the server is in the process of trying to restart and
generating errors in the log files, probably due to lack of space.

So I've cleaned out some cruft, but I think it's likely to recur. The
performance of the virtual disk is horribly slow and although the nexus jobs
are set up to purge the trash and remove unused items, I think that there's
a number of people hitting http://maven.eclipse.org for non-eclipse content.
There's also some data in /var which we can clean - there's a lot of data
in /var/messages, for example. However, the root password (was in ~admin/pw,
now gone) seems to have gone walkabout - anyone know where it went?
Finally, I'm concerned that the level of support (and management) that we
(as Dash) are able to provide doesn't fit the Eclipse organisation. Given
that we haven't succeeded in converting Eclipse to Maven hosting - and the
m4e tools never made it to production worthy status - I'm wondering if we
shouldn't shut the system down. Unfortunately, I don't know where the
repository is being used from hudson slaves, so I'm not sure if that's even
practical - or whether the repositories are being used by others.
Thoughts?
Alex
_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

Alex Blewitt

2012-08-15 22:34:57 UTC

Permalink

I mentioned to Igor that I'd be happy if they wanted to take it over but he didn't seem interested.

I do think it is at a stage that it needs more than just community ownership though, especially when problems occur during working hours when it isn't possible for us to intervene.

Alex

Sent from my iPhone 4S

Post by Jesse McConnell
+1 for CBI, I know they are planning on looking into the orbit mess as
well once SR1 is done
orbit mess being how to reconcile orbit with maven central in a sane fashion
jesse
--
jesse mcconnell

Personally, with CBI using maven. I'd actually recommend that Nexus server
fall under that group. Especially with Nexus 2.1.x able to support p2
repositories, and also make existing maven repositories p2 enabled. So
items could technically just be deployed to it and the basic p2 metadata is
generated as needed.
Dave

Post by Alex Blewitt

Post by Alex Blewitt
We've run out of disk space again - I'm trying to clean it up, but
there's something on the filesystem which appears to be taking a lot of
size. Right now the server is in the process of trying to restart and
generating errors in the log files, probably due to lack of space.

So I've cleaned out some cruft, but I think it's likely to recur. The
performance of the virtual disk is horribly slow and although the nexus jobs
are set up to purge the trash and remove unused items, I think that there's
a number of people hitting http://maven.eclipse.org for non-eclipse content.
There's also some data in /var which we can clean - there's a lot of data
in /var/messages, for example. However, the root password (was in ~admin/pw,
now gone) seems to have gone walkabout - anyone know where it went?
Finally, I'm concerned that the level of support (and management) that we
(as Dash) are able to provide doesn't fit the Eclipse organisation. Given
that we haven't succeeded in converting Eclipse to Maven hosting - and the
m4e tools never made it to production worthy status - I'm wondering if we
shouldn't shut the system down. Unfortunately, I don't know where the
repository is being used from hudson slaves, so I'm not sure if that's even
practical - or whether the repositories are being used by others.
Thoughts?
Alex
_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

Igor Fedorenko

2012-08-16 00:05:38 UTC

Permalink

I do not decide what is and is not part of CBI effort. You need to talk
to Andrew Ross to see if he feels they have resources and expertise
needed to take over maven.eclipse.org.

Personally, I do think that supported and well-maintained Maven/Nexus
hosted repository will benefit Eclipse community in general, but I also
think that Foundation (or should it be Planning Council?) will need to
make policy decisions before anything happens.

--
Regards,
Igor

Post by Alex Blewitt
I mentioned to Igor that I'd be happy if they wanted to take it over but he didn't seem interested.
I do think it is at a stage that it needs more than just community ownership though, especially when problems occur during working hours when it isn't possible for us to intervene.
Alex
Sent from my iPhone 4S

Personally, with CBI using maven. I'd actually recommend that Nexus server
fall under that group. Especially with Nexus 2.1.x able to support p2
repositories, and also make existing maven repositories p2 enabled. So
items could technically just be deployed to it and the basic p2 metadata is
generated as needed.
Dave

Post by Alex Blewitt

Post by Alex Blewitt
We've run out of disk space again - I'm trying to clean it up, but
there's something on the filesystem which appears to be taking a lot of
size. Right now the server is in the process of trying to restart and
generating errors in the log files, probably due to lack of space.

So I've cleaned out some cruft, but I think it's likely to recur. The
performance of the virtual disk is horribly slow and although the nexus jobs
are set up to purge the trash and remove unused items, I think that there's
a number of people hitting http://maven.eclipse.org for non-eclipse content.
There's also some data in /var which we can clean - there's a lot of data
in /var/messages, for example. However, the root password (was in ~admin/pw,
now gone) seems to have gone walkabout - anyone know where it went?
Finally, I'm concerned that the level of support (and management) that we
(as Dash) are able to provide doesn't fit the Eclipse organisation. Given
that we haven't succeeded in converting Eclipse to Maven hosting - and the
m4e tools never made it to production worthy status - I'm wondering if we
shouldn't shut the system down. Unfortunately, I don't know where the
repository is being used from hudson slaves, so I'm not sure if that's even
practical - or whether the repositories are being used by others.
Thoughts?
Alex
_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

Andrew Ross

2012-08-16 01:02:23 UTC

Permalink

Hi Everyone,

Sounds like fairly easy agreement that a well managed Nexus repository
would be of benefit to the community and that it makes sense to "reboot"
maven.eclipse.org. For what it's worth I can see obvious benefit and
synergy with the CBI efforts and the Long Term Support (LTS) program as
well - thus I can make a pretty clear business case that we simply need to
do this.

I'm hoping that we can lean on you all a bit longer for your help to
capture lessons learned from maven.eclipse.org to date, sort through bugs
and requirements, and help figure out how we should implement the new
system? If you feel like being part of this, please +1 to register your
interest.

In terms of timing, the work on the CBI build for the Eclipse platform is
taking a bit longer due to vacations and competing other demands so may not
be ready for SR1. This will slip when we can start picking up work with
Nexus by weeks. You can follow along on cbi-dev to track progress. I'm
cautiously optimistic the slip won't be too bad due in part by some great
work and persistence by Igor, Chris, Thanh, Paul, and others. We can always
use a few more hands testing if you're interested to help speed us up.

Also, I'm inclined to simply leave maven.eclipse.org for now and start a
new instance in parallel for fear of disruption to anyone using it.

Andrew

Post by Igor Fedorenko
I do not decide what is and is not part of CBI effort. You need to talk
to Andrew Ross to see if he feels they have resources and expertise
needed to take over maven.eclipse.org.
Personally, I do think that supported and well-maintained Maven/Nexus
hosted repository will benefit Eclipse community in general, but I also
think that Foundation (or should it be Planning Council?) will need to
make policy decisions before anything happens.
--
Regards,
Igor

Post by Alex Blewitt
I mentioned to Igor that I'd be happy if they wanted to take it over but
he didn't seem interested.
I do think it is at a stage that it needs more than just community
ownership though, especially when problems occur during working hours when
it isn't possible for us to intervene.
Alex
Sent from my iPhone 4S
+1 for CBI, I know they are planning on looking into the orbit mess as

Post by Jesse McConnell
well once SR1 is done
orbit mess being how to reconcile orbit with maven central in a sane fashion
jesse
--
jesse mcconnell

Personally, with CBI using maven. I'd actually recommend that Nexus server
fall under that group. Especially with Nexus 2.1.x able to support p2
repositories, and also make existing maven repositories p2 enabled. So
items could technically just be deployed to it and the basic p2 metadata is
generated as needed.
Dave

Post by Alex Blewitt

Post by Alex Blewitt
We've run out of disk space again - I'm trying to clean it up, but
there's something on the filesystem which appears to be taking a lot of
size. Right now the server is in the process of trying to restart and
generating errors in the log files, probably due to lack of space.

So I've cleaned out some cruft, but I think it's likely to recur. The
performance of the virtual disk is horribly slow and although the nexus jobs
are set up to purge the trash and remove unused items, I think that there's
a number of people hitting http://maven.eclipse.org for non-eclipse content.
There's also some data in /var which we can clean - there's a lot of data
in /var/messages, for example. However, the root password (was in ~admin/pw,
now gone) seems to have gone walkabout - anyone know where it went?
Finally, I'm concerned that the level of support (and management) that we
(as Dash) are able to provide doesn't fit the Eclipse organisation. Given
that we haven't succeeded in converting Eclipse to Maven hosting - and the
m4e tools never made it to production worthy status - I'm wondering if we
shouldn't shut the system down. Unfortunately, I don't know where the
repository is being used from hudson slaves, so I'm not sure if that's even
practical - or whether the repositories are being used by others.
Thoughts?
Alex
______________________________**_________________
dash-dev mailing list
https://dev.eclipse.org/**mailman/listinfo/dash-dev<https://dev.eclipse.org/mailman/listinfo/dash-dev>

______________________________**_________________
dash-dev mailing list
https://dev.eclipse.org/**mailman/listinfo/dash-dev<https://dev.eclipse.org/mailman/listinfo/dash-dev>

______________________________**_________________
dash-dev mailing list
https://dev.eclipse.org/**mailman/listinfo/dash-dev<https://dev.eclipse.org/mailman/listinfo/dash-dev>
______________________________**_________________

dash-dev mailing list
https://dev.eclipse.org/**mailman/listinfo/dash-dev<https://dev.eclipse.org/mailman/listinfo/dash-dev>

Aaron Digulla

2012-08-16 08:45:24 UTC

Permalink

Post by Andrew Ross
I'm hoping that we can lean on you all a bit longer for your help to
capture lessons learned from maven.eclipse.org to date, sort through bugs
and requirements, and help figure out how we should implement the new
system? If you feel like being part of this, please +1 to register your
interest.

+1 from me. I'm still actively developing mt4e but I'm also in a
production team which has completely different goals, so I only have a
couple of days every few months to enhance the tool set.

My current status: All important features work and I do convert
Eclipse releases in a couple of hours (mostly looking through the
error reports that mt4e generates) but the tool needs the usual
polishing (i.e. other people than me must try it and help improve the
parts where I'm blind).

Regards,

--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://www.pdark.de/ http://blog.pdark.de/

Aaron Digulla

2012-08-16 08:35:51 UTC

Permalink

Post by Jesse McConnell
+1 for CBI, I know they are planning on looking into the orbit mess as
well once SR1 is done
orbit mess being how to reconcile orbit with maven central in a sane fashion

Who is doing that? I already spent a lot of time in this area and I
have a solution that works better than anything else I've encountered
so far.

Regards,

Jesse McConnell

2012-08-16 11:57:34 UTC

Permalink

Post by Jesse McConnell
+1 for CBI, I know they are planning on looking into the orbit mess as
well once SR1 is done
orbit mess being how to reconcile orbit with maven central in a sane fashion

Who is doing that? I already spent a lot of time in this area and I have a
solution that works better than anything else I've encountered so far.

andrew ross of cbi I believe, was discussed on a cbi lts call a month
or two back

Igor Fedorenko

2012-08-15 23:56:57 UTC

Permalink

I would not use maven->p2 conversion functionality provided by nexus
extension plugins for any production purposes. This was part of research
project, which I do not believe was fully implemented due to
time/resources constraints (I was not directly involved, so may be wrong
about the reasons). Nexus p2 repository proxy functionality is the only
nexus/p2 functionality that is production ready, as far as I know.

--
Regards,
Igor

Post by David Carver
Personally, with CBI using maven. I'd actually recommend that Nexus
server fall under that group. Especially with Nexus 2.1.x able to
support p2 repositories, and also make existing maven repositories p2
enabled. So items could technically just be deployed to it and the
basic p2 metadata is generated as needed.
Dave

Post by Alex Blewitt

Post by Alex Blewitt
We've run out of disk space again - I'm trying to clean it up, but
there's something on the filesystem which appears to be taking a lot
of size. Right now the server is in the process of trying to restart
and generating errors in the log files, probably due to lack of space.

So I've cleaned out some cruft, but I think it's likely to recur. The
performance of the virtual disk is horribly slow and although the
nexus jobs are set up to purge the trash and remove unused items, I
think that there's a number of people hitting http://maven.eclipse.org
for non-eclipse content.
There's also some data in /var which we can clean - there's a lot of
data in /var/messages, for example. However, the root password (was in
~admin/pw, now gone) seems to have gone walkabout - anyone know where
it went?
Finally, I'm concerned that the level of support (and management) that
we (as Dash) are able to provide doesn't fit the Eclipse organisation.
Given that we haven't succeeded in converting Eclipse to Maven hosting
- and the m4e tools never made it to production worthy status - I'm
wondering if we shouldn't shut the system down. Unfortunately, I don't
know where the repository is being used from hudson slaves, so I'm not
sure if that's even practical - or whether the repositories are being
used by others.
Thoughts?
Alex
_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

_______________________________________________
dash-dev mailing list
https://dev.eclipse.org/mailman/listinfo/dash-dev

Aaron Digulla

2012-08-16 08:33:28 UTC

Permalink

Post by Alex Blewitt
So I've cleaned out some cruft, but I think it's likely to recur.
The performance of the virtual disk is horribly slow and although
the nexus jobs are set up to purge the trash and remove unused
items, I think that there's a number of people hitting
http://maven.eclipse.org for non-eclipse content.

Why does nexus mirror half of maven central? And why are there *two*
repos (repo1 and repo2)?

Post by Alex Blewitt
There's also some data in /var which we can clean - there's a lot of
data in /var/messages, for example. However, the root password (was
in ~admin/pw, now gone) seems to have gone walkabout - anyone know
where it went?

I have it. /var is a mere 100MB. Log rotation is making sure it
doesn't grow much beyond that. I could clean some of that but the bulk
is in repo1.

Post by Alex Blewitt
Finally, I'm concerned that the level of support (and management)
that we (as Dash) are able to provide doesn't fit the Eclipse
organisation. Given that we haven't succeeded in converting Eclipse
to Maven hosting - and the m4e tools never made it to production
worthy status - I'm wondering if we shouldn't shut the system down.
Unfortunately, I don't know where the repository is being used from
hudson slaves, so I'm not sure if that's even practical - or whether
the repositories are being used by others.
Thoughts?

We could simply shut it down and then ask people who come complaining
to help. After all, they have a vital interest to do something about
it. When I started the project, I just wanted to be able to provide
Eclipse bundles to non-OSGi projects. The interest seemed to be there
but frankly, I'm a software developer. I suck at organizing things.
What I need is a project manager.

Regards,

Alex Blewitt

2012-08-16 09:21:09 UTC

Permalink

Post by Alex Blewitt
So I've cleaned out some cruft, but I think it's likely to recur. The performance of the virtual disk is horribly slow and although the nexus jobs are set up to purge the trash and remove unused items, I think that there's a number of people hitting http://maven.eclipse.org for non-eclipse content.

Why does nexus mirror half of maven central? And why are there *two* repos (repo1 and repo2)?

Presumably because the build infra uses half of Maven central, or (more likely) random developers over the planet put 'maven.eclipse.org' at the front of their resolution profile and so we are acting as a cache for everything.

So Point 1 to learn; we need to have an internal Nexus instance which is for the purposes of the internal Hudson infra caching, and we need to have an external one that can contain published Eclipse content. Ideally members of the public should only be able to see the latter.

Point 2 to learn; the two repo1/repo2 were created to do some kind of load balance/mirror between the two (it was also created shortly after the repo1 DNS entry went wobbly a couple of years back). In retrospect we should have a single 'central' repo and then use the 'mirror' in the repository proxy configuration to utilise either of these. That would save storing a bunch of stuff twice. (In fact, we seem to be storing a third copy under 'central' which is the group, which doesn't help.)

Point 3 to learn is that disk space needs to be monitored and alerted, in a way (presumably) similar to the Hudson setup. Part time admins rarely work out well.

Point 4 is that whilst processing power for a VM isn't an issue, great gobs of virtual disk are a serious bottleneck to doing most of the work. Even a du -s * in the repositories folder takes significant amounts of time (tens of minutes) which really doesn't help when you are trying to solve the problems caused by lack of disk space.

Point 5 is that really, published Eclipse content should migrate to Maven Central instead of being hosted on Eclipse infra. The key challenge, it seems, is getting it in a suitable form. I believe that CBI may be the way to make this happen.

Alex

Aaron Digulla

2012-08-16 14:44:30 UTC

Permalink

Post by Alex Blewitt
Point 5 is that really, published Eclipse content should migrate to
Maven Central instead of being hosted on Eclipse infra. The key
challenge, it seems, is getting it in a suitable form. I believe
that CBI may be the way to make this happen.

I agree with the first statement and I love CBI but CBI won't solve
this; Tycho is just a thin wrapper over p2 - it doesn't create useful
POM files (main weakness: All the dependencies are missing because
Tycho just uses the POM to hook into Maven's build process).

Unless we can get every Eclipse project into using POM-first (instead
of Tycho's default manifest-first), the OSGi bundles need to be
converted to be usable by non-Tycho builds.

Regards,

Aaron Digulla

2012-08-16 08:24:10 UTC

Permalink

Post by Alex Blewitt
We've run out of disk space again - I'm trying to clean it up, but
there's something on the filesystem which appears to be taking a lot
of size. Right now the server is in the process of trying to restart
and generating errors in the log files, probably due to lack of space.

I posted a mail about this one week ago. It contains all the info you need.

Regards,