Setting up a YUM / RPM repository on Bintray

Bintray supports many kinds of repositories: Maven, Debian packages, etc. And this is very convenient for open source projects. I recently had to create a repository for Roboconf’s RPM packages. And the least I can say is that it did not work as expected. I made several attempts before it finally works.

The creation of a RPM repository is made through Bintray’s web interface. Create a new repository, set RPM as its type and make sure the YUM metadata folder depth is zero. Here, we assume there will be only one RPM repository contained in this « Bintray repository ».

Once created, add a new package in Bintray.
Packages are these entities that contain versions. In my case, there was only one package that I called main. So far, so good.

I once wrote an article about how to upload binaries on Bintray with their REST API. I will show CURL snippets for the following steps.

First, create a version.

curl -vvf -u${BINTRAY_USER}:${BINTRAY_API_KEY} -H "Content-Type: application/json" \
	-X POST ${BINTRAY_URL}/packages/<organization>/<repository-name>/<package-name>/versions \
	--data "{\"name\": \"${RELEASE_VERSION}\", \"github_use_tag_release_notes\": false }"

Then, upload binaries (RPM) to this package.

for f in $(find -name "*.rpm" -type f)
do
	echo
	echo "Uploading $f"
	curl -X PUT -T $f -u ${BINTRAY_USER}:${BINTRAY_API_KEY} \
		-H "X-Bintray-Version:${RELEASE_VERSION}" \
		-H "X-Bintray-Package:<package-name>" \
		-# -o "/tmp/curl-output.txt" \
		${BINTRAY_URL}/content/<organization>/<repository-name>/<package-name>/${RELEASE_VERSION}/
	
	echo
	echo "$(</tmp/curl-output.txt)"
	echo
done

To publish them, I generally recommend to do it through the web interface of Bintray. This is the manual step for verification before publishing. But you could use Bintray’s REST API too.

Once your binaries are published, I guess you will want to test them. Personnaly, I like to use Docker to test my installers (but whatever). Get a (virtual) machine and click the set me up link on Bintray’s page. There are not that many documentation on Bintray, but these set me up links generally show whatever you need.

In my case, it indicated the following steps.

There was indeed a file at https://bintray.com/roboconf/roboconf-rpm/rpm. But when I typed in yum install roboconf-dm, I got a 404 error (not found) about a file called repodata/repomd.xml.

Bintray indicates metadata files should be generated automatically. Obviously, it was not the case there. Hopefully, there is a REST command that allows to force the generation of these metadata.

curl -X POST -u ${BINTRAY_USER}:${BINTRAY_API_KEY} \
https://api.bintray.com/calc_metadata/<organization>/<repository-name>

Then, yum install roboconf-dm worked as expected.
I hope this article will help you. I would have saved 2 hours if I have had all these things right at the beginning.

Auto-repaired connections with RabbitMQ

Here is a short article about making RabbitMQ’s Java clients automatically repair their connections. Indeed, sometimes, network issues may result in broken connections.

Since version 3.3.0, default Java clients for RabbitMQ have options for this situation. Almost everything is in the factory that creates the channel and consumers.

ConnectionFactory factory = new ConnectionFactory();
factory.setUsername( messageServerUsername );
factory.setPassword( messageServerPassword );

// Timeout for connection establishment: 5s
factory.setConnectionTimeout( 5000 );

// Configure automatic reconnections
factory.setAutomaticRecoveryEnabled( true );

// Recovery interval: 10s
factory.setNetworkRecoveryInterval( 10000 );

// Exchanges and so on should be redeclared if necessary
factory.setTopologyRecoveryEnabled( true );

When an established connection gets broken, the client will automatically try to recover everything, including queues, exchanges and bindings. Please, refer to the user guide for more details. From my experience, the only part that can be a problem is about consumers.

Indeed, in my code, I used to rely on QueueingConsumers. This class is perfectly working, except it is deprecated and that it breaks connection recovery. So, if you are using it and you want connection recovery, you MUST replace it by your own consumer.

import java.io.IOException;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.logging.Logger;

import net.roboconf.core.utils.Utils;
import net.roboconf.messaging.api.messages.Message;
import net.roboconf.messaging.api.utils.SerializationUtils;

import com.rabbitmq.client.AMQP.BasicProperties;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Consumer;
import com.rabbitmq.client.DefaultConsumer;
import com.rabbitmq.client.Envelope;
import com.rabbitmq.client.ShutdownSignalException;

/**
 * Notice: QueueingConsumer is deprecated, hence this implementation that supports recovery.
 */
public class MyConsumer extends DefaultConsumer implements Consumer {

	private final Logger logger = Logger.getLogger( getClass().getName());


	/**
	 * Constructor.
	 * @param channel
	 */
	public MyConsumer( Channel channel ) {
		super( channel );
	}


	@Override
	public void handleDelivery( String consumerTag, Envelope envelope, BasicProperties properties, byte[] body )
	throws IOException {

		// Do what you have to do with your message.
		// Prefer a short processing...
	}


	@Override
	public void handleShutdownSignal( String consumerTag, ShutdownSignalException sig ) {

		if( sig.isInitiatedByApplication()) {
			this.logger.fine( "The connection to the messaging server was shut down." + id( consumerTag ));

		} else if( sig.getReference() instanceof Channel ) {
			int nb = ((Channel) sig.getReference()).getChannelNumber();
			this.logger.fine( "A RabbitMQ consumer was shut down. Channel #" + nb + ", " + id( consumerTag ));

		} else {
			this.logger.fine( "A RabbitMQ consumer was shut down." + id( consumerTag ));
		}
	}


	@Override
	public void handleCancelOk( String consumerTag ) {
		this.logger.fine( "A RabbitMQ consumer stops listening to new messages." + id( consumerTag ));
	}


	@Override
	public void handleCancel( String consumerTag ) throws IOException {
		this.logger.fine( "A RabbitMQ consumer UNEXPECTABLY stops listening to new messages." + id( consumerTag ));
	}


	/**
	 * @param consumerTag a consumer tag
	 * @return a readable ID of this consumer
	 */
	private String id( String consumerTag ) {

		StringBuilder sb = new StringBuilder();
		sb.append( " Consumer tag = " );
		sb.append( consumerTag );
		sb.append( ")" );

		return sb.toString();
	}
}

Before testing for real, and to be able to debug efficiently such code, you should add a recovery listener on your connections.

Channel channel = factory.newConnection().createChannel();

// Add a recoverable listener (when broken connections are recovered).
// Given the way the RabbitMQ factory is configured, the channel should be "recoverable".
((Recoverable) this.channel).addRecoveryListener( new MyRecoveryListener());

And a logging listener…

import java.util.logging.Logger;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Recoverable;
import com.rabbitmq.client.RecoveryListener;

public class MyRecoveryListener implements RecoveryListener {

	private final Logger logger = Logger.getLogger( getClass().getName());


	@Override
	public void handleRecovery( Recoverable recoverable ) {

		if( recoverable instanceof Channel ) {
			int channelNumber = ((Channel) recoverable).getChannelNumber();
			this.logger.fine( "Connection to channel #" + channelNumber + " was recovered." );
		}
	}
}

Now, you can test recovery for real.
Start your RabbitMQ server and then your client. Verify the connection is established. Then, turn off RabbitMQ and check your logs, the connection should appear as broken. Turn the server on. Wait few seconds and verify the connection was recovered.

If you do not want to turn off RabbitMQ, you can also play with your firewall (e.g. iptables) and disabled temporarily connections to the server. Both cases work with the code above.

Notice that connection recovery only works when a connection was successfully established. If you start the RabbitMQ server AFTER your client, recovery will not work. You may then consider using the Lyra project.