Tag Archives: ubuntu

Easy APT repo in S3

When running a number of Ubuntu or Debian servers, it can be extremely useful to have a custom APT repo for uploading your own packages, or third party packages that lack their own good repositories to subscribe to.

I recently found a nice Ruby utility called deb-s3 which allows easy uploading of dpkg files into an S3-hosted APT repository. It’s much easier than messing around with tools like reprepro and having to s3 cp or sync files up from a local disk into S3.

One main warning: This will create a *public* repo by default since it works out-of-the-box with the stock OS and (in my case) all the packages I’m serving are public open source programs that don’t need to be secured. If you want a *private* repo, you will need to use apt-transport-s3 to support authenticating with S3 to download files and configure deb-s3 for private upload.

Install like any other Ruby Gem:

gem install deb-s3

Adding packages is easy. First make sure your aws-cli is working OK and an S3 bucket has been created, then upload with:

deb-s3 upload \
--bucket example \
--codename codename \
--preserve-versions \
mypackage.deb

You can then add the repo to a Ubuntu or Debian server with:

# We trust HTTPS rather than GPG for this repo - but you can config
# GPG signing if you prefer.
cat > /etc/apt/sources.list.d/myrepo.list << EOF
deb [trusted=yes] https://example.s3.amazonaws.com codename main
EOF

# and ensure you update the package info on the server
apt-get update

Alternatively, here’s an example of how to add the repo with Puppet:

apt::source { 'myrepo':
 comment        => 'This is our own APT repo',
 location       => 'https://example.s3.amazonaws.com',
 release        => $::os["distro"]["codename"],
 repos          => 'main',
 allow_unsigned => true, # We don't GPG sign, HTTPS only
 notify_update  => true, # triggers apt-get update
}

Thunderbolt and other Macbook hardware issues with Linux

Having semi-recently switched to a Macbook Pro Retina 15″ at work, I decided to give MacOS a go. It’s been interesting, it’s not too bad an operating system and whilst it is something I could use on an ongoing basis, I quickly longed for the happy embrace of GNU/Linux where I have a bit more power and control over the system.

Generally the Linux kernel supports most of the Macbook hardware out-of-the-box (As of 3.15 anyway), but with a couple exceptions:

  • I believe support for the dual GPU mode switching is now fixed, however the model I’m using now is Intel only, so I can’t test this unfortunately.
  • The Apple Webcam does not yet have a driver. The older iSight driver doesn’t work, since the new gen of hardware is a PCIe connected device, not USB.
  • The WiFi requires a third party driver to be built for your kernel. You’ll want the latest Broadcom 802.11 STA driver in order for it to built with new kernel versions. Ubuntu users, get this version, or more recent.
  • If you’re having weird hangs where the Macbook just halts frequently waiting on on I/O, add “libata.force=noncq” kernel parameter. It seems that there is some bug with this SSD and some kernel versions that leads to weird I/O halts, which is fixed by this option.
  • Thunderbolt support is limited to only working on devices connected at boot up, no hotplug. Additionally, when using Thunderbolt, Suspend/Resume is disabled (although it works otherwise if there’s no Thunderbolt involved).

Of all these issues, the lack of Thunderbolt support was the one that was really frustrating me, since I need to use a Thunderbolt based Ethernet adaptor currently on a daily basis and I always rely on Suspend and Resume heavily.

Thankfully two kernel developers, Andreas Noever and Matthew J Garrett have been working on a series of kernel patches that introduce support for Thunderbolt hotplug and thus allow it to work on suspend and resume.

Sadly whilst this patch is awesome, it doesn't yet do wireless Thunderbolt for when the ethernet cable you want is too bloody short.

You too can now enjoy the shackles of a wired LAN connection like it’s 1990 all over again!

It doesn’t sound like it has been easy based on the posts on MJG’s blog which are well worth a read – essentially the Apple firmware does weird things with the Thunderbolt hardware when the OS doesn’t identify itself as Darwin (MacOS’s kernel) and likes to power stuff down after suspend/resume, so it’s taken some effort to debug and put in hardware-specific workarounds.
It will surely only be a matter of time before these awesome patches are merged, but if you need them right now and are happy to run rather beta kernel patches (who isn’t??) then the easiest way is to checkout their Git repo of 3.15 with all the patches applied. This repository should build cleanly via the usual means, and provide you with a new kernel module called “thunderbolt”.I’ve been testing it for a few days and it looks really good. I’ve had no kernel panics, freezes, devices failing to work or any issues with suspend/resume with these patches – the features that they claim to work, just work.  The only catches are:

  • If you boot the Macbook with the Thunderbolt device attached, it will be treated like a PCIe hotplug device… except that when you remove it, that Thunderbolt port won’t work again until the next restart. I recommend booting the Macbook with no devices attached, then hotplug once started to avoid this issue. I always remove before suspend and re-connect after resume as well (mostly because it’s a laptop and it’s easy to do so and avoid any issues).
  • The developers advise that Thunderbolt Displays don’t work at this time (however Mini DisplayPort connected screens work fine, even though they share the same socket).
  • The developers advise that chaining Thunderbolt devices is not yet supported. So stick to one device per port for now.

If you’re using Linux on a Macbook, I recommend grabbing the patched source and doing a build. Hopefully all these patches make their way into 3.16 or 3.17 and make this post irrelevant soon.

If you’re extra lazy and trust a random blogger’s binary packages, I’ve built deb packages for Ubuntu 13.10 (and should work just fine on 14.04 as well) for both the Thunderbolt enabled kernel as well as the Broadcom WiFi. You can download these packages here.

Ubuntu, the Windows of the Linux world

Sometimes I do wonder about if Ubuntu is actually the Windows of the Linux world, some of their design decisions, like non-closable restart windows…

Nice desktop you have there. Be a good fellow and reboot now ok?

Nice desktop you have there. Be a good fellow and reboot now ok?

Thankfully xkill closes *all* windows and leaves no survivors. It’s a shame that the general desktop environment on Ubuntu has been so cut down and over simplified over the last few years, since the server Ubuntu LTS releases are actually pretty damn good.

Updated Repositories

I’ve gone and updated my GNU/Linux repositories with a new home page – some of you may have been using this under my previous Amberdms branding, but it’s more appropriate that it be done under my own name these days and have it’s own special subdomain.

I want to unify the branding of a bit more of the stuff I have out there on the internet and also make sure I’m exposing it in a way that makes it easy for people to find and use, so I’m going through a process of improving site templates, linking between places and improving documentation/wording with the perspective of viewing as an outside user.

CSS3 shinyness! And it even mostly works in IE.

Been playing with new HTML5/CSS3 functionality for this site, have to say, it’s pretty awesome.

You can check out the new page at repos.jethrocarr.com, I’ve tried to make it as easy as possible to add my repositories to your servers -I’ll be refining this a little more in coming weeks, such as adding a decent package search function to the site to make it easier to grab some of the goodies hidden away in distribution directories.

I’m currently providing packages for RHEL & clones, Debian and Ubuntu. Whilst my RHEL repos are quite sizable now, the Debian & Ubuntu repositories are much sparser, so I’m going to make an effort to bring them to a level where they at least have all my public software (see projects.jethrocarr.com) available as well tested packages for current Debian Stable and Ubuntu LTS releases.

There’s some older stuff archived on the server if you go hunting as well, such as Fedora and ancient RHEL version packages, but I’m keeping them in the background for archival purposes only.

And yes, all packages are signed with my Amberdms/Jethro Carr GPG signing key. You should never be using any repositories without GPG signed packages, since they’re ideal attack vectors to use to install malicious content with a man-in-the-middle attack otherwise.