The fastest way to count columns in shell: AWK

Say you have (as I do) a file /tmp/myFile.txt containing a list of unrelated folders with their size, as follows:

1
2
3
4
5
6
7
7220    X03066659D/txt/20150109
1365 A30266659D/txt/20150112
9 X30626659D/txt/20150121
0 X30663659D/xml/20150102
5292 A30646659D/xml/20150105
10872 X30Q66659D/xml/20150107
7384 A30A66659D/xml/20150108

And you want to sum the first column of every line to know the total. AWK at rescue!

1
awk '{sum += $1} END {print sum} /tmp/myFile.txt'

If there were a discriminant you would like to use (imagine you only need those starting with A on second column) you can filter on AWK line (no need to grep)

1
awk '$2 ~ /^A/ {sum += $1} END {print sum}' /tmp/myFile.txt

Restore whole folders from Amazon S3 Glacier

Amazon Web Service it’s an incredible tool. And S3 it’s one of the best ways for storing your data and, of course, backups of your data.

If you use it for backup, you surely knows the Glacier feature: you can freeze your data getting a lower price… in money. Because the price you pay it’s that your data ain’t available immediatly when you need it, you have to wait to restore it before you can get it.

But a maybe worst issue is that you can’t restore a full folder. You only can restore files. So if you want to un-Glacier a folder you have to traverse all tree of folders, and check all individual files you want. One by one.

Althougt this sounds (and it is!) weird, it’s related with the way AWS S3 sees your files: there are no folders. A folder it’s just a visual representation of structure of files (using / as separator). But for S3 everything is a file.

Said that: I needed to restore a full directory with a lot of directories so doing the job one by one wasn’t an option. So I came to this solution on stackOverflow with credits to this other blog post.

I’ve modified the script just a bit because it didn’t work for me (maybe some parameters of aws-cli have changed). You’ll need the aws cli (Command Line Interface) to do this.

First of all, we get all Glaciered items on a bucket named here MYBUCKET, substitute it with your own name
(please note: I’ve updated it to allow spaces on file or folder names):

1
aws s3api list-objects-v2 --bucket MYBUCKET --query "Contents[?StorageClass=='GLACIER']" --output text | awk '{print substr($0, index($0, $2))}' | awk '{NF-=3};3' > glacier-restore.txt

This will get a list with all objects under Glacier rules on your bucket. Maybe you don’t want all of them so this is the moment to edit the glacier-restore.txt file in case you want to fine-tune the files you want to get.

Then, create (edit + chmod 755) the following script which takes the content of your file an do the restore:

1
2
3
4
5
6
7
8
#!/bin/sh

cat glacier-restore.txt | while read x
do
echo "Begin restoring $x"
aws s3api restore-object --restore-request Days=25 --bucket MYBUCKET --key "$x"
echo "Done restoring $x"
done

Remember: the process it’s not inmediate. With this command you instruct AWS to restore your files, but it will take between 3 and 5 hours to complete. You can do it faster (and more expensive) but if you need it so fast you’d maybe should consider standard storage.

Hope this helps!

Undo a pushed commit on git

We all know it. Shit happens.

Today, I screw a small piece of code ( thank you SonarLint! ;-) (*)). It was just a small optimization, but after checking with a coworker the code we couldn’t find what was going wrong but the reality was that something was broken so I decided to go back in time.

The commits (2 commits) was already pushed to our bitbucket and tried several approaches. I created a detached branch, modify and commit. No success. Then I tried some of features IntelliJ gaves me, but I could find a good one.

So finally I found (stackoverflow at rescue) this solution which help in my case. Be careful: my problem may be different than yours and so the solution. Even more, the solution I used it’s not the choosen as the best on previous link.

I had the following git history

1
2
3
4
5
6
7
8
9
10
11
commit 2071cf3cb8
Author: xxxxx
Date: Wed Nov 28 10:16:48 2018 +0100

commit 5ff2c4a3a1
Author: xxxxx
Date: Wed Nov 28 10:11:06 2018 +0100

commit 3a37cbc8e7
Author: yyyyy
Date: Wed Nov 28 09:58:33 2018 +0100

The two top commits was mine, and was wrong. I wanted to go back to 3a37cbc8e7 so I use the following sequence:

1
2
3
4
5
6
7
8
9
10
# Resets index to former commit; replace '3a37cbc8e7' with your commit code
git reset 3a37cbc8e7

# Moves pointer back to previous HEAD
git reset --soft HEAD@{1}

git commit -m "Revert commit to the past"

# Updates working copy to reflect the new commit
git reset --hard

Here is the point where you’re back to the commit you wanted to be. Check it. If you’re satisfied, just push it.

And additional benefit with this approach is that you don’t loose your history: there will be ever a culprit (myself, in this case) written on the chain.

Hope this helps.

(*) PS: Of course, the blame on SonarLint it’s just a joke. The fault was mine :-)

Convert file encoding, bulk way

I have an old java project, with all files enconded as ISO-8859-1 . It was made back in time where most of our development team worked with Windows (fortunately, those times are over ;-)). Now (almost 10 years later -yep that is a really legacy project-) we’re updating it, mostly in the frontend user interface.

So, when we started working with it we had a mix of old ISO-8859-1 files and new UTF-8. At some point we decided to change all files encoding to UTF-8. We made some changes using the IntelliJ features for this purpose but it was clear we needed some automatized way to do it, and this is the one-liner way we used:

1
find -type f -name "*.java" -exec file {} \; | grep ISO | cut -d ":" -f 1  | while read a ; do iconv -f ISO-8859-1 -t UTF-8 < $a > /tmp/tmp ; "mv" /tmp/tmp $a ; done

So read it as follows:

“Find all java files, check which of them are encoded as ISO-whatever and for all of them use the iconv command to create a new correctly encoded file as /tmp/tmp and then move it to the original location”

Of course: change the *”.java”* for whatever you need, and maybe it’s a good idea to run it in several steps to see what’s happening.

Run first just the find -type f -name "*.java" -exec file {} \; | grep ISO , then something like find -type f -name "*.java" -exec file {} \; | grep ISO | cut -d ":" -f 1 | while read a ; echo "gonna work with $a" ; done so you can be sure you don’t break anything (make a backup first!)

Error actualizando docker (mint 18)

Hoy, actualizando el sistema desde el entorno gráfico, me ha salido un error de un paquete que no podía actualizar. El sistema me sugería que buscara cual era el paquete que daba problemas usando el filtro rotos . Tras maldecirme por hacer estas cosas en el entorno gráfico en lugar de la consola, he ejecutado sudo apt-get update && sudo apt-get upgrade desde la consola para ver que es lo que estaba pasando realmente.

El resultado era:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Los siguientes paquetes tienen dependencias incumplidas:
docker-ce : Depende: containerd.io pero no está instalado
E: Dependencias incumplidas. Pruebe de nuevo utilizando -f.
jmiguel@mab ~ $ sudo apt-get -f install
Leyendo lista de paquetes... Hecho
Creando árbol de dependencias
Leyendo la información de estado... Hecho
Corrigiendo dependencias... Listo
Los paquetes indicados a continuación se instalaron de forma automática y ya no son necesarios.
libjs-openlayers libwireshark8 libwiretap6 libwscodecs1 libwsutil7
Utilice «sudo apt autoremove» para eliminarlos.
Se instalarán los siguientes paquetes adicionales:
containerd.io
Se instalarán los siguientes paquetes NUEVOS:
containerd.io
0 actualizados, 1 nuevos se instalarán, 0 para eliminar y 0 no actualizados.
1 no instalados del todo o eliminados.
Se necesita descargar 0 B/19,9 MB de archivos.
Se utilizarán 87,6 MB de espacio de disco adicional después de esta operación.
¿Desea continuar? [S/n] S
(Leyendo la base de datos ... 523472 ficheros o directorios instalados actualmente.)
Preparando para desempaquetar .../containerd.io_1.2.0-1_amd64.deb ...
Desempaquetando containerd.io (1.2.0-1) ...
dpkg: error al procesar el archivo /var/cache/apt/archives/containerd.io_1.2.0-1_amd64.deb (--unpack):
intentando sobreescribir `/usr/sbin/runc', que está también en el paquete runc 1.0.0~rc2+docker1.13.1-0ubuntu1~16.04.1
dpkg-deb: error: el subproceso copiado fue terminado por la señal (Broken pipe)
Se encontraron errores al procesar:
/var/cache/apt/archives/containerd.io_1.2.0-1_amd64.deb

Parece que había algun problema con la actulizacion de docker. He probado a parar docker antes de actualizar pero el resultado era el mismo, asi que la solución ha sido desinstalar docker-ce y runc , que parece ser antiguo y estar obsoleto.

Asi que tras ejecutar:

1
2
3
sudo apt-get remove docker-ce
sudo apt-get remove runc
sudo apt-get install docker-ce

Todo ha vuelto a la normalidad.

Espero que a alguien le sirva :-)

Several submit actions in one form

Sometimes you need to have several actions to be submitted on the same form. Usually you only have one button for this which points to the place you define on the FORM ACTION block.
So, how can you detect which button was clicked in plain HTML?

You can use the value variable on button, this way:

1
2
3
4
5
6
<FORM METHOD="whatever" ACTION="/your/endPoint/">


<button name="btnaction" value="accept" class="..." type="submit">Accept 1</button>"
<button name="btnaction" value="dontAccept" class="..." type="submit">Dont Accept</button>"
<button name="btnaction" value="maybe" class="..." type="submit">Maybe Accept</button>"

And in your code just check the btnAction parameter variable for accept, dontAccept or maybe values. Easy!

Show block id on Linux devices

This is the kind of things I have to seach everytime I need to install a new drive on a computer. As this is the kind of things I don’t do everyday, I forget it.

So… How list the devices IDs on a Linux system?. Just like this:

1
2
3
4
5
root@caronte:~# blkid
/dev/sda1: UUID="d5a0a58e-13bc-46bb-b106-de5219dd7259" TYPE="ext4" PARTUUID="92e8992d-01"
/dev/sda2: UUID="14fa763f-0861-41c3-ba52-ce25e94e009a" TYPE="ext4" PARTUUID="92e8992d-02"
/dev/sdb1: UUID="5bd7c629-74af-4714-b3f6-34bab7a988b0" TYPE="ext4" PARTUUID="dd4daff0-c29a-4489-ab93-06e3929e2cad"
/dev/sdc1: UUID="42123f82-e997-46b6-9dc7-d1d7f521379f" TYPE="ext4" PARTUUID="151b9380-01"

My Hello World

Welcome to my brand new blog!.

As I pretend to write about technologies I use, I cannot imagine a better way to start:

1
2
3
4
5
 _   _      _ _        __        __         _     _ 
| | | | ___| | | ___ \ \ / /__ _ __| | __| |
| |_| |/ _ \ | |/ _ \ \ \ /\ / / _ \| '__| |/ _` |
| _ | __/ | | (_) | \ V V / (_) | | | | (_| |
|_| |_|\___|_|_|\___/ \_/\_/ \___/|_| |_|\__,_|

Disclaimer:

I’m a non-english speaker. My spoken english is far to be fluent but I think I can make a good job trying to write it. And it should improve on future, as it’s part of what I want ot do with this new blog.

What can you find here?

I’ve been programmer (gwBasic, Clipper, Visual Basic, Perl, Java, Groovy) and systems analyst (or SysOp, SysAdmin, BOFH or whatever you want) since 1985. So I’ll write to whatever I find interesting but always related to technical questions. I have other blog for personal questions and for some time (maybe some day…) I had another one for aviation information: aerotrastornados.com

Tagging

All posts will have some tags. First for language [eng | esp]. Next for topic, and I’ll try to keep the list as short as possible: [programming | systems | java | groovy | grails | linux]

No more to write right now. Just let’s see if I can publish it without further problems before start customization, fonts, images and so on.