Technitribe

interesting problems (and a few solutions, too)

Technitribe
  • About the Authors
  • Log In
  • Log Out
  • Lost Password
  • Register
  • Reset Password
    • 9 Sep 2019

      Using jq to filter an array of objects from JSON

      Written by Tim Bielawa

      For some reason it took me an unreasonable amount of time to figure out how to filter an array (or list) of objects from a JSON stream. Every single example I found was a little too weird for me, or resulted in printing each object, but not in a final array format. Here’s what I came up with:

      Say for example you are parsing the AWS IP ranges JSON stream, you will receive an object like this:

      {
        "syncToken": "1567728788",
        "createDate": "2019-09-06-00-13-08",
        "prefixes": [
          {
            "ip_prefix": "18.208.0.0/13",
            "region": "us-east-1",
            "service": "AMAZON"
          },
          ... more objects here ...

      I was attempting to filter this down to ONLY objects where the service attribute was AMAZON. Using this jql I would get objects printed one after the other which is not what I wanted:

      $ jq -c '.prefixes[] | select(.service=="AMAZON")' < ip-ranges.json | head
      {"ip_prefix":"18.208.0.0/13","region":"us-east-1","service":"AMAZON"}
      {"ip_prefix":"52.95.245.0/24","region":"us-east-1","service":"AMAZON"}
      {"ip_prefix":"99.77.142.0/24","region":"ap-east-1","service":"AMAZON"}

      The correct syntax was ultimately very similar. 

      $ jq '.prefixes | map(. | select(.service=="AMAZON"))' < ip-ranges.json  | head
      [
      {
      "ip_prefix": "18.208.0.0/13",
      "region": "us-east-1",
      "service": "AMAZON"
      },

      Now we are getting each object returned as a member of an array. The difference is that we’re putting the .prefixes array objects into the map function and telling it to iterate every object through the select function. The map takes all of those matching objects and returns them as an array, whereas, previously we were only selecting objects that matched our select criteria. To get the objects back in a list we required the map. 

      0 Comments
    • 19 Jan 2017

      References in a sub-select

      Written by Alex Wood

      Have you ever had a sub-select where you really needed to reference a value in the outer query? I know I have! The naive way would be to run the outer query and then loop over the results running the inner query on each one. Luckily, there’s a better way. The Correlated subquery. Check it out! The example given is

      SELECT employee_number, name
        FROM employees AS Bob
        WHERE salary > (
          SELECT AVG(salary)
            FROM employees
            WHERE department = Bob.department);
      

      See how the sub-select references the outer query? It’s SQL magic.

      0 Comments
    • 23 Oct 2015

      SSL Renewal And You

      Written by Tim Bielawa

      RapidSSL_SEAL-90x50

      This post is about renewing SSL certificates. There’s not a lot of information I want to communicate here, so I’m going to keep it short.

      Yesterday the SSL certificate for https://blog.lnx.cx expired. I don’t know much about SSL, other than I find it more confusing/complicated than most things. I knew that I needed to renew the SSL certificate for the blog, but I did not know what that exactly meant. When I called my cert provider on the phone to renew, they told me that the renewal process begins with submitting a new Certificate Signing Request, or CSR in crypto parlance. We ended the call shortly thereafter and I set off to get started.

      I still had questions though. If I’m “renewing” my SSL certificate, does that mean my existing certificate is involved in some way? When I began reviewing the CSR generation procedure I saw no references to existing certificates. I did a bit of Internet research to try and figure this out.

      Eventually I found out that the idea of “renewing” a certificate is a bit of a misnomer. That is, nothing you have carries over with you. The process of “renewing” a certificate is actually the exact same process as getting an initial certificate. I’ll say that again for clarity:

      Renewing an SSL certificate is the exact same thing as getting your first SSL certificate.

      I hope this helps out other folks who are as confused as I was about the renewal process.

      0 Comments
    • 15 May 2015

      Getting consistent fingerprints from git-archive

      Written by Alex Wood

      The “git archive” man page states:

      git archive behaves differently when given a tree ID versus when given a commit ID or tag ID. In the first case the current time is used as the modification time of each file in the archive.

      By using the current time in this case, git-archive is dooming all of our tarballs to have constantly changing SHA256 hashes. A lot of build systems, including Fedora’s Koji, rely on source tarballs maintaining a consistent fingerprint. What is a person to do?

      Fix it of course! Below is a Python 2 program I wrote that addresses the issue. The code is well-commented (I hope) so you should be able to follow along. You give it the Unix timestamp you want the files to have, the git ref you want baked into the tar header, and the initial tarball. The result is printed to stdout so just redirect that to wherever you please (or pipe it into gzip). It also has some code in there to deal with tarballs created by the maven-assembly plugin, but it doesn’t surface that on the primitive CLI. I’m leaving that as an exercise for the reader I guess.

      0 Comments
    • 16 Mar 2015

      Open URL Bug with XDG on XFCE4

      Written by Alex Wood

      Recently I noticed that in my IRC client, when I right-click a URL and select “Open Link In Browser”, the system would open a new browser window (or tab if appropriate) but not pointed to the link I wanted to visit. It would just open the home page.

      What gives? Well, I happen to know from experience that in Linux most programs that need to use a “default” type service of which there are many implementations (such as a web-browser) use the xdg-open command. XDG associates different mime-types to default applications. Step one then is to figure out what’s going on with XDG.

      
      % xdg-mime query default text/html
      firefox.desktop firefox.desktop
      

      Here I’m asking XDG what applications are associated with the text/html mime-type. Seeing two firefox.desktop files was a bit of a surprise. Let’s find out more!

      
      % locate firefox.desktop
      /usr/share/applications/firefox.desktop
      /usr/share/xfce4/helpers/firefox.desktop
      

      So I open up those two files and the first file looks normal and that file actually belongs to the Firefox package according to rpm -qf. In the second file, I see

      
      X-XFCE-Commands=%B -remote "openURL(about:blank,new-window)";%B;
      X-XFCE-CommandsWithParameter=%B -remote "openURL(%s,new-window)";%B "%s";
      

      That looks strange to me. If I want to open a URL from the command line, I don’t use openURL. Let’s see what happens if I replace that with

      
      X-XFCE-Commands=%B;
      X-XFCE-CommandsWithParameter=%B "%s";
      

      Aha! It works! But why? Well, after a little searching I came across Mozilla Bug 1080319. Looks like the openURL was a legacy thing and got removed in Firefox 36. And a quick rpm -q firefox confirms that I’m running that version. Firefox 36.0.1 add support for openURL back in, but my hack will serve until that version hits Fedora.

      0 Comments
    • →
    Page 1 of 2
    • The Authors
    • Virtual Disk Guide

      Interested in virtualization? Do QCOWs rule your filesystem? Are you a libvirt or KVM+QEMU wizard? I wrote a book about virtual disk management. Check out the The Linux Sysadmin's Guide to Virtual Disks online for free at ScribesGuides.com.


      Consider supporting the author by purchasing a hard copy of the first edition for just $10.00 on Lulu.com.

    • bitmath

      bitmath is a Python library for dealing with file size units (GiB's, kB's, etc) in a sane way. bitmath supports arithmetic, rich comparison, conversion, automatic best human-readable representation, and many other utility functions. Read some examples on the docs site or check out the source on GitHub.

    • latest posts

      • Using jq to filter an array of objects from JSON September 9, 2019
      • Two Year Break — And we’re back! November 16, 2018
      • [Updated] GitHub + Gmail — Filtering for Review Requests and Mentions January 20, 2017
    • tags

      bitmath blog conference css dblatex DNS DocBook eclipse Emacs Erlang Fedora fedora 22 filter GNU Screen Haiku Introduction java jboss LCSEE Linux locale locales fix slicehost ubuntu Macports module nist nXML-Mode opengl open source OS X package packaging pki prefix units presentation project pypi Python scholarship si summit Tutorial ubuntu xcode XML XMPP
    • h4ck teh world

      tbielawatbielawa
      • Create
        aos-cd-jobs
        October 27, 2020 - 1:45 pm UTC
      • Pull Request
        openshift/aos-cd-jobs
        October 27, 2020 - 1:45 pm UTC

Creative Commons License
Technitribe by Tim Bielawa is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.