Technitribe

interesting problems (and a few solutions, too)

Technitribe
  • About the Authors
  • Log In
  • Log Out
  • Lost Password
  • Register
  • Reset Password
    • 4 Feb 2023

      Querying block device sizes in Python on Linux and Mac OS X

      Written by Tim Bielawa

      I drafted this blog post in 2016 (at least), but held off publishing it until I could have it fact checked. Well, 6 years have passed… I am 99% sure the information in this blog post is correct. But if you find an error with my explanation of the userspace-kernel-device dataflow then please send me an email so I can understand it better and update this post. Thank you!

      The Problem

      I’ve been experimenting with creating functionality within bitmath for reading the size of storage devices. This would provide a function similar to Python’s os.path.getsize, but for storage device capacity instead of file sizes.

      Unfortunately, it turns out that there is no out of the box (and cross-platform) solution in Python for reading the capacity of system storage devices. This meant some research was going to be required. Luckily, possible solutions for how to do this are abundant across the internet. Well, for Linux anyway. Figuring out how to make this work on Mac OS X was more challenging.

      And that’s where the story gets interesting.

      In the rest of this blog post we’ll learn the basics of how programs can interact with storage devices via the ioctl() system call. Then we’ll discuss the things we have to do and information we’ll need to have in order to implement an ioctl() request in Python. Next we’ll see how to gather all the necessary information (request codes and expected result sizes). Finally we’ll put all of this together into a runnable Python program.

      If you’re not familiar with that acronym, “ioctl” stands for “input/output control”.

      (more…)

      0 Comments
    • 9 Sep 2019

      Using jq to filter an array of objects from JSON

      Written by Tim Bielawa

      For some reason it took me an unreasonable amount of time to figure out how to filter an array (or list) of objects from a JSON stream. Every single example I found was a little too weird for me, or resulted in printing each object, but not in a final array format. Here’s what I came up with:

      Say for example you are parsing the AWS IP ranges JSON stream, you will receive an object like this:

      {
        "syncToken": "1567728788",
        "createDate": "2019-09-06-00-13-08",
        "prefixes": [
          {
            "ip_prefix": "18.208.0.0/13",
            "region": "us-east-1",
            "service": "AMAZON"
          },
          ... more objects here ...

      I was attempting to filter this down to ONLY objects where the service attribute was AMAZON. Using this jql I would get objects printed one after the other which is not what I wanted:

      $ jq -c '.prefixes[] | select(.service=="AMAZON")' < ip-ranges.json | head
      {"ip_prefix":"18.208.0.0/13","region":"us-east-1","service":"AMAZON"}
      {"ip_prefix":"52.95.245.0/24","region":"us-east-1","service":"AMAZON"}
      {"ip_prefix":"99.77.142.0/24","region":"ap-east-1","service":"AMAZON"}

      The correct syntax was ultimately very similar. 

      $ jq '.prefixes | map(. | select(.service=="AMAZON"))' < ip-ranges.json  | head
      [
      {
      "ip_prefix": "18.208.0.0/13",
      "region": "us-east-1",
      "service": "AMAZON"
      },

      Now we are getting each object returned as a member of an array. The difference is that we’re putting the .prefixes array objects into the map function and telling it to iterate every object through the select function. The map takes all of those matching objects and returns them as an array, whereas, previously we were only selecting objects that matched our select criteria. To get the objects back in a list we required the map. 

      1 Comment
    • 16 Nov 2018

      Two Year Break — And we’re back!

      Written by Tim Bielawa

      Where has Technitribe been for the last 2 years? The short answer: in domain limbo.

      Christmas Islands

      .cx is a Christmas Islands top-level domain (TLD). I did not have the domain set to auto-renew because I was trying to get away from my old registrar. At the time it was expiring I didn’t have the cash on hand to renew it. I will admit, I did not read all the Term & Conditions for the .cx TLD before registering. I guess I really screwed the pooch there.

      It turns out that if you have a .cxdomain, and it expires, then you have a problem. They will effectively hold it hostage for a period of 4 months. During this time you have the “option” to restore the domain, but at the cost of $400.00 USD.  Like I said, I was not very cash flush at the time.

      Waiting Period

      Shortly after the domain expired I moved across the country. While getting set up in my new location the domain left the restoration period and was open to general registration again. As is typical, a reseller had squatted the lnx.cx domain and bought it as soon as it became available. I resigned myself to my fate, I would have to let it go for a while.

      Return of lnx.cx

      The blog is now back! A few weeks ago I finally had the cash necessary to register it from a new registrar who isn’t evil (Netim). We have a new SSL Certificate issued by Let’s Encrypt, no more untrusted RapidSSL security alerts in your browser. And, as an aside, the setup for installing this new certificate was incredibly easy. I am genuinely surprised at how smooth it was using the certbot tool.

      What’s Next?

      I need to start writing more blog posts! I might shift the focus of the blog from being so Tech focused and open it up to capture some of my other interests, too. Such as woodworking. I really enjoy making and restoring furniture. I should write some blog posts about that.

      0 Comments
    • 20 Jan 2017

      [Updated] GitHub + Gmail — Filtering for Review Requests and Mentions

      Written by Tim Bielawa

      Update – 2017-01-27

      Just 3 days after publishing this blog post GitHub made a new blog post:

      Pull request reviews are a great way to share the weight of building software, and with review requests you can get the exact feedback you need.

      To make it easier to find the pull requests that need your attention, you can now filter by review status from your repository pull request index.

      Source: Filter pull request reviews and review requests

      I have tried this out and it’s great! Like most everything else on GitHub it’s very intuitive and simple to use. I won’t steal their thunder and describe it all here. So go check out the blog post for yourself and read up on the details (screenshots included!).

      Continue reading if you’re still interested in incorporating this kind of filtering and labeling into your Gmail account.

      The Problem

      I’ve been looking for a way to filter my GitHub Pull Request lists under the condition that a review is requested of me. The online docs didn’t show any filter options for this, so I checked out the @GitHubHelp twitter account. The answer was there on the front page — they don’t support filtering PRs by review-requested-by:me yet:

      @zaghnaboot Adding a filter for reviewers is definitely on our radar, though I don’t have a specific timeline to share. –SJ

      — GitHub Support (@GitHubHelp) January 19, 2017

      So what is one to do? I’m using Gmail so I began considering what filter options were available to me there. My objectives were to clearly label and highlight:

      •  PRs where review has been requested
      • Comments where I am @mention‘d

      Interested in knowing more? Read on after the break for all the setup details.

      (more…)

      2 Comments
    • 19 Jan 2017

      References in a sub-select

      Written by Alex Wood

      Have you ever had a sub-select where you really needed to reference a value in the outer query? I know I have! The naive way would be to run the outer query and then loop over the results running the inner query on each one. Luckily, there’s a better way. The Correlated subquery. Check it out! The example given is

      SELECT employee_number, name
        FROM employees AS Bob
        WHERE salary > (
          SELECT AVG(salary)
            FROM employees
            WHERE department = Bob.department);
      

      See how the sub-select references the outer query? It’s SQL magic.

      0 Comments
    • →
    Page 1 of 14
    • The Authors
    • Virtual Disk Guide

      Interested in virtualization? Do QCOWs rule your filesystem? Are you a libvirt or KVM+QEMU wizard? I wrote a book about virtual disk management. Check out the The Linux Sysadmin's Guide to Virtual Disks online for free at ScribesGuides.com.


      Consider supporting the author by purchasing a hard copy of the first edition for just $10.00 on Lulu.com.

    • bitmath

      bitmath is a Python library for dealing with file size units (GiB's, kB's, etc) in a sane way. bitmath supports arithmetic, rich comparison, conversion, automatic best human-readable representation, and many other utility functions. Read some examples on the docs site or check out the source on GitHub.

    • latest posts

      • Querying block device sizes in Python on Linux and Mac OS X February 4, 2023
      • Using jq to filter an array of objects from JSON September 9, 2019
      • Two Year Break — And we’re back! November 16, 2018
    • tags

      bitmath blog conference css dblatex DNS DocBook eclipse Emacs Erlang Fedora fedora 22 filter GNU Screen Haiku Introduction java jboss LCSEE Linux locale locales fix slicehost ubuntu Macports module nist nXML-Mode opengl open source OS X package packaging pki prefix units presentation project pypi Python scholarship si summit Tutorial ubuntu xcode XML XMPP
    • h4ck teh world

      tbielawatbielawa
      • Watch
        spotDL/spotify-downloader
        March 27, 2023 - 2:52 pm UTC
      • Watch
        exaloop/codon
        March 19, 2023 - 3:05 pm UTC
      • Issue Comment
        hashicorp/vault
        March 2, 2023 - 2:40 pm UTC
      • Issue Comment
        bitmath
        February 15, 2023 - 10:26 pm UTC
      • Issue Comment
        bitmath
        February 15, 2023 - 10:25 pm UTC

Creative Commons License
Technitribe by Tim Bielawa is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.