bitmath is a Python module I wrote which simplifies many facets of interacting with file sizes in various units as python objects. A few weeks ago version 1.3.1 was released with a few small updates.
Updates
- New function: bitmath.parse_string_unsafe(), a less strict version of bitmath.parse_string()
This new function accepts inputs using non-standard prefix units such as single-letter, or mis-capitalized units. For example,
parse_stringwill not accept a short unit like ‘100k‘, whereasparse_string_unsafewill gladly accept it:- Documentation Refresh: The project documentation has been thoroughly reviewed and refreshed.
Several broken, moved, or redirecting links have been fixed. Wording and examples are more consistent. The documentation also lands correctly when installed via package.
Getting bitmath-1.3.1
bitmath-1.3.1 is available through several installation channels:
Ubuntu builds have not been prepared yet due to issues I’ve been having with Launchpad and new package versions.
bitmath is a Python module I wrote for working with file size units (ex:
12GiB,64kB) as objects. You can use them just like you would use regular numbers in python. It’s full of other functionality as well. Objects have native ‘convert to $unit‘ methods, support native arithmetic, are sortable, and include a ‘best human readable prefix’ method.Since March 2014, bitmath had only been available via PyPi and Fedora/EPEL repositories. Now, as of July 2nd 2016, bitmath is natively available to Ubuntu users by means of a new Personal Package Archive (PPA) hosting bitmath builds for Xenial, Wily, Vivid, Trusty, and Precise.
Ubuntu users can install bitmath in the following way:
Ubuntu support wouldn’t have happened if GitHub user hkraal hadn’t submitted an issue. Thanks Henk for getting the fire lit!
- Check out bitmath on GitHub
I’m very excited (and proud) to announce that on March 3rd, 2016 I reached a long-term goal I set for myself 3½ years ago, by self-publishing my first book, The Linux Sysadmin’s Guide to Virtual Disks. The book is published under my new brand, Scribe’s Guides.
The first edition of The Virtual Disk Guide has been a long time coming. Nearly 7 years of on-and-off writing have gone into it. I’m relieved to have made it this far.
-
- Front cover
-
- Back cover
About the Virtual Disk Guide
I view the book as the definitive reference guide for virtual disk related activities — clear, concise, accurate, and approachable to readers of all skill levels— but that’s just my opinion. You can decide that for yourself.
The book is quite thoroughly cited and annotated with nearly 100 individual footnotes and references to additional learning resources. The book weighs in at around 80 pages, 7 chapters, and two technical appendices. Here’s the byline from the scribesguides.com website:
The Linux Sysadmin’s Guide to Virtual Disks demonstrates the core concepts of virtual disk management. Real-world problems are covered in the book’s “Cookbook” section. Other topics include: helper utilities, disk formats, troubleshooting tips, performance considerations, and comprehensive appendices.
GetBuy one (please?)- Buy a hard copy of the first edition for $10.00
- Read the latest build[1] of the PDF or HTML versions online for free
Or do both! Say “thanks!” by purchasing a copy, and then enjoy the latest builds online forever, for free!
[1] – The original first edition text is also available for free in PDF and HTML formats and is identical to the print copy
Open Source
The official publishing of The Virtual Disk Guide does not change anything about it’s openness or your freedom to remix it however you wish. The book is still freely licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0).
All of the source material used to build the book’s body material and cover images are still free and open source, covered under the same license. All digital media displayed in the book, such as figures and the cover art, was created using free/open source software. Each media item was created and saved in digital formats unencumbered by patents.
As ever, if you identify errors in the book or have thought of a way to improve it, please open a ticket on the GitHub issue tracker. If you’ve read a copy of the book already and would like to contribute a review or statement, feel free to reach out to me. Find my email in a github commit, or look at my other contact methods under the author highlight panel on scribesguides.com.
More Blog Posts Coming Soon
The experience of writing and publishing this book has taught me much, and it’s time to spread that information. Check back soon for a follow-up post I’m writing which covers more of the technical side of self-publishing. Specifically, self-publishing a DocBook 5 document at the on-demand printing website lulu.com.
Let me be explicitly clear, this is not a promotion for lulu.com.
Rather, the post will review some of the technical challenges I encountered (old examples: #1, #2, #3) during the publishing process, including challenges specific to Lulu. Such as, how I customized the PDF output from dblatex to look more personal and less generically academic, why I had to order three proof copies of the book before the cover matter printed in decent quality, and how to adjust your inner and outer page margins so there’s a reasonable amount of whitespace between the spine/binding and the body text.
I have a feeling that by the time I’m done with the blog posts I’m going to have written another book of documentation about how I wrote a book of documentation
-
It’s been quite a while since I’ve posted any bitmath updates (bitmath is a Python module I wrote which simplifies many facets of interacting with file sizes in various units as python objects) . In fact, it seems that the last time I wrote about bitmath here was back in 2014 when 1.0.8 was released! So here is an update covering everything post 1.0.8 up to 1.3.0.
New Features
- A command line tool,
bitmath, you can use to do simple conversions right in your shell [docs]! - New utility function bitmath.parse_string for parsing a human-readable string into a bitmath object
- New utility: argparse integration: bitmath.BitmathType. Allows you to specify arguments as bitmath types
- New utility: progressbar integration: bitmath.integrations.BitmathFileTransferSpeed. A more functional file transfer speed widget
- New bitmath module function: bitmath.query_device_capacity(). Create bitmath.Byte instances representing the capacity of a block device
- This my favorite enhancement
- In an upcoming blog post I’ll talk about just how cool I thought it was learning how to code this feature
- Conceptual and practical implementation topics included
- The bitmath.parse_string() function now can parse ‘octet’ based units
- Enhancement requested in #53 parse french unit names by walidsa3d.
- New utility function: bitmath.best_prefix()
- Return an equivalent instance which uses the best human-readable prefix-unit to represent it
- This is way cooler than it may sound at the surface, I promise you
Bug Fixes
- #49 – Fix handling unicode input in the bitmath.parse_string function. Thanks drewbrew!
- #50 – Update the setup.py script to be python3.x compat. Thanks ssut!
- #55 “best_prefix for negative values”. Now bitmath.best_prefix() returns correct prefix units for negative values. Thanks mbdm!
Misc
To help with the Fedora Python3 Porting project, bitmath now comes in two variants in Fedora/EPEL repositories (BZ1282560). The Fedora and EPEL updates are now in the repos. TIP:
python2-bitmathwill obsolete thepython-bitmathpackage. Do adnf/yum‘update‘ operation just to make sure you catch it.The PyPi release has already been pushed to stable.
Back in bitmath-1.0.8 we had 150 unit tests. The latest release has almost 200! Go testing!
:confetti:- A command line tool,
30 Oct 2015
Streaming the serial numbers from an X509 certificate revocation list
Written by Alex Wood
The project I work on uses X509 certificates with custom extensions to manage content access on the Red Hat CDN. The basic idea is that Candlepin issues X509 certificates with an extension saying what content the certificate is good for. Client systems then use that certificate for TLS client authentication when connecting to the CDN. If the content they are requesting (deduced from the request URL) matches the content available to them in the certificate, then access is granted.
This system works well in practice except for one problem: every time content for a particular product changes, the content data in the X509 extension becomes obsolete. We have to revoke the obsolete certificates and issue new ones. The result is an extremely large certificate revocation list (CRL).
For our cryptography needs, Candlepin uses the venerable Legion of the Bouncy Castle Java library. This library anticipates normal CRL usage so when building a CRL object from an existing file, the entire structure is read into memory at once. This approach doesn’t scale well with the numbers of revoked certificates we are dealing with, so we needed to devise a way to stream the CRL. Moreover, the only thing we really care about for our purposes is the revoked certificate’s serial number.
Streaming the CRL means we need to dissect the ASN1 that describes the CRL one piece at a time. RFC 5280 to the rescue! Looking at the description of the ASN1 for a CRL reveals that before the sequence containing the revocation entries, there will be a
thisUpdateand optionallynextUpdatefield of either type UTCTime or GeneralizedTime. We need to descend in the ASN1 until we get to thethisUpdatefield, look for and discard the optionalnextUpdatefield and then walk through therevokedCertificatessequence reading the serial numbers.That procedure is not exactly a walk in the park, so in the hope that someone else may find it useful, here is the solution I came up with. Keep in mind that the code does not check the signature on the CRL so this code should not be used for any CRL that you do not trust implicitly.
The end results are pretty dramatic. The benchmarking toolkit I’m using shows an improvement in execution time by an order of magnitude (from around 7 seconds to .7 seconds) and memory usage drops by about 30%. You can see the GC statistics in the graph below.
and the benchmarking results are
Benchmark Mode Cnt Score Error Units CRLBenchmark.inMemory avgt 20 7493.602 ± 941.592 ms/op CRLBenchmark.stream avgt 20 669.084 ± 91.382 ms/op
In writing this, A Layman’s Guide to a Subset of ASN.1, BER, and DER was of invaluable assistance to me as was the Wikipedia page on X.690. I recommend reading them both.
h4ck teh world
tbielawa - Issue Comment
openshift/openshift-ansible
September 29, 2016 - 6:10 pm UTC - Issues
openshift/openshift-ansible
September 29, 2016 - 6:10 pm UTC - Push
openshift/openshift-ansible
September 29, 2016 - 5:47 pm UTC - Delete
openshift-ansible
September 29, 2016 - 5:47 pm UTC - Pull Request
openshift/openshift-ansible
September 29, 2016 - 5:47 pm UTC
- Issue Comment
Technitribe by Tim Bielawa is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.