Tools/Services
SQLite as an Archive (or an Easy, Compressed Filesystem)
The sqlar project was an experiment to test the practicality of a SQLite archive format by the lead SQLite developer. I say “was” because, like sqlite, it seldom needs modification and appears to be stable.
What makes it fun is that the result is not internalized and inaccessible. Rather, it’s an intact SQLite database that you can readily inspect from the SQLite client. Therefore, the compression is per-blob and the purpose of the tool is merely to make it convenient to add records corresponding to files. The reverse should be true as well: if you create a sqlite DB and populate it with zlib-compressed data, you should simply be able to dump it using the sqlar tool.
To test sqlar out, either checkout or download a copy of the source (which embeds SQLite within it). You may download it here: SQLite Archiver.
Extract it and build:
$ mkdir sqlar $ cd sqlar/ $ tar xzf ../sqlar-src-15adeb2f9a.tar.gz $ cd sqlar-src-15adeb2f9a/ $ make gcc -g -I. -D_FILE_OFFSET_BITS=64 -Wall -Werror -c sqlite3.c -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION sqlite3.c gcc -g -I. -D_FILE_OFFSET_BITS=64 -Wall -Werror -o sqlar sqlar.c sqlite3.o -lz
Run a test:
$ mkdir test_root $ echo "test1" > test_root/test1 $ echo "test2" > test_root/test2 $ echo "test3" > test_root/test3 $ ./sqlar -v test_root.sqlar test_root added: test_root added: test_root/test1 added: test_root/test3 added: test_root/test2 $ mkdir output $ cd output/ $ ../sqlar -x -v ../test_root.sqlar test_root test_root/test1 test_root/test3 test_root/test2
Visually inspect the database file:
$ cd .. $ sqlite3 test_root.sqlar SQLite version 3.8.2 2013-12-06 14:53:30 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> .schema CREATE TABLE sqlar( name TEXT PRIMARY KEY, mode INT, mtime INT, sz INT, data BLOB ); sqlite> select * from sqlar; test_root|16893|1455064403|0| test_root/test1|33204|1455064393|6|test1 test_root/test3|33204|1455064403|6|test3 test_root/test2|33204|1455064399|6|test2 sqlite>
Notice that no compression was performed because the files are so trivial. zlib is used for compression every time unless you explicitly turn it off using “-n”.
You can also mount the archive using FUSE:
$ # This requires libfuse-dev to be installed. $ make sqlarfs gcc -g -I. -D_FILE_OFFSET_BITS=64 -Wall -Werror -o sqlarfs sqlarfs.c sqlite3.o -lz -lfuse $ mkdir mount $ ./sqlarfs test_root.sqlar `pwd`/mount
In another terminal, list the contents:
$ ls -l total 0 dr-xr-xr-x 1 dustin dustin 0 Feb 9 19:51 test_root $ cd test_root/ $ ls -l total 0 -r--r--r-- 1 dustin dustin 6 Feb 9 19:33 test1 -r--r--r-- 1 dustin dustin 6 Feb 9 19:33 test2 -r--r--r-- 1 dustin dustin 6 Feb 9 19:33 test3 -r--r--r-- 1 dustin dustin 576 Feb 9 19:51 test4 $ cat test4 Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
This was a later version of our earlier archive where I added a highly compressible text-file (test4).
It’s a fun little tool.
Split a Media File by a List of Time Offsets
We’ll split a single audio file containing the whole Quake soundtrack using SplitMedia.
The list file:
0:00:00 Quake Theme 0:05:08 Aftermath 0:07:34 The Hall of Souls ... 1:08:21 Scourge of Armagon 4 1:11:34 Scourge of Armagon 5 ... 1:39:57 Dissolution of Eternity 6 1:43:01 Dissolution of Eternity 7 1:46:07 Dissolution of Eternity 8
The command:
$ splitmedia Quake\ Soundtrack.m4a list_file.quake quake_output OFF 000:00:00.000 DUR 000308.000 01_QuakeTheme.m4a OFF 000:05:08.000 DUR 000146.000 02_Aftermath.m4a OFF 000:07:34.000 DUR 000500.000 03_TheHallofSouls.m4a ... OFF 001:08:21.000 DUR 000193.000 14_ScourgeofArmagon4.m4a OFF 001:11:34.000 DUR 000193.000 15_ScourgeofArmagon5.m4a ... OFF 001:39:57.000 DUR 000184.000 24_DissolutionofEternity6.m4a OFF 001:43:01.000 DUR 000186.000 25_DissolutionofEternity7.m4a OFF 001:46:07.000 DUR 000000.000 26_DissolutionofEternity8.m4a
Calculating a Hash for a Path (Recursively)
PathFingerprint allows you to recursively generate hashes for a directory structure. While doing this, it builds a catalog in a separate directory to serve as a cache. Subsequent runs of large directories will run much quicker. You can also do simple lookups against an existing catalog and generate/print a report of what has changed since the last run.
Build a test directory:
$ mkdir -p scan_path/subdir1 $ mkdir -p scan_path/subdir2 $ touch scan_path/subdir1/aa $ touch scan_path/subdir1/bb
Calculate the hash (with reporting enabled):
$ pfhash -s scan_path -c catalog_path -R - create file subdir1/aa create file subdir1/bb create path subdir1 create path subdir2 create path . 0df9bc5a7657b7d481c219656441f10d21fd5668
Run again with a couple of changes (with reporting enabled):
$ touch scan_path/subdir1/aa $ touch scan_path/subdir2/new_file $ pfhash -s scan_path -c catalog_path -R - update file subdir1/aa create file subdir2/new_file update path subdir2 update path . e700843c1b5c2f40a68098e1df96ef08b6081fe8
Lookup the hash using the lookup tool:
$ pflookup -c catalog_path e700843c1b5c2f40a68098e1df96ef08b6081fe8 $ pflookup -c catalog_path -r subdir1 426a98d313a0a740b8445daa5102b3ed6dd7f4ed $ pflookup -c catalog_path -r subdir1/aa da39a3ee5e6b4b0d3255bfef95601890afd80709
Creating an Installable Boot Disk for OSX
Courtesy of Apple:
Ephemeral/Disposable Email Accounts
These types of services give you temporary email-accounts that you can use for whatever purpose (like a one-time password or disposable phone). This one allows you to have more than one that points to the same inbox, and the emails are only kept for an hour:
Geonames API for RDF/Semantic Web Queries
Geonames has a sprawling set of APIs to search for all manner of geographical information.
Their Search API also has the option of returning the RDF-compatible XML. RDF is often used for Semantic Web-related applications.
The geonames_rdf Python library not only provides you the ability to query the API from your Python code but also provides a tool that can often perform identical queries from the command-line. Used as a library, you can get the raw RDF document, a list of XML nodes, or a simple 2-tuple list of keys and values. Used as a command-line tool, you can print the raw RDF result or a flat list of keys and values.
To install, use PyPI:
$ sudo pip install geonames_rdf
Sourcecode Example
Code fragment
sa = geonames.adapters.search.Search('username')
result = sa.query('detroit').country('us').max_rows(2).execute()
for id_, name in result.get_flat_results():
# make_unicode() is only used here for Python version-compatibility.
print(geonames.compat.make_unicode("[{0}]: [{1}]").format(id_, name))
Output
[http://sws.geonames.org/4990729/]: [Detroit] [http://sws.geonames.org/6955112/]: [Detroit-Warren-Livonia]
Command-line example
Simple list:
Pass the exact same parameter names and values to zero or more “-p” parameters:
$ gn_search dsoprea -p query detroit -p country us -p max_rows 2
Output:
[http://sws.geonames.org/4990729/]: [Detroit] [http://sws.geonames.org/6955112/]: [Detroit-Warren-Livonia]
Raw RDF response:
Pass the “-x” parameter:
$ gn_search dsoprea -p query detroit -p country us -p max_rows 2 -x
Output:
<rdf:RDF xmlns:cc="http://creativecommons.org/ns#" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:gn="http://www.geonames.org/ontology#" xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:wgs84_pos="http://www.w3.org/2003/01/geo/wgs84_pos#"> <gn:Feature rdf:about="http://sws.geonames.org/4990729/"> <rdfs:isDefinedBy rdf:resource="http://sws.geonames.org/4990729/about.rdf"/> <gn:name>Detroit</gn:name> <gn:alternateName xml:lang="af">Detroit</gn:alternateName> <gn:alternateName xml:lang="ar">ديترويت</gn:alternateName> <gn:alternateName xml:lang="az">Detroyt</gn:alternateName> <gn:alternateName xml:lang="be">Горад Дэтройт</gn:alternateName> <gn:alternateName xml:lang="bg">Детройт</gn:alternateName> <gn:alternateName xml:lang="bn">ডেট্রয়েট</gn:alternateName> <gn:alternateName xml:lang="bs">Detroit</gn:alternateName> <gn:alternateName xml:lang="ca">Detroit</gn:alternateName> <gn:alternateName xml:lang="ce">Детройт</gn:alternateName> <gn:alternateName xml:lang="cs">Detroit</gn:alternateName> <gn:alternateName xml:lang="da">Detroit</gn:alternateName> <gn:alternateName xml:lang="de">Detroit</gn:alternateName> <gn:alternateName xml:lang="el">Ντιτρόιτ</gn:alternateName> <gn:alternateName xml:lang="en">Detroit</gn:alternateName> <gn:alternateName xml:lang="eo">Detroit</gn:alternateName> <gn:alternateName xml:lang="es">Detroit</gn:alternateName> <gn:alternateName xml:lang="et">Detroit</gn:alternateName> <gn:alternateName xml:lang="fa">دیترویت</gn:alternateName> <gn:alternateName xml:lang="fi">Detroit</gn:alternateName> <gn:alternateName xml:lang="fr">Détroit</gn:alternateName> <gn:alternateName xml:lang="gl">Detroit</gn:alternateName> <gn:alternateName xml:lang="he">דטרויט</gn:alternateName> <gn:alternateName xml:lang="hi">डेट्राइट</gn:alternateName> <gn:alternateName xml:lang="hu">Detroit</gn:alternateName> <gn:alternateName xml:lang="hy">Դետրոյթ</gn:alternateName> <gn:alternateName xml:lang="id">Detroit</gn:alternateName> <gn:alternateName xml:lang="io">Detroit</gn:alternateName> <gn:alternateName xml:lang="is">Detroit</gn:alternateName> <gn:alternateName xml:lang="it">Detroit</gn:alternateName> <gn:alternateName xml:lang="ja">デトロイト</gn:alternateName> <gn:alternateName xml:lang="ka">დეტროიტი</gn:alternateName> <gn:alternateName xml:lang="kk">Детройт</gn:alternateName> <gn:alternateName xml:lang="ko">디트로이트</gn:alternateName> <gn:alternateName xml:lang="la">Detroitum</gn:alternateName> <gn:alternateName xml:lang="lt">Detroitas</gn:alternateName> <gn:alternateName xml:lang="lv">Detroita</gn:alternateName> <gn:alternateName xml:lang="mk">Детроит</gn:alternateName> <gn:alternateName xml:lang="mr">डेट्रॉईट</gn:alternateName> <gn:alternateName xml:lang="mrj">Детройт</gn:alternateName> <gn:alternateName xml:lang="mzn">دیترویت</gn:alternateName> <gn:alternateName xml:lang="nl">Detroit</gn:alternateName> <gn:alternateName xml:lang="nn">Detroit</gn:alternateName> <gn:alternateName xml:lang="no">Detroit</gn:alternateName> <gn:alternateName xml:lang="oc">Detroit</gn:alternateName> <gn:alternateName xml:lang="pl">Detroit</gn:alternateName> <gn:alternateName xml:lang="pt">Detroit</gn:alternateName> <gn:alternateName xml:lang="ro">Detroit</gn:alternateName> <gn:alternateName xml:lang="ru">Детройт</gn:alternateName> <gn:alternateName xml:lang="sah">Детройт</gn:alternateName> <gn:alternateName xml:lang="sk">Detroit</gn:alternateName> <gn:alternateName xml:lang="sr">Детроит</gn:alternateName> <gn:alternateName xml:lang="sv">Detroit</gn:alternateName> <gn:alternateName xml:lang="ta">டிட்ராயிட்</gn:alternateName> <gn:alternateName xml:lang="te">డెట్రాయిట్</gn:alternateName> <gn:alternateName xml:lang="tg">Детройт</gn:alternateName> <gn:alternateName xml:lang="th">ดีทรอยต์</gn:alternateName> <gn:alternateName xml:lang="tr">Detroit</gn:alternateName> <gn:alternateName xml:lang="ug">Détroyt</gn:alternateName> <gn:alternateName xml:lang="uk">Детройт</gn:alternateName> <gn:alternateName xml:lang="vi">Detroit</gn:alternateName> <gn:alternateName xml:lang="xmf">დეთროითი</gn:alternateName> <gn:alternateName xml:lang="yi">דעטרויט</gn:alternateName> <gn:alternateName xml:lang="zh">底特律</gn:alternateName> <gn:featureClass rdf:resource="http://www.geonames.org/ontology#P"/> <gn:featureCode rdf:resource="http://www.geonames.org/ontology#P.PPLA2"/> <gn:countryCode>US</gn:countryCode> <gn:population>713777</gn:population> <gn:postalCode>48258</gn:postalCode> <wgs84_pos:lat>42.33143</wgs84_pos:lat> <wgs84_pos:long>-83.04575</wgs84_pos:long> <wgs84_pos:alt>183</wgs84_pos:alt> <gn:parentCountry rdf:resource="http://sws.geonames.org/6252001/"/> <gn:nearbyFeatures rdf:resource="http://sws.geonames.org/4990729/nearby.rdf"/> <gn:locationMap rdf:resource="http://www.geonames.org/4990729/detroit.html"/> <gn:wikipediaArticle rdf:resource="http://en.wikipedia.org/wiki/Detroit"/> <rdfs:seeAlso rdf:resource="http://dbpedia.org/resource/Detroit"/> </gn:Feature> <gn:Feature rdf:about="http://sws.geonames.org/6955112/"> <rdfs:isDefinedBy rdf:resource="http://sws.geonames.org/6955112/about.rdf"/> <gn:name>Detroit-Warren-Livonia</gn:name> <gn:featureClass rdf:resource="http://www.geonames.org/ontology#L"/> <gn:featureCode rdf:resource="http://www.geonames.org/ontology#L.RGNE"/> <gn:countryCode>US</gn:countryCode> <gn:population>4425110</gn:population> <wgs84_pos:lat>42.34231</wgs84_pos:lat> <wgs84_pos:long>-83.07175</wgs84_pos:long> <gn:parentCountry rdf:resource="http://sws.geonames.org/6252001/"/> <gn:nearbyFeatures rdf:resource="http://sws.geonames.org/6955112/nearby.rdf"/> <gn:locationMap rdf:resource="http://www.geonames.org/6955112/detroit-warren-livonia.html"/> </gn:Feature> </rdf:RDF>
For more information, go to the project homepage.
Build Recursive Patches Against an Application Directory
PathManifest is a utility that deposits a manifest of an application directory into its root, and allows you to build differential patches over time. As the tools can also print JSON-encoded data on their completion, they can be readily integrated from other tools/applications.
Example usage:
$ pm_write_manifest /application/root $ pm_check_for_changes /application/root New --- new_directory/new_file2 new_file Updated ------- updated_file $ pm_make_differential_patch /application/root 201507282031 /tmp Created/Updated Files --------------------- new_directory/new_file2 updated_file new_file Patch file-path: /tmp/pm-patch-201507282031.tar.bz2
To apply the patch, simply expand. Note that this doesn’t support file removal but will in the future (still with a basic archive, but with the aid of an additional tool):
$ tar xjf /tmp/pm-patch-201507282031.tar.bz2
Display applied patches:
$ pm_show_applied_patches /application/root Applied Patches --------------- 201507282031 Affected Files -------------- new_directory/new_file2 updated_file new_file
Online Markdown Editor/Repository
Markable, a free, online service for editing and storage of Markdown documents with live preview.
Share Images with Temporary Short-Links (like with Pastebin)
A quick Google search rendered two respectable solutions:
- http://snag.gy: Allows you to paste an image and get a link. There are also some basic image-editing tools.
- http://pasteboard.co: It’s supposed to be the same as Snaggy, but also allows you to drag images. However, it failed (by way of stating that I was only allowed to drag-and-drop an image) with the four- or five-images that I attempted. I’m assuming that it detects whether you’re dragging an image based on the file-extension. If the file-extension isn’t present or there’s an inconvenient period in the filename, it will likely reject it.
Random Engineering 
You must be logged in to post a comment.