Various unrelated questions regarding Pedia products

[LibertyTrooper]

Off and on I've tried to catalog my library of books, audiobooks, and e-books (of various formats). What I've run into mostly has to do with getting ahold of a complete dataset in the first place. Luckily, I figured out how to do that with Amazon finally and have, with varying degrees of success, imported that list into several programs for suitability analysis.

Of all ( Book Collectorz, BookCrawler, BookPedia, Calibre, Data Crow, Delicious Library, Home Inventory, Numento, Readerware 3 Books, and TapForms), only BookPedia seems to be meeting some of my desired features. Which has, in turn, led me to further questions about the *Pedias.

The most glaring to me is why does not a single application exist to manage all of these assets? Its strange for me to have separate apps, and I assume separate data-stores, for what is essentially the same problem?

I'm also wondering about the flexibility of the plugin system. I have a bunch of code written in Python that I'd like to integrate directly into Bookpedia via pyobjc. Is such a thing even possible? If not, I'd welcome any options other than Applescript and Objective-C. I rather like the latter yet I detest Xcode. Javascript for Automation is something I'm not against either I suppose. Not much documentation or example exists it seems for the plugin points.

I'd like to use the plugin architecture to reformat data, exchange data between fields, retrieve information not otherwise supplied by existing update data sources such as Amazon, Google, etc. Examples would be integrating Goodreads and LibraryThing data.

I would like to use BookPedia to manage my reading, not simply collecting a list of books or managing loans to others. For example, if I have a Kindle book entry captured in the database, or an audible audio file, I'd like to launch the associated application and go to the content that I have selected. For Kindle books I suppose this may be as simple as figuring out the kindle:// protocol handler parameters. I just don't know how to go about doing this. For audible, I probably would have to point the app to a folder and launch from the folder (This actually sounds like a really good opportunity for a first plugin) For PDF's I might want to launch Margin Note Pro X or Liquid Text (on my iPad).

Is it possible for BookPedia to automatically import purchases of digital media such as e-books. Especially from iBooks and Amazon. I have tons of duplicated purchases which really annoys the stuffing out of me. Having BookPedia as the place to start when I want to buy a book would be very helpful. Especially if it can handle new editions and the like.

Can we customize the statistics page? It looks rather boring right now. If so, what are the methods?

Finally, why does Apple have such a ****on for Delicious Library? Its basically nothing but a Storefront for Delicious Monster to make associate sales at various book vendors (Amazon, Google, etc). Its a poorly maintained, hang and crash prone, unmaintained POS. But it looks kinda pretty to some. Too bad that 'pretty' intefere's with usability. So why doesn't the App Store feature BookPedia as prominently?

In summary and on I've tried to catalog my books in their various formats and existences. I've come to the conclusion that only BookPedia seems to work somewhat as I expect. Unfortunately, its not pushing the boundaries of my dreams for such a system. As I work through this, I'm sure to come up with more questions and, perhaps, even changed opinions. I thank you all in advance for bearing with me.

[Conor]

Thanks for considering Bookpedia on your extensive examination of a solution to your cataloging needs.

Bookpedia started out as just DVDpedia back in 2003, and almost 15 years later not only does it need a name change to Moviepedia but has developed Pedia brothers along the way. The short answer to the separate programs, is that it has always been technically easier to keep them distinct. They all do a few things different and would be hard to integrate. Also has their advantages for the user, specially in the complexity of use. Just keep items from mixing in a same collection would be complicated and a support pressure and if you allow mixing how do you handle the interface. Bookpedia alone is getting unwieldy with the columns menu, add a few tens of columns from the other programs and it becomes a complicated menu to navigate. Would need to be a re-write, a la Stuffpedia that you found on the other thread, so that items could share all attributes and a single database, with specifics added above for each item type.

I am in the middle of an update of the internals of the program to modernize it and stop supporting some iffy code for 10.5. We will drop support for anything prior to 10.12. As there is also all kinds of regressions with 10.13 that we need to address via this 10.12+ update. It's on my todo list to then update the plugin architecture as part of that, although it's worked untouched quite well for the last few years. It's quite flexible and would allow all the operations you describe. You could tie in the Python script or any other external code by simply making the plugin a wrapper around the Objective-C for data coming from the Python script. They are limited to a search plugin that gets a search term or an item passed to it and returns a dictionary (or hash depending on who you ask). There is also a menu command extension, that would let you do pretty much anything to the database when invoked, either on the selection or all the data. The Objective-C part can be simple as adding values into a dictionary as the database objects work the same way. The plugin page is from it's hay day, so it's actually easier to look at the sample code that has the most basic things you would do to an entry (or Title Case – source – that I made for my own needs that has more commands.)

The links section is quite flexible so that you can run any protocol and it will get passed on to the Finder to handle. But for more advance behavior you will have to turn to the plugin again, for example to leverage Handoff on iOS and send data over to the iPad assuming Liquid Text supports it.

You can import all new data into Bookpedia by dropping a folder on it and let it sift through all the subfolder looking for files it recognizes as digital books. The metadata on these files is always flimsy, but you can then add more data using the "Book -> Update from" command. Bookpedia will then skip field it already knows it's tracking on further imports. So you could consolidate info this way. There is no folder watching and automatic import triggering, but it's a feature I do want to add.

If you do "Show Package Contents" on Bookpedia, you will find a statistics.html file in the resource folder that you can copy into your data folder ~/Library/Application Support/Bookpedia (to avoid over writing with updates) and you can then customize the CSS and look. The individual pieces of bar graph and body are hard coded and get inserted in and can't be changed. You will also find statistics.xml that you can also copy, and will let you change what gets displayed – info on both here and here.

I couldn't agree more on Delicious Library, we should have been more insistent and aggressive in marketing ourselves. We were in the close running for the award the year before, but were a young company and developer learning the ways of the world. They also has Mike Matas, that is a super talented designer, that gave them an amazing edge when it came to flash and look; one that they been riding ever sense and the only thing worth admiring in that program.

I do hope Bookpedia serves you in scratching that cataloging itch, but when it dos fall short do let us know. As it's that feedback that keeps up improving the program and refining the edges.

[FineWine]

Finally, why does Apple have such a ****on for Delicious Library? Its basically nothing but a Storefront for Delicious Monster to make associate sales at various book vendors (Amazon, Google, etc). Its a poorly maintained, hang and crash prone, unmaintained POS. But it looks kinda pretty to some. Too bad that 'pretty' intefere's with usability. So why doesn't the App Store feature BookPedia as prominently?

I couldn't agree more on Delicious Library, we should have been more insistent and aggressive in marketing ourselves. We were in the close running for the award the year before, but were a young company and developer learning the ways of the world. They also has Mike Matas, that is a super talented designer, that gave them an amazing edge when it came to flash and look; one that they been riding ever sense and the only thing worth admiring in that program.

This is why I love the Bruji pedia's. When I moved from M$ Windoze in late 2007 I looked at Delicious Library and tried their trail version for a few weeks, cann't remember exactly why I ditched it per se but I found Bruji pedia's and as I love doing a little amateur coding I really appreciated the personal customability of the pedia's. See my Jungle Green & Moderator Detail View Templates and others here.

I do hope Bookpedia serves you in scratching that cataloging itch, but when it does fall short do let us know. As it's that feedback that keeps up improving the program and refining the edges.

It certainly does satisfy my cataloguing itch. Conor, Nora & team have always been very responsive to bugs & suggestions usually within 24 hours. Do join their beta downloads These are 99% stable.
Also you sound like the very person we (forum members) need for the Doghouse Moderator team.

Anyway welcome to Bruji and the fabulous pedia's and these forums. Keep up with the suggestions.

[LibertyTrooper]

I had wondered if the initial post was too long or too complicated. Thanks very much for the well articulated reply and the references to examine in greater depth.

One thing I wanted to mention is that I do have a problem with Bookpedia hanging quite often. Not sure if there is something I can provide to you to help track that down. I'm also wondering if its not a problem with High Sierra itself. I have a number of apps, ranging from Xcode to iBooks, that just hang for no apparent reason. Its easily overcome by force quitting when the loathsome spinning beach ball/pinwheel of death appears.

Now I'm off to look at the plugins so that I may figure out a way to correct the lastname ', ' firstname problem of my author field.

[FineWine]

One thing I wanted to mention is that I do have a problem with Bookpedia hanging quite often. Not sure if there is something I can provide to you to help track that down. I'm also wondering if its not a problem with High Sierra itself. I have a number of apps, ranging from Xcode to iBooks, that just hang for no apparent reason. Its easily overcome by force quitting when the loathsome spinning beach ball/pinwheel of death appears.

I have been sending Conor crash reports since High Sierra and this is his reply: (The latest being 7th November)

There are bunch of regressions in High Sierra that I am trying to get fixed, should have the beta ready in a week or two for testing. Although edge cases, they are frequent enough for High Sierra users.

These crash logs can be found here:~/Library/Logs/DiagnosticReports just attached them to an email and send them to Bruji at support@bruji.com with an explanation of what you were doing at the time of the crash or freeze.

For overall grease & oil change for smooth running of my macOS pre & post updates I use a FREE maintenance program called Onyx by Titanium

As suggested earlier please join their beta downloads

Now I'm off to look at the plugins so that I may figure out a way to correct the lastname ', ' firstname problem of my author field.

If you mean you want to swap lastname & firstname around have a read of this Help information on Sorting a Collection You can also sort 6 columns in any order you want.

[Conor]

I can indeed confirm it's a High Sierra issue, so far mainly tied to the details view. That is a HTML view and in High Sierra is all kinds of crazy. As you can imagine it can affect several apps, as it's common to use an HTML view for a lot of custom display. Working on working around then and should have a beta soon. But please do send me the crash reports, as it helps to confirm it's the same bug and not another High Sierra issue. Apple managed to even sneak them in between the beta and the actual release of the gold master, as normally I would have addressed them earlier prior to release.

[LibertyTrooper]

Apologies for the long post. It helps me organize my thoughts and may be a factor on deciding whether or not to write an article about this subject.

If you mean you want to swap lastname & firstname around have a read of this Help information on Sorting a Collection You can also sort 6 columns in any order you want.

Thanks for that information. The fix that I did was because BookPedia was interpreting the input as two distinct authors. I.e. for Heinlein, Robert I would end up with two authors, one Heinlein and the other Robert. I've updated my python script (really, the HumanNames lib is wonderful) to run on all author input, not just if I flag it.

I don't recall if my input single quoted the names as part of the field.

The two things I'm focusing on right now are to bring all my various lists together into a cohesive whole, and to massage data to expectations of each of the apps at which I"m looking. I have to integrate files from Pragmatic Bookshelf, listings from Amazon, lists of physical books barcode scanned years ago, listing from Audible (which I'm not sure if I have to use another Pedia to accurately capture), and, of course, iBooks and Kindle. (e-books are wonderful... no more dragging 50lbs of books with me when I'm on extended travel).

All that said, I'm down to BookPedia and Collectorz . Bookpedia works as its intended without jumping through too many hoops. Collectorz is still under consideration because I really like is the richness of the data that it could capture. That is, if its import wasn't so limited and it didn't look like a Filemaker app (oh I detest those).

The two main areas of functionality that I see needing improvements for all these apps is in import and integration of external data. Some just cannot be helpd at all and really shouldn't, if the authors had a conscience, continue being sold. Of those that are remaining, Bookpedia has the best and most intuitive import.

The remaining challenge is cleaning the data after import.

I truly am toying with the idea of writing up the results of, sometimes frustrating, odessey. However, to do so I would likely have to formalize the approach. Right now, I want to find the best that I may use so I'm a bit inattentive to some details.

One of the biggest positive influences on my decision is the interaction with product author. I've made posts on other forums (DL) that have gone unremarked. And, the posts were more and more provocative as the lack of feedback accumulated. I think DL is pretty much going to be a dead product in the near future. For what reasons, I cannot imagine. Perhaps there just isn't enough money in this market. I surmise that, for the folks here at Bruji, that perhaps this software was, and is, written because they love books. For me, its a crime to throw away a book. I'm interested in using software written by persons of like mind."

[Conor]

We do write it cause we enjoy it, luckily it puts food on the table and we can dedicate time developing the Pedias. Apple market has also exploded, so it's been boom, but there is also so much to do. Now we cover, Mac, iOS, TVOS and server side Doghouse. It's like doing ten jobs at once.

its a crime to throw away a book

I can't even watch someone dog ear a book.

The tag field use of the author field did have it's adjustment period, but we had to standardize to share data via Doghouse. Also most data sources use the first name last name approach. You can still sue the last name, first name approach if you desire. The tag field will break it up in the edit window, but since the separator is the comma, it will still work find in the list view as well for most features and export. But if you don't mind going first name first, we do strongly encourage it, as Bookpedia will take care of your needs better and will sort by last name correctly.

If you want to massage data and want to automate a lot of it, I can recommend the SQL backend. We want the data to be as open as possible and the database is an open SQLite file. It's not pretty, but if you know a little SQL it can be very powerful. There a bunch of examples here in the forum if you look around searching for SQLite. The only tricky ones are dates, that are stored as seconds since 1.1.2001.

Worth investigating as you can cut out Xcode (worse IDE ever) and not worry about building a plugin for Bookpedia. SQLite libraries are built into python directly, so you can run SQL statements from inside Python (possible random example taken from online).

Code: Select all

import sqlite3
conn = sqlite3.connect('/Users/me/Library/Application Support/Bookpedia/Database.bookpd')
c = conn.cursor()
crime = ('%crime%',)
c.execute('SELECT * FROM zentry WHERE zgenre like ?', crime)
print c.fetchone()

c.execute('UPDATE zentry SET zcutom1 = "Favorite Mystery" WHERE zgenre like ?', crime)
db.commit()
conn.close()

[LibertyTrooper]

I can't even watch someone dog ear a book.

This makes me feel really guilty.

. The tag field will break it up in the edit window, but since the separator is the comma, it will still work find in the list view as well for most features and export. But if you don't mind going first name first, we do strongly encourage it, as Bookpedia will take care of your needs better and will sort by last name correctly.

In most of the systems I've worked on, I've used semicolons as separators between 'array as string' elements. To me, from the perspective of grammar, that is exactly their purpose. Plus, its an edge case even more rare than your "Crosby Stills & Nash" example. To which a solution is simply to, by default, expand & to 'and'.

Worth investigating as you can cut out Xcode (worse IDE ever) and not worry about building a plugin for Bookpedia. SQLite libraries are built into python directly, so you can run SQL statements from inside Python (possible random example taken from online).

The Xcode development team seems to subscribe to the old Mac development idea that to make development difficult results in higher quality applications. Can't have all those easy to use IDE's like Windows does. Hell, Xcode isn't even up to old style Visual Basic much less others like Visual Studio and, dare I say, Delphi (a pascal language which originated on MacOS!) . Cannot even begin to think cross platform like old IB and Webobjects. (I wonder if Win32 WO projects still might compile under clang)

Currently, I'm using the IDE's from IntelliJ. I'm toying with buying Delphi since its now cross platform. Pascal isn't so bad (I'm dating myself aren't I; here's another clue: Dec 350 Pro running Venix).

I just assumed you were using CoreData which is why I didn't poke around the database itself. Indeed, with HumanNames and isbntools there is a whole lot that can be done by directly accessing the database. Stupid me: Assumptions. I made this assumption probably because I designed, a long time ago, an object oriented data model for inventory. Plus, I've been reading what LibraryThing is doing to their model currently.

The only tricky ones are dates, that are stored as seconds since 1.1.2001.

You have to wonder how much how much effort Apple and its ecosystem has had to spend on conversions to and from ISO8601 dates. And why would they want to do so? Sure their approach can support 10K year interval dates yet thats not the point. How much does 31 years difference actually make to that? When converting the listing from the Amazon kindle database, this was a bit of an annoyance because I just didn't consider that a BSD based OS would do it differently. Somewhere up in the guts of Foundation is a time-traveling translation gerbil driven mad by the inability gain the evidence necessary to prove that Richard Gere engaged in despicable crime against against nature, and a certain species in particular, during the drug induced haze of the late 80's.

Looking back over this post, its awfully ranty/whiney/negative. I'll try to do better in the future.

[Conor]

It made me laugh. Richard Gere's legacy will be marred by unsubstantiated and unverified claims, 2000 years from now it's the only thing that will remain on record. Like staring at hieroglyph of Richard Gere's posterior and a gerbil.

I wasn't even digging deep into the hell that is Xcode. Just on the surface I can't get it to auto complete my own stored snippets in a timely manner, let alone his intelligent auto complete. I have basically bolted Text Expander over it, in order to get some someplace of typing in a proper IDE. From there on out is the beginning of a slow spiral staircase into hell. But clang I do not fault, it was a fresh air at the time, it's now just lagged since the actual Xcode team took it over and no longer developed outside of the Apple walls.

[FineWine]

Oh dear you two need to get a motel room and here I thought we were going to have a "ménage à trois"

Now I used to have fun with visual basic (80's) mainly writting for my 10 pin bowling league stat's, scoring & positionings and then later HTML & CSS (2000's) but you two have left me for dead.

Though I have been thinking about learning Apples new Swift language for something to do in my soon to be retirement age

Please carry on the discussion and work around the topic of importing iBooks and especially Kindle Though I do have a large collection of iBooks, my Kindle collection is growing as well and anything to make cataloguing them efficiently into Bookpedia would be a blessing.

[LibertyTrooper]

It made me laugh. Richard Gere's legacy will be marred by unsubstantiated and unverified claims, 2000 years from now it's the only thing that will remain on record. Like staring at hieroglyph of Richard Gere's posterior and a gerbil.

Thus it will be then be known as the Rosetta stone that allowed them to decipher the dreaded Unicode Emoji.

I can't get it to auto complete my own stored snippets in a timely manner, let alone his intelligent auto complete. I have basically bolted Text Expander over it, in order to get some someplace of typing in a proper IDE.

Try AppCode. It might help. Plus, it has VIM emulation. If you are an Emacs guy... well, I'm sorry. Just can't take my hands from the home keys ESC ESC shift : wq is how I tap my fingers.

Please carry on the discussion and work around the topic of importing iBooks and especially Kindle Though I do have a large collection of iBooks, my Kindle collection is growing as well and anything to make cataloguing them efficiently into Bookpedia would be a blessing.

For iBooks, I recently found iBooks Export on github. It appears to be able to get at least a list of titles for downloaded books. I only buy the enhanced iBooks from iBooks. Everything else I get from Kindle. Open a terminal, change to a folder of your choice, type: "git clone --recursive https://github.com/yamadapc/iBooks-Export.git", change into the resulting directory, type: "xcodebuild" (you must have xcode), copy the app from the build directory, Launch. etc. etc. Its written in Swift. Have fun.

There are a bunch of other projects that extract notes, highlights, etc. I'll worry about that stuff later.

For Kindle, the process is more lengthy. You have to go into the Kindle Cloud Reader, scroll to the bottom of the list to make sure all titles are loaded, open the web inspector, select databases, find the database with the titles in the list, select all the rows in the table, right click, select copy, then paste into a spreadsheet, export, import, rinse and repeat. The result will be a bunch of self-explanatory information of which the most important are the ASIN's, Title, and Authors. The rest, thought I do not know why, were empty in my case. I know there has to be some way to get ahold of that database since its an SQLite database. I just haven't taken the time to figure it out yet. You can get your notes and highlights from read.amazon.com Hmm... that makes me wonder if one exists for normal amazon site or is it just the html5 kindle reading app. Eh, on the list of things I'll probably never get to.

I've been looking at Swift myself and, well, it does nothing for my sense of elegance. For most other languages I can read them without reading
the language spec. Not so with Swift. In the defense of Swift, though, I'm someone who still prefers K&R over ANSI.

Hopefully the stuff about iBooks and Kindle is useful.

[FineWine]

For Kindle, the process is more lengthy.

Did some research today and discovered the following treasure trove:

Read this wonderful set of instructions on Jan Drewniak's Blog "From Kindle to Calibre to iBooks" and all yours/mine/ours Kindle importation worries are over.

I did not do the Automator Script as I just copied all the .azw and .apnx files into a temp folder, then dragged the .azw file into the main work area of Calibre. When the instructions got to "Step 6: Getting from Calibre to iBooks" I just high lighted 3 books at a time and drag and dropped them onto the Bookpedia Dock icon and all went sweet. They were imported into the automatically created "Imported e-books" collection. (and yes the cover art came across too) I then did the usual "Update from". Then I high lighted all and drag & drop into my E-books collection which I have set up to be included in my main Library.

Conor as Calibre is open source would you be able to borrow, with permissions, the necessary azw & apnx to epub conversion code from Calibre and insert it into Bookpedia?

I will do a separate Topic in Pedia Talk with an obvious title for everyone to read.

Hope this helps people

[LibertyTrooper]

LOL. Sorry, I've been focused on list data, not the actual files. Plus, I didn't want to download several thousand Kindle and around a thousand iBooks. The iBooks ones are ridiculously huge.

Funny enough, much of Calibre's functionality is available in two python libraries. Its one of the reasons I was asking about embedding Python.

Almost a record for shortest post by me. Kurt 7.2K, the bionic cat afraid of thunder, is to thank..

[FineWine]

I didn't want to download several thousand Kindle and around a thousand iBooks. The iBooks ones are ridiculously huge.

WOW That is some serious library there. No matter which way you go it is going to be a long process of importation and/or conversion.
Good luck