Importing Kindle or iBooks into Bookpedia

[FineWine]

IMPORTANT - IF YOU ARE USING

Bookpedia Version: 5.9.3 or greater
and
Kindle Version: 1.26.1 or greater

PLEASE GOTO THIS TOPIC: Importing Kindle or iBooks into Bookpedia Part 2
-----------------------------------------------------------------------------------

As we know Bookpedia does not import Kindle books. After doing some Googling I came across a great blog article by Jan Drewniak - “From Kindle to Calibre to iBooks” which he published in January 2, 2014.

Following his set of instructions enabled me to import my Kindle library into my Bookpedia Library. I have refined his instructions and adapted them. Below are my adjusted instructions.

So the mission is to get your books from the Kindle for Mac app into Bookpedia and if you desire into iBooks as well. To do this we need a FREE third party app called Calibre. If you only have your books on your iPad or iPhone then you will need to download the Kindle for Mac app from the App Store. Then download all your books from your Kindle account cloud storage.

OK lets start!

Step 1: Locate the Kindle Data Folder
It's here:
~/Library/Containers/com.amazon.Kindle/Data/Library/Application Support/Kindle/My Kindle Content/

 You can easily go to this folder by pressing Command+Shift+G in the Finder, then copying and pasting the above path into the textfield and pressing 'Go'.

Within the “My Kindle Content” folder you will see Folders and Files. The files we are interested in are .azw and .apnx files. The folders will have the ASIN number of the book suffixed by _EBOK. These _EBOK folders are the new way Kindle is filing its books. Within each folder are the two files we are interested in .azw and .apnx These are your kindle books.

Step 2: Make a Temp folder and copy
We need to make a Temp folder, you choose your location, I made mine on the Desktop. Within the “My Kindle Content” folder copy across the .azw and .apnx files. Now open each ASIN number folder in turn and copy across their .azw and .apnx files. The reason we have to copy the .azw and .apnx files to a Temp folder is that on import Calibre deletes those files. A bit naughty IMHO.

Step 3: Download Calibre
Download Calibre from its parent site.

Step 4: DeDRMing Kindle books with Calibre
Download the DeDRM plugin for Calibre. The plugin will automatically strip DRM from Kindle books when they are added to Calibre.

Now to install the plugin into Calibre:
Open calibre's Preferences dialog. Click on the "Plugins" button on the “Advanced” tool line. Next, click on the button, "Load plugin from file". Navigate to the unzipped DeDRM_tools folder and, in the folder "DeDRM_calibre_plugin", find the file "DeDRM_plugin.zip". Click to select the file and select "Open". Click "Yes" in the "Are you sure?" dialog box. Click the "OK" button in the "Success" dialog box.

Now for some Calibre house keeping

Step 5: Automatically adding books into Calibre
First we need to set the preferred output format in Calibre, then we need to set up the automatic adding. These settings should already be set but no harm in checking.

To set the preferred output format. From the top menu, we go to: Calibre → Preferences → Behavior (button on the “Interface” tool line) And there we select EPUB as the preferred output format. Now for the automatic adding. From the top menu, we go to: Calibre → Preferences → Adding Books → Automatic Adding (button on the “Adding books” tool line) From here, we select the folder that we copied the kindle books into with Automator. My folder is named "Kindle 4 Mac Items". It's important that “automatically convert added files to the current output format” is checked. We click apply. Now after we hit apply, Calibre will automatically add the kindle books, strip the DRM off them, and convert them to ePubs. You can also Drag & Drop your .azw and .apnx files onto the central listing area.

Step 7: Adding Metadata and Cover Art
Unfortunately when Calibre brings in the Kindle books it lacks some metadata and sometimes the cover art. To refresh or add metadata right click and select Edit metadata > Download metadata and covers. This will take sometime depending on you library size. Once complete you now need to convert the your library into epub format again.

Step 8: Adding Metadata and Cover Art
Highlight all the books, right click and select Convert books > Create a catalog of the books in your calibre library. In the Catalog options > Catalog format select epub. Untick Add catalog to library if ticked. Click Apply then OK. Important final step is to restart the Calibre app course I found that the metadata did not update till I did.

NOTE: Steps 7 & 8 can take some time depending upon your Calibre library size. I personally believe that Bookpedia is quicker at adding/updating metadata. To Check on the import and conversion processes click on the word Jobs bottom right of the window.

Step 9: Getting from Calibre to Bookpedia
The Quick Way: I just highlight and drag two at a time onto the Bookpedia (Bookpedia prefers small quantities at a time) icon in the Dock from Calibre’s central listing area. Bookpedia will automatically create a collection called “Imported e-books”. Unfortunately I found using this method not all the metadata comes across. Of course you can fix this within Bookpedia.

The Best Way: Is to highlight all the books, right click and select Convert books > Create a catalog of the books in your calibre library. In the Catalog options > Catalog format select CSV. Choose a Catalog title. Untick Add catalog to library if ticked. Select CSV/XML options and and make sure every box is ticked. Click Apply then OK. I store the created file on my Desktop.Now drag this csv file on to the Bookpedia icon in the Dock. A list of all Bookpedia fields will open up. You can leave most of the settings as they are except for three: #mobi = ASIN, Cover = Image URL & Pubdate = Release

Step 10: Getting from Calibre to iBooks
Importing into iBooks is just the simple method of highlighting all your books in Calibre and drag and drop onto the iBooks icon in the dock.

And that is all folks.

[LibertyTrooper]

For those that, like I, only need a listing of their books from Kindle please follow these procedures:

Open Safari
Ensure "Show Develop menu in menu bar" is Preferences/Advanced tab is checked
{Develop Menu Item will show in menu bar}
Log into https://read.kindle.com
{Cover or list of your books will display}
Scroll to the end of your book list to ensure all titles are loaded.
Invoke Menu path Develop|Show Web Inspector
{The web inspector will display}
Select the "Storage" tab in the inspector
{A pane listing databases in a tree format will display}
In the navigator tree select the bookdata table in the K4W database
{The table pane will display a populated table with the following fields:}

asin,
contentType*,
title,
authors,
purchaseDate**,
maxPosition,
lastTimeRead,
lastPositionRead,
lastFPRSynctime***,
authorPronunciations,
titlePronunciation

Right click and select "Copy Table"
{The clipboard will contain your data}
Open a spreadsheet such as Numbers
Paste the data
{The spreadsheet should interpret your data as rows and columns matching the table you copied}

Most columns past purchaseDate appear to be empty, except for the last two which, in the case of some Japanese and Russian language books I have, contained data which seems to represent a phonetic transliteration of the author and title. Maybe for speech enabled books. Further notes on the columns are:

asin is Amazon's identifier for the product. You will find, however, that in some cases where the product is no longer available for sale, searching with that number will return a 404 result. Alternatively, you may enter the ASIN at http://www.goodreads.com and, in all cases I tried, the title will be found.

* The purchaseDate is in ISO8601 (unix time stamp) format and includes the Zulu time you actually purchased the book excluding microseconds. You'll have to massage it if you want to actually use the data since Apple uses their own time format. I used python.

**contentType has two apparent values: EBOK and EBSP. EBOK indicates a normal kindle title. EBSP indicates a Sample. Alas, there is no information regarding Prime titles you may have borrowed at one time or another. You must look at that through the normal "Manage your Kindle" site.

*** I believe this is supposed to be the "Furthest Page Read" (FPR) synctime

I will post a separate message about getting a list from iBooks which can be, in some cases, more of a PITA.

[FineWine]

All good info LibertyTrooper but might be a bit techie for those that are not like minded and just want to import Kindle into Bookpedia.

If you export from Numbers as a csv sheet then drag and drop onto the Bookpedia Dock icon Bookpedia does not recognise any of the data.

The only real benefit I see with your method is that it is a quick method to ascertain the ASIN number.
Your comment about Goodreads success with ASIN is quite correct. Conor has been trying to communicate with them so he can develop a search plug-in but they are not coming to the party.

EDIT: this link https://read.kindle.com is incorrect. It should be https://www.amazon.com/kindlecloudreader which takes you to your sign-in page.

[LibertyTrooper]

Good catch, I had meant to put read.amazon.com . I blame the dresser-diving, partially titanium, mid-night-mid-section impacting scared-of-thunder-cat.

I must confess I've never drag and dropped the CSV file. I always go the File->Import and set the fields accordingly. It imports just fine with asin, title, and author then. Anything else you have to massage.

I think both solutions are a bit 'techie'. Yours and mine. Just in different ways. Having both options allows choice until someone figures out how to automate it one or another.

I"m not going to hold my breath for Connor to support either given the potential troublesome questions they may raise.

On the Goodreads side, I applied and received my own key. It was far easier than Amazon and, given Amazon's current policies, much less murky from the afore-implied troublesome perspective. However, you can't cache some types of information which is probably what impedes Connor. I'll read the link you provided tomorrow probably. I'm rapidly, again due to the bionic cat, losing my steam for today.

[LibertyTrooper]

Oh my lord, its 18 hours later and I'm not even halfway through the download of all my Kindle books. Though, thank goodness for Keyboard Maestro, otherwise I'd still be clicking and selecting download 1 book at a time.

I wonder if I can sue Amazon for causing me carpal tunnel syndrome.

[LibertyTrooper]

If you are going to follow the instructions above with your books, not that I'm suggesting or condoning that you do, you should know that its best to use a version of the Kindle app between 1.14 and 1.19 . Otherwise, for some books you will run into a new format that is not able to be converted.