What is Scrapyard?

Scrapyard is a bookmarking extension where you can store and organize bookmarks, page fragments, complete HTML pages and PDF documents, or take notes. It is possible to export and import bookmarks in JSON or org-mode formats. Scrapyard also supports the import of RDF archives of the legacy ScrapBook addon.

Where to Find Scrapyard

Scrapyard is accessible through the dropdown menu of the Firefox sidebar as shown in the image below.

In Chrome, Scrapyard could be opened in the browser side panel through the button in the popup dialog. When there is no open side panel, Scrapyard may open itself in a separate window to display archived content or import RDF archives.

The Alt+Y keyboard shortcut allows to open Scrapyard directly. Keyboard shortcuts provided by any extension could be customized in the browser settings.

Getting Started

Scrapyard can store its content in one of three locations, chosen by the "Content location" option in the Settings tab:

Filesystem - content is stored in a directory on this computer by the backend application, which is started by the browser. When using this mode, the only way to share data across several computers is to use commercial or self-hosted cloud providers.
Server - content is stored by the backend application hosted on a server. In this case, everything could be immediately shared by several browsers and computers without any third-party solutions. Because of the limitations of plain-text storage, this mode is not designed for multi-user collaboration. Although Scrapyard tries its best to prevent data loss, it is strongly advised to set up automated backup of your data.
Browser internal storage - this option provides a limited set of features available without the backend application or server. Convenient when you only need to store bookmarks or notes and sometimes archive simple pages. If you switch to another storage type, content stored in the browser internal storage will be lost, although you may export and reimport it later. The cloud shelf is the only way to share bookmarks and archived pages in this mode.

Storing Content in the Filesystem

1. Installing the Backend Application

Scrapyard requires the installation of the backend application to access the filesystem. Please install the application using one of the download links provided in the Backend options tab.

2. Specifying the Content Directory Path

Specify the content directory path at the Settings tab.

Storing Content on a Server

1. Installing the Server

Download the CLI-based installer from the Backend options tab to the server (Linux or macOS), unpack it, and run ./install.sh --server. The installer creates a configuration file named .env with a randomly generated server key, and a service that automatically runs the server on startup. Alternatively, the server could be run in Docker. See the scrapyard_backend/server/DEPLOY.md file included in the installer for the details.

The most important configuration values in the .env file are:

DATA_PATH - the server directory where the content is stored.
AUTH_KEY - the server key that should be entered in the add-on settings.
SCHEME, HTTP_HOST, HTTP_PORT - the protocol (http or https), the network interface, and the port on which the server accepts connections.

It is strongly recommended to access the server only through a local network or Tailscale, which also provides valid HTTPS certificates. If the server uses a self-signed certificate, open the server URL in a browser tab once and accept the certificate before connecting.

2. Connecting to the Server

At the Settings tab, choose "Server" as the "Content location", enter the server URL (for example, https://my-server.my-tailnet.ts.net) and the server key, and click "Connect".

Scrapyard keeps a copy of the bookmark tree in the browser and synchronizes it with the server at startup, so the changes made in other browsers appear after the browser is restarted or the sidebar is reloaded. When the connection to the server is lost, a red server icon is displayed at the sidebar toolbar. Scrapyard reconnects automatically, the icon could also be clicked to reconnect immediately. If some changes could not be saved to the server, they are discarded, and the local data is updated from the server after the connection is restored.

Links to archives opened from the sidebar are valid for a limited time configurable at the server settings. If an old archive tab displays the "LINK EXPIRED" message, click the link on the page to open the archive again.

Main Features

Capturing Pages

To capture a web page or its fragment, select the part of the page you want to archive, open the Scrapyard popup dialog by clicking its icon on the browser toolbar, choose the destination shelf and folder, then press the "Bookmark" or "Archive" button. To capture multiple tabs, highlight them with Ctrl+click. The "Bookmark" button stores only web-page URLs as bookmarks. The "Archive" button captures the current selection, the whole page, if there is no active selection, or the whole document, if the opened link is not a web-page (for example, a PDF file).

In the bookmark tree, archived pages are marked with italic font, while ordinary bookmarks have the regular one. It is possible to additionally emphasize archives by enabling the corresponding add-on settings.

The following keyboard shortcuts allow to quickly capture the active tab into the default bookmark shelf:

Alt+Q - bookmark the current tab to the default shelf.
Alt+W - archive the current tab or selection to the default shelf.

If these shortcuts are already overridden by some other add-on, they could be customized in Firefox settings. There is an option to open the sidebar automatically when bookmarking from keyboard.

Capturing Sites

To create an offline copy of a website, enable the site capture mode by clicking on the white circle icon inside the "Archive" button.

A click on the "Archive Site" button will display the site capture options dialog:

Archive Toolbar

Every archived page contains a toolbar that offers several types of text markers to highlight sections of interest. The "Auto open" check makes the marker menu to open automatically when some text is selected. The "Edit document" button allows to clean the document of unnecessary elements by direct editing of its content. It is also possible to type in something, press F7 to display the caret. The DOM Eraser tool is able to remove any element on the page with a single click. The HTML code of a page may also be manually modified through the browser developer tools accessible through the F12 keyboard key. Press the "Save" button after you have finished document editing.

The size of a saved page in bytes and its creation date are available through the page info icon at the right-most toolbar side.

There is an option to not display the editing toolbar automatically. The toolbar always could be toggled by the Ctrl+Alt+T key combination.

Bookmark Shelves

It is possible to create an unlimited amount of bookmark shelves to structure your bookmarks. The shelf named "everything" allows to browse and search through all existing bookmarks. Scrapyard built-in shelves are listed in bold font and cannot be deleted or renamed.

If you mistakenly deleted a bookmark or even an entire shelf, it is possible to revert the deletion using the "Undo" item of the shelf menu . The undo operation is available until the browser is restarted or some other operation is performed. The "Sort by name" operation is also undoable.

TODO

It may be convenient to prioritize your bookmarks for processing with TODO states. Each bookmark may have one of the following TODO priorities: TODO, WAITING, or POSTPONED. You may find all your prioritized bookmarks at the built-in shelf named TODO. All bookmarks marked as DONE or CANCELLED are displayed on the DONE shelf.

If the path of the bookmark folder above its title provides not enough context, it is possible to fill the "Details" and "Date" fields at the bookmark property dialog. Only ISO (YYYY-MM-DD) date format is supported. Expired TODO items will be displayed first regardless of the assigned state.

Browser Bookmarks

Scrapyard seamlessly integrates with the browser bookmarks, so there is no need to switch to the built-in bookmark manager. Browser bookmarks are not included in the export or backups.

Cloud Bookmarking

Adding or copying/pasting bookmarks into the special shelf named Cloud makes them available across different browser instances that use a cloud provider with the same credentials. Cloud providers are configurable in the addon settings. It is possible to authorize in several cloud providers and quickly switch between them (currently only Dropbox and OneDrive are supported).

It is not necessary to install native cloud clients on any OS to use the cloud shelf. Just a Dropbox or a personal Microsoft account is required.

When using the server storage mode all your data is automatically available across all connected browsers, and the cloud fonder is usually unnecessary.

Browsing Local Files

The Files shelf allows browsing Org-mode, Markdown, and plain text files as notes in Scrapyard. Select "Add directory" in the context menu of the shelf to add a directory from the local filesystem. It is possible to open files from this shelf in an external text editor through the context menu of an item. Scrapyard will create a full-text search index for all added plain text files.

Add #+OPTIONS: toc:t or #+OPTIONS: toc:t num:t directive to your org-mode files to generate a table of contents.

Files other types could be included to the shelf by altering the file mask used during its creation. Non-textual files are opened with the associated application.

When the content is stored on a server (the Server content location in the settings), directory paths of the Files shelf refer to the filesystem of the server, and should be located inside its data directory (DATA_PATH in the server configuration):

An absolute path is used as is, for example, /srv/scrapyard/notes if DATA_PATH is /srv/scrapyard.
A relative path, for example, notes, is resolved against DATA_PATH.
Paths outside of DATA_PATH are not allowed. To add a directory located elsewhere, symlink or mount it into DATA_PATH on the server.
Opening files in an external editor or with the associated application is not available.

Sharing Bookmarks

A bookmark or archive could be shared to a cloud service through the "Share" context menu. Files that are shared to Dropbox or OneDrive appear at the "Apps/Scrapyard" folder. The "Cloud" menu item copies any selected items to the Scrapyard Cloud shelf.

Firefox Multi-Account Containers

Any Scrapyard bookmark could be opened in one of the Firefox Multi-Account containers, available through the "Open in Container" context menu of a bookmark, archive, or folder. Scrapyard will open the original URL of an archive with this menu, and all the items located inside a folder, if a folder is selected.

Additionally, it is possible to assign a container to a bookmark in its property dialog, as it is shown in the image below. Such bookmarks will automatically open in the corresponding container.

Bookmark Comments

While the "Details" field in the bookmark properties is intended for display at the TODO shelf, the bookmark property dialog also allows entering quick comments by clicking on the "callout" icon at its top-right corner.

Subsequent clicks on the icon alternate between the comment input field and bookmark properties.

The comment icon of bookmarks with filled-in comments takes the form of a "filled callout".

Notes

It is possible to attach more elaborate text notes with hyperlinks and images to every bookmark, or even create dedicated note-only bookmarks. Click "New" > "Notes" in the context menu of a folder, bookmark, or archive to create a note-only bookmark or attach notes. Items with non-empty notes are highlighted by the underlined text in the bookmark tree.

Notes could be entered in rich text format (using a visual editor), Markdown, Org-mode markup, or plain text. The format selector dropdown list is located in the bottom left corner of the "Edit" tab. Notes in Org format may have an automatically generated table of contents. Insert the example markup into the note editor to explore what options are available.

The text in the note editor is saved automatically, although you may save it at any time with the Ctrl+S keyboard shortcut.

Referencing to a bookmark or archive (Firefox only)

To make a reference to a Scrapyard bookmark or archive from the note markup, create a link with the following URL: ext+scrapyard://<BOOKMARK UUID>, for example:

ext+scrapyard://A4D409A0D1034D9BA0863E9DA8CE8FE7

When necessary, it is possible to add a hash with a link description:

ext+scrapyard://A4D409A0D1034D9BA0863E9DA8CE8FE7#short-bookmark-description

To refer the notes of a bookmark add notes: before the UUID:

ext+scrapyard://notes:A4D409A0D1034D9BA0863E9DA8CE8FE7

UUID of a bookmark is available from its property dialog:

Search

Sidebar Filtering

The input field at the Scrapyard sidebar allows filtering bookmarks of the current shelf by content and various other attributes:

Use the "Everything" search mode (default) to filter bookmarks by all available attributes. Use the "Everything" shelf to search through all bookmarks. Although the Sidebar filter is reasonably fast, it matches all items that contain every entered word longer than two characters in any order and distance. Use the full-text search if you need exact results.

To search a bookmark by its UUID, enter the UUID prefixed with uuid: into the input field.

Filtering by date supports simple queries:

2020-02-20 - the exact date.
before 2020-02-20 - before the specified date (the specified date is not included).
after 2020-02-20 - after the specified date (the specified date is not included).
between 2020-02-20 and 2020-03-20 - between the specified dates (the specified dates are included).

Search from URL-bar

It is possible to search through all bookmarks by title or URL from the browser URL-bar after entering the special "scr" keyword followed by a space:

If URL-bar search query begins with a plus sign (+), Scrapyard will interpret the rest of the text as a tag name and will search by tags instead. It is possible to activate browser URL-bar with the Ctrl+L keyboard shortcut.

Full-Text Search

Full-text search is accessible with the corresponding sidebar button: . It allows to find and highlight the exact occurrences of a phrase in the content of archived HTML pages.

Import/Export

Scrapyard allows importing bookmarks in Netscape HTML format which you may obtain from Firefox, Chrome, or other web-browsers. Its own collections of bookmarks and archived pages could also be exported or imported in JSON and ORG formats through the corresponding items of the shelf operations menu:

Importing Legacy ScrapBook RDF Archives

RDF archives could be imported from the corresponding options tab. There are two import modes:

Import RDF to Scrapyard. In this mode, the archived files are transferred into the scrapyard storage directory.
Open RDF. Import in this mode only recreates the RDF directory structure in the Scrapyard internal browser storage. The archived ScrapBook files are served from their original RDF folder. Shelves imported in the editing mode are marked with a tape reel icon. They vanish when the browser internal storage is reset, so they need to be reopened.

An opened RDF archive is writable: archived pages, folders and separators may be created, renamed, reordered, moved, copied and deleted in it, and the changes are stored in the original scrapbook.rdf file and its data directory, so the archive remains readable by the legacy ScrapBook add-on. Items may also be moved or copied between such a shelf and the other shelves. Notes, plain bookmarks and the TODO state are not a part of the ScrapBook format and are not stored in an RDF archive.

It is possible to edit the path of the imported RDF file through the "RDF Directory..." context menu item of a reel shelf.

Note: when Scrapyard is connected to a server, the RDF directory is read by the server backend, so a path on the server machine located inside the data directory (DATA_PATH) should be specified. A relative path is resolved against the data directory, and paths outside of it are rejected. Copy the RDF Scrapbook folder into the data directory on the server before importing it.

Link Checker

The automatic link checker allows finding broken links or URL duplicates. Its mode of operation may be selected through the "Check for" dropdown list. Since different archived page fragments may share the same original URL, link checker results in the "check for duplicates" mode may not always mean the duplication of content. Please check the duplicate bookmarks before deleting them.

Displaying Random Bookmarks

Random bookmarks from your collection could be displayed at the bottom of the Scrapyard sidebar. Check the corresponding option in the settings to enable this feature. The displayed bookmark is changed every 5 minutes, it also could be updated by a user request through the "Refresh" button. The "Find" button allows locating the bookmark at the Scrapyard sidebar by opening its shelf and folder.

Text Command Interface

It is possible to issue commands to Scrapyard through the text command interface offered by iShell Extension. To control Scrapyard with text commands just install iShell. In most cases, the command interface allows to significantly reduce the number of actions needed to create a bookmark or to archive a page.

The "shelf" Command

With the "shelf" command you can quickly switch to a shelf or folder, or even create a new one without using a mouse.

The "bookmark" and "archive" Commands

Use the "bookmark" or "archive" commands with the corresponding arguments to bookmark/archive a page into the specified destination. The arguments allow specifying bookmark title, path, details, tags, TODO state, and TODO deadline. Folders in the bookmark path will be autocompleted by iShell and created in Scrapyard if they do not exist. Note that due to the quirks of iShell parser spaces are not welcomed in folder names. It is possible to use dashes instead.

The first folder in the path is always interpreted as a shelf name. The tilde (~) character may be used in place of the "default" shelf. Contents of the Firefox "Bookmarks Menu" and "Other Bookmarks" bookmark folders may be accessed with the "@" and "@@" shortcuts respectively.

The "copy-to" and "move-to" Commands

Use these commands to quickly copy selected bookmarks to a desired shelf/folder without using the context menu and navigating through the shelf list and tree. It is possible to select bookmarks without opening them by holding Ctrl or Shift keys.

The commands accept an additional "by switching" argument that will open the destination folder after moving or copying.

The "scrapyard" Command

The "scrapyard" command allows browsing and search through the collection of bookmarks at the specified destination. See iShell command help for more details.

Creating Custom Capture Commands

If you often save bookmarks with similar properties to the same destination, it is possible to create custom capture commands with predefined parameters. The "CAPTURE" link at the bottom-right corner of iShell command editor allows to insert a template of such command. In addition to the predefined parameters, custom capture commands allow specifying comma-separated CSS selectors for elements that will automatically be retained or deleted. A custom CSS style could be added to the captured document through the "style" option.