issues

In: - https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347

This turned out to be the fix:

bash dogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml sqlite-utils rebuild-fts dogsheep-index.db

Hi, I'm trying to have a display function in Dogsheep's config.yml that includes something like this:

```

{{ display.title }} (source)

{{ display.snippet|safe }}

```

Unfortunately, rendering fails with a message 'urls is undefined'.

The same happens if I'm trying to build a row URL manually, using filters like quote_plus (as my keys are URLs).

Any hints?

Thanks!

When creating a dogsheep index from config.yml file on pocket.db (created using pocket-to-sqlite), I am getting this error

Traceback (most recent call last): File "/Users/khushmeeet/.pyenv/versions/3.11.2/bin/dogsheep-beta", line 8, in <module> sys.exit(cli()) ^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/cli.py", line 36, in index run_indexer( File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py", line 32, in run_indexer ensure_table_and_indexes(db, tokenize) File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py", line 91, in ensure_table_and_indexes table.add_foreign_key(*fk) File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py", line 2155, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File "/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py", line 1116, in add_foreign_keys cursor.execute( sqlite3.OperationalError: table sqlite_master may not be modified

Command I ran to get this error dogsheep-beta index pocket.db config.yml

Dogsheep version dogsheep-beta, version 0.10.2

Python version Python 3.11.2

This makes it so you can use Dogsheep if you're using Datasette with the --base-url /some-path/ setting.

Hi,

First of all, thank you for this wonderful project :smile:

I started to use dogsheep to make my personal data searchable, and by using the project I noticed an issue with the index command.

It always expects you are running it from the root folder from where the data is located, so I got some errors while trying to make it work on my setup.

I separate all databases inside a data folder (I published my setup to be easier to follow: https://github.com/humrochagf/my-dogsheep)

Before, I configured dogsheep.yml to add the data folder to its path like this:

yml data/twitter.db: tweets: sql: |- ...

And running the index command like this:

dogsheep-beta index data/dogsheep.db dogsheep.yml

It worked to the normal search feature with no problem this way, but when I started adding display_sql rules the app started to crash, because at datasette get_database it was looking for data/twitter and it only had a db called twitter there.

So my workaround to that was to cd into the data folder and run the indexer. You can check the way I'm doing it at this line of the makefile: https://github.com/humrochagf/my-dogsheep/blob/main/makefile#L3

It works but it would be nice to have an option to pass the path where the data is located to the index function.

Like on https://til.simonwillison.net/til/search?q=Snippet

https://datasette.io/-/beta?q=+ returns a 500

fts5: syntax error near ""

Refs https://github.com/simonw/datasette/issues/878

https://datasette.io/-/beta?q=github-to-sqlite&sort=relevance&type=blog.db%2Fentries - "no such column: to"

``` results = beta_db["search_index"].search("run") if use_porter: assert results == [ ( "dogs.db/dogs", "1", "Cleo", "2020-08-22 04:41:33", 1, 0, "running", None, None, ) ] else:

      assert results == []

E assert <generator ob...x7f3c080a3e08> == [] E +<generator object Table.search at 0x7f3c080a3e08> E -[] E Full diff: E - [] E + <generator object Table.search at 0x7f3c080a3e08> ``` This was caused by a backwards incompatible change in sqlite-utils 3.0: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v3-0

datasette.client is designed for this kind of thing, to replace this code: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/init.py#L223-L232

A link to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish should be to /-/beta?category=1&timestamp__date=2018-08-01&q=swedish&sort=newest

Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all.

Ability to browse (and facet) by date.

I can bring these over from my personal Dogsheep.

Following #21

Got this on a search results page.

I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible. Originally posted by @simonw in https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584

In regular timeline view sort defaults to newest, not relevance - so this UI is incorrect:

Sometimes I want to sort by date, not by relevance.

quantified-self habit-tracking google-fit time-tracking wearables quantifiedself for example

I think "table" is the wrong name for the concept I'm using it for here.

Two reasons: firstly, table is a reserved word in SQLite. More importantly, it turns out there's not a direct mapping from tables to types of search result. In particular, for GitHub I ended up having two different "tables" of repositories - one for repos created by me, another for repos that I have starred.

As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck.

Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53

That should probably do DELETE FROM index1.search_index WHERE [table] = ? first.

Searching for eg #dogfest currently throws an FTS exception - but I want to support advanced FTS query tricks as seen in #13.

https://dogsheep.simonwillison.net/-/beta?q=%23dogfest

fts5: syntax error near "#"

Idea: catch that error and re-run the query with FTS escaping applied!

It would be cool if the display_sql could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a class="geojson" element output by the display template) - ala https://github.com/simonw/datasette-leaflet-geojson

Then I could render workout routes on a map, or Swarm checkin points.

simon willison NOT screenshot for example.

Part of #3 - in particular I want to make sure my photos are displayed with a thumbnail.

This will be a page at /-/beta which renders using a custom template.

It will offer a default timeline view plus search and facet by type/date.

I sometimes like to configure titles that are things like "Comment on issue X" or "Photo in Golden Gate Park" - these shouldn't be included in the search index but should be stored so they can be displayed to provide context.

Add a column for this - probably called context - and make it so it can be populated.

A category for things that were sent to me: DMs, emails etc. Follows #7.

Some of the data in Dogsheep is stuff that was written publicly - tweets, blog posts, GitHub commits to public repos.

Some of it is private data - emails, photos, direct messages, Swarm checkins, commits to private repos.

Being able to filter for just one or the other (or both) would be useful. Especially when giving demos!

Some of the content I'm indexing is by me - photos I've taken, tweets I wrote, commits, comments I posted.

Some of it is stuff that I've "liked" or "bookmarked" in some way - favourited tweets, Pocket articles, starred GitHub repos.

It woud be useful to be able to differentiate between the two.

sql select search_index_fts.rank, search_index.rowid, search_index.[table], search_index.key, search_index.title, search_index.timestamp, search_index.search_1 from search_index join search_index_fts on search_index.rowid = search_index_fts.rowid order by search_index_fts.rank, search_index.timestamp desc

This can be on by default. You can turn it off for a table in the config file using stemming: none - or maybe tokenize: none to match the terminology used by SQLite and sqlite-utils: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#enabling-full-text-search

Once I've got the Datasette plugin to a state where it's worth building a demo: #3

I can use data from my public https://github-to-sqlite.dogsheep.net/ demo plus the Pocket data subset I use for the demo in https://github.com/dogsheep/pocket-to-sqlite/issues/5 - I could pull in the https://dogsheep-photos.dogsheep.net/ photos data too.

Since default view will likely be ordered by timestamp descending.