• build: use poetry for package management, asdf for python version
• build: cleanup poetry config, add keywords, ignore dist
• ci: migrate circleci to gh actions
• fix: dup method definition

1. The code uses two different ways of writing encoding formats, mac_roman and macroman. It is uncertain whether there are any typo errors.
2. When there are Chinese characters in the content, exporting it results in garbled code. Changing it to utf8 can fix the issue.

Closes #2095.

:books: Documentation preview :books:: https://datasette--2159.org.readthedocs.build/en/2159/

This adds a brand link to the top navbar if 'brand' key is populated in metadata.json. The link will be either '#' or use the contents of 'brand_url' in metadata.json for href.

I was able to get this done on my own site by replacing templates/_crumbs.html with a custom version, but I thought it would be nice to incorporate this in the tool directly.

:books: Documentation preview :books:: https://datasette--2158.org.readthedocs.build/en/2158/

:books: Documentation preview :books:: https://datasette--2100.org.readthedocs.build/en/2100/

Gotta tell it where to look

G'day! I got confused while debugging, earlier today. That's on me, but it does strike me a little repetition in the metadata documentation might help those flicking around it rather than reading it from top to bottom. No worries if you think otherwise.

:books: Documentation preview :books:: https://datasette--2096.org.readthedocs.build/en/2096/

Just like facet's suggest() is trapping QueryInterrupted for facet columns, we also need to trap get_row_count(), which can reach timeout if database tables are big enough.

I've included get_columns() inside the block as that's just another query, despite it's a really cheap one and might never raise the exception.

:books: Documentation preview :books:: https://datasette--2082.org.readthedocs.build/en/2082/

:books: Documentation preview :books:: https://datasette--2076.org.readthedocs.build/en/2076/

serving multiple database files and getting tired by the default sort, changes so the sort order puts the latest changed databases to be on top of the list so don't have to scroll down, lazy as i am ;)

:books: Documentation preview :books:: https://datasette--2074.org.readthedocs.build/en/2074/

This PR adds a new section about packaging a plugin using poetry within the "Writing plugins" page of the documentation.

:books: Documentation preview :books:: https://datasette--2061.org.readthedocs.build/en/2061/

This is a version of #2031 for the 0.64.x branch.

:books: Documentation preview :books:: https://datasette--2044.org.readthedocs.build/en/2044/

...while maintaining given order.

Fixes #1975 (if I'm understanding correctly).

:books: Documentation preview :books:: https://datasette--2026.org.readthedocs.build/en/2026/

Fixes #2016

:books: Documentation preview :books:: https://datasette--2025.org.readthedocs.build/en/2025/

Not sure when this changed (iOS 16 maybe?), but the WorkoutStatistics now has a whole bunch of information about workouts, e.g. for runs it contains the distance (as a <WorkoutStatistics type="HKQuantityTypeIdentifierDistanceWalkingRunning ...> element).

Adding it as another column at leat allows me to pull these out (using SQLite's JSON support). I'm running with this patch on my own data now.

Given the code here, it seems like in the past this file was named "tweet.js". In recent exports, it's named "tweets.js". The archive importer needs to be modified to take this into account. Existing logic is reused for importing this table. (However, the resulting table name will be different, matching the different file name -- archive_tweets, rather than archive_tweet).

Previously, twitter-to-sqlite only supported OAuth1 authentication, and the token must be on behalf of a user. However, Twitter also supports application-only bearer tokens, documented here: https://developer.twitter.com/en/docs/authentication/oauth-2-0/bearer-tokens This PR adds support to twitter-to-sqlite for using application-only bearer tokens. To use, the auth.json file just needs to contain a "bearer_token" key instead of "api_key", "api_secret_key", etc.

• add to checkin_details view new column for a calculated photo links
• supported multiple links split by newline
• create events table if there's no events in the history to avoid SQL errors

Fixes #9.

Tested with:

 hacker-news-to-sqlite user hacker-news.db fernand0

Result: Traceback (most recent call last): File "/home/ftricas/.pyenv/versions/3.10.6/bin/hacker-news-to-sqlite", line 8, in <module> sys.exit(cli()) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/home/ftricas/.pyenv/versions/3.10.6/lib/python3.10/site-packages/hacker_news_to_sqlite/cli.py", line 27, in user submitted = user.pop("submitted", None) or [] AttributeError: 'NoneType' object has no attribute 'pop'

There is a problem of style with the patch (but not sure what to do) because with the new inicialization ( submitted = []) the part

 or []

is not needed. Maybe there is a more adequate way of doing this.

Recent versions of setuptools can run most things from setup.cfg so one can have a simpler version that does not require executing code on install.

The bulk of the changes were automated by running https://pypi.org/project/setup-py-upgrade/ with a few minor edits for the bits that it can not auto convert (the initial get_long_description() and get_version() can not be automatically converted)

Looks like a previous commit missed a photo-to-sqlite→ dogsheep-photos replacement.

Fix typo

I'm not sure if this is due to a new format or just because my ageinfo file is blank, but trying to import an archive would crash when it got to that file. This PR adds a guard clause in the ageinfo transformer and sets a default value that doesn't throw an exception. Seems likely to be the same issue mentioned by danp in https://github.com/dogsheep/twitter-to-sqlite/issues/54, my ageinfo file looks the same. Added that same ageinfo file to the test archive as well to help confirm my workaround didn't break anything.

Let me know if you want any changes!

Twitter pushes you heavily towards v2 projects instead of v1 apps – I know the README mentions v1 API compatibility at the top, but I still nearly got turned around here.

Some repos organize network dependents by a Package toggle. This PR adds the ability to page through those options and scrape underlying dependents.

File name from google takeout has no space. The code only runs without error if filename is "MyActivity.json" and not "My Activity.json". Is it a new change by Google?

When parsing the Evernote <note> elements, the script will now also parse any nested <tag> elements, writing them out into a separate sqlite table.

Here is an example of how to query the data after the script has run: select notes.*, (select group_concat(tag) from notes_tags where notes_tags.note_id=notes.id) as tags from notes;

My .enex source file is 3+ years old so I am assuming the structure hasn't changed. Interestingly, my notebook names show up in the tags list where the tag name is prefixed with notebook_, so this could maybe help work around the first limitation mentioned in the evernote-to-sqlite blog post.

Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table team_members beteween teams and users.

Currently the _through clause can only work if the FK relationship is defined in a specific direction. I don't think there is any reason for this limitation, as an FK allows joining in both directions.

This is an admittedly hacky change to implement bidirectional joins using _through. It does work for our use-case, but I don't know if there are other implications that I haven't thought of. Also if this change is desirable we probably want to make the code a little nicer.

Fixes https://github.com/simonw/datasette/issues/1459

Adds support for --open --get /path to be used in combination.

If --open is provided alone, datasette will open a web page to a default URL. If --get <url> is provided alone, datasette will output the result of doing a GET to that URL and then exit. If --open --get <url> are provided together, datasette will open a web page to that URL.

TODO items: - [ ] update documentation - [ ] print out error message when --root --open --get <url> is used - [ ] adjust code to require that <url> start with a / when -o --get <url> is used - [ ] add test(s)

note, '@CTB' is used in this PR to flag code that needs revisiting.

This makes it so you can use Dogsheep if you're using Datasette with the --base-url /some-path/ setting.

Workflows that have an id in any step result in the following error when running workflows:

e.g.github-to-sqlite workflows github.db nixos/nixpkgs

Traceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invokeTraceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in call return self.main(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 763, in invoke return __callback(args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py", line 601, in workflows utils.save_workflow(db, repo_id, filename, content) File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py", line 865, in save_workflow db["steps"].insert_all( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2596, in insert_all self.insert_chunk( File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2378, in insert_chunk result = self.db.execute(query, params) File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 419, in execute return self.conn.execute(sql, parameters) sqlite3.IntegrityError: datatype mismatch ```

• Information about the ID key in a step for GHA
• An example workflow from a public repo

Changes

I'm proposing that the key for id in step is replaced with step_id so that it no longer interferes with the table id for tracking the record.

Special thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this 😄

The tool covers only chat imports at the time of opening this PR but I'm planning to import everything else that I feel inquisitive about

ref: https://github.com/gavindsouza/instagram-to-sqlite

Closes #197. I have added support for sorting by multiple parameters as mentioned in the issue above, and together with that, a suggestion on how to implement such sorting in the user interface.

Hey .. Awesome work developing this project, that I found very useful to me and saved me some work.. Thanks.. :)

Some background to this PR... I've been searching around for a tool allowing me to transforming my personal collection of Evernote notes to a format easier to search and potentially easier import to future services.

Now I discovered problem processing my large data ~5GB using the existing source using Pythons builtin xml-parser that unfortunately was unable to succeed without exception breaking the process.

My first attempt I tried to adapt to more robust lxml package allowing huge data and with "recover", but even if it worked better it also failed processing the whole data. Even using the memory efficient etree.iterparse() it also unfortunately got into trouble.

And with no luck finding any other libraries successfully parsing this enormous file I instead chose to build a "hugexmlparser" module that allows parsing this huge file using yield (on a byte-to-byte-level) and allows you to set a maximum size for <note> to cater for potential malformed or undesirable large attachments to export, should succeed covering potential exceptions. Some cases found where the parses discover malformed XML within <content> so also in those cases try to save as much as possible by escaping (to be dealt at a later stage, better than nothing), and if a missing end </note> before new (malformed?) it would add this after encounter a new start-tag.

The code for the recovery process is a bit rough and for certain room for refactoring, but at the moment is seem to achieve what I wanted.

Now with the above we pass this a minor changed version of save_note_recovery() assure the existing works. Also adding this as a new recover-enex command to click and kept the original options. A couple of new tests was added as well to check against using this command.

Now this currently works to me, but thought I might share a PR in such as you find use for this yourself or found useful to others finding this repository.

As a second step .. When the time allows it would have been nice to also be able to easily export from SQLite to formatted HTML/MD and attachments saved... but that might perhaps be better a separate project ... or if you or someone else have something that might shared to save some trouble, I would be interested ;-)

Parses the datetime columns so they are subsequently properly recognized as datetime.

Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/17

Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/14

Refs #239. Still needs documentation and CLI implementation.

to keep the data write from failing if it came across an error during processing. In particular when trying to convert my HealthKit zip file (and that of my wife's) it would consistently error out with the following:

``` db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: too many SQL variables

db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: too many SQL variables

db.py 1709 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: table rBodyMass has no column named metadata_HKWasUserEntered

healthkit-to-sqlite 8 <module> sys.exit(cli())

core.py 829 call return self.main(args, *kwargs)

core.py 782 main rv = self.invoke(ctx)

core.py 1066 invoke return ctx.invoke(self.callback, **ctx.params)

core.py 610 invoke return callback(args, *kwargs)

cli.py 57 cli convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)

utils.py 42 convert_xml_to_sqlite write_records(records, db)

utils.py 143 write_records db[table].insert_all(

db.py 1899 insert_all self.insert_chunk(

db.py 1720 insert_chunk self.insert_chunk(

db.py 1720 insert_chunk self.insert_chunk(

db.py 1714 insert_chunk result = self.db.execute(query, params)

db.py 226 execute return self.conn.execute(sql, parameters)

sqlite3.OperationalError: table rBodyMass has no column named metadata_HKWasUserEntered ```

Adding the try / except in the write_records seems to fix that issue.

Finally got around to trying dogsheep-photos but I want to use backblaze's b2 service instead of AWS S3. Had to add a way to optionally specify the endpoint_url to connect to. Then with the bucket being empty the initial key retrieval would fail. Probably a better way to see that the bucket is empty than doing a test inside the paginator loop.

Also probably a better way to specify the endpoint_url as we get and test for it twice using the same code in two different places but did not want to spend too much time worrying about it.

This adds a straightforward set of classes to aid in the construction of SQL queries.

My plan for this was to allow plugins to manipulate the Datasette-generated SQL in a more structured way. I'm not sure that's going to work, but I feel like this is still a step forward - it reduces the number of intermediate variables in TableView.data which aids readability, and also factors out a lot of the boring string concatenation.

There are a fair number of minor structure changes in here too as I've tried to make the ordering of TableView.data a bit more logical. As far as I can tell, I haven't broken anything...

This PR is related to #293, but now all commits have been merged.

The purpose is to support other file formats that aren't SQLite, like files with PyTables format. I've tried to accomplish that using external connectors published with entry points.

The modifications in the original datasette code are minimal and many are in a separated file.