What are you doing this week?

16 points by caius a day ago on lobsters | 19 comments

Enjoying Scotland. Quite a large amount of it over a few days.

: munificent | 4 hours ago
I hope you have a great time. I took a trip to the UK with my family this summer and Scotland was lovely. We didn't manage to make it out to the Isle of Skye, but we did do a day trip out to the highlands and it was an experience I'll never forget. Simultaneously desolate and beautiful.

Working on Janet infrastructure.

Janet's original package manager JPM is being depricated, so we need to better document the replacements' alternative workflows. I'd previously written this. Janet now does the hard work internally, allowing people to easily build package managers with different workflows on top, working with the same data.

I'm expanding the our support for sqlite3's extended API. First I did some refactoring and now I'm building a bunch of random projects to highlight/explore those extended features, with demos/experiments of:

code in the DB (or compiled functions)
shadow REGEXP with janet PEGs using sqlite3_create_function_v2
letting you use and move sqlite3 DBs across threads and images (via serialize)
integrating statements and such as Janet "abstract types" allowing you to stream rows or do like (each row sttmt ...)

Related to the packaging experiments, I worked out a good workflow (related code) for building a whole tool chain on different commitments, with different flags etc. letting me test optimizations. Due to a mistake, I was comparing my global -O3 native build with unoptimized ones for a bit. So far, I improved equality. Currently, declarative-dsls's test suite runs 25% faster on master and many other whole programs by ~10% compared to the last release!

Basically, I'm twiddling my thumbs waiting for the results of a business trip I just returned from, to bear fruit.

: tonyarkles | 23 hours ago
Very cool! I don't get to use Janet nearly as much as I'd like, but I have had good fun with it.

stig | 22 hours ago

Off to Barcelona with my wife for our 25th anniversary on Thursday. It’ll be our first holiday without kid in 15 years. Wednesday will be packing, and preparing the house for in-laws to come and house/babysit.

icefox | 7 hours ago

Making up with my partner. I think we'll be okay. :-)

: thirdtruck | 6 hours ago
Aww! Hoping that works out.

regedit | a day ago

I've been getting NetBSD 11.0 running on a Netgate SG-1000, the little AM335x box Netgate used to sell as a pfSense appliance.

The stock armv7 image, plus a modified device tree, boots from SD using Netgate's U-Boot without needing any changes to the eMMC. Netgate's Fatboot boot script checks for a UserFatboot variable and runs it if it's set.

The BeagleBone Black device tree sets up the shared pins for MII, but the SG-1000 is wired for RGMII. That seems to have been why I got gigabit link but no inbound traffic. Stripping out the pinctrl setup and fixing the PHY mode and address got one port working.

The second PHY shows up fine in U-Boot, but NetBSD still won't attach it. That's what I'm digging into this week. Reading through the CPSW driver.

I'd like to get a proper board DTS together and build an SD image that other SG-1000 owners can just write to a card and boot.

landon | 15 hours ago

Adopting 2 cats 🐱😸. Also work.

thirdtruck | 8 hours ago

Continuing the job hunt. Even McDonald's didn't get back to us so yeah, it's bad out there. Switching focus back to tech work for a spell.

Prepping for our birthday next week! Also hosting board games both personal and public.

Looking for any financial assistance we can find in the meantime since a combination of the market, misfortune, and morals have left us nearly zeroed out.

kryptiskt | a day ago

It's crunch time at work, so I'll be using my brain power on that for this week, and I'll spend my free time on not-so-demanding things like chill video games and reading.

This weekend I looked at editors as related in the weekend thread and I'm going to try out Sublime Text (as jbauer suggested) for a while, it seems to tick most of my boxes. So far, so good. I like open source, but I don't mind spending money on software, especially so for a small developer-driven company.

Loup-Vaillant | a day ago

I'm currently adding multi-thread support for Argon2 in Monocypher. Without depending on any multi-thread API at all. I'm sticking to the "pure portable C99 with zero dependency" that makes this little library so easy to integrate pretty much everywhere.

Instead I chopped up the API into an init() update() final() structure that let users implement parallelism however they want:

void crypto_argon2_init(crypto_argon2_ctx *ctx, uint32_t hash_size,
                        void *work_area,
                        crypto_argon2_config config,
                        crypto_argon2_inputs inputs,
                        crypto_argon2_extras extras);
void crypto_argon2_segment(const crypto_argon2_ctx *ctx,
                           uint32_t pass, uint32_t slice, uint32_t lane);
void crypto_argon2_final(const crypto_argon2_ctx *ctx, uint8_t *hash);

The single thread implementation goes like this:

void crypto_argon2(u8 *hash, u32 hash_size, void *work_area,
                   crypto_argon2_config config,
                   crypto_argon2_inputs inputs,
                   crypto_argon2_extras extras)
{
	crypto_argon2_ctx ctx;
	crypto_argon2_init(&ctx, hash_size, work_area, config, inputs, extras);

	for (u32 pass = 0; pass < config.nb_passes; pass++) {
		for (u32, slice, 0, 4) {
			// May be done in parallel, one thread per lane.
			for (u32 lane = 0; lane < config.nb_lanes; lane++) {
				crypto_argon2_segment(&ctx, pass, slice, lane);
			}
			// Sync point: wait for all lanes to complete.
		}
	}

	crypto_argon2_final(&ctx, hash);
}

Early experiments with pthreads suggest this should easily accommodate any multi-thread API.

Next step is documentation, and provide a production worthy code example.

david_chisnall | 13 hours ago

How do you do the sync point in C99? C11 has mechanisms for establishing the happens-before relationship here, C99 does not.

Loup-Vaillant | an hour ago

How do you do the sync point in C99?

That's the thing: I don't.

The constraints for Monocypher are more or less, dependable C with zero dependencies. For me that's C89 with volatile, // style comments, and declaring local variables anywhere. My only dependencies are stddef.h and stdint.h, from which I take size_t and the fixed size integers. I have no explicit runtime dependency, not even LibC — though compilers do sometime turn a loop of mine into a memcpy(3) call.

There's not much I can do under such drastic constraints:

No networking.
No I/O.
No explicit program aborts.
No heap allocation.
No multi-threading.

Monocypher does not, and will not, do multithreading. Only support it.

The code you see above is what I plan to ship:

A toolkit for users to write their own implementation of Argon2.
An single-thread implementation that uses that toolkit. The comments there aren't pseudocode, they're just comments, meant to document how one may parallelise Argon2.

Basically, Monocypher will "support" multi-threading by leaving it as an exercise to the user. That's cheating, I know, but as far as I can tell multi-threading is the easy part. Here's my pthreads code, minus error handling to reduce clutter:

#include <monocypher.h>
#include <pthread.h>
#include <stdlib.h>

typedef struct {
	pthread_t          thread;
	pthread_barrier_t *barrier;
	crypto_argon2_ctx *ctx;
	uint32_t           lane;
} thread_data;

static void *argon2_lane(void *ptr)
{
	thread_data *data      = ptr;
	uint32_t     lane      = data->lane;
	uint32_t     nb_passes = data->ctx->config.nb_passes;
	for (uint32_t pass = 0; pass < nb_passes; pass++) {
		for (uint32_t slice = 0; slice < 4; slice++) {
			crypto_argon2_segment(data->ctx, pass, slice, lane);
			pthread_barrier_wait(data->barrier);
		}
	}
	return NULL;
}

static int argon2_phtreads(uint8_t *hash, uint32_t hash_size,
                           crypto_argon2_config config,
                           crypto_argon2_inputs inputs,
                           crypto_argon2_extras extras)
{
	size_t       work_size = config.nb_blocks * 1024;
	size_t       data_size = config.nb_lanes  * sizeof(thread_data);
	void        *work_area = malloc(work_size);
	thread_data *lane_data = malloc(data_size);

	crypto_argon2_ctx ctx;
	crypto_argon2_init(&ctx, hash_size, work_area,
	                   config, inputs, extras);

	pthread_barrier_t barrier;
	pthread_barrier_init(&barrier, NULL, config.nb_lanes);

	for (size_t lane = 0; lane < config.nb_lanes; lane++) {
		lane_data[lane].barrier = &barrier;
		lane_data[lane].ctx     = &ctx;
		lane_data[lane].lane    = lane;
		pthread_create(&lane_data[lane].thread, NULL,
		               argon2_lane, lane_data + lane);
	}
	for (size_t lane = 0; lane < config.nb_lanes; lane++) {
		pthread_join(lane_data[lane].thread, NULL);
	}
	pthread_barrier_destroy(&barrier);

	crypto_argon2_final(&ctx, hash);
	free(work_area);
	free(lane_data);
	return 0;
}

I'll need to add proper error handling, account for EAGAIN where appropriate, and perhaps add a couple asserts to guard the functions that are supposed to never fail. The result will go to the user manual, for users to copy & paste, or adapt to their own multi-threading API.

I use a similar approach with EdDSA, by the way. In Monocypher3 I had direct support for an incremental signature API, but the only way I could make it work with both BLAKE2b EdDSA and Ed25519 was a full blows v-table. It did the job, but it ultimately proved very inflexible.

Then one day a company approached me to ad Ed25519ph to Monocypher. Pretty quickly it was clear to me that my incremental interface with swappable hash couldn't do it. I had to take my source code apart to assemble what they needed. It took me but a day, which they graciously paid full rate for.

That work gave me an idea: what if instead of using a v-table to swap in the parts I want into the full algorithm, I instead cut the algorithm into coarse constituent parts, and then use those parts to assemble whatever variation of the algorithm I, or my users, may need? Sure it falls short of direct support, but who cares, as long as it gives us the capability?

I think I managed to make it work. Heres' Ed25519 and Ed25519ph, from the toolkit and SHA-512:

void crypto_ed25519_key_pair(u8 secret_key[64], u8 public_key[32], u8 seed[32])
{
	u8 a[64];
	COPY(a, seed, 32);                      // a[ 0..31]  = seed
	crypto_wipe(seed, 32);
	COPY(secret_key, a, 32);                // secret key = seed
	crypto_sha512(a, a, 32);                // a[ 0..31]  = scalar
	crypto_eddsa_trim_scalar(a, a);         // a[ 0..31]  = trimmed scalar
	crypto_eddsa_scalarbase(public_key, a); // public key = [trimmed scalar]B
	COPY(secret_key + 32, public_key, 32);  // secret key includes public half
	WIPE_BUFFER(a);
}

static void hash_reduce(u8 h[32],
                        const u8 *a, size_t a_size,
                        const u8 *b, size_t b_size,
                        const u8 *c, size_t c_size,
                        const u8 *d, size_t d_size)
{
	u8 hash[64];
	crypto_sha512_ctx ctx;
	crypto_sha512_init  (&ctx);
	crypto_sha512_update(&ctx, a, a_size);
	crypto_sha512_update(&ctx, b, b_size);
	crypto_sha512_update(&ctx, c, c_size);
	crypto_sha512_update(&ctx, d, d_size);
	crypto_sha512_final (&ctx, hash);
	crypto_eddsa_reduce(h, hash);
}

static void ed25519_dom_sign(u8 signature[64], const u8 secret_key[64],
                             const u8 *dom,     size_t dom_size,
                             const u8 *message, size_t message_size)
{
	u8 a[64];  // secret scalar and prefix
	u8 r[32];  // secret deterministic "random" nonce
	u8 h[32];  // publicly verifiable hash of the message (not wiped)
	u8 R[32];  // first half of the signature (allows overlapping inputs)
	const u8 *pk = secret_key + 32;

	crypto_sha512(a, secret_key, 32);
	crypto_eddsa_trim_scalar(a, a);
	hash_reduce(r, dom, dom_size, a + 32, 32, message, message_size, 0, 0);
	crypto_eddsa_scalarbase(R, r);
	hash_reduce(h, dom, dom_size, R, 32, pk, 32, message, message_size);
	COPY(signature, R, 32);
	crypto_eddsa_mul_add(signature + 32, h, a, r);

	WIPE_BUFFER(a);
	WIPE_BUFFER(r);
}

void crypto_ed25519_sign(u8 signature [64], const u8 secret_key[64],
                         const u8 *message, size_t message_size)
{
	ed25519_dom_sign(signature, secret_key, 0, 0, message, message_size);
}

int crypto_ed25519_check(const u8 signature[64], const u8 public_key[32],
                         const u8 *msg, size_t msg_size)
{
	u8 h_ram[32];
	hash_reduce(h_ram, signature, 32, public_key, 32, msg, msg_size, 0, 0);
	return crypto_eddsa_check_equation(signature, public_key, h_ram);
}

static const u8 domain[34] = "SigEd25519 no Ed25519 collisions\1";

void crypto_ed25519_ph_sign(uint8_t signature[64], const uint8_t secret_key[64],
                            const uint8_t message_hash[64])
{
	ed25519_dom_sign(signature, secret_key, domain, sizeof(domain),
	                 message_hash, 64);
}

int crypto_ed25519_ph_check(const uint8_t sig[64], const uint8_t pk[32],
                            const uint8_t msg_hash[64])
{
	u8 h_ram[32];
	hash_reduce(h_ram, domain, sizeof(domain), sig, 32, pk, 32, msg_hash, 64);
	return crypto_eddsa_check_equation(sig, pk, h_ram);
}

It's not trivial. It's not even easy, which is why it (optionally) ships with Monocypher. But it's possible: one can look at the Ed25519 and Ed25519ph specs, look at Monocypher's API (which you will note hides away the elliptic curve stuff), and come up with pretty much this code.

Argon2 by contrast is genuinely easy. I'm very happy about that, because it removes a major weakness from Monocypher. With that, it will get the compatibility and security that comes with processing parallel lanes in parallel. (One thing that makes password hashing & key derivation so tricky, is how its security is directly proportional to the speed of its implementation.)

Now I need to do some vectorisation... In pure C99. Can't have it all, but I did manage to get significant gains from auto-vectorisation alone. With any luck, I may even saturate the memory bus.

brn | 8 hours ago

Released a couple of "big" redesigns for Uruky yesterday (settings and landing pages), and I have a couple of job interviews scheduled, which I need to focus on (Uruky's doing great, but it's very far away from replacing a full time job for us).

Meanwhile, our small/indie-web index is growing and I've trimmed like 150k few low quality URLs in the last few days as well. Fun!

matthiasportzel | a day ago

My current programming project is an implementation of the beautifully complex chess-like Veney.

: ansible-rs | 23 hours ago

Veney

In case anyone else is interested: https://veney.xyz/

I have never heard of this before. And despite the comparison to chess, just a brief read through the rules makes it feel very different than any other board game I've heard of.

yiyu | 20 hours ago

This week I’m working on Folune, a minimal, local-first desktop editor for text and Markdown, built with Zig, a native webview, and CodeMirror.

evanrelf | 17 hours ago

Mostly working on building my text editor, like usual. Thank goodness it's fun to work on, because it sure is taking a long time to finish!

My current architecture has served me well for a long time. But I think I need to tweak some of the underlying state management stuff to facilitate proper transactional changes. I'm very inspired by the architecture of CodeMirror 6.

I feel relatively confident I can get what I want using some eventual consistency stuff. I say more about this here if you're curious.